Build internals

From postmarketOS
Jump to: navigation, search

To build a package with pmbootstrap, you will only need the command pmbootstrap build hello-world. Optionally with --force to build even if the package was already built, and with --arch=armhf to build for another architecture, replace armhf accordingly. This wiki page lists some more information about what is going on in the background.

initramfs

The initramfs shows the boot splash images, and allows the root partition to be unlocked (currently via telnet). You can add a hook to inspect the initramfs running on the device, as described in Inspecting the initramfs.

To rebuild the initramfs, run mkinitfs inside the chroot with the right parameters (or - much easier - pmbootstrap initfs build). This gets done automatically, whenever a new kernel gets installed, or when the postmarketos-mkinitfs package gets installed the first time.

pmbootstrap vs. abuild

pmbootstrap wraps around abuild (Alpine's official program to build apks), but it does a few things different than abuild (which internally often calls apk):

  • pmbootstrap can cross-compile out of the box, utilizing different chroots as needed (see below for details)
  • pmbootstrap does dependency parsing on its own (so it works across the aports folder and the binary repository, also with timestamp based rebuilds and can detect across chroots when a package is outdated and explicitly install them).
  • pmbootstrap does not honor operators in dependencies, such as: <, >, =, !. These simply get ignored (! packages don't count as dependencies). This may lead to errors, if it does please report them. However, since we're calling apk to install the packages, it does the real dependency checking and so far it's working well enough.
  • pmbootstrap parses APKINDEX and APKBUILD files on its own.
  • APKINDEX parsing is considered to be pretty good (because the format is dead simple!)
  • APKBUILD parsing would require a shell to be done perfectly (which would in turn kill performance). The way it is implemented right now, is that the variables we care about are hardcoded inside the pbm/config/__config.py or if not possible otherwise directly in pmb/parse/apkbuild.py. That is really fast and works for all packages we care about. If it breaks somewhere, it should be easy to patch. See #351 for a better implementation.
  • pmbootstrap does not remove build dependencies after a build is done. This is for performance reasons - if you want a clean stat, run pmbootstrap zap.
  • pmbootstrap has a hack right now, that gzip always uses weak compression (also for speed)
  • pmbootstrap only installs the makedepends before compiling a package. This is a performance hack, because most of the time that is enough and saves compiling and installing bigger packages (e.g. device-packages depending on linux packages). (non-strict mode only)

Cross-compile types

There are two cross-compile types supported. pmb.build.autodetect.crosscompile() figures out, which one is the right one for each build.

  • None: The package does not have any binaries (noarch), it ends in -repack (such as qemu-user-static-repack) or the target architecture is the same as the "native" architecture (e.g. compiling heimdall for x86_64 on an x86_64 system).
  • "native": the build system of the package understands cross-compiling, like all kernel packages. We can use the native chroot with the cross-compiler inside that chroot for maximum speed.
  • "distcc": the build system of the package does not understand cross-compiling, so we run the whole compiling process inside a chroot with the target architecture (with qemu user mode emulation, this is slow). We avoid using qemu emulation for the compiler though, because we use the cross-compiler inside the native chroot through distcc (which was originally meant to share compiling efforts across the network, but it works for this case. Arch Linux ARM also does this and has a detailed description of how to set this up manually.)

Performance related issues

Caches

pmbootstrap uses various caches. They can all be found inside the work folder, and start with cache_. All cache folders get mounted to the appropriate chroots, depending on $ARCH. They are shared among the chroots, when it makes sense (e.g. cache_distfiles).

  • cache_apk_$ARCH: APK files from binary repositories (see also: Local APK cache)
  • cache_ccache_$ARCH: ccache: Whenever you compile something with pmbootstrap, the output gets cached in this folder (depending on the architecture). When you compile the same code for the second time, the cached output gets used, thus saving you a lot of time (think of re-compiling kernels, because you want to test another kernel config option etc.)
  • cache_distfiles: Whenever you build a package, abuild (which pmbootstrap wraps) will download the source files to the distfiles cache (and skip these downloads, when they already exist). The exact file name can be controlled inside the APKBUILD (more info).
  • cache_git: pmbootstrap can download git repositories. This gets used in pmbootstrap aportgen, which copies a package (aka. aport) from Alpine Linux and customizes it (for example: gcc-armhf), so we inherit all patches and changes automatically, without much maintenance work. The git repos get stored inside this folder.
  • cache_http: This stores files, that get downloaded with pmb.helpers.http.download(), so they don't need to be downloaded again every time. Currently, this gets used for the initial download of Alpine Linux' main repositories APKINDEX.tar.gz and apk-tools-static (which is a static build of the apk package manager, used to set up the chroot).