Build internals

From postmarketOS Wiki

To build a package with pmbootstrap, you will only need the command pmbootstrap build hello-world. Optionally with --force to build even if the package was already built, and with --arch=armhf to build for another architecture, replace armhf accordingly. This wiki page lists some more information about what is going on in the background.

initramfs

The initramfs shows the boot splash images, and allows the root partition to be unlocked (currently via telnet). You can add a hook to inspect the initramfs running on the device, as described in Inspecting the initramfs.

To rebuild the initramfs, run mkinitfs inside the chroot with the right parameters (or - much easier - pmbootstrap initfs build). This gets done automatically, whenever a new kernel gets installed, or when the postmarketos-mkinitfs package gets installed the first time.

pmbootstrap vs. abuild

pmbootstrap wraps around abuild (Alpine's official program to build apks), but it does a few things different than abuild (which internally often calls apk):

  • pmbootstrap can cross-compile out of the box, utilizing different chroots as needed (see below for details)
  • pmbootstrap does dependency parsing on its own (so it works across the aports folder and the binary repository and can detect across chroots when a package is outdated and explicitly install them).
  • pmbootstrap does not honor operators in dependencies, such as: <, >, =, !. These simply get ignored (! packages don't count as dependencies). This may lead to errors, if it does please report them. However, since we're calling apk to install the packages, it does the real dependency checking and so far it's working well enough.
  • pmbootstrap parses APKINDEX and APKBUILD files on its own.
  • APKINDEX parsing is considered to be pretty good (because the format is dead simple!)
  • APKBUILD parsing would require a shell to be done perfectly (which would in turn kill performance). The way it is implemented right now, is that the variables we care about are hardcoded inside the pbm/config/__config__.py or if not possible otherwise directly in pmb/parse/apkbuild.py. That is really fast and works for all packages we care about. If it breaks somewhere, it should be easy to patch.
  • pmbootstrap does not remove build dependencies after a build is done (except when using --strict). This is for performance reasons - if you want a clean start, run pmbootstrap zap.
  • pmbootstrap has a hack right now, that gzip always uses weak compression (also for speed)

Cross-compile types

These are the cross-compile types supported. pmb.build.autodetect.crosscompile() figures out, which one is the right one for each build.

  • None: The target architecture is the same as the "native" architecture (e.g. compiling heimdall for x86_64 on an x86_64 system).
  • "native": the build system of the package understands cross-compiling, like all kernel packages. We can use the native chroot with the cross-compiler inside that chroot for maximum speed.
  • "crossdirect": the native chroot gets mounted in the foreign arch chroot, and with some magic we can redirect everything to the cross compilers (#1731). This is the default method, however some packages fail with this method, you can revert to the slower but more reliable QEMU emulation with pmbootstrap --no-cross ....

See pmbootstrap: Cross Compiling for details.

Performance related issues

  • #659 Use "native" for (almost) all packages?

Caches

pmbootstrap uses various caches. They can all be found inside the work folder, and start with cache_. All cache folders get mounted to the appropriate chroots, depending on $ARCH. They are shared among the chroots, when it makes sense (e.g. cache_distfiles).

  • cache_apk_$ARCH: APK files from binary repositories (see also: Local APK cache)
  • cache_ccache_$ARCH: ccache: Whenever you compile something with pmbootstrap, the output gets cached in this folder (depending on the architecture). When you compile the same code for the second time, the cached output gets used, thus saving you a lot of time (think of re-compiling kernels, because you want to test another kernel config option etc.)
  • cache_distfiles: Whenever you build a package, abuild (which pmbootstrap wraps) will download the source files to the distfiles cache (and skip these downloads, when they already exist). The exact file name can be controlled inside the APKBUILD (more info).
  • cache_git: pmbootstrap can download git repositories. This gets used in pmbootstrap aportgen, which copies a package (aka. aport) from Alpine Linux and customizes it (for example: gcc-armhf), so we inherit all patches and changes automatically, without much maintenance work. The git repos get stored inside this folder.
  • cache_http: This stores files, that get downloaded with pmb.helpers.http.download(), so they don't need to be downloaded again every time. Currently, this gets used for the initial download of Alpine Linux' main repositories APKINDEX.tar.gz and apk-tools-static (which is a static build of the apk package manager, used to set up the chroot).

Why noarch packages are built for each architecture

For a noarch package, we could in theory compile it once and then create symlinks to all arches pointing to the package where we have created it. And previous versions of pmbootstrap used to do that. But that makes dependency parsing hard.

For example: device packages are always noarch, so in theory you could compile the device package for your armhf device in x86_64, and then symlink it to the armhf package folder. But it would still depend on the kernel, which is armhf only for that device (assuming it is not mainlined, like most devices are right now). So it would not make sense to install it in x86_64 anyway, and if you would want to build it, you could only really do it in armhf anyway, because that's where the kernel is available.

So to simplify it, it is handled like in Alpine now; noarch packages just get built for each arch independently.

pmbootstrap specific APKBUILD options

For various reasons, we have added support for the following pmbootstrap specific values in APKBUILD options.

  • !pmb:crossdirect: do not use the crossdirect method (!1922). pmbootstrap will just build completely in qemu (slow).
  • !pmb:kconfigcheck: for linux-* packages only. When set, the given linux package does not fail "pmbootstrap kconfig check" (!1753).
  • pmb:cross-native: build the package using the "native" cross compilation method !1920
  • pmb:gpu-accel: used for UI specific packages to define GPU acceleration requirement !2043
  • pmb:strict: always build the package in strict mode, even if --strict is not passed to "pmbootstrap build" (!1771).