Build internals: Difference between revisions
document APKBUILD options !pmb:kconfigcheck and pmb:strict |
DolphinChips (talk | contribs) mNo edit summary |
||
Line 30: | Line 30: | ||
=== Performance related issues === | === Performance related issues === | ||
* | * {{issue|659}} Use "native" for (almost) all packages? | ||
* | * {{issue|384}} Distcc: use pump mode for possible performance improvement | ||
* | * {{issue|480}} Distcc: has integrated timeout, slowing down our builds ._. | ||
=== Caches === | === Caches === | ||
Line 42: | Line 42: | ||
* <code>cache_distfiles</code>: Whenever you build a package, <code>abuild</code> (which pmbootstrap wraps) will download the source files to the <code>distfiles</code> cache (and skip these downloads, when they already exist). The exact file name can be controlled inside the APKBUILD ([https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#source more info]). | * <code>cache_distfiles</code>: Whenever you build a package, <code>abuild</code> (which pmbootstrap wraps) will download the source files to the <code>distfiles</code> cache (and skip these downloads, when they already exist). The exact file name can be controlled inside the APKBUILD ([https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#source more info]). | ||
* <code>cache_git</code>: <code>pmbootstrap</code> can download git repositories. This gets used in <code>pmbootstrap aportgen</code>, which copies a package (aka. aport) from Alpine Linux and customizes it (for example: <code>gcc-armhf</code>), so we inherit all patches and changes automatically, without much maintenance work. The git repos get stored inside this folder. | * <code>cache_git</code>: <code>pmbootstrap</code> can download git repositories. This gets used in <code>pmbootstrap aportgen</code>, which copies a package (aka. aport) from Alpine Linux and customizes it (for example: <code>gcc-armhf</code>), so we inherit all patches and changes automatically, without much maintenance work. The git repos get stored inside this folder. | ||
* <code>cache_http</code>: This stores files, that get downloaded with [https:// | * <code>cache_http</code>: This stores files, that get downloaded with [https://gitlab.com/postmarketOS/pmbootstrap/-/blob/master/pmb/helpers/http.py <code>pmb.helpers.http.download()</code>], so they don't need to be downloaded again every time. Currently, this gets used for the initial download of Alpine Linux' main repositories <code>APKINDEX.tar.gz</code> and [https://pkgs.alpinelinux.org/package/edge/main/armhf/apk-tools-static apk-tools-static] (which is a static build of the <code>apk</code> package manager, used to set up the chroot). | ||
=== Why noarch packages are built for each architecture === | === Why noarch packages are built for each architecture === |
Revision as of 15:23, 3 April 2020
To build a package with pmbootstrap
, you will only need the command pmbootstrap build hello-world
. Optionally with --force
to build even if the package was already built, and with --arch=armhf
to build for another architecture, replace armhf
accordingly. This wiki page lists some more information about what is going on in the background.
initramfs
The initramfs shows the boot splash images, and allows the root partition to be unlocked (currently via telnet). You can add a hook to inspect the initramfs running on the device, as described in Inspecting the initramfs.
To rebuild the initramfs, run mkinitfs
inside the chroot with the right parameters (or - much easier - pmbootstrap initfs build
). This gets done automatically, whenever a new kernel gets installed, or when the postmarketos-mkinitfs
package gets installed the first time.
pmbootstrap vs. abuild
pmbootstrap
wraps around abuild
(Alpine's official program to build apks), but it does a few things different than abuild
(which internally often calls apk
):
pmbootstrap
can cross-compile out of the box, utilizing different chroots as needed (see below for details)pmbootstrap
does dependency parsing on its own (so it works across theaports
folder and the binary repository and can detect across chroots when a package is outdated and explicitly install them).pmbootstrap
does not honor operators in dependencies, such as:<
,>
,=
,!
. These simply get ignored (!
packages don't count as dependencies). This may lead to errors, if it does please report them. However, since we're callingapk
to install the packages, it does the real dependency checking and so far it's working well enough.pmbootstrap
parsesAPKINDEX
andAPKBUILD
files on its own.APKINDEX
parsing is considered to be pretty good (because the format is dead simple!)APKBUILD
parsing would require a shell to be done perfectly (which would in turn kill performance). The way it is implemented right now, is that the variables we care about are hardcoded inside thepbm/config/__config__.py
or if not possible otherwise directly inpmb/parse/apkbuild.py
. That is really fast and works for all packages we care about. If it breaks somewhere, it should be easy to patch.pmbootstrap
does not remove build dependencies after a build is done (except when using--strict
). This is for performance reasons - if you want a clean start, runpmbootstrap zap
.pmbootstrap
has a hack right now, thatgzip
always uses weak compression (also for speed)
Cross-compile types
These are the cross-compile types supported. pmb.build.autodetect.crosscompile()
figures out, which one is the right one for each build.
None
: The target architecture is the same as the "native" architecture (e.g. compilingheimdall
forx86_64
on anx86_64
system)."native"
: the build system of the package understands cross-compiling, like all kernel packages. We can use the native chroot with the cross-compiler inside that chroot for maximum speed."distcc"
: the build system of the package does not understand cross-compiling, so we run the whole compiling process inside a chroot with the target architecture (with qemu user mode emulation, this is slow). We avoid using qemu emulation for the compiler though, because we use the cross-compiler inside the native chroot through distcc (which was originally meant to share compiling efforts across the network, but it works for this case. Arch Linux ARM also does this and has a detailed description of how to set this up manually.) This method became a lot slower, as we had to run distcc behind a locally running SSH server, in order to avoid a security issue (!1649). This method is legacy, replaced by "crossdirect" below."crossdirect"
: similar to distcc, but the native chroot gets mounted in the foreign arch chroot, and with some magic we can redirect everything to the cross compilers without having distcc running (#1731). Therefore we side-step the security issue and the performance problems. As of writing, this method is brand new and only available in pmbootstrap master. If any problems arise, you can switch to "distcc-sshd" again by runningpmbootstrap --no-crossdirect ...
.
- #659 Use "native" for (almost) all packages?
- #384 Distcc: use pump mode for possible performance improvement
- #480 Distcc: has integrated timeout, slowing down our builds ._.
Caches
pmbootstrap
uses various caches. They can all be found inside the work
folder, and start with cache_
. All cache folders get mounted to the appropriate chroots, depending on $ARCH. They are shared among the chroots, when it makes sense (e.g. cache_distfiles
).
cache_apk_$ARCH
: APK files from binary repositories (see also: Local APK cache)cache_ccache_$ARCH
: ccache: Whenever you compile something withpmbootstrap
, the output gets cached in this folder (depending on the architecture). When you compile the same code for the second time, the cached output gets used, thus saving you a lot of time (think of re-compiling kernels, because you want to test another kernel config option etc.)cache_distfiles
: Whenever you build a package,abuild
(which pmbootstrap wraps) will download the source files to thedistfiles
cache (and skip these downloads, when they already exist). The exact file name can be controlled inside the APKBUILD (more info).cache_git
:pmbootstrap
can download git repositories. This gets used inpmbootstrap aportgen
, which copies a package (aka. aport) from Alpine Linux and customizes it (for example:gcc-armhf
), so we inherit all patches and changes automatically, without much maintenance work. The git repos get stored inside this folder.cache_http
: This stores files, that get downloaded withpmb.helpers.http.download()
, so they don't need to be downloaded again every time. Currently, this gets used for the initial download of Alpine Linux' main repositoriesAPKINDEX.tar.gz
and apk-tools-static (which is a static build of theapk
package manager, used to set up the chroot).
Why noarch packages are built for each architecture
For a noarch
package, we could in theory compile it once and then create symlinks to all arches pointing to the package where we have created it. And previous versions of pmbootstrap used to do that. But that makes dependency parsing hard.
For example: device packages are always noarch
, so in theory you could compile the device package for your armhf
device in x86_64
, and then symlink it to the armhf
package folder. But it would still depend on the kernel, which is armhf
only for that device (assuming it is not mainlined, like most devices are right now). So it would not make sense to install it in x86_64
anyway, and if you would want to build it, you could only really do it in armhf
anyway, because that's where the kernel is available.
So to simplify it, it is handled like in Alpine now; noarch packages just get built for each arch independently.
pmbootstrap specific APKBUILD options
For various reasons, we have added support for the following pmbootstrap specific values in APKBUILD options.