pmbootstrap/Troubleshooting

< pmbootstrap‎

This page contains tips for troubleshooting issues with pmbootstrap. If you're installing postmarketOS with pmbootstrap, this section may be helpful.

Things to check first

Make sure that you're on the latest pmbootstrap version. The latest pmbootstrap version is 3.1.0. Run:

$ pmbootstrap -V

This should print out the currently installed version of pmbootstrap. If you're on an older version, upgrade to the latest version first (how to do this depends on how pmbootstrap was installed).

Once you're on the latest pmbootstrap version, run the following to update the repositories:

$ pmbootstrap update  # updates package repositories in chroots
$ pmbootstrap pull    # updates the cloned pmaports repository

Clearing the cache and zapping chroots

Many issues can be resolved by clearing the cache or zapping the chroots used for building images/packages. pmbootstrap automates this with the pmbootstrap zap command.

The following will zap all chroots (make sure to answer "y" to every prompt):

$ pmbootstrap zap

If you're encountering issues with package building, deleting the HTTP cache (-hc flag), distfiles cache (-d flag) and previous locally-compiled packages (-p flag) might help:

$ pmbootstrap zap -hc -d -p

You can force-delete all caches by adding the -a flag:

$ pmbootstrap zap -a

Getting logs

When using pmbootstrap, you can run pmbootstrap log to print the log live as your commands are run. In another terminal window:

$ pmbootstrap log

Note that if you want to get logs *after* completing an operation, the above command will only print the last few lines; sometimes this can be enough, but usually you'll need to open the full log file.

The log file is located in the pmbootstrap work directory; by default, its location is ~/.local/var/pmbootstrap/log.txt. If you're unsure where your work directory is, you can use the following command:

$ pmbootstrap config work
/home/user/.local/var/pmbootstrap

Compatibility issues with host system

Running pmbootstrap on non-x86_64 architectures

pmbootstrap should run out-of-the-box on non-x86_64 64-bit architectures. However, you may need to modify the cross-compilation packages for some packages to build correctly.

See the Run pmbootstrap on non-x86_64 arches article for more information.

Using pmbootstrap behind a proxy

pmbootstrap will attempt to copy proxy variables from the host system to the created chroots. In some cases, however, this can fail.

If your proxy environment variables use a hostname defined in /etc/hosts, you need to change them to use the IP address.

sh: can't create /proc/sys/fs/binfmt_misc/register: nonexistent directory

This issue is known to happen in Alpine Virt hardened and Void Linux, and is caused by the binfmt_misc module not being loaded. The following commands can fix it:

# modprobe binfmt_misc
# mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

If it still does not work (or if you get an error), it is probably not supported by your host Linux distribution's kernel (CONFIG_BINFMT_MISC). Try to use another kernel.

Errors with grsec-based kernel on the host system

grsec and patchsets based on it are not supported by pmbootstrap - some of the security hardening features they provide interfere with its operation. It may be possible to get it partially working by disabling some features - see Troubleshooting:grsec.

Errors with ecryptfs

TODO: This may no longer be an issue with newer versions of ecryptfs; if you use it, please test this and update this page!

When your "work" folder is inside an ecryptfs mounted folder, /dev/random may not work inside the chroot and you may have other issues. Try to move your work folder to another location, you can use /tmp to test if this resolves the issue. You can change the location in pmbootstrap init. See also: #61.

Specific issues

ERROR: This device does not exist anymore, check <https://postmarketos.org/renamed> to see if it was renamed

One case where this might happen is if you remove pmbootstrap's work path, and then run a pmbootstrap init. As you might notice it is still remembering values and devices from previous executions.

To solve this you need to remove (or rename) $XDG_CONFIG/pmbootstrap.cfg (i.e ~/.config/pmbootstrap.cfg). After doing so pmbootstrap init should work fine.

"1 error; XXXX MiB in XXX packages" but no visible error, "Command failed (exit code 1): apk add ..."

Sometimes if package installation fails on a previous command, the error carries over to all subsequent apk operations - that's the "1 error" mentioned in the line. Check the log further back to see if you can't find the exact error there.

To fix this, try running pmbootstrap chroot apk fix (you may need to pass e.g. the --rootfs flag or the --arch=... flag depending on which chroot the error happened in). This will attempt to fix the encountered errors, and if it fails, it will print the error message again.

Alternatively, you could try zapping the relevant chroot with pmbootstrap zap (see #Clearing the cache and zapping chroots) and re-doing your command.

WARNING: No provider for dependency...

Normal behavior. it means, that your "user" repository (the repository with all custom compiled packaged from pmOS, such as your device package and kernel) does not have the dependencies from the official Alpine Linux repositories (such as cryptsetup, iw, htop, ...) when indexing it. But these dependencies are in the official repositories, so they will be found when you try to install something. Usually the package indexing is done on all repositories at once in Alpine (then you wouldn't get the error), but pmOS does it only on the "user" repository.

ERROR: /mnt/pmbootstrap-packages/x86_64/qemu-user-static-repack-2.8-r6.apk: UNTRUSTED signature

You don't have your own package signing key imported, which gets generated when you compile a package for the first time. pmbootstrap should automatically import it, but when you hit ^C during that, it may go wrong.

Your generated key is in ~/.local/var/pmbootstrap/config_abuild, all imported keys are in ~/.local/var/pmbootstrap/config_apk_keys.

A quick fix is deleting all packages:

./pmbootstrap zap -p

See issue #270 for more information and don't hesitate to reopen if this didn't work for you.

APK package index (APKINDEX.tar.gz) is missing required key

Check the file downloaded right.

Use https:// in the mirror section package mirror URLs with "pmbootstrap config".

Installed version newer than the version in the repositories

You have a program installed in one of your chroots with a higher version, than what is available in the binary package repositories (either from upstream or the packages you have built yourself).

Example how this can happen:

Git branch switch to one which increased the version of postmarketos-mkinitfs
Build postmarketos-mkinitfs
Install it in a chroot (e.g. by running pmbootstrap install)
Branch switched to master
All packages got zapped, but not the chroots
Build postmarketos-mkinitfs again (this time it has a lower version count!)

Solution: Simply zap your chroot! When it gets set up again, it will use the current package version.

Newer version in binary package repositories than in aports folder

You have built a package earlier, and the branch you are currently working in has a lower version in the aports folder.

Example how this can happen:

Git branch switch to a feature branch which increased the version of postmarketos-mkinitfs
Build postmarketos-mkinitfs
Switch to master branch
Run just about any pmbootstrap command, it will print this warning:

 Package 'postmarketos-mkinitfs' in your aports folder has version X, but the binary package repositories already have version Y

Solution: Run pmbootstrap zap -m to delete the mismatched binary packages.

Note: If that does not help it, you can manually delete that specific packages from your aports folder: sudo rm ~/.local/var/pmbootstrap/packages/*/postmarketos-mkinitfs*.apk and run pmbootstrap index to re-index all packages. See also: #466, !474

If you want to use older package built locally from pmaports:

Build the package using pmbootstrap build <pkg> --force to ignore newer version present in binary repos
Enter chroot, e.g. device's rootfs: pmbootstrap chroot -r
Install package in specific version, e.g. apk add linux-postmarketos-qcom-sdm845==6.2.0-r3

The warning (package <pkg>: aport version <v1> is lower than <v2> from the binary repository. <v2> will be used when installing <pkg>.) will still appear during e.g. pmbootstrap install, but your package will stay at specified version until chroot is wiped.

Permission denied: .../pmbootstrap/work/chroot_native/proc/1/cwd'

It looks like you have created your work folder as: pmbootstrap/work, where pmbootstrap is the folder with the checked out source code. That is not supported. Please run pmbootstrap shutdown, then delete that folder, then run pmbootstrap init again and choose another work folder (e.g. ~/.local/var/pmbootstrap).

After that it should work - if this still does not resolve it, please open up an issue in the GitLab issue tracker.

can't create `/dev/null`: Permission denied

Make sure that your working directory is mounted correctly. This can happen when working over NFS, or when you mounted the drive your working directory is on through the file explorer (if that's the case, umount it, and mount it manually). This can probably happen on every drive mounted with nosuid or similar.

cp: write error: No space left on device (during file transfer from chrooted environment to target image)

This issue is triggered when pmbootstrap determines the wrong size for the target image, as it is unfortunately very hard to calculate the correct size independent of the filesystem on which the chroot resides. Some examples for this: #1916; #928 where more elaborate information can be found.

While fixing the problem seems impossible for every configuration, there is a workaround available (#1904 and its associated merge request !1989): Use the -E/--extra-space option of pmbootstrap to specify extra space on the resulting image file in MB. Just play around with the value until it works for you. You can specify the value also through pmbootstrap init in its Additional options.

Cannot download packages from mirrors: "temporary error (try again later)"

Another symptom of this error is the message: "alpine-base (no such package)."

This is because of a network error - APK cannot connect to any of the package mirrors. pmbootstrap will copy your /etc/resolv.conf into the chroot. If your resolv.conf is empty or misconfigured, APK will not be able to connect to a DNS server. See: resolv.conf(5)

ERROR: ... BAD archive

If you pmbootstrap log shows something similar to the following:

(005853) [22:25:47] (native) % apk --no-progress add -u qemu-user-static-repack qemu-user-static-repack-binfmt
WARNING: Ignoring /mnt/pmbootstrap-packages/x86_64/APKINDEX.tar.gz: No such file or directory
(1/2) Installing qemu-user-static-repack (2.8-r9)
ERROR: qemu-user-static-repack-2.8-r9: BAD archive
(2/2) Installing qemu-user-static-repack-binfmt (2.8-r9)
1 error; 20 MiB in 36 packages
(005853) [22:25:53] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

...then downloading a package (in this case qemu-user-static-repack) failed. Please delete the broken package and try it again:

$ cd ~/.local/var/pmbootstrap/ # go into your work folder, as specified in 'pmbootstrap init'
$ sudo rm -v cache_apk_*/qemu-user-static-repack-* # adjust the failing package name accordingly

Build errors

It's not installing my changed package

You have changed a package but it's using the one from the repository and not your changed build.

You probably didn't build it or not for the correct architecture. pmbootstrap build defaults to the host architecture (usually x86_64), if you need that package on your device, use the --arch flag.

`musl-`/`busybox-static-` fails to build with a 404 error

Run pmbootstrap aportgen musl-armhf musl-aarch64 musl-x86_64. The reason why this fails from time to time is, that we use Alpine's binary musl lib c package for armhf/aarch64 in the native chroot, to get the cross-compiler working. So the download URL inside the aports/cross/musl-*/APKBUILD file must be in sync with the upstream version. This pmbootstrap command safely gets the new version, uses apk to verify the signature of the package, generates the hash, and updates the APKBUILD file. To make sure, that pmbootstrap aportgen receives the latest musl version, delete your HTTP cache right before by executing this command: pmbootstrap zap -hc. Don't bother making a pull-request for this, as the fix is really trivial (see this commit). Just report it as issue, if it happens.

boot-deploy exits with ERROR: "deviceinfo_header_version is 2, but 'deviceinfo_dtb' is missing. Set 'deviceinfo_dtb'" when compiling downstream kernel

Deviceinfo file is generated with deviceinfo_header_version set to 2 and boot-deploy expects a dtb file to be to be specified in deviceinfo. Verify that the deviceinfo_dtb entry in your deviceinfo is not empty or points to a non-existent dtb/dts in your kernel source.

BusyBox does something different than the standard tool for the job, and this breaks building my package!

Example: unzip in BusyBox doesn't extract symlinks like the normal unzip implementation (#82).

Solution: Simply add unzip to the makedepends in the APKBUILD, and the "real" unzip will be used (or in that case, try to get the sources in another archive format, such as .tar.bz2, .tar.xz, .tar.gz).

Kernel building issues

TODO: Move these to Troubleshooting:kernel?

mkbootimg: error: argument --dt: can't open '/boot/dt.img'

This happens when you have deviceinfo_bootimg_qcdt enabled (which is correct when pmbootstrap generated it that way), but your kernel does not generate the /boot/dt.img file. Please adjust your kernel APKBUILD as follows:

add dtbtool to makedepends
extend your build() function like this
and your package() function like that

See #688 for a plan to automatically generate the linux APKBUILD right.

Packaging for aarch64: ./config.guess: unable to guess system type

This happens in packages built with autoconf (./configure; make; make install), when it ships an outdated config.guess file in the source code. Adjust your APKBUILD's prepare() function like done in !698.

"ERROR: Is $builddir set correctly?" for linux-* package

Check where the kernel sources archive has the kernel-related code stored and change the builddir value accordingly.

Flashing problems

Invalid sparse file format at header magic

In some devices (e.g.: bullhead, titan), when flashing the system (pmbootstrap flasher flash_rootfs), it works but fastboot complains of Invalid sparse file format at header magic.

$ ./pmbootstrap.py flasher flash_rootfs
[23:02:58] (native) install android-tools
[23:02:58] (native) flash rootfs image
target reported max download size of 536870912 bytes
Invalid sparse file format at header magic
sending sparse 'system' 1/1 (274101 KB)...
OKAY [  9.452s]
writing 'system' 1/1...
OKAY [ 12.584s]
finished. total time: 22.036s
[23:03:20] Done

It happens when your device expects a sparse system image. The workaround is as follows:

Set deviceinfo_flash_sparse=true in deviceinfo

To do this, edit the file in ./local/var/pmbootstrap/cache_git/pmaports/device/your_device_name/deviceinfo and add deviceinfo_flash_sparse="true". When finished, run "pmbootstrap install" to rebuild the file system

For more information, see issue #299

Heimdall: Flashing fails at 7%

You might also be getting the error "ERROR: Failed to confirm end of file transfer sequence"

It might be, that your device only accepts "sparse images" (#1643). Try setting deviceinfo_flash_sparse="true" in the deviceinfo.

FAILED (remote: Unknown chunk type)

This is a bootloader limitation. Flash with netcat as a workaround, or use this wiki's search feature to come up with more related information. Feel free to organize this in a better way, but this should be a good starting point. See also #1735.

flasher (fastboot) is stuck in < waiting for any device >

Maybe you need to install udev rules on your host Linux distribution. Arch has the android-udev package for example, which uses the rules from here.

If that does not help, it might be necessary to set the flash_fastboot_vendor_id in your deviceinfo (this was necessary for the Amazon Kindle Fire HDX). Try to search the web for your device name and "fastboot vendor id".

If you were previously able to use fastboot and it stopped working, sometimes pmbootstrap shutdown to restart the chroot helps (#1633). Also try to reboot the device into fastboot again.