Troubleshooting

From postmarketOS

If you're troubleshooting development issues, please see the Developer information section below.

If your device is a smartphone, tablet, or other embedded device, you should be able to retrieve boot logs by attaching it to your PC and accessing the "PMOS_LOGS" storage device. Please see the README file for information on opening a new issue. You may also find some of the following tips useful. The most likely causes of boot failure are a bug or regression due to a kernel upgrade or a change to the initramfs, or a corruption issue during flashing. In the former case please do the following:

1. Check the pmaports issue tracker to see if someone has reported a similar issue

2. If so, then leave a comment that you encountered the same bug, attaching the log archive from your device if available.

3. If you're encountering this failure on your first boot, try re-flashing and double checking the installation instructions for your device. If flashing via fastboot, use the "fasboot reboot" command rather than rebooting via the device controls - this will ensure that any pending writes are correctly flushed.

4. If your device still doesn't boot, open a new issue and provide as much information as you can about the steps you took prior to encountering the problem (SEE THE LIST OF IMPORTANT INFORMATION BELOW). This will help the developers a lot to properly reproduce and debug the issue. postmarketOS is run by volunteers largely in their free time, and it can be time consuming to have to go back and forth to get the necessary information.

Due to the vast number of configurations and usecases supported by postmarketOS, it's difficult to begin to triage an issue without knowing more information about your device and installation method. We hope to continue improving how we handle booting issues and appreciate any feedback.

Important information to provide

Please include this information when opening an issue about a boot failure. Copy/paste ALL of the following including the header/footer into the issue description and fill it out by writing below the questions and putting x's between the [ ] for the relevant options.

<!--PMOS BOOT ISSUE FORM START-->
## When did the issue occur? (Check all that apply)

* [ ] First boot
* [ ] After an upgrade
* [ ] After installing a new package
* [ ] After editing a system configuration file (Under `/etc`, `/usr`, or `/boot`)
* [ ] After inserting/removing an SD card

## If the issue occured after an upgrade, how did you perform the upgrade?

* [ ] By running `apk upgrade` in a terminal
* [ ] By running `apk upgrade -a` in a terminal
* [ ] Via the app store (GNOME software, KDE discover, etc)

## If the issue occured after installing a package, what was it and how did you install it?

<!--Please also specify if it was installed from FlatHub / FlatPak-->

<!--PMOS BOOT ISSUE FORM END-->

Boot partition not found

This is caused by either:

  • A kernel bug (failing to detect storage devices)
  • A corruption issue
  • A bug in the initramfs code
  • A configuration issue (deviceinfo/kernel cmdline)

Root partition not found

This is more likely to be caused by a bug in the initramfs code or a configuration issue.

Developer information

  • Always look at the log file for a detailed error message: ~/.local/var/pmbootstrap/log.txt
  • Make sure, that you have the latest version. Your bug may already be fixed.
  • Please report the bugs you have found in the issues, so we can fix them.
  • Syslog is enabled, and will log to shared memory once the OS has booted. It can be read with the logread tool in BusyBox. The syslog is currently not preserved across reboots.

See also: How to report issues

Specific troubleshooting pages

Port/device troubleshooting

References

Porting process, preparation and various information

See also: Troubleshooting#pmbootstrap and build related errors

My device runs the Linux kernel (e.g. Android), but the vendor does not provide the source

  1. Ask kindly via e-mail.
  2. If they don't provide the source, team up with gpl-violations.org. You have the legal right to get the source, because the Linux kernel is under the GPLv2!
  3. Consider building generic kernel for your device.
  4. Another option is using prebuilt kernels

My Android device does not have partition labels, how do I know where to put the boot.img file etc?

  • Search online for the partitions, they are probably documented somewhere.
  • Try to dump all partitions. Run `file` on them, it should roughly tell you the contents. If that does not help, run binwalk.
    • file output: check if it is a kernel (vmlinuz), boot.img file or just "data".
    • In case you have "data", don't flash anything there, we probably can't generate that format yet. Please open an issue in GitHub or ask on the channel.
  • You can also look at recovery zips from other Android ROMs, that were released for your device. Extract it and look at META-INF/com/google/android/updater-script, search for commands starting with run_program("/sbin/busybox", "dd", they are for flashing partitions.
    • In one example, there was run_program("/sbin/busybox", "dd", "if=/tmp/boot.blob", "of=/dev/block/mmcblk0p4");, which means that a boot.blob file gets flashed to /dev/block/mmcblk0p4. That boot.blob file is also in the recovery zip - run the file program on it (see above).

Serial Console Debug Cable (early boot debugging)

For some devices it's possible to get early boot output using a debug cable that provides access to the UART. Unfortunately manufacturers do not use to publish the details and it is end users who in some cases have figured it out:

On fastboot devices, you may have to enable UART debugging with fastboot oem uart-on.

By default, all output from the initramfs get redirected to /pmOS_init.log. If you want the output through the debug cable you need to add PMOS_NO_OUTPUT_REDIRECT to your kernel command line.

See Serial debugging

Proprietary blobs

Many devices require proprietary blobs for enabling certain peripherals (wifi/bluetooth/cellular/GPS). These repositories appear to contain blobs for a large percentage of devices out there.

Force reboot

On some phones, you can hold down the power button for ten seconds to force a reboot. If that does not work: On other phones, you have to hold the power button down for a full minute (or hold it together with volume down). On some phones, it might take even longer (3 minutes or so).

You can also give from console echo b > /proc/sysrq-trigger.

Errors on the device

Touch screen device present but not outputting any touch events

For some devices (at least motorola-shamu and motorola-lux), the touch screen I2C device is present but not activated at boot, and must be activated before it produces any output.

For the motorola-lux, this is done in the following way:

echo 1 > /sys/devices/soc.0/78b8000.i2c/i2c-4/4-0020/drv_irq

The path to the correct touch device depends on the device. This can be added to an initfs hook to activate the touch screen automatically at boot.

CHARGING MODE: error starting charging-sdl

When attaching the power cable while the device is off, Android devices boot into a special "charging mode". postmarketOS's charging-sdl display does not work yet on all devices.

To debug the issue, follow instructions at Inspecting the initramfs including activating the initramfs-extra, then execute start_charging_mode in the shell.

During initramfs generation: "modprobe: WARNING: Module drm not found in directory /lib/modules/..."

This means, that a module can not be put in an initramfs. When you are not using a mainline kernel, it is likely that there are no modules configured, but the code is built directly into the kernel. This means, it is expected and can be ignored.

Phone shows the telnet message, a black screen, or doesn't leave the bootloader or Samsung/Google/... splash screen

See: Troubleshooting:boot

I have ported postmarketOS to my device and the screen doesn't get updated

There is a known bug where if you execute cat /sys/class/graphics/fb0/modes > /sys/class/graphics/fb0/mode exactly one frame is drawn, and the screen isn't updated otherwise.

Device reboots

  • If your device always reboots after the same time (e.g. 2 minutes) after booting, you probably need to feed (or disable) a watchdog. Try to search your kernel config for watchdog, and also the internet for watchdog + your phone name.
  • Android: The kernel config option CONFIG_ANDROID_LOW_MEMORY_KILLER may cause reboots. Please check if you have disabled it. If this helps you, please let us know, so we can consider checking for this option.
  • Android: it is possible to get the dmesg log from just before the crash after the reboot if you enable some kernel options.

User interface is unresponsive

  • If you have a situation where an application or display manager user interface is unresponsive, you can use TTYescape to kill the offending process, power cycle the device, or otherwise access helpful command line utilities.

SSH / udev / touch screen doesn't work

  • Make sure that DEVTMPFS is enabled in your kernel config. Do not enable DEVTMPFS_MOUNT.

I can't access the framebuffer device

"if running weston from ssh, use --tty to specify a tty"

Specify a tty in weston's commandline. The correct syntax is: weston --tty=1.

Weston and fbsplash -s <image> do not work

If Weston returns an error because it could not access /dev/tty1, even when you specify it on the command line as above, you may need to enable Virtual Terminals in the kernel config.
You can enable CONFIG_VT using pmbootstrap menuconfig and checking under Device Drivers -> Character devices -> Virtual terminal

ERROR: unsupported 'crypto_LUKS' filesystem (/dev/mmcblk2p2)

See postmarketos-release-upgrade#15.

Flashing problems

Invalid sparse file format at header magic

In some devices (e.g.: bullhead, titan), when flashing the system (pmbootstrap flasher flash_rootfs), it works but fastboot complains of Invalid sparse file format at header magic.

$ ./pmbootstrap.py flasher flash_rootfs
[23:02:58] (native) install android-tools
[23:02:58] (native) flash rootfs image
target reported max download size of 536870912 bytes
Invalid sparse file format at header magic
sending sparse 'system' 1/1 (274101 KB)...
OKAY [  9.452s]
writing 'system' 1/1...
OKAY [ 12.584s]
finished. total time: 22.036s
[23:03:20] Done

It happens when your device expects a sparse system image. The workaround is as follows:

  • Set deviceinfo_flash_sparse=true in deviceinfo

To do this, edit the file in ./local/var/pmbootstrap/cache_git/pmaports/device/your_device_name/deviceinfo and add deviceinfo_flash_sparse="true". When finished, run "pmbootstrap install" to rebuild the file system

For more information, see issue #299

Login screen hangs

When switching from an installation on the system-partition to an installation on the userdata-partition, or when switching from a downstream-kernel to a mainline-kernel, it can happen, that the system boots up but hangs before login.

Logread via SSH may print a loop of

Jan 1 02:45:10 device-xy daemon.info init: can't open /dev/ttyHSL0: No such file or directory
Jan 1 02:45:10 device-xy daemon.info init: process '/sbin/getty -L ttyHSL0 115200 vt100' (pid 9119) exited. Scheduling for restart.

Try to erase the partitions before flashing: fastboot erase system && fastboot erase userdata && fastboot erase cache

Heimdall: Flashing fails at 7%

You might also be getting the error "ERROR: Failed to confirm end of file transfer sequence"

It might be, that your device only accepts "sparse images" (#1643). Try setting deviceinfo_flash_sparse="true" in the deviceinfo.

FAILED (remote: Unknown chunk type)

This is a bootloader limitation. Flash with netcat as a workaround, or use this wiki's search feature to come up with more related information. Feel free to organize this in a better way, but this should be a good starting point. See also #1735.

flasher (fastboot) is stuck in < waiting for any device >

Maybe you need to install udev rules on your host Linux distribution. Arch has the android-udev package for example, which uses the rules from here.

If that does not help, it might be necessary to set the flash_fastboot_vendor_id in your deviceinfo (this was necessary for the Amazon Kindle Fire HDX). Try to search the web for your device name and "fastboot vendor id".

If you were previously able to use fastboot and it stopped working, sometimes pmbootstrap shutdown to restart the chroot helps (#1633). Also try to reboot the device into fastboot again.

ERROR: Failed to confirm end of file transfer sequence!

This message can mean anything. To get more information you can instead create an android-recovery-zip and install that from recovery. The installation will probably fail now as well, but at least this time you will have logs in /tmp/postmarketos/.

One possible reason for the error is simply that the image is too large for the partition, in which case you need to decrease the size, or maybe re-partition your device.

The whole error message might look something like

[...]
Uploading BOOT
100%
ERROR: Failed to confirm end of file transfer sequence!
ERROR: BOOT upload failed!

Ending session...
ERROR: Failed to send end session packet!
Releasing device interface...

pmbootstrap and build related errors

Alpine, chroots and pmbootstrap

pmbootstrap is broken! But it worked yesterday!

Sorry! Revert to the last commit, that was known to work (git checkout $commit) and report an issue, if there is not already one. Consult with others in the IRC/matrix channel to find out, which commit was the last one that worked.

ERROR: This device does not exist anymore, check <https://postmarketos.org/renamed> to see if it was renamed

One case where this might happen is if you remove pmbootstrap's work path, and then run a pmbootstrap init. As you might notice it is still remembering values and devices from previous executions.

To solve this you need to remove (or rename) $XDG_CONFIG/pmbootstrap.cfg (i.e ~/.config/pmbootstrap.cfg). After doing so pmbootstrap init should work fine.

BusyBox does something different than the standard tool for the job, and this breaks building my package!

Example: unzip in BusyBox doesn't extract symlinks like the normal unzip implementation (#82).

Solution: Simply add unzip to the makedepends in the APKBUILD, and the "real" unzip will be used (or in that case, try to get the sources in another archive format, such as .tar.bz2, .tar.xz, .tar.gz).

WARNING: No provider for dependency...

Normal behavior. it means, that your "user" repository (the repository with all custom compiled packaged from pmOS, such as your device package and kernel) does not have the dependencies from the official Alpine Linux repositories (such as cryptsetup, iw, htop, ...) when indexing it. But these dependencies are in the official repositories, so they will be found when you try to install something. Usually the package indexing is done on all repositories at once in Alpine (then you wouldn't get the error), but pmOS does it only on the "user" repository.

ERROR: busybox-1.27.0-r1.post-install: script exited with error 127

Programs running in (foreign arch) chroots cannot be executed. See #1972.

ERROR: /mnt/pmbootstrap-packages/x86_64/qemu-user-static-repack-2.8-r6.apk: UNTRUSTED signature

You don't have your own package signing key imported, which gets generated when you compile a package for the first time. pmbootstrap should automatically import it, but when you hit ^C during that, it may go wrong.

Your generated key is in ~/.local/var/pmbootstrap/config_abuild, all imported keys are in ~/.local/var/pmbootstrap/config_apk_keys.

A quick fix is deleting all packages:

./pmbootstrap zap -p

See issue #270 for more information and don't hesitate to reopen if this didn't work for you.

Installed version newer than the version in the repositories

You have a program installed in one of your chroots with a higher version, than what is available in the binary package repositories (either from upstream or the packages you have built yourself).

Example how this can happen:

  • Git branch switch to one which increased the version of postmarketos-mkinitfs
  • Build postmarketos-mkinitfs
  • Install it in a chroot (e.g. by running pmbootstrap install)
  • Branch switched to master
  • All packages got zapped, but not the chroots
  • Build postmarketos-mkinitfs again (this time it has a lower version count!)

Solution: Simply zap your chroot! When it gets set up again, it will use the current package version.

It's not installing my changed package

You have changed a package but it's using the one from the repository and not your changed build.

You probably didn't build it or not for the correct architecture. pmbootstrap build defaults to x86_64 architecture, if you need that package on your device, use the --arch flag.

Newer version in binary package repositories than in aports folder

You have built a package earlier, and the branch you are currently working in has a lower version in the aports folder.

Example how this can happen:

  • Git branch switch to a feature branch which increased the version of postmarketos-mkinitfs
  • Build postmarketos-mkinitfs
  • Switch to master branch
  • Run just about any pmbootstrap command, it will print this warning:
 Package 'postmarketos-mkinitfs' in your aports folder has version X, but the binary package repositories already have version Y

Solution: Run pmbootstrap zap -m to delete the mismatched binary packages.

Note: If that does not help it, you can manually delete that specific packages from your aports folder: sudo rm ~/.local/var/pmbootstrap/packages/*/postmarketos-mkinitfs*.apk and run pmbootstrap index to re-index all packages. See also: #466, !474


If you want to use older package built locally from pmaports:

  1. Build the package using pmbootstrap build <pkg> --force to ignore newer version present in binary repos
  2. Enter chroot, e.g. device's rootfs: pmbootstrap chroot -r
  3. Install package in specific version, e.g. apk add linux-postmarketos-qcom-sdm845==6.2.0-r3

The warning (package <pkg>: aport version <v1> is lower than <v2> from the binary repository. <v2> will be used when installing <pkg>.) will still appear during e.g. pmbootstrap install, but your package will stay at specified version until chroot is wiped.

mkbootimg: error: argument --dt: can't open '/boot/dt.img'

This happens when you have deviceinfo_bootimg_qcdt enabled (which is correct when pmbootstrap generated it that way), but your kernel does not generate the /boot/dt.img file. Please adjust your kernel APKBUILD as follows:

  • add dtbtool to makedepends
  • extend your build() function like this
  • and your package() function like that

See #688 for a plan to automatically generate the linux APKBUILD right.

Permission denied: .../pmbootstrap/work/chroot_native/proc/1/cwd'

It looks like you have created your work folder as: pmbootstrap/work, where pmbootstrap is the folder with the checked out source code. That is not supported. Please run pmbootstrap shutdown, then delete that folder, then run pmbootstrap init again and choose another work folder (e.g. ~/.local/var/pmbootstrap).

After that it should work - if this still does not resolve it, please open up an issue in the GitLab issue tracker.

can't create /dev/null: Permission denied

Make sure that your working directory is mounted correctly. This can happen when working over NFS, or when you mounted the drive your working directory is on trough the file explorer (if that's the case, umount it, and mount it manually). This can probably happen on every drive mounted with nosuid or similar.

cp: write error: No space left on device (during file transfer from chrooted environment to target image)

This issue is triggered when pmbootstrap determines the wrong size for the target image, as it is unfortunately very hard to calculate the correct size independent of the filesystem on which the chroot resides. Some examples for this: #1916; #928 where more elaborate information can be found.

While fixing the problem seems impossible for every configuration, there is a workaround available (#1904 and its associated merge request !1989): Use the -E/--extra-space option of pmbootstrap to specify extra space on the resulting image file in MB. Just play around with the value until it works for you. You can specify the value also through pmbootstrap init in its Additional options.

"ERROR: Is $builddir set correctly?" for linux-* package

Check where the kernel sources archive has the kernel-related code stored and change the builddir value accordingly.

BAD signature errors from apk

If apk throws BAD signature errors, oftentimes the dl-cdn mirror from Alpine didn't cache the package properly. The solution is then to send a PURGE to the complete URL of the package (open dl-cdn.alpinelinux.org/alpine in your browser and find the complete URL). Like this:

curl -X PURGE https://dl-cdn.alpinelinux.org/alpine/edge/testing/x86_64/plasma-dialer-0.2-r0.apk

This also works if bpo is failing with a BAD signature error.

Build errors

Errors with grsec-based kernel on the host system

See: Troubleshooting:grsec

Errors with ecryptfs

When your "work" folder is inside an ecryptfs mounted folder, /dev/random may not work inside the chroot and you may have other issues. Try to move your work folder to another location, you can use /tmp to test if this resolves the issue. You can change the location in pmbootstrap init. See also: #61. (This doesn't seem to be an issue with newer versions of ecryptfs anymore.)

musl-*/busybox-static-* fails to build with a 404 error

Run pmbootstrap aportgen musl-armhf musl-aarch64 musl-x86_64. The reason why this fails from time to time is, that we use Alpine's binary musl lib c package for armhf/aarch64 in the native chroot, to get the cross-compiler working. So the download URL inside the aports/cross/musl-*/APKBUILD file must be in sync with the upstream version. This pmbootstrap command safely gets the new version, uses apk to verify the signature of the package, generates the hash, and updates the APKBUILD file. To make sure, that pmbootstrap aportgen receives the latest musl version, delete your HTTP cache right before by executing this command: pmbootstrap zap -hc. Don't bother making a pull-request for this, as the fix is really trivial (see this commit). Just report it as issue, if it happens.

sh: can't create /proc/sys/fs/binfmt_misc/register: nonexistent directory

This issue is known to happen in Alpine Virt hardened and Void Linux. These commands should fix it:

sudo modprobe binfmt_misc
sudo mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

If it still does not work, it is probably not supported by your host Linux distribution's kernel (CONFIG_BINFMT_MISC). Try to use another kernel.

du: unrecognized option: summarize

Update pmbootstrap.

Packaging for aarch64: ./config.guess: unable to guess system type

This happens in packages built with autoconf (./configure; make; make install), when it ships an outdated config.guess file in the source code. Adjust your APKBUILD's prepare() function like done in !698.

ERROR: ... BAD archive

If you pmbootstrap log shows something similar to the following:

(005853) [22:25:47] (native) % apk --no-progress add -u qemu-user-static-repack qemu-user-static-repack-binfmt
WARNING: Ignoring /mnt/pmbootstrap-packages/x86_64/APKINDEX.tar.gz: No such file or directory
(1/2) Installing qemu-user-static-repack (2.8-r9)
ERROR: qemu-user-static-repack-2.8-r9: BAD archive
(2/2) Installing qemu-user-static-repack-binfmt (2.8-r9)
1 error; 20 MiB in 36 packages
(005853) [22:25:53] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

...then downloading a package (in this case qemu-user-static-repack) failed. Please delete the broken package and try it again:

$ cd ~/.local/var/pmbootstrap/ # go into your work folder, as specified in 'pmbootstrap init'
$ sudo rm -v cache_apk_*/qemu-user-static-repack-* # adjust the failing package name accordingly

CROSS_COMPILE_ARM32 not defined or empty, the compat vDSO will not be built

Disable CONFIG_COMPAT_VDSO in kernel config.

../tools/dct/DrvGen: No such file or directory

This error is specific to Mediatek kernels and it occures when DrvGen tries to load libc6.so library it has been linked against but Alpine chroot doesn't support libc. One of the solutions is to copy python3dct.patch to your linux-... directory and activate it in the APKBUILD. This patch replaces precompiled DrvGen with a python3 version.

boot-deploy exits with ERROR: "deviceinfo_header_version is 2, but 'deviceinfo_dtb' is missing. Set 'deviceinfo_dtb'" when compiling downstream kernel

Deviceinfo file is generated with deviceinfo_header_version set to 2 and boot-deploy expects a dtb file to be to be specified in deviceinfo. Verify that the deviceinfo_dtb entry in your deviceinfo is not empty or points to a non-existent dtb/dts in your kernel source.

Errors while running pmbootstrap on non-x86_64 arches

If you get Can't build 'binutils-or1k-elf' for architecture aarch64, see Run pmbootstrap on non-x86_64 arches.

Using pmbootstrap behind a proxy

If you rely on proxy variables like HTTP_PROXY to use pmbootstrap, make sure you use at least version 2.0.0. If your proxy environment variables are set to a proxy hostname that in in /etc/hosts, you will need to replace it with the IP address it resolves to.

Troubleshooting steps kept around for historical reference

Note You're most likely not going to encounter these errors. If you do, file an issue in the appropriate repository.

pmbootstrap builds packages that it should download from the binary repo/that are already built

Note Most likely, this is issue #1023!

Try to disable the "timestamp based rebuilds" feature in "pmbootstrap init" as a workaround and please report this as a bug. We need the following for the bug report:

  • *enable* timestamp based rebuilds
  • pmbootstrap -v install (let it run until it starts building the package it should not build)
  • pmbootstrap log (we need the log from here!)

Invalid pkgver '${_pkgbase}_git${_pkgsnap}'

If you get the following error, your pmbootstrap needs to be upgraded:

ERROR: Invalid pkgver '${_pkgbase}_git${_pkgsnap}' in APKBUILD: /home/user/.local/var/pmbootstrap/cache_git/pmaports/cross/gcc-aarch64/APKBUILD