Troubleshooting on-device issues/Booting problems

< Troubleshooting on-device issues‎

This page describes ways to debug new postmarketOS device ports in which the kernel compiles, but shows no output on the screen, fails to respond to telnet/SSH requests, or doesn't boot at all.

As always, please edit this page to make corrections and add useful tips.

Figuring out booting problems

Add rc_logger=YES to /etc/rc.conf to have openrc log all its steps to /var/log/rc.log.

Determine whether boot failed

If the device automatically reboots, then the kernel is not loading, or there is a fatal bug in the kernel. However, even if your device only displays the OEM logo or a black screen, the kernel may have actually loaded successfully.

To discover whether the kernel loaded, connect the device via USB, then run dmesg or lsusb. If the device is listed there, then it booted successfully. (See here for sample output.)

An alternate method is to set an initramfs hook that turns on an LED or the vibration motor when the device boots successfully. One way to do this is by booting the device using pmbootstrap:

$ pmbootstrap initfs hook_add maximum-attention
$ pmbootstrap flasher boot

Note that the device will loop in the initramfs stage with this turned on, so make sure to disable it when you're finished.

You can also check for the init log, pmOS_init.log, in the root directory on the device.

If it turns out that your kernel loaded and your device has an IP address, then you probably only need to get your display working.

If boot fails

If the tests above indicate that boot failed, read /proc/last_kmsg or /sys/fs/pstore/console-ramoops on the device to try to find the point of failure. If your device has TWRP installed, you can access them from the command line either by opening a terminal in TWRP, or by using TWRP's ADB interface. For more details, see here and here.

If you find console-ramoops corrupted, cool the device by putting it in a fridge or holding it in front of an AC vent for a few minutes then reboot and check it again. Depending on your device console-ramoops might also get overwritten by a bootloader that you are leveraging to access TWRP right after a crash. A possible workaround is to flash TWRP to the boot partition and your kernel to the recovery partition. To test your kernel you can now boot from recovery and after crashing the device will reboot normally, thereby executing TWRP from where you can access console-ramoops.

Maybe your device requires a special string in boot.img file for booting like this or something like this. You can use a hex editor to see the special string.

If booting from a microSD card

If the initramfs cannot find the boot partition, it will display an error splash screen and wait forever. If the display driver does not work yet, the device will appear to hang with a blank screen, and the SSH server will not come up.

If you get a "bad boot image header" error when booting from a microSD card, the device's fastboot mode might not allow fastboot boot usage. Run fastboot boot boot.img, where boot.img is a recovery image or a working boot image from a known-good ROM. If you get the same error, flash the kernel to the device with pmbootstrap flasher flash_kernel.

Check your device configuration

You may need to revise your device's configuration files and recompile the kernel.

Make sure that you are using the correct defconfig.
If your kernel's APKBUILD file uses arch=armhf, be aware that Alpine Linux' armhf architecture requires ARMv6 with a floating point unit (FPU). (See issue #1203.)
The bootimg file in different ROMs may have different deviceinfo values. Run pmbootstrap bootimg_analyze filename on a bootimg file for your device, and compare the output with the contents of your deviceinfo file. In Android ROMs, this file is called boot.img. Recovery ROMs are themselves bootimg files. If you want to analyze the bootimg of your pmOS ROM, you can extract it with the unpackbootimg command.
If TWRP has been ported to your device, extract the TWRP image file with unpackbootimg, and the initramfs inside it with cpio. Check the rc files in initramfs for useful comments or settings (such as write /sys/class/xxx/yyy/enable 1).
If a similar device already has a postmarketOS port, take a look at its aport to see if it does anything special.
If you have a QCDT device, QCDT#Debugging may be helpful. Use extract-dtb to extract the dtb files appended to the kernel. Compare yours with the dtb files from a known working ROM or recovery OS (such as TWRP).

If the configuration already appears to be correct, try building the kernel with an older version of GCC. If possible, determine which version of GCC would have been used to build the kernel for Android.

Kernel panics with CONFIG_VT enabled

Some Qualcomm devices with a 4.4 release kernel panic on boot with CONFIG_VT=y. If you think you're affected by this simply test booting with CONFIG_VT=n and see if it yields a blank screen (instead of e.g. just rebooting). Now the kernel should have booted without issues and you'll get telnet / SSH. If disabling VT helped you, re-enable it and follow the steps below:

Revert the following 4 commits from your kernel tree after finding their commit hashes using git log for example and creating patch files for use with your kernel package:

tty: check before stopping kthread (example)
tty: move tty_port workqueue to be a kthread (example)
tty: add tty_port_set_policy function (example)
msm_serial_hs: make the Bluetooth tty thread RT (example)

Original source

Try a prebuilt kernel

If your kernel isn't working and you are desperate to boot postmarketOS on your device, you can try loading initramfs with a prebuilt kernel. Hypothetically, this may help you to debug the issue. Also, after booting with a prebuilt kernel you can also try loading the postmarketOS kernel with kexec. See also the guide for building the kernel with Android's build system.

Try a different codebase

If all else fails, it may help to try a different branch or fork. If your device runs Android but the LineageOS kernel does not work, try another branch, or even another AOSP distribution. You can find other compatible ROMs on GitLab, GitHub, the XDA Forums, and various websites.

Network troubleshooting

To check for a network interface, Run ip a or ifconfig. If the pmOS device's network interface is up but it has no IP address, try manually running a DHCP client. If that fails, check your kernel configuration.

Some downstream kernels require writing a value to sysfs before the network comes up. Example from device-htc-bravo:

echo 4 > /sys/devices/platform/android_usb/usb_function_switch

By setting an initramfs hook, you can dump the sysfs to a partition on the device, in order to compare it with another system (such as TWRP). See the initramfs hooks examples.

Can ping device, but SSH says "connection refused"

First, make sure you are pinging the right device (ssh <user>@172.16.42.1)

pmOS generates SSH keys during first boot. If the device loses power before it fully boots, you are left with empty files instead of keys. If that happens, reflash the device and make sure that the initramfs script completes before you power-off or reboot.

If the initramfs script completes and you can ping the device, but SSH still doesn't work, log OpenRC's output to kmesg (see issue #1582).

One possible cause is a failed fsck. Try this:

Telnet into initramfs
Mount the root partition
Comment out the boot partition in /etc/fstab
Try to boot (#1511).

If boot succeeds, try reflashing the postmarketOS rootfs to correct the error.

General debugging

/proc/last_kmsg and /sys/fs/pstore/console-ramoops are useful sources of debugging data.

On boot, postmarketOS generates an init log, pmOS_init.log, in the root directory on the device. The init system is OpenRC. See issue #1582 for steps to log OpenRC's output to kmesg.

It's also possible to step through the initramfs script line by line. Install the debug shell initramfs hook to get a telnet shell directly from the initramfs. As soon as you are in the initramfs shell, open init.sh in your browser, and execute it one line at a time to figure out where it fails.

$ . /init_functions.sh
$ . /etc/deviceinfo
(the lines until 'mount_boot_partition' can be skipped, as they have already been executed when the usb-shell hook is running)
$ mount_boot_partition /boot
$ extract_initramfs_extra /boot/initramfs-extra
$ start_charging_mode
…

The functions are all defined in init_functions.sh.

Serial debugging

A serial cable makes it much easier to follow the output of the bootloader and to get the output of dmesg. Unfortunately, there is no standard way to access the UART on mobile devices. An off-the-shelf USB-to-TTL adapter is unlikely to work, so you would probably have to build an interface yourself. Making a serial adapter takes some time, and it isn't even known how to do it for all devices.

If you have a functioning serial interface, you can log OpenRC's output to the serial port by setting RC_LOGGING to true in /etc/rc.conf.

Netconsole debugging

In some cases the kernel may be booting but hanging/crashing before PostmarketOS is fully initialized. If you do not have a serial cable for your device it is possible to get somewhat early boot logs from the kernel over netconsole. To enable netconsole for your device

Enable CONFIG_NETCONSOLE in your kernel configuration to enable netconsole
Enable CONFIG_USB_ETH in your kernel configuration to force the USB port to ethernet mode early in the boot process
Add netconsole parameters to the deviceinfo_kernel_cmdline for your device as described here. Use usb0 for the device name
Execute nc -u -l -p <port> and boot your device

Boot partition not found

If your device uses UFS, check that deviceinfo_rootfs_image_sector_size is set to 4096.

Document your progress

If you haven't already done so, please create a wiki page for your device. On this page, document everything you tried so that others can build on the work you have already done.