Jump to content

Troubleshooting on-device issues/Booting problems: Difference between revisions

From postmarketOS Wiki
Wfranken (talk | contribs)
No edit summary
Knuxify (talk | contribs)
No edit summary
 
(29 intermediate revisions by 19 users not shown)
Line 1: Line 1:
This page is about systematically debugging new postmarketOS device ports, where the kernel successfully compiles, but booting it appears not to be working at all. As always, please extend with everything you find out.
{{subpages|Troubleshooting on-device issues}}


== Preparation ==
This page describes ways to debug new postmarketOS device ports in which the kernel compiles, but shows no output on the screen, fails to respond to telnet/SSH requests, or doesn't boot at all.
* Make sure you have created a [[Help:Device_Page|wiki page for your device]]
* Document everything you have tried out there, so when the next person with the same device comes along, it is possible to start from the work you have already done.


== General tips ==
As always, please edit this page to make corrections and add useful tips.
* When everything fails: '''Try different kernel sources for your device'''
 
** Ideally you would use the kernel sources, that have been used to build a known working (Android) ROM or recovery image (e.g. TWRP) for your device
=== Figuring out booting problems ===
*** Make sure that you are really able to boot that known working kernel with Android/TWRP etc.
Add <code>rc_logger=YES</code> to <code>/etc/rc.conf</code> to have openrc log all its steps to <code>/var/log/rc.log</code>.
** For Android devices, if the usual LineageOS kernel does not work, try to find another one through GitHub search and on XDA
 
* [https://wiki.postmarketos.org/wiki/Matrix_and_IRC Ask in IRC/Matrix]
== Determine whether boot failed ==
* [https://github.com/postmarketOS/pmbootstrap/issues Open an issue] (and link to it from your device's wiki page)
If the device automatically reboots, then the kernel is not loading, or there is a fatal bug in the kernel. However, even if your device only displays the OEM logo or a black screen, the kernel may have actually loaded successfully.
* Make a [[serial cable]] and look at the output of the bootloader
 
* If there is a similar device, that already has a postmarketOS port, take a look at its aport to see if it does anything special.
To discover whether the kernel loaded, connect the device via USB, then run <tt>dmesg</tt> or <tt>lsusb</tt>. If the device is listed there, then it booted successfully. (See [[USB_Network#Making_sure_the_USB_is_detected|here]] for sample output.)
* Extract the TWRP boot.img, and the initramfs inside it (with <code>unpackbootimg</code> and [https://superuser.com/a/813899 cpio])
 
** Look through all the <code>rc</code> files, there might be some useful comment or some line like <code>write /sys/class/xxx/yyy/enable 1</code>
An alternate method is to set an initramfs hook that turns on an LED or the vibration motor when the device boots successfully. One way to do this is by booting the device using pmbootstrap:


== Does the kernel boot? ==
Even if it looks like your device does nothing (e.g. still displaying the OEM logo or displaying a black screen), it can be that the kernel is actually booted. So unless you know for sure that it does not boot at all (because it directly reboots), try the following to get clarity:
* Check if you have a new network interface (run <code>ip a</code> or <code>ifconfig</code>)
** If you have the network interface, but no IP, try [[USB_Network#Troubleshooting|manually running a DHCP client]]
* Check the output of <code>lsusb</code> (before plugging the device in, then while it is in the bootloader mode, and finally after you have booted pmOS)
* Check what <code>dmesg</code> says after booting
** [[USB_Network#Troubleshooting|Example output for successfully booted device]]
* Run with an initramfs hook, that signalizes a successful boot with LEDs and the vibrator:
<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
$ pmbootstrap initfs hook_add maximum-attention
$ pmbootstrap initfs hook_add maximum-attention
$ pmbootstrap flasher boot
$ pmbootstrap flasher boot
(or however you boot the kernel/initramfs)
</syntaxhighlight>
</syntaxhighlight>
* Try to read dmesg after rebooting:
** Immediately entering flash mode after the reboot
** Boot a custom recovery such as TWRP
** read <code>/proc/last_kmsg</code> or <code>/sys/fs/pstore/console-ramoops</code> to see if it booted at all, and what caused the reboot
** more on that topic is [[Mainlining_FAQ#Writing_dmesg_to_RAM_and_reading_it_out_after_reboot|here]] and [http://docs.halium.org/en/latest/porting/debug-build/dmesg.html here]


Depending on whether your kernel is booted or not, read on in the appropriate section below. If it turns out, that your kernel is booted, and you have a network interface and get an IP address, then you probably only need to [[Porting_to_a_new_device#Display|get your display working]].
Note that the device will loop in the initramfs stage with this turned on, so make sure to disable it when you're finished.
 
You can also check for the init log, <tt>pmOS_init.log</tt>, in the root directory on the device.  
 
If it turns out that your kernel loaded ''and'' your device has an IP address, then you probably only need to [[Porting to a new device#Display|get your display working]].


== Kernel boots without network interface ==
=== If boot fails ===
* [[USB_Network#Kernel_configuration_parameters|Check your kernel configuration]].
If the tests above indicate that boot failed, read <code>/proc/last_kmsg</code> or <code>/sys/fs/pstore/console-ramoops</code> on the device to try to find the point of failure. If your device has TWRP installed, you can access them from the command line either by opening a terminal in TWRP, or by using TWRP's ADB interface. For more details, see [[Mainlining_FAQ#Writing_dmesg_to_RAM_and_reading_it_out_after_reboot|here]] and [http://docs.halium.org/en/latest/porting/debug-build/dmesg.html here].
* Some downstream kernels require writing a value to sysfs before the network comes up. Example from <code>device-htc-bravo</code>:
 
If you find <code>console-ramoops</code> corrupted, cool the device by putting it in a fridge or holding it in front of an AC vent for a few minutes then reboot and check it again.
Depending on your device <code>console-ramoops</code> might also get overwritten by a bootloader that you are leveraging to access TWRP right after a crash. A possible workaround is to flash TWRP to the boot partition and your kernel to the recovery partition. To test your kernel you can now boot from recovery and after crashing the device will reboot normally, thereby executing TWRP from where you can access <code>console-ramoops</code>.
 
Maybe your device requires a special string in boot.img file for booting like [https://gitlab.postmarketos.org/postmarketOS/boot-deploy/-/merge_requests/67/diffs this] or something like this. You can use a hex editor to see the special string.
 
==== If booting from a microSD card ====
If the initramfs cannot find the [[Partition-layout|boot partition]], it will display an error splash screen and wait forever. If the display driver does not work yet, the device will appear to hang with a blank screen, and the SSH server will not come up.
 
If you get a "<code>bad boot image header</code>" error when booting from a microSD card, the device's fastboot mode might not allow <code>fastboot boot</code> usage. Run <code>fastboot boot ''boot.img''</code>, where <code>''boot.img''</code> is a recovery image or a working boot image from a known-good ROM. If you get the same error, flash the kernel to the device with <code>pmbootstrap flasher flash_kernel</code>.
 
==== Check your device configuration ====
You may need to revise your device's configuration files and recompile the kernel.
 
* Make sure that you are using the correct [[Porting to a new device#Defconfig|defconfig]].
* If your kernel's APKBUILD file uses <code>arch=armhf</code>, be aware that Alpine Linux' <tt>armhf</tt> architecture requires ARMv6 with a floating point unit (FPU). (See issue {{issue|1203}}.)
* The bootimg file in different ROMs may have different [[deviceinfo]] values. Run <code>pmbootstrap bootimg_analyze ''filename''</code> on a bootimg file for your device, and compare the output with the contents of your deviceinfo file. In Android ROMs, this file is called <code>boot.img</code>. Recovery ROMs are themselves bootimg files. If you want to analyze the bootimg of your pmOS ROM, you can extract it with the <code>unpackbootimg</code> command.
* If TWRP has been ported to your device, extract the TWRP image file with <code>unpackbootimg</code>, and the initramfs inside it with [https://superuser.com/a/813899 cpio]. Check the <tt>rc</tt> files in initramfs for useful comments or settings (such as <code>write /sys/class/xxx/yyy/enable 1</code>).
* If a similar device already has a postmarketOS port, take a look at its aport to see if it does anything special.
* If you have a [[QCDT#Does_my_device_need_a_QCDT_boot.img_file.3F|QCDT device]], [[QCDT#Debugging]] may be helpful. Use <code>extract-dtb</code> to extract the [[dtb]] files appended to the kernel. Compare yours with the dtb files from a known working ROM or recovery OS (such as TWRP).
 
If the configuration already appears to be correct, try building the kernel with an [[Vendor kernel specific package#GCC_version|older version of GCC]]. If possible, determine which version of GCC would have been used to build the kernel for Android.
 
==== Kernel panics with CONFIG_VT enabled ====
Some Qualcomm devices with a 4.4 release kernel panic on boot with <code>CONFIG_VT=y</code>. If you think you're affected by this simply test booting with <code>CONFIG_VT=n</code> and see if it yields a blank screen (instead of e.g. just rebooting). Now the kernel should have booted without issues and you'll get telnet / SSH. If disabling <code>VT</code> helped you, re-enable it and follow the steps below:
 
Revert the following 4 commits from your kernel tree after finding their commit hashes using <code>git log</code> for example and creating patch files for use with your kernel package:
 
* <code>tty: check before stopping kthread</code> ([https://github.com/moto-SDM660/android_kernel_motorola_sdm660/commit/2fabaace47075980d0250b0459925bab05adb30a example])
* <code>tty: move tty_port workqueue to be a kthread</code> ([https://github.com/moto-SDM660/android_kernel_motorola_sdm660/commit/6e9ee368bf929a8c9ddafff1d2bb8d0bcca55117 example])
* <code>tty: add tty_port_set_policy function</code> ([https://github.com/moto-SDM660/android_kernel_motorola_sdm660/commit/3024fed4d868368c5e92063ecd5a93e34fd65bed example])
* <code>msm_serial_hs: make the Bluetooth tty thread RT</code> ([https://github.com/moto-SDM660/android_kernel_motorola_sdm660/commit/f96334676cd13f3fa0bbeecda3b97f1bf6254c2d example])
 
[https://github.com/ubports/porting-notes/wiki/Halium-9#qualcomm-devices-with-linux-kernel-44-panic-with-config_vt-enabled Original source]
 
==== Try a prebuilt kernel ====
If your kernel isn't working and you are desperate to boot postmarketOS on your device, you can try loading initramfs with a [[Using prebuilt kernels|prebuilt kernel]]. Hypothetically, this may help you to debug the issue. Also, after booting with a prebuilt kernel you can also try loading the postmarketOS kernel with <tt>kexec</tt>. See also the guide for [[Troubleshooting:kernel#Building_the_kernel_with_Android.27s_build_system|building the kernel with Android's build system]].
 
==== Try a different codebase ====
If all else fails, it may help to try a different branch or fork. If your device runs Android but the LineageOS kernel does not work, try another branch, or even another AOSP distribution. You can find other compatible ROMs on GitLab, GitHub, the [https://forum.xda-developers.com/ XDA Forums], and various websites.
 
== Network troubleshooting ==
To check for a network interface, Run <code>ip a</code> or <code>ifconfig</code>. If the pmOS device's network interface is up but it has no IP address, try [[USB_Network#Troubleshooting|manually running a DHCP client]]. If that fails, [[USB Network#Kernel configuration parameters|check your kernel configuration]].
 
Some downstream kernels require writing a value to <tt>sysfs</tt> before the network comes up. Example from <code>device-htc-bravo</code>:
<syntaxhighlight lang="shell">
<syntaxhighlight lang="shell">
echo 4 > /sys/devices/platform/android_usb/usb_function_switch
echo 4 > /sys/devices/platform/android_usb/usb_function_switch
</syntaxhighlight>
</syntaxhighlight>
* It is possible to dump the sysfs from an initramfs hook to a partition on the device, and read it out and comparing with another system (such as TWRP).
* See the [[Troubleshooting:boot:initfshooks|initramfs hooks examples]].


== postmarketOS is stuck somewhere in the boot process ==
By setting an initramfs hook, you can dump the sysfs to a partition on the device, in order to compare it with another system (such as TWRP). See the [[Troubleshooting:boot:initfshooks|initramfs hooks examples]].
* Do you have the system partition flashed to your device (or put it on an SD card and inserted it)?
 
** If the initramfs can not find the [[Partition-layout|boot partition]], it will display an error splash screen and wait forever.
=== Can ping device, but SSH says "connection refused" ===
** That means, if your screen does not work yet, it will just appear to be stuck and the SSH server will not come up.
 
First, make sure you are pinging the right device (<code>ssh <user>@172.16.42.1</code>)
 
pmOS generates SSH keys during first boot. If the device loses power before it fully boots, you are left with empty files instead of keys. If that happens, reflash the device and make sure that the initramfs script completes before you power-off or reboot.
 
If the initramfs script completes and you can ping the device, but SSH still doesn't work, log OpenRC's output to <code>kmesg</code> (see issue {{issue|1582}}).
 
One possible cause is a failed fsck. Try this:
 
# [[Inspecting_the_initramfs|Telnet into initramfs]]
# Mount the root partition
# Comment out the boot partition in <code>/etc/fstab</code>
# Try to boot ({{issue|1511}}).
 
If boot succeeds, try reflashing the postmarketOS rootfs to correct the error.
 
== General debugging ==
<code>/proc/last_kmsg</code> and <code>/sys/fs/pstore/console-ramoops</code> are useful sources of debugging data.
 
On boot, postmarketOS generates an init log, <code>pmOS_init.log</code>, in the root directory on the device. The init system is OpenRC. See issue {{issue|1582}} for steps to log OpenRC's output to <code>kmesg</code>.
 
It's also possible to step through the initramfs script line by line. Install the [[Inspecting the initramfs|debug shell]] initramfs hook to get a telnet shell directly from the initramfs. As soon as you are in the initramfs shell, open [https://gitlab.postmarketos.org/postmarketOS/pmaports/-/blob/master/main/postmarketos-initramfs/init.sh init.sh] in your browser, and execute it one line at a time to figure out where it fails.


=== Run the initramfs script line by line ===
Install the [[Inspecting_the_initramfs|debug shell]] initramfs hook to get a telnet shell directly from the initramfs. As soon as you are in the initramfs shell, open [https://gitlab.com/postmarketOS/pmaports/blob/master/main/postmarketos-mkinitfs/init.sh.in init.sh.in] in your browser, and execute it line by line to figure out where it fails. The functions you are calling (<code>mount_boot_partition</code> etc.) are defined [https://gitlab.com/postmarketOS/pmaports/blob/master/main/postmarketos-mkinitfs/init_functions.sh init_functions.sh].
<syntaxhighlight lang="shell-session">
<syntaxhighlight lang="shell-session">
$ . /init_functions.sh
$ . /init_functions.sh
$ . /etc/deviceinfo
$ . /etc/deviceinfo
(the lines until 'mount_boot_partition' can be skipped, as they have already been executed when the usb-shell hook is running)
(the lines until 'mount_boot_partition' can be skipped, as they have already been executed when the usb-shell hook is running)
$ mount_boot_partition
$ mount_boot_partition /boot
$ extract_initramfs_extra /boot/initramfs-*-extra
$ extract_initramfs_extra /boot/initramfs-extra
$ start_charging_mode
$ start_charging_mode
...
</syntaxhighlight>
</syntaxhighlight>


=== Can ping to device, but SSH says "connection refused" ===
The functions are all defined in [https://gitlab.postmarketos.org/postmarketOS/pmaports/blob/master/main/postmarketos-mkinitfs/init_functions.sh init_functions.sh].


It might be that fsck fails on a partition: try telnetting into initramfs and comment out the boot partition, then try booting (https://github.com/postmarketOS/pmbootstrap/issues/1511)
=== Serial debugging ===
A [[Serial debugging|serial cable]] makes it much easier to follow the output of the bootloader and to get the output of <tt>dmesg</tt>. Unfortunately, there is no standard way to access the UART on mobile devices. An off-the-shelf USB-to-TTL adapter is unlikely to work, so you would probably have to build an interface yourself. Making a serial adapter takes some time, and it isn't even known how to do it for all devices.


=== If that does not help ===
If you have a functioning serial interface, you can log OpenRC's output to the serial port by setting <tt>RC_LOGGING</tt> to <tt>true</tt> in <code>/etc/rc.conf</code>.
* If you have powered off the device while it was not fully booted yet, it could be that you killed it while it was generating SSH keys. That means, it may have empty files instead of actual keys, so they will not get generated again, but also SSHD fails to start. When this could be the case with your issue, try to flash again and make sure that you do not turn off the device during the boot process this time.
* If the initramfs script seems to be working fine, chances are that the error happens as OpenRC runs. See {{github|1582}} for steps to log OpenRC's output to <code>kmesg</code>


== Kernel doesn't even boot ==
=== Netconsole debugging ===
* Check the output of the bootloader with a [[serial debugging|serial cable]] if you have one.
In some cases the kernel may be booting but hanging/crashing before PostmarketOS is fully initialized. If you do not have a serial cable for your device it is possible to get somewhat early boot logs from the kernel over netconsole. To enable netconsole for your device
** Making such a cable takes some time, and it isn't even known how to do it for all devices. So if you don't have such a cable, try other methods.
 
* Double-check that you have used the right [[Porting_to_a_new_device#Defconfig|defconfig]]
# Enable CONFIG_NETCONSOLE in your kernel configuration to enable netconsole
* Android bootimg issues:
# Enable CONFIG_USB_ETH in your kernel configuration to force the USB port to ethernet mode early in the boot process
** Make sure that <code>pmbootstrap bootimg_analyze</code> outputs the same values, that you have put in your [[deviceinfo]]
# Add netconsole parameters to the deviceinfo_kernel_cmdline for your device as described [https://www.kernel.org/doc/html/latest/networking/netconsole.html here]. Use usb0 for the device name
** Analyze a bootimg file, for which you have verified that it boots on your device (with TWRP or Android).
# Execute <code>nc -u -l -p <port></code> and boot your device
** If you [[QCDT#Does_my_device_need_a_QCDT_boot.img_file.3F|have a QCDT device]], check out [[QCDT#Debugging|QCDT debugging]]
 
** use <code>extract-dtb</code> and <code>unpackbootimg</code> (both packaged in postmarketOS) to compare the [[dtb]] files appended to the kernel and boot.img file (compare the postmarketOS version against the one from a known working ROM or recovery OS such as TWRP)
== Boot partition not found ==
* A less systematic approach is booting the postmarketOS initramfs with a [[Using_prebuilt_kernels|prebuilt kernel]]. This is listed for the sake of completeness, but building another kernel from source (like in the [[Porting_to_a_new_device|porting guide]]) is probably more fruitful and over a much cleaner approach.
 
** If that works, maybe you can debug the issue from that point somehow
If your device uses UFS, check that <code>deviceinfo_rootfs_image_sector_size</code> is set to 4096.
** Or even load the postmarketOS kernel with kexec
 
** There's a also a guide for [[Troubleshooting:kernel#Building_the_kernel_with_Android.27s_build_system|building the kernel with Android's build system]]
== Document your progress ==
* Alpine Linux' <code>armhf</code> architecture requires armv6 with FPU {{github|1203}}
If you haven't already done so, please [[Help:Device_Page|create a wiki page for your device]]. On this page, document everything you tried so that others can build on the work you have already done.


== See also ==
== See also ==
* [[Boot process]]
* [[Boot process]]
* {{github|1206}} HTC One M8 not booting
* [[Mainlining FAQ]]
* {{github|1203}} HTC Wildfire not booting
* [[Matrix and IRC]] (to get help via live chat)
* {{github|99}} HTC Droid DNA (dlx) not booting
* [[Vendor kernel specific package]]
* [https://developer.qualcomm.com/qfile/28821/lm80-p0436-1_little_kernel_boot_loader_overview.pdf Qualcomm LK Boot Loader Overview]
 
== Further reading ==
* [https://developer.qualcomm.com/qfile/28821/lm80-p0436-1_little_kernel_boot_loader_overview.pdf Qualcomm Little Kernel Boot Loader Overview] (PDF)
* Issue {{issue|99}} HTC Droid DNA (dlx) not booting
* Issue {{issue|1206}} HTC One M8 not booting
* Issue {{issue|1203}} HTC Wildfire not booting
 
[[Category:Troubleshooting|boot]]

Latest revision as of 20:13, 16 December 2024

This page describes ways to debug new postmarketOS device ports in which the kernel compiles, but shows no output on the screen, fails to respond to telnet/SSH requests, or doesn't boot at all.

As always, please edit this page to make corrections and add useful tips.

Figuring out booting problems

Add rc_logger=YES to /etc/rc.conf to have openrc log all its steps to /var/log/rc.log.

Determine whether boot failed

If the device automatically reboots, then the kernel is not loading, or there is a fatal bug in the kernel. However, even if your device only displays the OEM logo or a black screen, the kernel may have actually loaded successfully.

To discover whether the kernel loaded, connect the device via USB, then run dmesg or lsusb. If the device is listed there, then it booted successfully. (See here for sample output.)

An alternate method is to set an initramfs hook that turns on an LED or the vibration motor when the device boots successfully. One way to do this is by booting the device using pmbootstrap:

$ pmbootstrap initfs hook_add maximum-attention
$ pmbootstrap flasher boot

Note that the device will loop in the initramfs stage with this turned on, so make sure to disable it when you're finished.

You can also check for the init log, pmOS_init.log, in the root directory on the device.

If it turns out that your kernel loaded and your device has an IP address, then you probably only need to get your display working.

If boot fails

If the tests above indicate that boot failed, read /proc/last_kmsg or /sys/fs/pstore/console-ramoops on the device to try to find the point of failure. If your device has TWRP installed, you can access them from the command line either by opening a terminal in TWRP, or by using TWRP's ADB interface. For more details, see here and here.

If you find console-ramoops corrupted, cool the device by putting it in a fridge or holding it in front of an AC vent for a few minutes then reboot and check it again. Depending on your device console-ramoops might also get overwritten by a bootloader that you are leveraging to access TWRP right after a crash. A possible workaround is to flash TWRP to the boot partition and your kernel to the recovery partition. To test your kernel you can now boot from recovery and after crashing the device will reboot normally, thereby executing TWRP from where you can access console-ramoops.

Maybe your device requires a special string in boot.img file for booting like this or something like this. You can use a hex editor to see the special string.

If booting from a microSD card

If the initramfs cannot find the boot partition, it will display an error splash screen and wait forever. If the display driver does not work yet, the device will appear to hang with a blank screen, and the SSH server will not come up.

If you get a "bad boot image header" error when booting from a microSD card, the device's fastboot mode might not allow fastboot boot usage. Run fastboot boot boot.img, where boot.img is a recovery image or a working boot image from a known-good ROM. If you get the same error, flash the kernel to the device with pmbootstrap flasher flash_kernel.

Check your device configuration

You may need to revise your device's configuration files and recompile the kernel.

  • Make sure that you are using the correct defconfig.
  • If your kernel's APKBUILD file uses arch=armhf, be aware that Alpine Linux' armhf architecture requires ARMv6 with a floating point unit (FPU). (See issue #1203.)
  • The bootimg file in different ROMs may have different deviceinfo values. Run pmbootstrap bootimg_analyze filename on a bootimg file for your device, and compare the output with the contents of your deviceinfo file. In Android ROMs, this file is called boot.img. Recovery ROMs are themselves bootimg files. If you want to analyze the bootimg of your pmOS ROM, you can extract it with the unpackbootimg command.
  • If TWRP has been ported to your device, extract the TWRP image file with unpackbootimg, and the initramfs inside it with cpio. Check the rc files in initramfs for useful comments or settings (such as write /sys/class/xxx/yyy/enable 1).
  • If a similar device already has a postmarketOS port, take a look at its aport to see if it does anything special.
  • If you have a QCDT device, QCDT#Debugging may be helpful. Use extract-dtb to extract the dtb files appended to the kernel. Compare yours with the dtb files from a known working ROM or recovery OS (such as TWRP).

If the configuration already appears to be correct, try building the kernel with an older version of GCC. If possible, determine which version of GCC would have been used to build the kernel for Android.

Kernel panics with CONFIG_VT enabled

Some Qualcomm devices with a 4.4 release kernel panic on boot with CONFIG_VT=y. If you think you're affected by this simply test booting with CONFIG_VT=n and see if it yields a blank screen (instead of e.g. just rebooting). Now the kernel should have booted without issues and you'll get telnet / SSH. If disabling VT helped you, re-enable it and follow the steps below:

Revert the following 4 commits from your kernel tree after finding their commit hashes using git log for example and creating patch files for use with your kernel package:

  • tty: check before stopping kthread (example)
  • tty: move tty_port workqueue to be a kthread (example)
  • tty: add tty_port_set_policy function (example)
  • msm_serial_hs: make the Bluetooth tty thread RT (example)

Original source

Try a prebuilt kernel

If your kernel isn't working and you are desperate to boot postmarketOS on your device, you can try loading initramfs with a prebuilt kernel. Hypothetically, this may help you to debug the issue. Also, after booting with a prebuilt kernel you can also try loading the postmarketOS kernel with kexec. See also the guide for building the kernel with Android's build system.

Try a different codebase

If all else fails, it may help to try a different branch or fork. If your device runs Android but the LineageOS kernel does not work, try another branch, or even another AOSP distribution. You can find other compatible ROMs on GitLab, GitHub, the XDA Forums, and various websites.

Network troubleshooting

To check for a network interface, Run ip a or ifconfig. If the pmOS device's network interface is up but it has no IP address, try manually running a DHCP client. If that fails, check your kernel configuration.

Some downstream kernels require writing a value to sysfs before the network comes up. Example from device-htc-bravo:

echo 4 > /sys/devices/platform/android_usb/usb_function_switch

By setting an initramfs hook, you can dump the sysfs to a partition on the device, in order to compare it with another system (such as TWRP). See the initramfs hooks examples.

Can ping device, but SSH says "connection refused"

First, make sure you are pinging the right device (ssh <user>@172.16.42.1)

pmOS generates SSH keys during first boot. If the device loses power before it fully boots, you are left with empty files instead of keys. If that happens, reflash the device and make sure that the initramfs script completes before you power-off or reboot.

If the initramfs script completes and you can ping the device, but SSH still doesn't work, log OpenRC's output to kmesg (see issue #1582).

One possible cause is a failed fsck. Try this:

  1. Telnet into initramfs
  2. Mount the root partition
  3. Comment out the boot partition in /etc/fstab
  4. Try to boot (#1511).

If boot succeeds, try reflashing the postmarketOS rootfs to correct the error.

General debugging

/proc/last_kmsg and /sys/fs/pstore/console-ramoops are useful sources of debugging data.

On boot, postmarketOS generates an init log, pmOS_init.log, in the root directory on the device. The init system is OpenRC. See issue #1582 for steps to log OpenRC's output to kmesg.

It's also possible to step through the initramfs script line by line. Install the debug shell initramfs hook to get a telnet shell directly from the initramfs. As soon as you are in the initramfs shell, open init.sh in your browser, and execute it one line at a time to figure out where it fails.

$ . /init_functions.sh
$ . /etc/deviceinfo
(the lines until 'mount_boot_partition' can be skipped, as they have already been executed when the usb-shell hook is running)
$ mount_boot_partition /boot
$ extract_initramfs_extra /boot/initramfs-extra
$ start_charging_mode

The functions are all defined in init_functions.sh.

Serial debugging

A serial cable makes it much easier to follow the output of the bootloader and to get the output of dmesg. Unfortunately, there is no standard way to access the UART on mobile devices. An off-the-shelf USB-to-TTL adapter is unlikely to work, so you would probably have to build an interface yourself. Making a serial adapter takes some time, and it isn't even known how to do it for all devices.

If you have a functioning serial interface, you can log OpenRC's output to the serial port by setting RC_LOGGING to true in /etc/rc.conf.

Netconsole debugging

In some cases the kernel may be booting but hanging/crashing before PostmarketOS is fully initialized. If you do not have a serial cable for your device it is possible to get somewhat early boot logs from the kernel over netconsole. To enable netconsole for your device

  1. Enable CONFIG_NETCONSOLE in your kernel configuration to enable netconsole
  2. Enable CONFIG_USB_ETH in your kernel configuration to force the USB port to ethernet mode early in the boot process
  3. Add netconsole parameters to the deviceinfo_kernel_cmdline for your device as described here. Use usb0 for the device name
  4. Execute nc -u -l -p <port> and boot your device

Boot partition not found

If your device uses UFS, check that deviceinfo_rootfs_image_sector_size is set to 4096.

Document your progress

If you haven't already done so, please create a wiki page for your device. On this page, document everything you tried so that others can build on the work you have already done.

See also

Further reading