Troubleshooting on-device issues

From postmarketOS Wiki
Hint If you get stuck or need assistance, you can join our Matrix/IRC chats (use the #postmarketOS channel for general support questions).

This page provides an overview of methods for troubleshooting on-device issues, with a focus on existing ports.

Navigating this page

You can browse the different sections of this page by clicking the relevant links in the table of contents, located on the sidebar, or click one of the following quick links:

#General information · #Boot issues · #Interface issues · #Application/general usage issues · #Package management (apk) issues

Other troubleshooting pages

Searching for specific issues

If you want to search for a specific section or error, you can use your browser's "find in page" option:

  • Firefox: ≡ Menu → Find in page
  • Chromium: ≡ Menu → Find and edit → Find

If you have a keyboard connected, you can also use Ctrl+F to open the search bar.

Note about command examples

On this wiki, you may find commands prefixed with $ or #. These characters shouldn't be copied - they indicate whether that command should be run as a regular user ($), or as root (#; in which case, you may need to prefix the command with sudo).

For example, the following tells you to execute the command "ls" as a regular user: 

$ ls

The following tells you to execute the command "apk update" as root (sudo apk update): 

# apk update

General information

Checking your device's wiki page

The wiki page for your device contains information about supported features, and could have information about the issue you're experiencing. Find your device's wiki page by looking through the Devices page or using the search feature on the wiki.

The page for the device's chipset/SoC can often be useful as well, as it contains information that more broadly applies to all devices with the chipset. The chipset page is linked on your device's wiki page; click the SoC name in the device infobox.

Making sure your system is up-to-date

In some cases, the issue you're dealing with may have been fixed in a newer version. For this reason, it's worth making sure that your system is up-to-date before you try to troubleshoot your issue.

To upgrade your packages, run: 

# apk upgrade -a
# apk fix

You should also reboot after performing an upgrade.

The -a switch allows for package downgrades, in case a package was downgraded in the upstream repository or if you were testing a package version from outside the upstream repository. The second command fixes any installation errors, if any appear (sometimes this can happen e.g. when a new package replaces another one).

If you're on a stable release, make sure you're on the latest available release. Currently, the latest stable release is v24.06. You can check which release you're on by running the following command: 

$ cat /etc/os-release | grep VERSION

If the version is edge, you're on the edge branch. If you see a version number, check if there isn't a newer version available; if there is, see Upgrade to a newer postmarketOS release.

Looking through reported issues

Most issues in postmarketOS are reported to the pmaports issue tracker. It's worth checking if somebody hasn't already encountered the same issue as you.

If your issue hasn't been reported yet, you are encouraged to report it; this way, developers can be notified about issues they may have missed, and other users will have an easier time troubleshooting their devices. For an overview of where and how to report issues, as well as what information to include, see How to report issues.

Connecting to the device

postmarketOS can set up a network over USB; the device's IP address on this network is 172.16.42.1. This network can be used to connect to the device over SSH. To do so, plug the device into your computer via USB, and run the following command on the computer (replace user with the username you set on the device): 

$ ssh user@172.16.42.1

If you have trouble connecting to your device, see USB Network#Troubleshooting.

You can also upload files over SFTP. Use an SFTP client like FileZilla; your file manager of choice (Nautilus/Files, Dolphin, Thunar...) may also have support for SFTP connections.

Running apps over SSH

Note that you may need to set some environment variables if you're trying to run graphical applications on the device over SSH. For a full list of environment variables set on your system, you can run env in a terminal on the device.

For graphical apps, you'll likely need to set at least the following: 

$ export DISPLAY=:0
$ export WAYLAND_DISPLAY=wayland-0
$ export XDG_RUNTIME_DIR="/run/user/$(id -u)"

TTYescape

Full article: TTYescape

With TTYescape, if your device is frozen, and you need to access the command line without connecting to the device remotely, you can switch to the TTY by holding down Volume Down and pressing Power 3 times.

TTYescape is installed by default on Phosh, Plasma Mobile, GNOME Mobile and sxmo (sway). On other UIs, you can follow the instructions in TTYescape#Installation to install and enable it.

Force reboot

For phones and tablets, you can force-reboot the device by holding down Power for at least 10 seconds. On some devices (e.g. most Samsung devices after 2014) you may also need to hold down the Volume Down button at the same time. In some rarer cases, you may need to hold the power button for a full minute or longer.

You can also run the following command in the terminal: 

$ echo b > /proc/sysrq-trigger

Getting logs

Depending on the issue you're facing, there are different ways to get the logs (messages from apps/services/the kernel etc.):

  • Kernel logs - contain messages printed by the Linux kernel; useful in debugging driver-related issues. Run sudo dmesg or sudo cat /proc/kmsg if the former doesn't work.
    • You can also watch the logs update live with the -w flag: sudo dmesg -w.
  • System log - contains messages from system services. Run logread (note that this also contains kernel logs).
    • You can also watch the system log update live by passing the -F flag (or -f to not show old messages).
    • logread truncates its output by default; you can pass the -n 0 flag to it to show all logs.

There are also other log files that may be of interest depending on what you're troubleshooting; see #Getting logs during boot, #Getting app/interface logs and the linked articles in #Issues with particular components.

Boot issues

See also: Troubleshooting on-device issues/Booting problems

The exact methods of troubleshooting non-booting devices depend on the stage of the boot process that it fails on.

Getting logs during boot

The initramfs is able to expose a PMOS_LOGS storage device when the device is plugged into a PC. This uses the USB mass storage gadget, and as such may not be supported by all kernels (it should work for devices with (close to) mainline kernels, likely won't work on downstream). See the README.txt file in the mounted folder, or read pmaports!4646 for more information.

If not, check if you're able to SSH into the device as described in #Connecting to the device, and fetch the logs that way.

If the boot process fails early enough that SSH is not yet available, it's also possible to enable the debug shell by holding down the Volume Down button during boot - this will enable a telnet shell. See Initramfs/Inspecting for full instructions and more information.

Serial debugging

Full article: Serial debugging

For debugging very early boot issues, including kernel crashes, it's useful to have a way to do serial debugging. This usually involves putting together a special cable that, when connected to a USB-UART converter, will get logs from the device's bootloader and the kernel.

This is mostly meant for more advanced users and developers; in most cases, such cables can't be purchased ready-made. If you're a regular user, this option is an absolute last resort.

Kernel logs from previous boot

On some devices, it is possible to retrieve the kernel log from the previous boot. Depending on the kernel used on the device, this can be accessed from another working install (or from anything running the same kernel, e.g. in TWRP) in one of two ways:

  • By reading /proc/last_kmsg.
  • By mounting the pstore partition and reading its contents: 
    # mkdir /tmp/pstore
# mount -t pstore -o kmsg_bytes=16000 - /tmp/pstore

Device is stuck on manufacturer logo / doesn't get to postmarketOS boot splash

  • Check if the display is listed as working on your device's wiki page.
  • Plug the device into your computer. Are there any messages in dmesg when you plug it in? Does it appear in lsusb?
  • If a device appears, check if it establishes a network connection with your computer. Run ip a on your computer - you should see a "usb0" or similar device, with IP addresses in the range of 172.16.42.XXX.
  • Report an issue to the pmaports repo. If you managed to connect to the device, attach logs; see #Getting logs during boot.

Device is stuck on postmarketOS boot splash

A tablet displaying the "ERROR: Boot partition not found, retrying..." message.

"ERROR: boot partition not found, retrying..."

This usually happens when postmarketOS cannot find the partition that the rootfs has been installed to, or after a broken kernel upgrade (if the module necessary to mount the filesystem on the boot partition is missing). The "boot partition" in this message refers to the boot subpartition of the pmOS partition (see Partition layout).

Usually, this happens if the rootfs hasn't been flashed or was corrupted. Make sure that the rootfs has been flashed correctly, reflash if needed.

If you flashed the rootfs to an SD card or another external storage medium, make sure the correct medium is plugged into the device. If you flashed a pre-built image to the SD card, note that some devices may need additional steps to prepare the system image before flashing the standard image to an SD card - check your device's wiki page.

In some rare cases, this may be a bug in the initramfs code; if so, open an issue in the pmaports repository, and provide logs (see #Getting logs during boot).

UI does not boot, device is stuck on TTY "login:" screen or blank screen with blinking cursor

After the splash screen disappears, the user interface attempts to load. If it returns to a TTY login screen, or you get a blank screen with a blinking/static cursor in the top left corner, then the UI has failed to launch. See #Interface issues for debugging steps.

Application/general usage issues

Issues with particular components

There are separate wiki pages for troubleshooting various components:

If you do not find an answer to your problems there, read on for more generic troubleshooting instructions.

Getting app/interface logs

Logs from the desktop environment/user interface and some running services/apps are stored in the session log. Depending on the display manager, this log is located in:

  • tinydm (Phosh, Plasma Mobile, Sxmo and more) - ~/.local/state/tinydm.log (or $XDG_STATE_HOME/tinydm.log)
  • lightdm (Xfce4, MATE) - ~/.xsession-errors (and ~/.xsession-errors.old)
  • gdm (GNOME, GNOME Mobile) - ~/.cache/gdm/session.log

Errors from background services (bluetoothd for Bluetooth, wpa_supplicant/iwd for WiFi, ModemManager/ofono for the modem, PulseAudio/Pipewire for audio, etc.) can be found in logread (see notes in main #Getting logs session).

Display manager log

For troubleshooting issues with user interfaces not starting up, it's useful to look at the display manager log. Depending on the display manager, this log is located in:

  • tinydm - does not have such a log (it is shared with the session log mentioned above).
  • lightdm - files in /var/log/lightdm/
  • gdm - /var/log/gdm/greeter.log (older logs in /var/log/gdm/greeter.log.X)

For Xorg-based environments (Xfce4, MATE, Sxmo), the Xorg log in /var/log/Xorg.0.log is useful for figuring out startup issues.

Getting logs from apps

Many applications will print out various information when run from the command line.

You can pipe the output of the app/command into a file by doing: 

$ your-command &> logfile

The & in &> ensures that both the standard output and error streams are captured.

Note that when capturing logs with this method, you will no longer see the logs printed out live as the app runs. If you need to see the logs while the app is running, you can run: 

$ your-command 2>&1 | tee logfile

2>&1 here is necessary for tee to pick up messages from the error stream.

If the application closes with a "Segmentation fault" or similar, you'll need to get a backtrace so that the developers can tell what went wrong; see the guide in Troubleshooting/Capturing a backtrace with GDB.

"File not found" while running binary from terminal, despite it being there

This is caused by the binary being linked to a library that isn't installed, and usually appears when trying to run a binary compiled for glibc (the GNU libc) instead of musl (the libc that Alpine Linux/postmarketOS uses).

If it's the latter, try to recompile the software yourself, or ask the developer for a version of the binary compiled against musl, or install gcompat.

Glitchy graphics in apps

Sometimes, due to bugs in the GPU drivers, applications may appear to "glitch out" or crash entirely. This often happens with Firefox, as well as any games or apps requiring 3D graphics.

In particular, Adreno 300 series GPUs (used in e.g. the Qualcomm Snapdragon 410 (MSM8916)) are known to struggle with this.

To work around these issues, you can try the following:

  • Forcing an older OpenGLES version; run the app from the command line with MESA_GLES_VERSION_OVERRIDE=2.0 appname
  • Forcing software rendering; run the app from the command line with LIBGL_ALWAYS_SOFTWARE=true appname

You can find a full list of relevant environment variables in the Mesa documentation.

Persisting custom variables

If setting one of those variables fixes your app, you can persist the changes by adding a manual override in the app's .desktop file.

  • Find the .desktop file for the application; see the files in /usr/share/applications.
  • Copy your app's .desktop file to ~/.local/share/applications/ (you may need to create this directory first).
  • Open the .desktop file in a text editor; find the line that starts with Exec and modify it to match the format of Exec=env VARIABLE_NAME=value appname.
    • Example: Exec=/usr/lib/firefox/firefox %u becomes Exec=env LIBGL_ALWAYS_SOFTWARE=true /usr/lib/firefox/firefox %u.

Glitchy graphics in Firefox

In particular, Firefox can struggle with hardware acceleration, making it completely unusable. To disable hardware acceleration:

  • Run Firefox from the command line, disabling software rendering: LIBGL_ALWAYS_SOFTWARE=true firefox-esr (or firefox if you use the non-ESR version)
  • Go into the settings, search for "hardware acceleration"
  • Uncheck "Use hardware acceleration when available"

ERROR: unsupported 'crypto_LUKS' filesystem (/dev/mmcblk2p2)

See postmarketos-release-upgrade#15.

Package management (apk) issues

See also: Terminal cheat sheet#Package management

This section details troubleshooting steps related to working with the apk package manager. apk is the package manager used by postmarketOS and the distribution it extends, Alpine Linux. The name stands for "Alpine package keeper" (no relation to the Android .apk format).

Icon WARNING: Never install packages from untrusted sources!

Things to check

  • Make sure you have a working internet connection, and that your system clock/date and timezone are set up correctly.
  • Run sudo apk update to update the repo index.
  • Upgrade your system with sudo apk upgrade -a. The -a flag allows for downgrades to get your device to match the packages available in the repositories; as such, using it is recommended.

"ERROR: temporary error (try again later)" when trying to add a package

This is usually indicative of a network issue; check your internet connection.

In some cases, it could also be caused by the used package mirror being down.

"ERROR: unable to select packages" when trying to add/upgrade a package

See also: Troubleshooting package dependency issues

You may see an error similar to this: 

ERROR: unable to select packages:
  package-1.2.3-r1:
    breaks: app1-3.4.5-r0[package~0.1.2] app2-3.4.5-r0[package~0.1.2]
    satisfies: app3-3.4.5-r0[package~1.2.3] app4-3.4.5-r0[package~1.2.3]

This happens when a library is upgraded, but the packages that depend on it haven't been rebuilt yet. Common examples of libraries that often cause this issue include webkit2gtk-*, icu and python3. Sometimes, the process of rebuilding the affected packages can take a while; try again in a few hours.

Make sure to run sudo apk update to update the package index from the repository.

Usually, large rebuilds affecting many packages are mentioned on the edge blog. You can also see the Alpine Linux builder status for an overview of what's currently being built.

If the issue persists for longer, try upgrading your system with sudo apk upgrade -a.

"1 error; XXXX MiB in XXX packages" but no visible error

Sometimes, if a package installation/upgrade fails on a previous command, the error carries over to all subsequent apk operations - that's the "1 error" mentioned in the line.

To try to fix this, run sudo apk fix. This command will attempt to reinstall the package(s) affected by the error, and if it fails, it will print the error again.

In some cases, an upgrade may also help; run sudo apk upgrade -a.

"package not found", even though it is in the repositories

You can search for packages on pkgs.alpinelinux.org (or pkgs.postmarketos.org for pmOS-specific packages). If you see that a package is listed there, but you can't install it:

  • Make sure you have a working internet connection.
  • Run sudo apk update to update the repo index.
  • Check which architectures the package is available for - it may not be available for your device. (Your device's architecture is listed in the infobox on its device page.)
  • Check which category the package belongs to. Packages in the testing category are only available on postmarketOS edge.

"ERROR: BAD signature" when trying to add a package

This can sometimes happen if a repo mirror hasn't been fully updated yet. Run sudo apk update and try again, or try in a few minutes.

"WARNING: opening from cache ...: UNTRUSTED signature"

  • Make sure that your system clock is set up correctly
  • Run sudo apk update

See also

Retrieved from "https://wiki.postmarketos.org/index.php?title=Troubleshooting_on-device_issues&oldid=67545"