User:Knuxify/Draft:Troubleshooting

From postmarketOS Wiki
Hint If you get stuck or need assistance, you can join our Matrix/IRC chats (use the #postmarketOS channel for general support questions).

This page provides an overview of methods for troubleshooting on-device issues, with a focus on existing ports.

If you are looking for support for developing your own port, or if you're dealing with a more technical issue with an unmaintained port and need to fix it, see Troubleshooting/Development issues.

Navigating this page

You can browse the different sections of this page by clicking the relevant links in the table of contents, located on the sidebar, or click one of the following quick links:

#General information · #pmbootstrap issues · #Boot issues · #Interface issues · #Application/general usage issues · #Package management (apk) issues

Searching for specific issues

If you want to search for a specific section or error, you can use your browser's "find in page" option:

  • Firefox: ≡ Menu → Find in page
  • Chromium: ≡ Menu → Find and edit → Find

If you have a keyboard connected, you can also use Ctrl+F to open the search bar.

Note about command examples

On this wiki, you may find commands prefixed with $ or #. These characters shouldn't be copied - they indicate whether that command should be run as a regular user, or as root (in which case, you may need to prefix the command with sudo).

For example, the following tells you to execute the command "ls" as a regular user: 

$ ls

The following tells you to execute the command "apk update" as root (sudo apk update): 

# apk update

General information

Making sure your system is up-to-date

In some cases, the issue you're dealing with may have been fixed in a newer version. For this reason, it's worth making sure that your system is up-to-date before you try to troubleshoot your issue

To upgrade your packages, run: 

# apk upgrade -a
# apk fix

You should also reboot after performing an upgrade.

The -a switch allows for package downgrades, in case a package was downgraded in the upstream repository or if you were testing a package version from outside the upstream repository. The second command fixes any installation errors, if any appear (sometimes this can happen e.g. when a new package replaces another one).

If you're on a stable release, make sure you're on the latest available release. Currently, the latest stable release is v24.06. You can check which release you're on by running the following command: 

$ cat /etc/os-release | grep VERSION

If the version is edge, you're on the edge branch. If you see a version number, check if there isn't a newer version available; if there is, see Upgrade to a newer postmarketOS release.

Looking through reported issues

Most issues in postmarketOS are reported to the pmaports issue tracker. It's worth checking if somebody hasn't already encountered the same issue as you.

If your issue hasn't been reported yet, you are encouraged to report it; this way, developers can be notified about issues they may have missed, and other users will have an easier time troubleshooting their devices. For an overview of where and how to report issues, as well as what information to include, see How to report issues.

Connecting to the device

postmarketOS can set up a network over USB; the device's IP address on this network is 172.16.42.1. This network can be used to connect to the device over SSH. To do so, plug the device into your computer via USB, and run the following command on the computer (replace user with the username you set on the device): 

$ ssh user@172.16.42.1

If you have trouble connecting to your device, see USB Network#Troubleshooting.

You can also upload files over SFTP. Use an SFTP client like FileZilla; your file manager of choice (Nautilus/Files, Dolphin, Thunar...) may also have support for SFTP connections.

Running apps over SSH

Note that you may need to set some environment variables if you're trying to run graphical applications on the device over SSH. For a full list of environment variables set on your system, you can run env in a terminal on the device.

For graphical apps, you'll likely need to set at least the following: 

$ export DISPLAY=:0
$ export WAYLAND_DISPLAY=wayland-0
$ export XDG_RUNTIME_DIR="/run/user/$(id -u)"

TTYescape

Full article: TTYescape

With TTYescape, if your device is frozen, and you need to access the command line without connecting to the device remotely, you can switch to the TTY by holding down Volume Down and pressing Power 3 times.

TTYescape is installed by default on Phosh, Plasma Mobile, GNOME Mobile and sxmo (sway). On other UIs, you can follow the instructions in TTYescape#Installation to install and enable it.

Force reboot

For phones and tablets, you can force-reboot the device by holding down Power for at least 10 seconds. On some devices (e.g. most Samsung devices after 2014) you may also need to hold down the Volume Down button at the same time. In some rarer cases, you may need to hold the power button for a full minute or longer.

You can also run the following command in the terminal: 

$ echo b > /proc/sysrq-trigger

Getting logs

Depending on the issue you're facing, there are different ways to get the logs (messages from apps/services/the kernel etc.):

  • Kernel logs - contain messages printed by the Linux kernel; useful in debugging driver-related issues. Run sudo dmesg or sudo cat /proc/kmsg if the former doesn't work.
    • You can also watch the logs update live with the -w flag: sudo dmesg -w.
  • System log - contains messages from system services. Run logread (note that this also contains kernel logs).
    • You can also watch the system log update live by passing the -F flag (or -f to not show old messages).
    • logread truncates its output by default; you can pass the -n 0 flag to it to show all logs.

There are also other log files that may be of interest depending on what you're troubleshooting; see #Getting logs during boot, #Getting app/interface logs and the linked articles in #Issues with particular components.

Boot issues

See also: Troubleshooting:boot

The exact methods of troubleshooting non-booting devices depend on the stage of the boot process that it fails on.

Getting logs during boot

The initramfs is able to expose a PMOS_LOGS storage device when the device is plugged into a PC. This uses the USB mass storage gadget, and as such may not be supported by all kernels (it should work for devices with (close to) mainline kernels, likely won't work on downstream). See the README.txt file in the mounted folder, or read pmaports!4646 for more information.

If not, check if you're able to SSH into the device as described in #Getting logs, and fetch the logs that way.

If the boot process fails early enough that SSH is not yet available, it's also possible to enable the debug shell by holding down the Volume Down button during boot - this will enable a telnet shell. See Initramfs/Inspecting for full instructions and more information.

Serial debugging

Full article: Serial debugging

For debugging very early boot issues, including kernel crashes, it's useful to have a way to do serial debugging. This usually involves putting together a special cable that, when connected to a USB-UART converter, will get logs from the device's bootloader and the kernel.

This is mostly meant for more advanced users and developers; in most cases, such cables can't be purchased ready-made. If you're a regular user, this option is an absolute last resort.

Device is stuck on manufacturer logo / doesn't get to postmarketOS boot splash

  • Check if the display is listed as working on your device's wiki page.
  • Plug the device into your computer. Are there any messages in dmesg when you plug it in? Does it appear in lsusb?
  • If a device appears, check if it establishes a network connection with your computer. Run ip a on your computer - you should see a "usb0" or similar device, with IP addresses in the range of 172.16.42.XXX.
  • Report an issue to the pmaports repo. If you managed to connect to the device, attach logs; see #Getting logs during boot.

Device is stuck on postmarketOS boot splash

A tablet displaying the "ERROR: Boot partition not found, retrying..." message.

"ERROR: boot partition not found, retrying..."

This usually happens when postmarketOS cannot find the partition that the rootfs has been installed to, or after a broken kernel upgrade (if the module necessary to mount the filesystem on the boot partition is missing). The "boot partition" in this message refers to the boot subpartition of the pmOS partition (see Partition layout).

Usually, this happens if the rootfs hasn't been flashed or was corrupted. Make sure that the rootfs has been flashed correctly, reflash if needed.

If you flashed the rootfs to an SD card or another external storage medium, make sure the correct medium is plugged into the device. If you flashed a pre-built image to the SD card, note that some devices may need additional steps to prepare the system image before flashing the standard image to an SD card - check your device's wiki page.

In some rare cases, this may be a bug in the initramfs code; if so, open an issue in the pmaports repository, and provide logs (see #Getting logs during boot).

UI does not boot, device is stuck on TTY "login:" screen or blank screen with blinking cursor

After the splash screen disappears, the user interface attempts to load. If it returns to a TTY login screen, or you get a blank screen with a blinking/static cursor in the top left corner, then the UI has failed to launch. See #Interface issues for debugging steps.

Application/general usage issues

Issues with particular components

There are separate wiki pages for troubleshooting various components:

If you do not find an answer to your problems there, read on for more generic troubleshooting instructions.

Getting app/interface logs

Logs from the desktop environment/user interface and some running services/apps are stored in the session log. Depending on the display manager, this log is located in:

  • tinydm (Phosh, Plasma Mobile, Sxmo and more) - ~/.local/state/tinydm.log (or $XDG_STATE_HOME/tinydm.log)
  • lightdm (Xfce4, MATE) - ~/.xsession-errors (and ~/.xsession-errors.old)
  • gdm (GNOME, GNOME Mobile) - ~/.cache/gdm/session.log

Errors from background services (bluetoothd for Bluetooth, wpa_supplicant/iwd for WiFi, ModemManager/ofono for the modem, PulseAudio/Pipewire for audio, etc.) can be found in logread (see notes in main #Getting logs session).

Display manager log

For troubleshooting issues with user interfaces not starting up, it's useful to look at the display manager log. Depending on the display manager, this log is located in:

  • tinydm - does not have such a log (it is shared with the session log mentioned above).
  • lightdm - files in /var/log/lightdm/
  • gdm - /var/log/gdm/greeter.log (older logs in /var/log/gdm/greeter.log.X)

For Xorg-based environments (Xfce4, MATE, Sxmo), the Xorg log in /var/log/Xorg.0.log is useful for figuring out startup issues.

Getting logs from apps

Many applications will print out various information when run from the command line.

You can pipe the output of the app/command into a file by doing: 

$ your-command &> logfile

The & in &> ensures that both the standard output and error streams are captured.

Note that when capturing logs with this method, you will no longer see the logs printed out live as the app runs. If you need to see the logs while the app is running, you can run: 

$ your-command 2>&1 | tee logfile

2>&1 here is necessary for tee to pick up messages from the error stream.

If the application closes with a "Segmentation fault" or similar, you'll need to get a backtrace so that the developers can tell what went wrong; see the guide in Troubleshooting/Capturing a backtrace with GDB.

"File not found" while running binary from terminal, despite it being there

This is caused by the binary being linked to a library that isn't installed, and usually appears when trying to run a binary compiled for glibc (the GNU libc) instead of musl (the libc that Alpine Linux/postmarketOS uses).

If it's the latter, try to recompile the software yourself, or ask the developer for a version of the binary compiled against musl, or install gcompat.

Package management (apk) issues

See also: Terminal cheat sheet#Package management

This section details troubleshooting steps related to working with the apk package manager. apk is the package manager used by postmarketOS and the distribution it extends, Alpine Linux. The name stands for "Alpine package keeper" (no relation to the Android .apk format).

Icon WARNING: Never install packages from untrusted sources!

Things to check

  • Make sure you have a working internet connection, and that your system clock/date and timezone are set up correctly.
  • Run sudo apk update to update the repo index.
  • Upgrade your system with sudo apk upgrade -a. The -a flag allows for downgrades to get your device to match the packages available in the repositories; as such, using it is recommended.

"ERROR: temporary error (try again later)" when trying to add a package

This is usually indicative of a network issue; check your internet connection.

In some cases, it could also be caused by the used package mirror being down.

"ERROR: unable to select packages" when trying to add/upgrade a package

See also: Troubleshooting:dependencies

You may see an error similar to this: 

ERROR: unable to select packages:
  package-1.2.3-r1:
    breaks: app1-3.4.5-r0[package~0.1.2] app2-3.4.5-r0[package~0.1.2]
    satisfies: app3-3.4.5-r0[package~1.2.3] app4-3.4.5-r0[package~1.2.3]

This happens when a library is upgraded, but the packages that depend on it haven't been rebuilt yet. Common examples of libraries that often cause this issue include webkit2gtk-*, icu and python3. Sometimes, the process of rebuilding the affected packages can take a while; try again in a few hours.

Make sure to run sudo apk update to update the package index from the repository.

Usually, large rebuilds affecting many packages are mentioned on the edge blog. You can also see the Alpine Linux builder status for an overview of what's currently being built.

If the issue persists for longer, try upgrading your system with sudo apk upgrade -a.

"1 error; XXXX MiB in XXX packages" but no visible error

Sometimes, if a package installation/upgrade fails on a previous command, the error carries over to all subsequent apk operations - that's the "1 error" mentioned in the line.

To try to fix this, run sudo apk fix. This command will attempt to reinstall the package(s) affected by the error, and if it fails, it will print the error again.

In some cases, an upgrade may also help; run sudo apk upgrade -a.

"package not found", even though it is in the repositories

You can search for packages on pkgs.alpinelinux.org (or pkgs.postmarketos.org for pmOS-specific packages). If you see that a package is listed there, but you can't install it:

  • Make sure you have a working internet connection.
  • Run sudo apk update to update the repo index.
  • Check which architectures the package is available for - it may not be available for your device. (Your device's architecture is listed in the infobox on its device page.)
  • Check which category the package belongs to. Packages in the testing category are only available on postmarketOS edge.

"ERROR: BAD signature" when trying to add a package

This can sometimes happen if a repo mirror hasn't been fully updated yet. Run sudo apk update and try again, or try in a few minutes.

"WARNING: opening from cache ...: UNTRUSTED signature"

  • Make sure that your system clock is set up correctly
  • Run sudo apk update

pmbootstrap issues

See also: pmbootstrap#Troubleshooting

This section contains tips for troubleshooting issues with pmbootstrap. If you're installing postmarketOS with pmbootstrap, this section may be helpful.

Things to check

Make sure that you're on the latest pmbootstrap version. The latest pmbootstrap version is 2.3.1. Run: 

$ pmbootstrap -V

This should print out the currently installed version of pmbootstrap. If you're on an older version, upgrade to the latest version first (how to do this depends on how pmbootstrap was installed).

Note If you downloaded pmbootstrap from git, note that the current main branch of the repo contains the work-in-progress code for the upcoming 3.0.0 release, and many things don't yet work correctly. Before continuing, switch to a stable release: 
$ cd path/to/pmbootstrap
$ git checkout 2.3.1

Once you're on the latest pmbootstrap version, run the following to update 

$ pmbootstrap update  # updates package repositories in chroots
$ pmbootstrap pull    # updates the cloned pmaports repository

Clearing the cache and zapping chroots

Many issues can be resolved by clearing the cache or zapping the chroots used for building images/packages. pmbootstrap automates this with the pmbootstrap zap command.

The following will zap all chroots (make sure to answer "y" to every prompt): 

$ pmbootstrap zap

If you're encountering issues with package building, deleting the HTTP cache (-hc flag), distfiles cache (-d flag) and previous locally-compiled packages (-p flag) might help: 

$ pmbootstrap zap -hc -d -p

You can force-delete all caches by adding the -a flag: 

$ pmbootstrap zap -a

Getting logs

When using pmbootstrap, you can run pmbootstrap log to print the log live as your commands are run. In another terminal window: 

$ pmbootstrap log

Note that if you want to get logs *after* completing an operation, the above command will only print the last few lines; sometimes this can be enough, but usually you'll need to open the full log file.

The log file is located in the pmbootstrap work directory; by default, its location is ~/.local/var/pmbootstrap/log.txt. If you're unsure where your work directory is, you can use the following command: 

$ pmbootstrap config work
/home/user/.local/var/pmbootstrap

Running pmbootstrap on non-x86_64 architectures

pmbootstrap should run out-of-the-box on non-x86_64 64-bit architectures. However, you may need to modify the cross-compilation packages for some packages to build correctly.

See the Run pmbootstrap on non-x86_64 arches article for more information.

Using pmbootstrap behind a proxy

pmbootstrap will attempt to copy proxy variables from the host system to the created chroots. In some cases, however, this can fail.

If your proxy environment variables use a hostname defined in /etc/hosts, you need to change them to use the IP address.

sh: can't create /proc/sys/fs/binfmt_misc/register: nonexistent directory

This issue is known to happen in Alpine Virt hardened and Void Linux, and is caused by the binfmt_misc module not being loaded. The following commands can fix it: 

# modprobe binfmt_misc
# mount -t binfmt_misc none /proc/sys/fs/binfmt_misc

If it still does not work (or if you get an error), it is probably not supported by your host Linux distribution's kernel (CONFIG_BINFMT_MISC). Try to use another kernel.

See also

Retrieved from "https://wiki.postmarketos.org/index.php?title=User:Knuxify/Draft:Troubleshooting&oldid=65883"