Troubleshooting on-device issues
If you get stuck or need assistance, you can join our Matrix/IRC chats (use the #postmarketOS channel for general support questions). |
This page provides an overview of methods for troubleshooting on-device issues, with a focus on existing ports.
You can browse the different sections of this page by clicking the relevant links in the table of contents, located on the sidebar, or click one of the following quick links:
#General information · #Boot issues · #Interface issues · #Application/general usage issues · #Package management (apk) issues
Other troubleshooting pages
- For help with pmbootstrap (e.g. if you're installing postmarketOS with pmbootstrap), see pmbootstrap/Troubleshooting.
Searching for specific issues
If you want to search for a specific section or error, you can use your browser's "find in page" option:
- Firefox: ≡ Menu → Find in page
- Chromium: ≡ Menu → Find and edit → Find
If you have a keyboard connected, you can also use Ctrl+F to open the search bar.
Note about command examples
On this wiki, you may find commands prefixed with $
or #
. These characters shouldn't be copied - they indicate whether that command should be run as a regular user ($
), or as root (#
; in which case, you may need to prefix the command with sudo
).
For example, the following tells you to execute the command "ls" as a regular user:
$ ls
The following tells you to execute the command "apk update" as root (sudo apk update
):
# apk update
General information
Checking your device's wiki page
The wiki page for your device contains information about supported features, and could have information about the issue you're experiencing. Find your device's wiki page by looking through the Devices page or using the search feature on the wiki.
The page for the device's chipset/SoC can often be useful as well, as it contains information that more broadly applies to all devices with the chipset. The chipset page is linked on your device's wiki page; click the SoC name in the device infobox.
Making sure your system is up-to-date
In some cases, the issue you're dealing with may have been fixed in a newer version. For this reason, it's worth making sure that your system is up-to-date before you try to troubleshoot your issue.
To upgrade your packages, run:
# apk upgrade -a
# apk fix
You should also reboot after performing an upgrade.
The -a
switch allows for package downgrades, in case a package was downgraded in the upstream repository or if you were testing a package version from outside the upstream repository. The second command fixes any installation errors, if any appear (sometimes this can happen e.g. when a new package replaces another one).
If you're on a stable release, make sure you're on the latest available release. Currently, the latest stable release is v24.06. You can check which release you're on by running the following command:
$ cat /etc/os-release | grep VERSION
If the version is edge
, you're on the edge branch. If you see a version number, check if there isn't a newer version available; if there is, see Upgrade to a newer postmarketOS release.
Looking through reported issues
Most issues in postmarketOS are reported to the pmaports issue tracker. It's worth checking if somebody hasn't already encountered the same issue as you.
If your issue hasn't been reported yet, you are encouraged to report it; this way, developers can be notified about issues they may have missed, and other users will have an easier time troubleshooting their devices. For an overview of where and how to report issues, as well as what information to include, see How to report issues.
Connecting to the device
postmarketOS can set up a network over USB; the device's IP address on this network is 172.16.42.1
. This network can be used to connect to the device over SSH. To do so, plug the device into your computer via USB, and run the following command on the computer (replace user
with the username you set on the device):
$ ssh user@172.16.42.1
If you have trouble connecting to your device, see USB Network#Troubleshooting.
You can also upload files over SFTP. Use an SFTP client like FileZilla; your file manager of choice (Nautilus/Files, Dolphin, Thunar...) may also have support for SFTP connections.
Running apps over SSH
Note that you may need to set some environment variables if you're trying to run graphical applications on the device over SSH. For a full list of environment variables set on your system, you can run env
in a terminal on the device.
For graphical apps, you'll likely need to set at least the following:
$ export DISPLAY=:0
$ export WAYLAND_DISPLAY=wayland-0
$ export XDG_RUNTIME_DIR="/run/user/$(id -u)"
TTYescape
With TTYescape, if your device is frozen, and you need to access the command line without connecting to the device remotely, you can switch to the TTY by holding down Volume Down and pressing Power 3 times.
TTYescape is installed by default on Phosh, Plasma Mobile, GNOME Mobile and sxmo (sway). On other UIs, you can follow the instructions in TTYescape#Installation to install and enable it.
Force reboot
For phones and tablets, you can force-reboot the device by holding down Power for at least 10 seconds. On some devices (e.g. most Samsung devices after 2014) you may also need to hold down the Volume Down button at the same time. In some rarer cases, you may need to hold the power button for a full minute or longer.
You can also run the following command in the terminal:
$ echo b > /proc/sysrq-trigger
Getting logs
Depending on the issue you're facing, there are different ways to get the logs (messages from apps/services/the kernel etc.):
- Kernel logs - contain messages printed by the Linux kernel; useful in debugging driver-related issues. Run
sudo dmesg
orsudo cat /proc/kmsg
if the former doesn't work.- You can also watch the logs update live with the
-w
flag:sudo dmesg -w
.
- You can also watch the logs update live with the
- System log - contains messages from system services. Run
logread
(note that this also contains kernel logs).- You can also watch the system log update live by passing the
-F
flag (or-f
to not show old messages). logread
truncates its output by default; you can pass the-n 0
flag to it to show all logs.
- You can also watch the system log update live by passing the
There are also other log files that may be of interest depending on what you're troubleshooting; see #Getting logs during boot, #Getting app/interface logs and the linked articles in #Issues with particular components.
Boot issues
The exact methods of troubleshooting non-booting devices depend on the stage of the boot process that it fails on.
Getting logs during boot
The initramfs is able to expose a PMOS_LOGS
storage device when the device is plugged into a PC. This uses the USB mass storage gadget, and as such may not be supported by all kernels (it should work for devices with (close to) mainline kernels, likely won't work on downstream). See the README.txt file in the mounted folder, or read pmaports!4646 for more information.
If not, check if you're able to SSH into the device as described in #Connecting to the device, and fetch the logs that way.
If the boot process fails early enough that SSH is not yet available, it's also possible to enable the debug shell by holding down the Volume Down button during boot - this will enable a telnet shell. See Initramfs/Inspecting for full instructions and more information.
Serial debugging
For debugging very early boot issues, including kernel crashes, it's useful to have a way to do serial debugging. This usually involves putting together a special cable that, when connected to a USB-UART converter, will get logs from the device's bootloader and the kernel.
This is mostly meant for more advanced users and developers; in most cases, such cables can't be purchased ready-made. If you're a regular user, this option is an absolute last resort.
Kernel logs from previous boot
On some devices, it is possible to retrieve the kernel log from the previous boot. Depending on the kernel used on the device, this can be accessed from another working install (or from anything running the same kernel, e.g. in TWRP) in one of two ways:
- By reading
/proc/last_kmsg
. - By mounting the
pstore
partition and reading its contents:# mkdir /tmp/pstore # mount -t pstore -o kmsg_bytes=16000 - /tmp/pstore
Device is stuck on manufacturer logo / doesn't get to postmarketOS boot splash
- Check if the display is listed as working on your device's wiki page.
- Plug the device into your computer. Are there any messages in
dmesg
when you plug it in? Does it appear inlsusb
? - If a device appears, check if it establishes a network connection with your computer. Run
ip a
on your computer - you should see a "usb0" or similar device, with IP addresses in the range of172.16.42.XXX
.- If you see a device in
ip a
, check if you can connect to it withssh 172.16.42.1
, ortelnet 172.16.42.1
. - If you see a device but can't connect to it, see Troubleshooting on-device issues/Booting problems#Network troubleshooting.
- If you see a device in
- Report an issue to the pmaports repo. If you managed to connect to the device, attach logs; see #Getting logs during boot.
Device is stuck on postmarketOS boot splash
"ERROR: boot partition not found, retrying..."
This usually happens when postmarketOS cannot find the partition that the rootfs has been installed to, or after a broken kernel upgrade (if the module necessary to mount the filesystem on the boot partition is missing). The "boot partition" in this message refers to the boot subpartition of the pmOS partition (see Partition layout).
Usually, this happens if the rootfs hasn't been flashed or was corrupted. Make sure that the rootfs has been flashed correctly, reflash if needed.
If you flashed the rootfs to an SD card or another external storage medium, make sure the correct medium is plugged into the device. If you flashed a pre-built image to the SD card, note that some devices may need additional steps to prepare the system image before flashing the standard image to an SD card - check your device's wiki page.
In some rare cases, this may be a bug in the initramfs code; if so, open an issue in the pmaports repository, and provide logs (see #Getting logs during boot).
UI does not boot, device is stuck on TTY "login:" screen or blank screen with blinking cursor
After the splash screen disappears, the user interface attempts to load. If it returns to a TTY login screen, or you get a blank screen with a blinking/static cursor in the top left corner, then the UI has failed to launch. See #Interface issues for debugging steps.
Application/general usage issues
Issues with particular components
There are separate wiki pages for troubleshooting various components:
- Audio: TODO (Audio, but no concrete troubleshooting section other than a hint; ALSA UCM in Alsa UCM, nothing about Pulseaudio or Pipewire...)
- Bluetooth: Bluetooth#Troubleshooting
- Display: TODO (Display, points to Display/Troubleshooting which is mostly downstream-related; also Screen Calibration though not entirely related, but should be integrated into the former?)
- GPS: GPS
- Touchscreen: TODO (Touchscreen)
- WiFi: WiFi#Troubleshooting, WiFi#Debugging (TODO - needs cleanups. Most of the WiFi page is for bringing it up on downstream...)
If you do not find an answer to your problems there, read on for more generic troubleshooting instructions.
Getting app/interface logs
Logs from the desktop environment/user interface and some running services/apps are stored in the session log. Depending on the display manager, this log is located in:
- tinydm (Phosh, Plasma Mobile, Sxmo and more) -
~/.local/state/tinydm.log
(or$XDG_STATE_HOME/tinydm.log
) - lightdm (Xfce4, MATE) -
~/.xsession-errors
(and~/.xsession-errors.old
) - gdm (GNOME, GNOME Mobile) -
~/.cache/gdm/session.log
Errors from background services (bluetoothd for Bluetooth, wpa_supplicant/iwd for WiFi, ModemManager/ofono for the modem, PulseAudio/Pipewire for audio, etc.) can be found in logread
(see notes in main #Getting logs session).
Display manager log
For troubleshooting issues with user interfaces not starting up, it's useful to look at the display manager log. Depending on the display manager, this log is located in:
- tinydm - does not have such a log (it is shared with the session log mentioned above).
- lightdm - files in
/var/log/lightdm/
- gdm -
/var/log/gdm/greeter.log
(older logs in/var/log/gdm/greeter.log.X
)
For Xorg-based environments (Xfce4, MATE, Sxmo), the Xorg log in /var/log/Xorg.0.log
is useful for figuring out startup issues.
Getting logs from apps
Many applications will print out various information when run from the command line.
You can pipe the output of the app/command into a file by doing:
$ your-command &> logfile
The &
in &>
ensures that both the standard output and error streams are captured.
Note that when capturing logs with this method, you will no longer see the logs printed out live as the app runs. If you need to see the logs while the app is running, you can run:
$ your-command 2>&1 | tee logfile
2>&1
here is necessary for tee to pick up messages from the error stream.
If the application closes with a "Segmentation fault" or similar, you'll need to get a backtrace so that the developers can tell what went wrong; see the guide in Troubleshooting/Capturing a backtrace with GDB.
"File not found" while running binary from terminal, despite it being there
This is caused by the binary being linked to a library that isn't installed, and usually appears when trying to run a binary compiled for glibc (the GNU libc) instead of musl (the libc that Alpine Linux/postmarketOS uses).
If it's the latter, try to recompile the software yourself, or ask the developer for a version of the binary compiled against musl, or install gcompat
.
Glitchy graphics in apps
Sometimes, due to bugs in the GPU drivers, applications may appear to "glitch out" or crash entirely. This often happens with Firefox, as well as any games or apps requiring 3D graphics.
In particular, Adreno 300 series GPUs (used in e.g. the Qualcomm Snapdragon 410 (MSM8916)) are known to struggle with this.
To work around these issues, you can try the following:
- Forcing an older OpenGLES version; run the app from the command line with
MESA_GLES_VERSION_OVERRIDE=2.0 appname
- Forcing software rendering; run the app from the command line with
LIBGL_ALWAYS_SOFTWARE=true appname
You can find a full list of relevant environment variables in the Mesa documentation.
Persisting custom variables
If setting one of those variables fixes your app, you can persist the changes by adding a manual override in the app's .desktop
file.
- Find the
.desktop
file for the application; see the files in/usr/share/applications
. - Copy your app's
.desktop
file to~/.local/share/applications/
(you may need to create this directory first). - Open the
.desktop
file in a text editor; find the line that starts withExec
and modify it to match the format ofExec=env VARIABLE_NAME=value appname
.- Example:
Exec=/usr/lib/firefox/firefox %u
becomesExec=env LIBGL_ALWAYS_SOFTWARE=true /usr/lib/firefox/firefox %u
.
- Example:
Glitchy graphics in Firefox
In particular, Firefox can struggle with hardware acceleration, making it completely unusable. To disable hardware acceleration:
- Run Firefox from the command line, disabling software rendering:
LIBGL_ALWAYS_SOFTWARE=true firefox-esr
(orfirefox
if you use the non-ESR version) - Go into the settings, search for "hardware acceleration"
- Uncheck "Use hardware acceleration when available"
ERROR: unsupported 'crypto_LUKS' filesystem (/dev/mmcblk2p2)
See postmarketos-release-upgrade#15.
Package management (apk) issues
This section details troubleshooting steps related to working with the apk package manager. apk is the package manager used by postmarketOS and the distribution it extends, Alpine Linux. The name stands for "Alpine package keeper" (no relation to the Android .apk format).
WARNING: Never install packages from untrusted sources! |
Things to check
- Make sure you have a working internet connection, and that your system clock/date and timezone are set up correctly.
- Run
sudo apk update
to update the repo index. - Upgrade your system with
sudo apk upgrade -a
. The-a
flag allows for downgrades to get your device to match the packages available in the repositories; as such, using it is recommended.
"ERROR: temporary error (try again later)" when trying to add a package
This is usually indicative of a network issue; check your internet connection.
In some cases, it could also be caused by the used package mirror being down.
"ERROR: unable to select packages" when trying to add/upgrade a package
You may see an error similar to this:
ERROR: unable to select packages: package-1.2.3-r1: breaks: app1-3.4.5-r0[package~0.1.2] app2-3.4.5-r0[package~0.1.2] satisfies: app3-3.4.5-r0[package~1.2.3] app4-3.4.5-r0[package~1.2.3]
This happens when a library is upgraded, but the packages that depend on it haven't been rebuilt yet. Common examples of libraries that often cause this issue include webkit2gtk-*
, icu
and python3
. Sometimes, the process of rebuilding the affected packages can take a while; try again in a few hours.
Make sure to run sudo apk update
to update the package index from the repository.
Usually, large rebuilds affecting many packages are mentioned on the edge blog. You can also see the Alpine Linux builder status for an overview of what's currently being built.
If the issue persists for longer, try upgrading your system with sudo apk upgrade -a
.
"1 error; XXXX MiB in XXX packages" but no visible error
Sometimes, if a package installation/upgrade fails on a previous command, the error carries over to all subsequent apk operations - that's the "1 error" mentioned in the line.
To try to fix this, run sudo apk fix
. This command will attempt to reinstall the package(s) affected by the error, and if it fails, it will print the error again.
In some cases, an upgrade may also help; run sudo apk upgrade -a
.
"package not found", even though it is in the repositories
You can search for packages on pkgs.alpinelinux.org (or pkgs.postmarketos.org for pmOS-specific packages). If you see that a package is listed there, but you can't install it:
- Make sure you have a working internet connection.
- Run
sudo apk update
to update the repo index. - Check which architectures the package is available for - it may not be available for your device. (Your device's architecture is listed in the infobox on its device page.)
- Check which category the package belongs to. Packages in the
testing
category are only available on postmarketOS edge.
"ERROR: BAD signature" when trying to add a package
This can sometimes happen if a repo mirror hasn't been fully updated yet. Run sudo apk update
and try again, or try in a few minutes.
"WARNING: opening from cache ...: UNTRUSTED signature"
- Make sure that your system clock is set up correctly
- Run
sudo apk update