Debugging the GNOME stack

This article is about debugging the GNOME stack on postmarketOS. That could be either a crashing Phosh or GNOME mobile session, or a GTK or other glib application that doesn't work. Parts of this are probably also useful for other stacks.

Getting verbose log output

Use the G_MESSAGES_DEBUG variable to enable more logging. You can either set it to a specific component to get its debug log messages, or to all to enable the most detailed logging from all components. Export the variable before starting the application you want to debug, e.g.:

export G_MESSAGES_DEBUG=all

One application

Run the app over SSH with the environment variables set to the same as if the app was running inside your GNOME session (so it can talk to the Wayland compositor etc.). One way to do this, is opening a terminal on the Phone, starting tmux, and then attaching over SSH to the tmux session with tmux a.

Full session (Phosh and other interfaces that use tinydm)

If you want to debug a whole session (e.g. if Phosh is not starting up), then write above export line to ~/.profile, which is the file that tinydm sources before running a session.

In case of Phosh, it may also be useful to enable debugging output of libwayland-server, which is used through wlroots in Phoc (Phoc is the Wayland compositor that Phosh is using). Do this with:

export WAYLAND_DEBUG=server

Find the tinydm log file in ~/.local/state/tinydm.log. The previous log is in ~/.local/state/tinydm.log.old, which is useful if your Phosh session is over and over since it contains a full crash log.

Other useful environment variables are PHOC_DEBUG and PHOSH_DEBUG. See the manual pages phosh(1) and phoc(1) on your system.

Full session (GNOME/GNOME Shell on Mobile)

GNOME uses gdm as the login manager, so the tinydm log file isn't present.

If the error appears before you can log in, there's a chance that you will be able to find the root cause in the greeter logs. gdm's logs are at /var/log/gdm/greeter.log. If gdm restarts, the previous log is moved to /var/log/gdm/greeter.log.1. /var/log/Xorg.0.log may also be of some use.

Some general errors can be spotted by running logread, but they usually don't include full tracebacks.

If you see an error like "Application 'org.gnome.Shell.desktop' killed by signal 11", here's how you can debug it:

Open a new SSH session.
Add the following packages: gdb gtk4.0-dbg glib-dbg mutter-dbg gnome-shell-dbg gjs-dbg libffi-dbg
Restart gdm
Run sudo gdb -p $(pidof gnome-shell)
If you see a backtrace but no "Thread 1 "gnome-shell" received signal SIGSEGV, Segmentation fault.", it's usually non-fatal and can just be skipped with continue (you can still run bt full if you want).
If you do see that message, run bt full and press c when prompted ("--Type <RET> for more, q to quit, c to continue without paging--").
Run continue and repeat the above for any subsequent backtraces, until the program quits.
Quit gdb with quit.
Copy the backtrace and report to the most relevant repository based on where the failing code is (usually that'll be gnome-shell or mutter repository (the latter also includes clutter and cogl, if those error out)).

See also: the GNOME wiki article on debugging the GNOME shell

Understanding GNOME session startup

The commands that run when starting a wayland session are defined in /usr/share/wayland-sessions/, e.g. for Phosh there is a phosh.desktop. In case of Phosh, this starts /usr/bin/phosh, a shell script, which then starts /usr/bin/phoc and instructs it to run gnome-session with a --session=phosh argument. To understand what this runs, we need to look at /usr/share/gnome-session/sessions/phosh.session where we will find a long RequiredComponents line. As of writing:

RequiredComponents=org.gnome.SettingsDaemon.A11ySettings;org.gnome.SettingsDaemon.Color;org.gnome.SettingsDaemon.Datetime;org.gnome.SettingsDaemon.Housekeeping;org.gnome.SettingsDaemon.Keyboard;org.gnome.SettingsDaemon.MediaKeys;org.gnome.SettingsDaemon.Power;org.gnome.SettingsDaemon.PrintNotifications;org.gnome.SettingsDaemon.Rfkill;org.gnome.SettingsDaemon.ScreensaverProxy;org.gnome.SettingsDaemon.Sharing;org.gnome.SettingsDaemon.Smartcard;org.gnome.SettingsDaemon.Sound;org.gnome.SettingsDaemon.UsbProtection;org.gnome.SettingsDaemon.Wacom;org.gnome.SettingsDaemon.Wwan;org.gnome.SettingsDaemon.XSettings;sm.puri.Phosh;sm.puri.OSK0;

For each of these, there is a .desktop file in /usr/share/applications. Most importantly for Phosh: /usr/share/applications/sm.puri.Phosh.desktop. And in that file you will actually find the line that runs Phosh:

Exec=/usr/libexec/phosh

Now you can modify it as needed, e.g. add /usr/bin/strace before it or change the command to run a local build that you didn't install into the system.