Debugging the GNOME stack

From postmarketOS

This article is about debugging the GNOME stack on postmarketOS. That could be either a crashing Phosh or GNOME mobile session, or a GTK or other glib application that doesn't work. Parts of this are probably also useful for other stacks.

Getting verbose log output

Use the G_MESSAGES_DEBUG variable to enable more logging. You can either set it to a specific component to get its debug log messages, or to all to enable the most detailed logging from all components. Export the variable before starting the application you want to debug, e.g.:

export G_MESSAGES_DEBUG=all

One application

Run the app over SSH with the environment variables set to the same as if the app was running inside your GNOME session (so it can talk to the Wayland compositor etc.). One way to do this, is opening a terminal on the Phone, starting tmux, and then attaching over SSH to the tmux session with tmux a.

Full session (Phosh and other interfaces that use tinydm)

If you want to debug a whole session (e.g. if Phosh is not starting up), then write above export line to ~/.profile, which is the file that tinydm sources before running a session.

In case of Phosh, it may also be useful to enable debugging output of libwayland-server, which is used through wlroots in Phoc (Phoc is the Wayland compositor that Phosh is using). Do this with:

export WAYLAND_DEBUG=server

Find the tinydm log file in ~/.local/state/tinydm.log. The previous log is in ~/.local/state/tinydm.log.old, which is useful if your Phosh session is over and over since it contains a full crash log.

Full session (GNOME/GNOME Shell on Mobile)

GNOME uses gdm as the login manager, so the tinydm log file isn't present.

If the error appears before you can log in, there's a chance that you will be able to find the root cause in the greeter logs. gdm's logs are at /var/log/gdm/greeter.log. If gdm restarts, the previous log is moved to /var/log/gdm/greeter.log.1. /var/log/Xorg.0.log may also be of some use.

Some general errors can be spotted by running logread, but they usually don't include full tracebacks.

See also: the GNOME wiki article on debugging the GNOME shell

Understanding GNOME session startup

The commands that run when starting a wayland session are defined in /usr/share/wayland-sessions/, e.g. for Phosh there is a phosh.desktop. In case of Phosh, this starts /usr/bin/phosh, a shell script, which then starts /usr/bin/phoc and instructs it to run gnome-session with a --session=phosh argument. To understand what this runs, we need to look at /usr/share/gnome-session/sessions/phosh.session where we will find a long RequiredComponents line. As of writing:

RequiredComponents=org.gnome.SettingsDaemon.A11ySettings;org.gnome.SettingsDaemon.Color;org.gnome.SettingsDaemon.Datetime;org.gnome.SettingsDaemon.Housekeeping;org.gnome.SettingsDaemon.Keyboard;org.gnome.SettingsDaemon.MediaKeys;org.gnome.SettingsDaemon.Power;org.gnome.SettingsDaemon.PrintNotifications;org.gnome.SettingsDaemon.Rfkill;org.gnome.SettingsDaemon.ScreensaverProxy;org.gnome.SettingsDaemon.Sharing;org.gnome.SettingsDaemon.Smartcard;org.gnome.SettingsDaemon.Sound;org.gnome.SettingsDaemon.UsbProtection;org.gnome.SettingsDaemon.Wacom;org.gnome.SettingsDaemon.Wwan;org.gnome.SettingsDaemon.XSettings;sm.puri.Phosh;sm.puri.OSK0;

For each of these, there is a .desktop file in /usr/share/applications. Most importantly for Phosh: /usr/share/applications/sm.puri.Phosh.desktop. And in that file you will actually find the line that runs Phosh:

Exec=/usr/libexec/phosh

Now you can modify it as needed, e.g. add /usr/bin/strace before it or change the command to run a local build that you didn't install into the system.

See also