Troubleshooting/Capturing a backtrace with GDB

If the application crashes with a "Segmentation fault" or similar error, you may be able to get more information about the crash from a backtrace. Developers often require this information to be able to more easily find the exact issue that causes the app to crash.

Getting gdb and debug symbols

To get a backtrace, install the gdb package, as well as the -dbg subpackages for the app and the libraries it uses. The -dbg subpackages contain debug symbols, which are used to identify the name and parameters of a function in GDB's backtraces. Not all packages will have debug symbols available - in that case, you can proceed without them (though you may end up with missing symbols), or try modifying the package to add them: #Adding a -dbg subpackage.

For GTK apps, you'll usually need at least glib-dbg gtk4.0-dbg libadwaita-dbg and potentially others, depending on the language the app is written in. For QT apps, you'll usually need at least qt6-qtbase-dbg. If there are any other -dbg subpackages you'll need to install, you'll spot them when getting the backtrace.

Getting the backtrace

Running GDB

Run GDB with one of the following commands:

• gdb my-app;
• gdb --args my-app arg1 arg2 if you need to provide arguments;
• gdb -p $(pidof my-app) to connect to a running process. You may need to run this command as root.

The GDB prompt will show up.

GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "armv7-alpine-linux-musleabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) 

In the case of the first two commands, you will need to type run to get the program to run:

(gdb) run

Getting the crash

Perform the action that causes the program to crash. The program execution should pause, and you should see something like this:

Thread 1 "gnome-shell" received signal SIGSEGV, Segmentation fault.
param_spec_pool_hash (key_spec=<optimized out>) at ../gobject/gparam.c:958
warning: 958	../gobject/gparam.c: No such file or directory
(gdb) 

GDB stops on all signals by default. There are some cases where it will pause execution, but which aren't fatal to the program: Warnings from syscall_cp.s SIGILL in libcrypto.so.3 (this is how OpenSSL figures out hardware capabilities) In those cases, just skip ahead with continue. You may need to do so multiple times (especially in the OpenSSL case, there will be multiple such checks).

This tells us that a segmentation fault happened, and it tells us where and in which file it happened. The "no such file or directory" error can be safely ignored - GDB tries to look for the source code in this location, but it's not required.

Now, get the full backtrace by running bt full:

(gdb) bt full
#0  param_spec_pool_hash (key_spec=<optimized out>) at ../gobject/gparam.c:958
        key = <optimized out>
        p = 0x2 <error: Cannot access memory at address 0x2>
        h = 2759989744
#1  0xb6ab101e in g_hash_table_lookup_node (hash_return=<synthetic pointer>, key=0xbeeca6bc, hash_table=0xb6835b20 = {...})
    at ../glib/ghash.c:416
        node_hash = <optimized out>
        (...some more values...)
        node_key = <optimized out>
#2  g_hash_table_lookup (hash_table=hash_table@entry=0xb6835b20 = {...}, key=key@entry=0xbeeca6bc) at ../glib/ghash.c:1483
        node_index = <optimized out>
        node_hash = <optimized out>
        __func__ = "g_hash_table_lookup"
        (...and so on...)

If at the bottom you see the following prompt:

--Type <RET> for more, q to quit, c to continue without paging--

then type "c" and hit Return/Enter to display the full backtrace.

You can also get the backtrace for all threads by prefixing the bt full command with thread apply all:

(gdb) thread apply all bt full

If in your backtrace you see ??? instead of the function name:

#0  0xb2506ece in ??? () at /usr/lib/gio/modules/libdconfsettings.so

then you're missing the debug symbols for that library. You can find the main package for the library by copying the library name into the "File" box at pkgs.alpinelinux.org/contents. Find the -dbg package for this library. If there isn't one, you can skip it, although this will mean that you only get a partial backtrace. Sometimes this is enough for the developer (if the #0 function is visible, then it should be fine), but you may want to get more debug symbols - in that case, see #Adding a -dbg subpackage.

After that, type continue, and if another backtrace appears, repeat the above steps. Repeat this until the program quits:

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

Press Return/Enter, then type quit to exit GDB.

Once you're done, copy the backtraces you just got and attach them to the issue.

Make sure that the logs/backtrace don't contain any sensitive data (MAC/IP address, passwords, etc.)!

Adding a -dbg subpackage

Some packages may not have a -dbg subpackage. In those cases, you can add one manually by forking the package APKBUILD, modifying the subpackages variable and rebuilding the package with pmbootstrap.

• Run pmbootstrap aportgen --fork-alpine package-name (replace package-name with the name of the package you want to build debug symbols for). See pmbootstrap#Building packages.
• Go to the directory that the new APKBUILD is located in.
• Open the APKBUILD in your favorite code editor. Find the pkgrel variable and increase it by 1. Then, find the subpackages variable, and add $pkgname-dbg:

  subpackages="$pkgname-dev $pkgname-dbg"

  • If there is no subpackages variable, add it somewhere around the other variables.
• Run pmbootstrap build --arch=your-arch package-name to build the package. Replace your-arch with your device's architecture, package-name with the name of the package you want to rebuild.
• Run pmbootstrap sideload --arch=your-arch package-name package-name-dbg to sideload the new package to your device.

If you're adding a -dbg subpackage to a library, you may want to make an upstream merge request to Alpine's aports repo to get the subpackage added there too, so that other people can benefit from it as well.