Serial debugging

From postmarketOS Wiki

To get kernel or bootloader logs on a system without a working display/full USB configuration, a serial cable may be used, which allows us to preview the serial console on the device.

To debug your device with a serial cable, you will need:

  • A serial debugging cable for your device. You will need to make this yourself, so prior experience with soldering and soldering tools is recommended. (You might be able to make a cable without soldering if you find the right parts, though.)
  • A USB-UART adapter that will allow you to connect the UART cable from your phone/tablet to your computer.
  • A piece of software like microcom, minicom, screen, PuTTY or CuteCom that will be used to access the output from the USB-UART adapter on the computer.

For the purpose of this page, the device is the phone/tablet you want to debug, and the computer is the device you want to access the logs from.

USB Serial gadget

If your device is already booting, but failing somewhere in the kernel or init, you might have some luck with USB serial gadget.

Working USB-UART adapters

Adapters based on FT232 (for example this one on sparkfun), CH340 and CP2102 are known to work. All adapters should work though. Try to verify if your device uses 3.3 V or 1.8 V and get an adapter with the correct voltage, so you do not have to use a voltage divider.

Finding the UART ports on your device

The serial port can be accessed by different means, depending on the device - through the microUSB port with a special cable, through the headphone jack, or only through UART ports on the main board. As such, the first step will be finding the proper way to connect to the UART ports on your device.

First, look at your device's wiki page and the Serial debugging/Cable schematics page. You may find your device on there already - in which case, this section can be skipped. If you can't find any info, or you're unsure, continue with this section.

Identifying the MUIC

In some devices (for example, most Samsung phones) serial access can be gained through the microUSB port.

Start by identifying the Micro-USB Interface Controller used in your device. The MUIC is what handles things like USB-OTG - it usually figures out what a cable is supposed to do by checking the resistance between the ground pin and the ID pin. Most MUICs implement an UART jig mode as part of the Carkit standard (see the Micro-USB Interface Controller page), which allows the USB port to be used as a serial input/output port.

You can try looking for kernel options with "MUIC", "USB" or "UART" in the name in the downstream kernel's defconfig - if you find a seemingly SOC-specific option, grep through the kernel source to find the driver (hint: don't include the CONFIG_ part). Sometimes the MUIC is integrated into the PMU/PMIC or handled by the device's chipset, in which case using generic values from the MUIC page may work.

Once you've found the model of your MUIC, look for similar chips in the MUIC page to find the resistance values you should use for your cable. If you're unsure, look up your MUIC chip's datasheet; if you can't find the MUIC used in your device, or the datasheet does not have the resistance value table, use the default values from the MUIC page.

Finding the UART ports on your device's main board

Sometimes there is no other solution but to poke at random motherboard pins to find the serial port. This is mostly a lucky process, and you need to have access to an oscilloscope (maybe a sensitive multimeter would work as well?). With such an instrument you try to find TX (and GND) by measuring between different pins when you:

  • send either garbage, or a repeating pattern to the serial port at /dev/ttysomething (if you have a kernel where it's working)
  • reboot your phone, hopefully it writes something during boot (but that's really trying luck)

Locating TX is usually good enough for debugging, but RX should be quite close, with a high impedance, or more likely a pulling resistor. A pull-up or pull-down resistor will make the pin look like gnd or vcc.

Most modern Qualcomm devices will dump bootloader logs via UART, measuring with a standard multimeter and looking for a pin which sits around 0.9V during early boot and then goes up to 1.8V or down to 0V is a good indicate that it's TX.

See also

Mainboard UART on Exynos devices

On Samsung's Exynos devices (and maybe some other Samsung devices), UART is available as part of a 12 or 20 pin connector. The connector may or may not be soldered onto the board. If it is not you can find just the set of pads. On more recent devices the pads might also be covered by the PCB coating/solder mask. In this case you would need to physically peel off the top layer to expose the pads, and then solder the wires or connector onto it. Chapter 22, "JTAG (Joint Test Action Group)", of the book Seeking the Truth from Mobile Evidence show an example where this is done for some other Samsung Qualcomm device, for the purpose of getting JTAG access (check out the book if you have access).

To peel off, the mask you need a small flat screwdriver and a steady hand (stackexchange recommends a fiberglass brush but that does not seem to cut through the plastic/epoxy surface layer of the pcb).

Exynos4 devices, or at least the Galaxy S3, has the 20 pin molex connector, from which UART and JTAG can be accessed. In the xda thread gt-i9305 hardware hacking, the different pads are identified, and the part number of the socket revealed to be AXE620124, a socket from Panasonic. The corresponding male, which we need to mount onto some sort of breakout board to easily access the pins, has part number AXE520127. The 12-pin molex connector found on other devices seem to be part of the same product family, and the female then has part number AXE612124 with the corresponding male AXE512127.

In the figures below the pads are shown on a few Exynos devices, with or without the connector, and with or without the pads being covered by solder mask.

On the devices where the connector is missing it would need to be soldered for easy access to UART. To do that you would need a hot air soldering station. Below can be seen that AXE620124 and AXE612124 fit onto the pads:

Building the cable

Depending on the cable you need to make (microUSB, headphone jack, USB-C or just pins on the board) the process might be different. It's helpful to have a breadboard and some goldpin connector cables to go with it, as it makes putting together and tweaking the cable easier and reduces the amount of soldering that has to be done. You can find the build instructions for your cable on the Serial debugging/Cable schematics page.

microUSB-based cables

There are several ways to connect wires to a microUSB plug:

Before you try to modify an existing USB-microUSB cable, note that the microUSB plugs in USB-microUSB cables often don't have a connection for the ID pin, which makes them unsuitable for debugging cables.

USB-C based cables

USB-C has 24 pins compared to the 5 pins in microUSB. For most device models it is not known how to make a debug cable, and most likely some special ("vendor defined messages") USB-C PD commands need to be sent to enable UART. As an alternative it might be possible to find the UART pins on the main board of the device, see the finding the UART ports on your device's main board section.

A Google engineer has also published schematics for a USB-C UART cable. It should be compatible with Google Pixel and possibly other devices, if (and only if) they are designed so that UART is available through the SBU1/2 pins. From the GitHub repository: "USB-Cereal is a debugging and development tool for designs that use USB-C port and bring out serial TX/RX lines into SBU1/2 pins. USB-Cereal is made compatible with both 1.8V and 3.3V signalling." -- https://github.com/google/usb-cereal

Soldering a connector on your device

This can be a bit involved, but is generally necessary if you want to use it:

  • This procedure is dangerous for your device, even more so if you are not very experienced
    • Prefer to ask help in a hackerspace, makerspace, fablab or similar you'll practice another day on something easier
    • If you have to solder it yourself, it's probably a good idea to practice on an old motherboard before
  • Prefer soldering short cables to the PCB instead of full-length ones
    • The smaller (diameter) the cables, the better, generally
    • too long or too thick and the pads you have soldered risk being teared apart by the cables
    • too small and it will be a hassle to work with
    • The best is to have them a couple (4-8) centimeters long, to solder them, solder a connector on the other end, and scotch tape them to the pcb
    • Please, don't forget to pick insulated wires

Kernel configuration

In your device tree (or defconfig's CONFIG_CMDLINE setting) add console=ttyXXX,115200n8 to your cmdline. (Make sure that CONFIG_CMDLINE_EXTEND is enabled.)

Replace ttyXXX with the TTY used in your system for serial output. If you're unsure, check the device trees of similar devices, or your device's configuration in downstream; if still in doubt, you can try sending random data to the TTY through /dev/ttyXXX and seeing what works.

Some chipsets also require an additional defconfig setting to be set to enable the serial console (e.g. for OMAP: CONFIG_SERIAL_OMAP and CONFIG_SERIAL_OMAP_CONSOLE).

Plugging in the cable

Once your cable is assembled, it is time to plug it into your USB-UART converter. The most important pins you need to know about are:

  • Ground - usually a black cable.
  • Power/VCC - usually a red cable.
  • TX - The transmission line, aka the output.
  • RX - The receive line, aka the input.
Icon WARNING: DO NOT CONNECT THE POWER PIN IF YOU DON'T KNOW YOUR DEVICE'S UART OUTPUT VOLTAGE! Most UART adapters will send 5V or 3.3V by default and phones will usually use 1.8V (occasionally 3.3V). Plugging the power/VCC pin in when the voltages don't match on your adapter and device may kill it. See the "3.3V vs 1.8V" section at the bottom of this page.

To receive logs, you will only need to connect the ground and TX wires of your cable (TX from device goes to RX on adapter). To send information through the serial console, you will also need to connect the VCC and RX wires to VCC and TX on your adapter (RX from device goes to TX on adapter).

Note that on most schematics "TX" and "RX" refer to the TX/RX port on the UART board. Wire up your cable accordingly.

If this is not the case, you'll have to plug the TX (output) wire from your cable to the RX (input) line on your adapter. (You'll need to do the reverse - RX to TX - if you want to send data.)

Then, plug the other end of the cable into your device. Some devices may automatically boot up when the jig is plugged in - if your device starts up without you pressing the power button when you plug the cable in, it means that the resistance is being detected and the serial mode has been triggered.

Connecting to the serial console from your computer

To connect to the serial console, you'll need a program that can do so. For this you can use:

  • microcom (included in BusyBox) - microcom -s <baud_rate> /dev/ttyXXX
  • minicom - minicom -D /dev/ttyXXX -b <baud_rate>
  • screen - screen /dev/ttyXXX <baud_rate>
  • PuTTY (GUI)
  • CuteCom (GUI)
  • ...or any other program that supports connecting to the serial console.

The baud rate used for the serial console will be stored in your device's device tree (assuming you are using mainline, otherwise you may have some luck looking around downstream's board and defconfig files).

If you're using an USB serial adapter, the path to your TTY will be /dev/ttyUSB[X], where [X] is a number. You can also look through the output of ls /dev/tty*; this will come up with a few different TTYs though, including ones that are used by the system.

Note that on most Linux distributions only root and/or a special group (usually tty or dialout) can access serial ports; to access them as a regular user, see which group owns your TTY:

$ ls -l /dev/ttyUSB0
0 crw-rw---- 1 root dialout 4, 64 Aug 28 08:32 /dev/ttyUSB0

...then add yourself to that group by doing sudo usermod -a -G dialout <username> (replace dialout with the group you saw in the ls -l command above).

Troubleshooting

  1. Make sure you are in the dialout group, see #Connecting to the serial console from your computer
  2. Make sure you are using the correct connection settings: baud rate, parity, etc.
  3. Make sure your ground connection is stable. If in doubt, replace the cables.
  4. If you're only getting partial output, you may need to use a different resistance between the ID and Ground pins. Some devices return different things depending on the resistance given.
  5. If you're not getting any output (or if you're only getting some line breaks), try swapping the RX and TX cables.
  6. If there's no serial cable schematic for your phone, you may try to port a chainloaded bootloader using linux, which will force MUIC to multiplex uart out.

On Samsung devices

On some Samsung devices, the bootloader will only print out very limited information (and in some rarer cases, nothing at all). In that case, you may need to do one of the following:

  • While booting the device, mash the Enter key on your PC to abort auto-boot and get into the bootloader config prompt. (There may not be any visible indication that you got into the prompt!). Then, type log and hit Enter again.
    • You may also need to change the kernel log level; get your current level with printenv and save it somewhere, and use setenv KERNEL_LOG_LEVEL 17740 to change it.
  • On certain devices it is necessary to hold down the Volume Down button while booting for kernel logs to show up; without it, the bootloader only shows a very small portion of the bootloader logs.

See also: Samsung Galaxy Core Plus (samsung-cs02)#Bootloader / S-Boot 4.0

On Google devices

On the Pixel 6, you may need to run fastboot oem uart enable in order to enable UART output.

No kernel output

If you see Uncompressing Linux... done, booting the kernel., that means your kernel is booting. (In some cases, and on some platforms, this message might not appear. The best way to make sure that the kernel is booting is to try the procedures described below.)

Sometimes the kernel crashes before the serial console can be properly brought up. However, it's possible to print out the kernel log messages before that happens, using early printk.

  1. Enable CONFIG_DEBUG_LL (in menuconfig: Kernel hacking > arm Debugging > Kernel low-level debugging functions (read help!))
  2. Under this option in menuconfig, a new sub-option will appear, named Kernel low-level debugging port; set the correct debug port option for your architecture, if applicable.
  3. Select CONFIG_EARLY_PRINTK (in menuconfig: Early printk)
  4. Add earlyprintk to your kernel's CMDLINE parameters.

In some cases however, this will not be enough, and you may get partial logs or no logs at all. In those scenarios, you'll have to use the infamous printascii hack.

Due to the kernel changing very rapidly, the exact implementation of this hack differs; it generally involves finding the function which has the text buffer for the logs, and then adding code to print said buffer using the printascii() command provided by CONFIG_EARLY_PRINTK.

Here's such a patch for the v5.15 kernel. You can apply it by downloading the patch, then running git am path/to/patch in your kernel tree.

Note With the printascii hack, once the serial console is brought up, you might get double (or triple) outputs; in some cases, some of these duplicate lines will appear garbled.

Once you're done debugging, revert the hack to keep your kernel logs clean.

Alternatives to using a serial cable

You may have some luck with using RAM-storage-based methods of serial debugging. Tools like pstore and ramoops make this easy - they usually just need to be enabled in mainline and downstream, and a downstream kernel can be used to access the pstore-reserved RAM region where logs are dumped. The path for this is /sys/fs/pstore/console-ramoops, although older devices may use /proc/last_kmsg.

See Enabling pstore and ramoops for a work-in-progress attempt at documenting this.

Icon TODO: Verify this and elaborate.

3.3V vs 1.8V UARTs

Schematic for a 3.3v->1.8v UART voltage divider

Most phones expose their UART pins on 1.8 V voltage, and most USB-UART converters expect UART side to have 3.3 V. This works well for receiving characters from device; this is enough to read bootloader logs or kernel boot logs (dmesg). However sending 3.3 V to device that expects 1.8 V is dangerous; so you can not use shell over UART without an additional effort. To check if your phone uses 3.3 V or 1.8 V you can connect an oscillioscope to the TX and GND on the phone (or maybe a sensitive multimeter?).

You will need a voltage divider at least on converter->device TX line, which can be set up as in this stack exchange answer: https://electronics.stackexchange.com/a/333230

Use also this https://ohmslawcalculator.com/voltage-divider-calculator

See also