Rescuing/accessing a broken system booting from external medium

This article explains step-by-step how you can get into a broken postmarketOS install from a second storage medium, and recover data from there or fix the installation. One use case is fixing postmarketOS on the eMMC of the PinePhone with an SD card.

Boot postmarketOS from second storage medium

Download a postmarketOS image and put it on the second storage medium. If there are no pre-built images for your device, build one yourself with pmbootstrap. Boot from the storage medium (in case of the PinePhone, it will automatically boot from the SD card once inserted.)

Get shell access

Attach a serial cable (remember to set killswitches accordingly), or
Attach an USB cable and use SSH, or
Connect to Wi-Fi and use SSH, or
Use the device itself (e.g. using a build with the console UI selected in pmbootstrap

Mount the eMMC's root partition

Use lsblk to figure out which storage device is eMMC (if there are only two, it's the one where nothing is mounted). In this example, it is /dev/mmcblk2.

$ lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0      179:0    0 59.5G  0 disk 
├─mmcblk0p1  179:1    0  243M  0 part /boot
└─mmcblk0p2  179:2    0 59.2G  0 part /
mmcblk2      179:32   0 29.1G  0 disk 
├─mmcblk2p1  179:33   0  121M  0 part 
└─mmcblk2p2  179:34   0   29G  0 part 
mmcblk2boot0 179:64   0    4M  1 disk 
mmcblk2boot1 179:96   0    4M  1 disk

You can also use cgpt show to figure out which partition is which.

NOTE: This is a different example from the above, this shows a separate kernel partition in addition to the boot and root filesystems.

$ sudo cgpt show /dev/mmcblk1
WARNING: Primary GPT header is being ignored
       start        size    part  contents
           0           1          PMBR (Boot GUID: C3DADED4-C48D-D045-804D-38C6663496ED)
           1           1 IGNORED  Pri GPT header
    30777311          32          Sec GPT table
       24576       32768       1  Label: "pmOS_kernel"
                                  Type: ChromeOS kernel
                                  UUID: 6898A103-B1ED-6745-B62E-A61CC72D60CD
                                  Attr: priority=10 tries=5 successful=1 
       57344      524288       2  Label: "pmOS_boot"
                                  Type: Linux data
                                  UUID: 2BC8C937-EAE-A648-94C9-CD5C13A5F47E
      581632    30194688       3  Label: "pmOS_root"
                                  Type: Linux data
                                  UUID: 66765F29-1744-F240-942B-F9771D66A55AC
          64       16384      11  Label: "RWFW"
                                  Type: ChromeOS firmware
                                  UUID: 08432388-CADD-7147-86A8-7BDD112C9544
    30777343           1          Sec GPT header

The key information here is, that /dev/mmcblk1p1 is the kernel partition , /dev/mmcblk1p2 is the boot partition, and finally /dev/mmcblk1p3 contains the root filesystem (see why all this matters a few sections further below).

a) Encrypted

$ sudo cryptsetup luksOpen /dev/mmcblk2p2 emmc
Enter passphrase for /dev/mmcblk2p2:
$ sudo mount /dev/mapper/emmc /mnt

b) Not Encrypted

$ sudo mount /dev/mmcblk2p2 /mnt

Access your data

If you only want to access (and presumably save/retrieve) the data on the broken system, after the above you'll find the broken system's root partition in /mnt:

$ cd /mnt
$ ls
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

"Logging" into the broken system

Apart from the above, you'd want to mount your /boot partition too (you can find the device name by the cgpt labels, but the smallest partition is probably a good guess too (also, you've already found the root partition).

You'll also want to make your running kernel's virtual filesystems accessible within the /mnt tree and then finally chroot to that directory. This means, you'll have the kernel running from the external device, but the filesystem tree (root all the rest) in use will be that of the broken system -- you're essentially making /mnt being your new /.

$ mount /dev/mmcblk2p1 /mnt/boot
$ sudo mount -t proc proc /mnt/proc/
$ sudo mount -t sysfs sys /mnt/sys/
$ sudo mount --rbind /dev /mnt/dev/
$ sudo mount --rbind /run /mnt/run/ # Not strictly needed

After preparing the filesystem tree in /mnt as described above, you can enter the broken system with:

$ sudo chroot /mnt

At this point you'll be able to work as if you've booted the broken system: you an install/remove packages, fix broken packages, edit configuration files, and so on.

NOTE: The package installer (apk), more specifically the kernel package setup scripts, will refuse to flash the kernel partition when running within a chroot, so if you want to fix your kernel on the broken system, after (re)installing the appropriate kernel package in the chroot, you'd also want to flash the kernel image by hand: dd if=/boot/vmlinuz.kpart of=/dev/mmcblk<n>p<m>.

After this, you can leave the chroot and reboot:

$ exit
$ reboot