fds-os/docs/developer/recovery.md

Recovery and rollback

Development reference and historical context. For current operating instructions, use the user manual. Acceptance applies only to the source and artifacts identified in each record.

Documentation index · Boot images · Media tools · EEPROM

Recovery is a separate, read-only FDS image. It contains native s6, Bash, GNU utilities, XBPS, filesystem tools, the static FDS tools, and the base Dasung controller. It does not need files or programs from a SYSTEM cartridge. The software workflow is covered by make recovery-test; physical Pi recovery remains deferred. The complete internal disk includes this recovery image; its guide separates verified image construction from deferred physical installation. The build commands below create ordinary files and do not write a host disk.

Build the recovery image

Use the prepared x86_64 build host described in Your first build. From the repository root, run these commands sequentially:

make recovery
make rootfs-test
make recovery-test

The full VM test also needs the Pi kernel/initramfs and a known-good CLI SYSTEM image. On a fresh checkout, build those first with make rootfs PROFILE=cli, make rootfs-test, make initramfs, and make system-card PROFILE=cli, then run the recovery sequence above. Run builds and VM tests sequentially because they share the same project-local Void environment.

make recovery builds the recovery rootfs profile independently, configures all packages and caches at image construction time, then creates EROFS. Its outputs are:

File	Meaning
out/rootfs-recovery.tar	Complete recovery rootfs, including package database and compiled s6 configuration
out/fds-recovery.img	Raw EROFS payload for the internal GPT partition named FDS_RECOVERY
out/recovery-build.*/manifest.json	Rootfs hash, filesystem hash, profile and size
out/manifests/rootfs-latest/	Exact selected package archives and build input records

The latest-rootfs pointers also select this recovery build. Profile-specific rootfs pointers preserve earlier CLI/development builds. A recovery EROFS payload is not a whole-disk image and is not a SYSTEM cartridge. The SYSTEM builder rejects a recovery rootfs to prevent confusing those roles.

Enter recovery

Stage0 can enter recovery in two ways:

1. At its missing-SYSTEM or ambiguous-SYSTEM console, type recovery and Enter.
2. For a deliberate maintenance boot, use fds.boot=recovery in the boot partition's single-line cmdline.txt, replacing fds.boot=normal if present. Restore normal mode when maintenance is complete.

Stage0 requires exactly one readable FDS_RECOVERY partition and mounts it read-only. It does not silently choose between duplicate partitions. The ARM VM suite supplies disposable virtual partitions; physical firmware/NVMe/display behavior must still be checked on the Pi.

The local prompt is:

FDS RECOVERY — LOCAL MAINTENANCE CONSOLE
RECOVERY#

This is an explicit root maintenance console. It reads startup files from the immutable recovery image and uses a temporary home under /run/fds. The root password remains locked; no SSH or network login service is started. Normal CLI and development images continue to use the ordinary fds account and FDS>.

Recovery inspects cartridges read-only. It does not automatically mount DATA writable, activate an ENVIRONMENT desktop, or enable Ethernet. Those actions require explicit commands. The Dasung controller remains in the base boot bundle.

Inspect a failed system

fds info
fds bays
fds bay 2
fds inspect BAY02
fds topology
fds boot-profile
lsusb -t
lsblk -o NAME,MAJ:MIN,MODEL,SERIAL,SIZE,RO,TYPE,FSTYPE,PARTLABEL,MOUNTPOINTS
dmesg

Use the calibrated bay number and the displayed model/serial to identify a cartridge. Do not infer a bay from /dev/sda or from discovery order. A valid inactive SYSTEM mounts read-only under /run/fds/media/NN; fds bay N reports the actual path. A bad filesystem or manifest produces an error instead of running anything from the cartridge.

An empty bay configuration produces UNCONFIGURED, not guessed bay numbers. See Cartridges for calibration. Permanent machine configuration on FDS_INTERNAL is part of the remaining M12 work.

Check and repair DATA

Start with a read-only check. For example, for DATA in bay 2:

fds recovery check BAY02

This command is available only to root in the recovery image. It identifies one USB disk and one FDS_DATA partition, unmounts its read-only view, reserves the disk exclusively, verifies GPT and the kernel partition identity, and invokes e2fsck -f -n. It does not repair the filesystem. A clean result leaves DATA unmounted and reports SAFE TO REMOVE.

If DATA was explicitly activated writable, eject that session first. Close any shell whose current directory is on DATA and any other reader before checking; an ordinary busy-unmount failure is reported, never bypassed with lazy unmount.

If the check reports problems, review its log and preserve a backup where possible. Preview the repair with:

fds recovery repair BAY02

The preview displays the model, serial, capacity and a confirmation command. Copy that exact command only after checking the intended cartridge. Its token binds the bay, kernel insertion number and current boot. A token from another insertion or boot is rejected. The preview makes no filesystem changes.

Confirmed repair runs e2fsck -f -p, which performs conservative automatic repairs and stops when manual judgement is required. It then flushes the device, invalidates its block cache, and runs a second e2fsck -f -n. Only a successful verification produces DATA REPAIRED AND VERIFIED and SAFE TO REMOVE.

The checker uses a temporary kernel loop device backed by the already verified partition descriptor. This lets e2fsck take its own exclusive device claim while FDS retains the physical whole-disk reservation. The loop is removed automatically when its last descriptor closes. This uses the existing kernel loop driver, libc crate and base e2fsprogs; no new package or Rust dependency is introduced.

Failed or interrupted checks/repairs retain a quarantine record across cartridge daemon restarts for the same insertion. They do not inherit an earlier SAFE status. The checker holds the disk reservation and is killed if its supervising daemon dies. The command runs synchronously: bay operations and orderly shutdown wait for it to finish; the local shell remains usable. A client timeout is not success and does not authorize removing media. Inspect the log and wait for the service to finish before retrying. Logs are in /run/fds/recovery/ and are limited to 16 MiB per invocation.

This narrow recovery command requires a readable GPT and primary ext filesystem signature. It deliberately does not guess partition boundaries, recreate a broken GPT, or force answers to destructive e2fsck questions. Such cases require an offline backup and expert use of the included filesystem tools. It cannot recover data that was never written to storage.

For structured results, prefix the command with fds --json. After a successful check or repair, remove/reinsert DATA to inspect it again. Recovery still keeps it read-only until fds data use N is explicitly requested.

Prepare replacement SYSTEM media

A known-good SYSTEM image must be supplied on separate source media. Do not use a damaged image as an update source. For example, source DATA in bay 2 and an unmounted replacement cartridge in bay 4:

fds bay 2
fds inspect image /run/fds/media/02/fds-system-cli.img
fds inspect BAY04
fds burn system /run/fds/media/02/fds-system-cli.img BAY04

Use the actual source mount shown by fds bay 2. If the destination already contains a mounted cartridge, run fds eject 4 first. The burn preview identifies the source checksum, destination capacity/model/serial and confirmation command. Follow Media creation and writing for confirmation and status commands. Completion requires device flush, readback and GPT verification. Recovery uses the same protected writer as the main system: it refuses mounted destinations, active root storage and disks containing internal FDS partition names.

Keep the previous known-good SYSTEM cartridge. After a successful replacement write, use fds poweroff, swap SYSTEM while powered off, and boot normally. Rollback means restoring the previous SYSTEM cartridge; it does not undo DATA file changes, application migrations, internal boot updates or EEPROM changes.

Capture diagnostics and shut down

bash /usr/share/fds/capture-hardware /tmp/recovery-capture
cat /tmp/recovery-capture/status.tsv
fds power status
fds poweroff

The collector records identities, USB topology, mounts, kernel messages, packages and boot events with checksums. Its output is temporary unless copied to a healthy DATA cartridge after explicitly activating that DATA with fds data use N. Do not activate the damaged cartridge merely to save a report. Eject writable DATA after copying, or let fds poweroff perform the normal verified shutdown. See Shutdown when shutdown reports a blocking DATA error.

Recovery is independent of SYSTEM, but it still depends on working internal boot storage, the kernel and firmware. An external rescue medium is required when that layer fails. EEPROM preparation and rollback are described separately in EEPROM configuration; no EEPROM is changed by these recovery tools.