# Recovery and rollback Development reference and historical context. For current operating instructions, use the [user manual](../README.md). Acceptance applies only to the source and artifacts identified in each record. [Documentation index](README.md) · [Boot images](boot.md) · [Media tools](media-tools.md) · [EEPROM](eeprom.md) Recovery is a separate, read-only FDS image. It contains native s6, Bash, GNU utilities, XBPS, filesystem tools, the static FDS tools, and the base Dasung controller. It does not need files or programs from a SYSTEM cartridge. The software workflow is covered by `make recovery-test`; physical Pi recovery remains deferred. The [complete internal disk](internal-storage.md) includes this recovery image; its guide separates verified image construction from deferred physical installation. The build commands below create ordinary files and do not write a host disk. ## Build the recovery image Use the prepared x86_64 build host described in [Your first build](getting-started.md). From the repository root, run these commands sequentially: ```sh make recovery make rootfs-test make recovery-test ``` The full VM test also needs the Pi kernel/initramfs and a known-good CLI SYSTEM image. On a fresh checkout, build those first with `make rootfs PROFILE=cli`, `make rootfs-test`, `make initramfs`, and `make system-card PROFILE=cli`, then run the recovery sequence above. Run builds and VM tests sequentially because they share the same project-local Void environment. `make recovery` builds the `recovery` rootfs profile independently, configures all packages and caches at image construction time, then creates EROFS. Its outputs are: | File | Meaning | | --- | --- | | `out/rootfs-recovery.tar` | Complete recovery rootfs, including package database and compiled s6 configuration | | `out/fds-recovery.img` | Raw EROFS payload for the internal GPT partition named `FDS_RECOVERY` | | `out/recovery-build.*/manifest.json` | Rootfs hash, filesystem hash, profile and size | | `out/manifests/rootfs-latest/` | Exact selected package archives and build input records | The latest-rootfs pointers also select this recovery build. Profile-specific rootfs pointers preserve earlier CLI/development builds. A recovery EROFS payload is not a whole-disk image and is not a SYSTEM cartridge. The SYSTEM builder rejects a recovery rootfs to prevent confusing those roles. ## Enter recovery Stage0 can enter recovery in two ways: 1. At its missing-SYSTEM or ambiguous-SYSTEM console, type `recovery` and Enter. 2. For a deliberate maintenance boot, use `fds.boot=recovery` in the boot partition's single-line `cmdline.txt`, replacing `fds.boot=normal` if present. Restore normal mode when maintenance is complete. Stage0 requires exactly one readable `FDS_RECOVERY` partition and mounts it read-only. It does not silently choose between duplicate partitions. The ARM VM suite supplies disposable virtual partitions; physical firmware/NVMe/display behavior must still be checked on the Pi. The local prompt is: ```text FDS RECOVERY — LOCAL MAINTENANCE CONSOLE RECOVERY# ``` This is an explicit **root maintenance console**. It reads startup files from the immutable recovery image and uses a temporary home under `/run/fds`. The root password remains locked; no SSH or network login service is started. Normal CLI and development images continue to use the ordinary `fds` account and `FDS>`. Recovery inspects cartridges read-only. It does not automatically mount DATA writable, activate an ENVIRONMENT desktop, or enable Ethernet. Those actions require explicit commands. The Dasung controller remains in the base boot bundle. ## Inspect a failed system ```sh fds info fds bays fds bay 2 fds inspect BAY02 fds topology fds boot-profile lsusb -t lsblk -o NAME,MAJ:MIN,MODEL,SERIAL,SIZE,RO,TYPE,FSTYPE,PARTLABEL,MOUNTPOINTS dmesg ``` Use the calibrated bay number and the displayed model/serial to identify a cartridge. Do not infer a bay from `/dev/sda` or from discovery order. A valid inactive SYSTEM mounts read-only under `/run/fds/media/NN`; `fds bay N` reports the actual path. A bad filesystem or manifest produces an error instead of running anything from the cartridge. An empty bay configuration produces `UNCONFIGURED`, not guessed bay numbers. See [Cartridges](cartridges.md) for calibration. Permanent machine configuration on `FDS_INTERNAL` is part of the remaining M12 work. ## Check and repair DATA Start with a read-only check. For example, for DATA in bay 2: ```sh fds recovery check BAY02 ``` This command is available only to root in the recovery image. It identifies one USB disk and one `FDS_DATA` partition, unmounts its read-only view, reserves the disk exclusively, verifies GPT and the kernel partition identity, and invokes `e2fsck -f -n`. It does not repair the filesystem. A clean result leaves DATA unmounted and reports `SAFE TO REMOVE`. If DATA was explicitly activated writable, eject that session first. Close any shell whose current directory is on DATA and any other reader before checking; an ordinary busy-unmount failure is reported, never bypassed with lazy unmount. If the check reports problems, review its log and preserve a backup where possible. Preview the repair with: ```sh fds recovery repair BAY02 ``` The preview displays the model, serial, capacity and a confirmation command. Copy that exact command only after checking the intended cartridge. Its token binds the bay, kernel insertion number and current boot. A token from another insertion or boot is rejected. The preview makes no filesystem changes. Confirmed repair runs `e2fsck -f -p`, which performs conservative automatic repairs and stops when manual judgement is required. It then flushes the device, invalidates its block cache, and runs a second `e2fsck -f -n`. Only a successful verification produces `DATA REPAIRED AND VERIFIED` and `SAFE TO REMOVE`. The checker uses a temporary kernel loop device backed by the already verified partition descriptor. This lets `e2fsck` take its own exclusive device claim while FDS retains the physical whole-disk reservation. The loop is removed automatically when its last descriptor closes. This uses the existing kernel loop driver, `libc` crate and base `e2fsprogs`; no new package or Rust dependency is introduced. Failed or interrupted checks/repairs retain a quarantine record across cartridge daemon restarts for the same insertion. They do not inherit an earlier SAFE status. The checker holds the disk reservation and is killed if its supervising daemon dies. The command runs synchronously: bay operations and orderly shutdown wait for it to finish; the local shell remains usable. A client timeout is not success and does not authorize removing media. Inspect the log and wait for the service to finish before retrying. Logs are in `/run/fds/recovery/` and are limited to 16 MiB per invocation. This narrow recovery command requires a readable GPT and primary ext filesystem signature. It deliberately does not guess partition boundaries, recreate a broken GPT, or force answers to destructive `e2fsck` questions. Such cases require an offline backup and expert use of the included filesystem tools. It cannot recover data that was never written to storage. For structured results, prefix the command with `fds --json`. After a successful check or repair, remove/reinsert DATA to inspect it again. Recovery still keeps it read-only until `fds data use N` is explicitly requested. ## Prepare replacement SYSTEM media A known-good SYSTEM image must be supplied on separate source media. Do not use a damaged image as an update source. For example, source DATA in bay 2 and an unmounted replacement cartridge in bay 4: ```sh fds bay 2 fds inspect image /run/fds/media/02/fds-system-cli.img fds inspect BAY04 fds burn system /run/fds/media/02/fds-system-cli.img BAY04 ``` Use the actual source mount shown by `fds bay 2`. If the destination already contains a mounted cartridge, run `fds eject 4` first. The burn preview identifies the source checksum, destination capacity/model/serial and confirmation command. Follow [Media creation and writing](media-tools.md) for confirmation and status commands. Completion requires device flush, readback and GPT verification. Recovery uses the same protected writer as the main system: it refuses mounted destinations, active root storage and disks containing internal FDS partition names. Keep the previous known-good SYSTEM cartridge. After a successful replacement write, use `fds poweroff`, swap SYSTEM while powered off, and boot normally. Rollback means restoring the previous SYSTEM cartridge; it does not undo DATA file changes, application migrations, internal boot updates or EEPROM changes. ## Capture diagnostics and shut down ```sh bash /usr/share/fds/capture-hardware /tmp/recovery-capture cat /tmp/recovery-capture/status.tsv fds power status fds poweroff ``` The collector records identities, USB topology, mounts, kernel messages, packages and boot events with checksums. Its output is temporary unless copied to a healthy DATA cartridge after explicitly activating that DATA with `fds data use N`. Do not activate the damaged cartridge merely to save a report. Eject writable DATA after copying, or let `fds poweroff` perform the normal verified shutdown. See [Shutdown](power.md) when shutdown reports a blocking DATA error. Recovery is independent of SYSTEM, but it still depends on working internal boot storage, the kernel and firmware. An external rescue medium is required when that layer fails. EEPROM preparation and rollback are described separately in [EEPROM configuration](eeprom.md); no EEPROM is changed by these recovery tools.