# Recovery and rollback [Documentation index](README.md) · [Boot images](developer/boot.md) · [Write cartridges](workstation.md#write-a-cartridge-to-usb) · [EEPROM](eeprom.md) FDS recovery is an independent maintenance system on internal storage. It can inspect a failed SYSTEM, check DATA and help prepare replacement cartridges. ## Build the recovery image Use the prepared x86_64 build host described in [Your first build](getting-started.md). From the repository root, run these commands sequentially: ```sh make recovery ``` `make recovery` builds the `recovery` rootfs profile independently, configures all packages and caches at image construction time, then creates EROFS. Its outputs are: | File | Meaning | | --- | --- | | `out/rootfs-recovery.tar` | Complete recovery rootfs, including package database and compiled s6 configuration | | `out/fds-recovery.img` | Raw EROFS payload for the internal GPT partition named `FDS_RECOVERY` | | `out/recovery-build.*/manifest.json` | Rootfs hash, filesystem hash, profile and size | | `out/manifests/rootfs-latest/` | Exact selected package archives and build input records | The latest-rootfs pointers also select this recovery build. Profile-specific rootfs pointers preserve earlier CLI/development builds. A recovery EROFS payload is not a whole-disk image and is not a SYSTEM cartridge. The SYSTEM builder rejects a recovery rootfs to prevent confusing those roles. ## Enter recovery Stage0 can enter recovery in two ways: 1. At its missing-SYSTEM or ambiguous-SYSTEM console, type `recovery` and Enter. 2. For a deliberate maintenance boot, use `fds.boot=recovery` in the boot partition's single-line `cmdline.txt`, replacing `fds.boot=normal` if present. Restore normal mode when maintenance is complete. Stage0 requires exactly one readable `FDS_RECOVERY` partition and mounts it read-only. It does not silently choose between duplicate partitions. The local prompt is: ```text FDS RECOVERY — LOCAL MAINTENANCE CONSOLE RECOVERY# ``` This is an explicit **root maintenance console**. It reads startup files from the immutable recovery image and uses a temporary home under `/run/fds`. The root password remains locked; no SSH or network login service is started. Normal CLI and development images continue to use the ordinary `fds` account and `FDS>`. Recovery inspects cartridges read-only. It does not automatically mount DATA writable, activate an ENVIRONMENT desktop, or enable Ethernet. Those actions require explicit commands. The Dasung controller remains in the base boot bundle. ## Inspect a failed system ```sh fds info fds bays fds bay 2 fds inspect BAY02 fds topology fds boot-profile lsusb -t lsblk -o NAME,MAJ:MIN,MODEL,SERIAL,SIZE,RO,TYPE,FSTYPE,PARTLABEL,MOUNTPOINTS dmesg ``` Use the calibrated bay number and the displayed model/serial to identify a cartridge. Do not infer a bay from `/dev/sda` or from discovery order. A valid inactive SYSTEM mounts read-only under `/run/fds/media/NN`; `fds bay N` reports the actual path. A bad filesystem or manifest produces an error instead of running anything from the cartridge. ## Check and repair DATA Start with a read-only check. For example, for DATA in bay 2: ```sh fds recovery check BAY02 ``` This command is available only to root in the recovery image. It identifies one USB disk and one `FDS_DATA` partition, unmounts its read-only view, reserves the disk exclusively, verifies GPT and the kernel partition identity, and invokes `e2fsck -f -n`. It does not repair the filesystem. A clean result leaves DATA unmounted and reports `SAFE TO REMOVE`. If the check reports problems, review its log and preserve a backup where possible. Preview the repair with: ```sh fds recovery repair BAY02 ``` The preview displays the model, serial, capacity and a confirmation command. Copy that exact command only after checking the intended cartridge. Its token binds the bay, kernel insertion number and current boot. A token from another insertion or boot is rejected. The preview makes no filesystem changes. Confirmed repair runs `e2fsck -f -p`, which performs conservative automatic repairs and stops when manual judgement is required. It then flushes the device, invalidates its block cache, and runs a second `e2fsck -f -n`. Only a successful verification produces `DATA REPAIRED AND VERIFIED` and `SAFE TO REMOVE`. Failed or interrupted checks/repairs retain a quarantine record across cartridge daemon restarts for the same insertion. They do not inherit an earlier SAFE status. The checker holds the disk reservation and is killed if its supervising daemon dies. The command runs synchronously: bay operations and orderly shutdown wait for it to finish; the local shell remains usable. A client timeout is not success and does not authorize removing media. Inspect the log and wait for the service to finish before retrying. Logs are in `/run/fds/recovery/` and are limited to 16 MiB per invocation. This narrow recovery command requires a readable GPT and primary ext filesystem signature. It deliberately does not guess partition boundaries, recreate a broken GPT, or force answers to destructive `e2fsck` questions. Such cases require an offline backup and expert use of the included filesystem tools. It cannot recover data that was never written to storage. For structured results, prefix the command with `fds --json`. After a successful check or repair, remove/reinsert DATA to inspect it again. Recovery still keeps it read-only until `fds data use N` is explicitly requested. ## Prepare replacement SYSTEM media A known-good SYSTEM image must be supplied on separate source media. Do not use a damaged image as an update source. For example, source DATA in bay 2 and an unmounted replacement cartridge in bay 4: ```sh fds bay 2 fds inspect image /run/fds/media/02/fds-system-cli.img fds inspect BAY04 fds burn system /run/fds/media/02/fds-system-cli.img BAY04 ``` Use the actual source mount shown by `fds bay 2`. If the destination already contains a mounted cartridge, run `fds eject 4` first. The burn preview identifies the source checksum, destination capacity/model/serial and confirmation command. Follow [Media creation and writing](developer/media-tools.md) for confirmation and status commands. Completion requires device flush, readback and GPT verification. Recovery uses the same protected writer as the main system: it refuses mounted destinations, active root storage and disks containing internal FDS partition names. Keep the previous known-good SYSTEM cartridge. After a successful replacement write, use `fds poweroff`, swap SYSTEM while powered off, and boot normally. Rollback means restoring the previous SYSTEM cartridge; it does not undo DATA file changes, application migrations, internal boot updates or EEPROM changes. ## Capture diagnostics and shut down ```sh bash /usr/share/fds/capture-hardware /tmp/recovery-capture cat /tmp/recovery-capture/status.tsv fds power status fds poweroff ``` The collector records identities, USB topology, mounts, kernel messages, packages and boot events with checksums. Its output is temporary unless copied to a healthy DATA cartridge after explicitly activating that DATA with `fds data use N`. Do not activate the damaged cartridge merely to save a report. Eject writable DATA after copying, or let `fds poweroff` perform the normal verified shutdown. See [Shutdown](developer/power.md) when shutdown reports a blocking DATA error. Recovery is independent of SYSTEM, but it still depends on working internal boot storage, the kernel and firmware. An external rescue medium is required when that layer fails. EEPROM preparation and rollback are described separately in [EEPROM configuration](eeprom.md); no EEPROM is changed by these recovery tools.