fds-os/docs/m12-validation.md

# M12 production preparation: software evidence

[Implementation ledger](implementation-status.md) · [EEPROM](eeprom.md) · [Recovery](recovery.md)

M12 **software acceptance is complete**, including the signed local FDS/OS
0.1.0 release. Physical Pi tests, EEPROM application, display behavior and
storage power-loss behavior are deferred. Earlier checkpoints below preserve
the development history; the final acceptance identifies the released bytes.

## Final local release acceptance

Completed on 2026-09-21. The local release is
[`out/fds-os-0.1.0/README.md`](../out/fds-os-0.1.0/README.md).
It contains 38 signed manifest entries, versioned images and packages,
profile inventories, source and complete build-input archives, guides and
reproducibility evidence. It has not been published or installed on hardware.

- Frozen source SHA-256: `8947e57552f5db834fbb6bcce6d79623bf0605072b42b4ccf04d1e4fe7bb9f29`.
- Input-lock SHA-256: `6fa630446f91a7a0a2b85324b7a258ea6351978ff8f8a4cf9970f73f8d24fe2c`.
- Manifest SHA-256: `0c223b4996258efbe33db780ab0d05019c35e36be291499a9548c27371c451a4`.
- Signing public-key fingerprint (decoded 32 bytes): `d5c95e8c1a079ed523b859c0ee78b8e849bbf387b3719dcdae1661ae4f7eb267`.
- Signing public-key text-file SHA-256: `30d832747e95dca2c7ad3f5a117d2069a2d94e6d3eee2dd385caea688159e677`.

Two newly restored trees, `out/rebuild-m12-v5-a` and `out/rebuild-m12-v5-b`,
ran `tools/in-frozen-build make all` sequentially with no network access.
Both started without FDS build outputs. The complete 27-artifact comparison
passed, covering all five installable images, three rootfs tars, four initramfs
formats, kernel/DTB, four EEPROM preview files and all nine FDS packages.
Logs: `out/logs/m12-offline-v5-{a,b}.log`; comparison:
[`out/m12-reproducibility-v5.json`](../out/m12-reproducibility-v5.json).

Every one of those artifacts also matches the normal build used by the complete
[Clap runtime acceptance](clap-validation.md). The identity proof is
[`out/m12-tested-to-release-v5.json`](../out/m12-tested-to-release-v5.json),
and `out/m12-clap-acceptance-index.json` records the 21 acceptance logs and hashes.
This establishes exact-byte coverage of the released images without repeating
the same VM suite against identical images. Physical tests remain deferred.

The first restored tree additionally passed `make smoke-test`, `make check`
and `make signing-test` inside the isolated wrapper. A second full comparison
after those checks still passed. Logs are `out/logs/m12-v5-{smoke-test,check,signing-test}.log`;
the comparison is `out/m12-reproducibility-v5-after-checks.json`.
The optional direct host `qemu-aarch64` smoke run is reported as skipped when
absent; the separate ARM signing and OS runtime suites execute real ARM code.

The assembler verified source identity and rootfs build inputs before signing.
Verification against the separately retained public key checked every manifest
entry. OpenSSL independently verified the actual release signature. Both actual
release archives were extracted with mode-preserving flags; the complete input
lock and standalone source identity passed. Evidence:
[`out/m12-release-acceptance-v5.json`](../out/m12-release-acceptance-v5.json),
with logs in `out/logs/m12-v5-*.log` and extracted files under
`out/m12-release-acceptance.mzkfsvev`.

The private signing key is outside the checkout, snapshot and release output.
Its corresponding trusted public key is
`/home/felis/.local/share/fds/signing-0.1.0-20260921/release-01.pub`.
On this workstation, from the checkout:

```sh
out/rebuild-m12-v5-a/target/x86_64-unknown-linux-gnu/release/fds-release verify \
  out/fds-os-0.1.0 \
  --key /home/felis/.local/share/fds/signing-0.1.0-20260921/release-01.pub
```

| Image | Bytes | SHA-256 |
| --- | ---: | --- |
| `fds-boot-0.1.0.img` | 536,870,912 | `1466bbac3f782115bce78b48376ab90a7364416379700e944c529280b7253703` |
| `fds-internal-0.1.0.img` | 1,881,145,344 | `5197cd5edbbb9f689f12d233371cf0079ac50b36a4d3ca3cfe4c38cd320fc468` |
| `fds-recovery-0.1.0.img` | 491,593,728 | `4b0c9dad4336535d5b90c35cacd1500528836008be3384bdcd22b078b8a25f5b` |
| `fds-system-cli-0.1.0.img` | 493,879,296 | `03233a1118471cd8550ef2813fa99d371ef00b23edde5a3ade161d371e36c177` |
| `fds-system-development-0.1.0.img` | 1,989,148,672 | `9110f966212b59186e60ae7f88b6500fb0909478e6cbbb17735eebc969092c1e` |
| `kernel_2712-0.1.0.img` | 26,751,488 | `9e811bc55aa8e0b6a7233c6c84978244f2affee7bc808f3913dcd2fab3cd8013` |

The [67-section ledger](implementation-status.md) records implementation,
evidence and physical deferrals for the entire master plan. Main-checkout status
documentation was updated after acceptance; the frozen trees and signed archives
retain the exact checkpoint documentation used for their source identity.
Consequently, their historical “verification in progress” wording predates this
record. Runtime code, configuration and image bytes were not changed by this
final documentation update. Use the preserved inputs to reproduce the exact
release images. The signed `RELEASE-NOTES.md` also corrects an archived guide's
fingerprint terminology: the tool hashes decoded key bytes, not the text file.
The standalone release verification guide includes that correction. The original
assembler output is retained at `out/fds-os-0.1.0-checkpoint/`; final packaging
adds these signed release notes and updated standalone guidance without changing
any image, package, source archive or input archive.

## EEPROM configuration and rollback

`make eeprom-test` passed against the actual pinned 2 MiB Pi 5 EEPROM image.
Evidence is in `out/m12-eeprom.5b97ph36/`; the public run is recorded in
`out/logs/m12-eeprom-final.log`. `make check` also passed afterward.

The tests cover both production and development settings, exact configuration
readback, unchanged firmware payloads, identical repeated outputs, preservation
of unrelated conditional settings, exact rollback configuration, existing-output
refusal, and rejection of invalid firmware/device/config inputs. The tool never
invokes an update/apply command. The default firmware is a preparation input,
not a backup of the user's Pi.

See the [EEPROM guide](eeprom.md) for the source commit, hashes, complete profile
settings, generated files, and the deferred physical application/rollback steps.

## Independent recovery

The recovery implementation has its own rootfs profile, EROFS builder, local
maintenance console, read-only cartridge defaults, and explicit DATA check/repair
commands. It retains native s6 and the base Dasung controller. No additional Rust
crate or target package was needed: repair uses the existing e2fsprogs and kernel
loop support while preserving the physical disk reservation.

`make recovery`, `make rootfs-test` and the complete `make recovery-test` passed.
The final runtime evidence is `out/m12-recovery.rlvmytrz/`, linked from
`out/m12-recovery-latest`. Logs are
`out/logs/m12-recovery-identity-build.log`,
`out/logs/m12-recovery-identity-rootfs.log`, and
`out/logs/m12-recovery-final-acceptance.log`.

| Artifact | Verified result |
| --- | --- |
| Recovery rootfs | `out/rootfs-build.D5TCGs/rootfs-aarch64.tar`, 499,609,600 bytes; 180 packages and 1,394 AArch64 ELF files |
| Rootfs SHA-256 | `0611c0f8f24d04a733401063f9f52c7bc08f6389cb707601df083390fe063ae1` |
| Recovery EROFS | `out/recovery-build.9ukiaknh/recovery.erofs`, 490,598,400 bytes |
| EROFS SHA-256 | `7a219ca803325e4d60c3acf1b4cffe8717e7a7ec4e65377f48c900c131910bcd` |

The tests use the packaged recovery image and Pi kernel. The only rootfs change
for virtual cartridge tests is the observed virtual bay map; no replacement
runtime binaries or simulated success responses are inserted.

Verified cases:

- Two independent EROFS constructions from the same tar produce identical bytes.
- Recovery boots with SYSTEM absent and with an unreadable SYSTEM. Its root is
  read-only, PID 1 is native s6, the maintenance console is root, and Dasung is
  supervised without needing the physical monitor.
- DATA remains read-only; ENVIRONMENT and Ethernet insertion do not automatically
  start the desktop or DHCP. An ordinary user cannot invoke recovery repairs.
- A real ext4 inode reference-count error is detected without changing DATA.
  Wrong confirmations and busy mounts are rejected. Confirmed repair fixes the
  error, flushes/invalidate caches, and passes another read-only check. The
  original test file survives, and a separate host e2fsck also passes.
- Failed checks survive daemon restart as quarantine. Confirmation from a previous
  insertion is rejected. Clean checks preserve every byte of the DATA image.
- Recovery writes a complete known-good SYSTEM image to a disposable virtual USB
  device, verifies it by readback, inspects it, and excludes it from DATA repair.
  A separate VM boots that written image to the ordinary UID 1000 FDS console.
  The source SYSTEM is the previously verified M10 CLI image; this proves the
  recovery replacement workflow, not a new release-signing claim.
- Killing the daemon during a real checker kills the child and preserves DATA.
  An explicit retry succeeds. Twenty partition-table rereads and a subsequent
  service restart preserve the unmounted SAFE state; temporary loops are gone.
- The diagnostic collector completes all 17 commands and verifies its output
  checksums. Successful cases power off through native s6.

The interruption/rescan case also passed three consecutive focused runs in
`out/m12-recovery-rescan.rlr9u2aq/`, `out/m12-recovery-rescan.cvzye8z0/`, and
`out/m12-recovery-rescan.wj0jlfkq/`. This caught and fixed a real bug: temporary
partition disappearance during a reread previously cleared the eject record.
The daemon now retains that record while the same physical disk insertion is
present; a new insertion still invalidates it.

The normal CLI rebuild and regression checks also passed: `make rootfs PROFILE=cli`,
`make rootfs-test`, `make system-card PROFILE=cli`, `make init-test`,
`make console-test`, `make data-test`, `make tooling-test`, and `make check`.
Evidence is in `out/logs/m12-cli-*.log`, `out/logs/m12-tooling.log`, and
`out/logs/m12-check.log`. The normal-image root check emitted
`FDS_M12_RECOVERY_BOUNDARY`, confirming that recovery repair remains unavailable
outside the recovery profile, even to root.

That CLI rootfs is `out/rootfs-build.HqBtpz/rootfs-aarch64.tar`
(SHA-256 `af132511a6784eced9e52e971f010df6717e3caed751d14017ca9e924196e8b3`),
and its SYSTEM image is `out/system-build.psymtwbr/system.img`
(SHA-256 `5bc69f31d74a4c07596b696a72138cd24327ff4bc467f428717a56ce44c6bd27`).
Native init passed in `out/m2-vm.GyJsNg`, console checks in `out/m5-vm.h5ygfe1j`,
and DATA checks in `out/m7-vm.nt_tsdfs`. Console readiness was 1,853.221941 ms,
32.182153 ms above the earlier M10 CLI baseline of 1,821.039788 ms; this is below
the 100 ms investigation threshold. These artifacts precede the internal-settings
implementation below. No physical timing or media-durability claim follows from
these VM measurements.

## Internal NVMe and machine settings

Implementation and current usage are described in [Internal storage](internal-storage.md).
`make internal-test` passed in `out/m12-internal.vqr74fcs/`, linked from
`out/m12-internal-latest`. The public log is
`out/logs/m12-internal-final-acceptance.log`; rootfs evidence is in
`out/logs/m12-machine-final-rootfs.log`. The test uses packaged CLI and recovery
images, the actual Pi kernel, and disposable virtual NVMe devices. No runtime
binary or configuration file is injected into the booted OS.

Verified cases:

- Two complete 1,881,145,344-byte internal GPT images are byte-identical, with
  independent FAT, EROFS, ext4, GPT and payload checks. Their SHA-256 is
  `0d464ba18b0988a79a75ea54b6d81c5fa01acfea567b3d0a79f9faf2a428dfc6`.
- `make internal-image` publishes the same hash at `out/internal-build.xxzvw_3l/internal.img`,
  linked from `out/fds-internal.img`; its log is `out/logs/m12-internal-published.log`.
- The actual loader selects one correctly laid-out NVMe and leaves every byte
  unchanged during read-only loading. The private mount is gone afterward.
- Root recovery installs validated settings atomically, preserves previous bytes,
  flushes/unmounts storage, and keeps this boot's active mapping unchanged even
  across a cartridge-daemon restart.
- A separate reboot loads the saved machine name, bay map and hardware catalog.
  The real virtual keyboard is recognized in its assigned bay with its saved name.
- Boot diagnostics survive reboot and are retrieved byte-for-byte. Non-root
  writes, path traversal, duplicates, existing retrieval outputs and oversized
  diagnostic files are rejected. Independent post-write e2fsck passes.
- BOOT and RECOVERY payload hashes remain unchanged after settings/diagnostic
  writes. Ordinary user data is not placed on internal storage.
- A complete USB lookalike cannot supply settings. Two eligible NVMes boot the
  normal UID 1000 console with explicit ambiguity/fallback status.
- Invalid JSON, writable settings, symlinks and unclean ext4 leave the disk bytes
  unchanged, record an error and retain a usable recovery console with image
  defaults. No automatic filesystem repair or journal replay is performed.

An earlier test used unsupported PCI hotplug on QEMU's root PCI bus; the final
ambiguity fixture instead boots the ordinary SYSTEM with both NVMes present.
Earlier partial runs remain available, but are not counted as acceptance.

The settings trust checks exposed omitted `/` metadata in the rootfs archive.
EROFS had synthesized a host-owned mode-0777 root directory. The archive now
records root:root mode 0755 explicitly, both OS builders reject old archives,
and rootfs/VM tests check the result. `out/m12-root-metadata-repro-audit.json`
compares two otherwise-identical recovery builds; only the new `.` entry differs.
The build-specific auxiliary linker cache has been removed.

The tested recovery rootfs is `out/rootfs-build.P4WWwz/rootfs-aarch64.tar`
(SHA-256 `9d0e54533399da79616853e8233d2abb8efbc6b2d51772bd86f2669134c957b4`),
and its EROFS is `out/recovery-build.1ukd0xgl/recovery.erofs`
(SHA-256 `1c1930c86960f8e208408f4a0c244e4e5ae876263bab234b25307d4154ef3f91`).
The updated CLI rootfs is `out/rootfs-build.BP1Ylv/rootfs-aarch64.tar`
(SHA-256 `58eb153416972dca1c29a87d7b356638b8b1c08df56231470c18ee50fbad2d29`),
and SYSTEM is `out/system-build.o9d6affu/system.img`
(SHA-256 `cb544eb25f18b6572f78fc245818bd2cbfa18423b44d57b5af251069f2e6498e`).

The complete recovery regression also passed against these updated artifacts in
`out/m12-recovery.idrwxslu/` (`out/logs/m12-machine-recovery-regression.log`). It
repeats the missing/damaged SYSTEM, actual DATA repair, interrupted-checker,
partition-reread, full SYSTEM burn/readback, diagnostics and replacement-boot
checks described above. This time the source SYSTEM is the updated CLI image
listed here, including the settings loader and corrected filesystem-root metadata.

The updated normal CLI has already passed rootfs, SYSTEM construction, native
init, console independence and DATA regression checks. Logs are
`out/logs/m12-machine-cli-*.log`; native init evidence is `out/m2-vm.5FHPFv`,
console evidence is `out/m5-vm.ifuo9ei6`, and DATA evidence is
`out/m7-vm.6xcpnj97`. The measured VM console time was 1,865.948488 ms,
44.908700 ms above the earlier M10 CLI baseline, below the 100 ms review threshold.

## Build-host regression

After the host image-tool changes, `make bootstrap`, `make smoke-test`, and
`make check` all passed. Logs are `out/logs/m12-foundation-bootstrap.log`,
`out/logs/m12-foundation-smoke.log`, and `out/logs/m12-foundation-check.log`.
The smoke test cross-built and inspected GNU hello as aarch64 glibc and the Rust
smoketest as static aarch64-musl. Optional direct ARM execution was skipped because
the workstation lacks `qemu-aarch64`; the separate ARM VM/runtime suites above
still executed the actual OS tools. Upstream tracked Void files remain unchanged.

## Development profile

The native GNU C/C++ and Rust toolchains, build systems, Git, GDB, strace and Vim
are selected and documented in [Development](development.md). The expanded image
passed `make development-test` in `out/m12-development.r6mrgm0b/`.
The rootfs is `out/rootfs-build.p8DKvd/rootfs-aarch64.tar` (2,012,047,360 bytes;
245 packages and 1,731 AArch64 ELF files), and its SYSTEM is
`out/system-build.jnubdw1v/system.img`. Logs are
`out/logs/m12-development-clock-{build,rootfs,system,acceptance}.log`.

The ordinary UID 1000 console compiled and ran C/C++ with CMake/Ninja, C with
Meson/Ninja and Make, and a dependency-free Rust project with offline Cargo.
GDB stopped at `main`, reported a backtrace and continued; strace captured the
program's write; a local Git commit and headless Vim edit also succeeded.
SYSTEM remained read-only and unchanged, optional services stayed off, and
native shutdown completed. Desktop/network acceptance then passed in
`out/m8-vm.4c6_jj4h/`, and the complete shutdown/DATA error suite passed in
`out/m10-vm.q_ewzl4g/`; see `out/logs/m12-development-{desktop,power}.log`.

This test found a real startup issue: the VM's wall clock started at Unix second
1 while CMake's immutable inputs had the source timestamp 1,789,909,701.
Ninja consequently tried to regenerate its manifest repeatedly. Native startup
now advances only an older clock to the fixed image timestamp, without a time
server or waiting. The compiler acceptance verifies that path; a separate
`make clock-test` passed in `out/m12-clock.l39g3uhx/`, preserving a guest clock
set to 2030 before the actual startup code runs. Non-root clock changes are
rejected and boot measurements still use `Linux CLOCK_BOOTTIME`.
`out/logs/m12-clock-final.log` records that check. This is not physical RTC
validation and the floor does not supply the accurate current date.

The first corrected-clock run also exposed a test transport mistake: interactive
Bash consumed the literal tab in a generated Makefile. Test files now cross the
serial console as base64, preserving their exact bytes.

## Release signature tool

`make signing-test` passed in `out/m12-signing.md4r2kbo/`, recorded in
`out/logs/m12-signing-public.log`. It builds both x86_64 and static-musl ARM
executables, checks the RFC 8032 Ed25519 vector, and independently signs/verifies
the exact manifest domain with OpenSSL. Host and ARM signatures match OpenSSL's
bytes. Wrong and weak keys, altered signature/manifest/artifact bytes, wrong
sizes, missing files, symlinks, FIFOs, path traversal, duplicates, unknown fields,
oversized manifests, insecure private-key permissions and output overwrites are
rejected. These are disposable test keys, not a signed OS release.

The verifier is included in the base CLI package for all profiles. Its guide and
dependency rationale are in [Release signatures](releases.md), and the build now
collects license notices for all 68 resolved third-party ARM workspace crates,
including the vendored libusb notices. `make bootstrap`, `make smoke-test` and
`make check` passed after adding the signing dependencies, with logs under
`out/logs/m12-signing-{bootstrap,smoke,check}.log`. The later package-notice and
offline-build interface changes are being rebuilt and checked separately.

## Repeated console timing checkpoint

After the clock floor, packaged signer/notices and deterministic package epoch
changes, one console run measured 2,103.545197 ms. This exceeded the 100 ms
investigation threshold against the earlier single M10 sample. Five alternating
pairs then compared the preceding internal-settings CLI image with the updated
CLI image, using the same kernel and initramfs for every run. Evidence and image
hashes are in `out/m12-console-pairs.yuoz7unr/comparison.json`; all ten serial logs
are retained beside it. The execution log is `out/logs/m12-console-pairs.log`.

| Image | Five console-ready samples, ms | Median, ms |
| --- | --- | ---: |
| Before | 1884.195231, 1909.087577, 1960.273690, 2055.400846, 1908.411769 | 1909.087577 |
| Updated | 1938.522887, 1913.188789, 1935.906888, 1958.748315, 1943.363749 | 1938.522887 |

The median increased by 29.435310 ms, below the threshold. The initial updated
sample was outside the repeated updated range. This is workstation/QEMU evidence
including scheduling noise, not physical Pi performance; no artificial delay or
boot barrier was added. This checkpoint precedes Rust source-path normalization.

## Complete build and release interface checkpoint

`make packages` now uses the same complete base-package builder as rootfs
assembly. It passed in `out/logs/m12-packages-public.log`, including a fresh
cross-build of the Pi kernel. The preceding `make bootstrap`, `make smoke-test`
and `make check` passed in `out/logs/m12-offline-foundation-{bootstrap,smoke,check}.log`.
`make all` now sequences the complete image set. Its first complete run passed
in `out/logs/m12-all-first.log`. Subsequent rootfs and foundation checks passed,
as did the complete boot matrix (`out/m4-vm.kn9Hku/`), native development tools
(`out/m12-development.hpk59nul/`) and internal-storage acceptance
(`out/m12-internal.8z6nffpa/`). The corresponding logs are
`out/logs/m12-all-{recovery-rootfs,check,boot,development,internal}.log`.
These results precede source-path normalization in Rust release builds.

The frozen-input, restore, network-isolation, comparison and signed-assembly
interfaces are documented in [Offline rebuilds](reproducible-builds.md).
`make release-contract-test` passed in `out/m12-release-contracts.yg33ry5r/`
(`out/logs/m12-release-contracts.log`). Its small fixtures verify rejection of
changed input bytes/modes/paths/symlinks, invalid lock metadata, differing or
missing artifacts, comparison of one tree with itself, changed input locks,
existing output reports/directories and private-key symlinks. Snapshot creation
also refuses an active base-package build. These contract checks do not stand
in for two actual complete offline builds.

A settings-size regression test also passed: individually valid TOML comments
can expand during JSON escaping. The machine configuration now rejects an
encoded bundle exceeding its loader limit before installation writes any data.
All 15 shared-contract tests passed in `out/logs/m12-machine-encoding-tests.log`.

## Earlier release checkpoints

The user-requested [Clap migration](clap-validation.md) is complete: 48 Rust
tests, ARM CLI/alias execution, the complete image build, all runtime suites,
signing and Dasung checks, and repeated startup measurements passed. All Rust
command-line handling now uses the shared locked Clap dependency. The v4
offline candidate was deliberately stopped during kernel compilation to include
this change; its second tree was not built and it has no accepted comparison.
The accepted v5 pair uses this updated source checkpoint.

`make release-contract-test` also passed with the actual archive writer and the
documented extraction flags in `out/logs/m12-release-contracts-v4.log`: repeated
archives match, and extraction preserves ordinary executable modes, sticky
directories, symlinks and the complete frozen lock. Default tar extraction had
stripped those permissions in a small diagnostic fixture. The v3 candidate was
stopped during kernel compilation before acceptance so that the corrected
archive instructions and the completed user-guide audit could be frozen
together. It is not a failed runtime test or an accepted release build.

The first real snapshot exposed two restore defects, now corrected: shallow
Void bundles need their Git boundary file, and xbps-src's cross repository
configuration retains only custom filenames containing `local`. The dependency
that revealed the second issue was already captured; no network fallback was
added. `out/logs/m12-offline-v1-a.log` and `out/logs/m12-offline-v2-a.log` retain
the failed attempts. The resumed v2 tree is a diagnostic probe, not a clean
release acceptance run.

The fresh offline v2 compilation matched the normal build byte-for-byte for all
seven core FDS binaries, Dasung, the Pi kernel/DTB, and the kernel/Dasung packages.
Records are `out/m12-offline-v2-rust-comparison.json` and
`out/m12-offline-v2-native-comparison.json`. Personal absolute build paths are
absent from all eight release executables after path normalization; see
`out/m12-remap-binaries.json`. Bootstrap, smoke-test and checks passed in
`out/logs/m12-remap-{bootstrap,smoke,check}.log`.

The documented larger-disk GPT relocation was checked on a disposable regular
file in `out/m12-internal-relocation.j29f1rx6/`. After extending the image by
512 MiB and relocating its backup GPT, independent partition-table verification
passed and every partition's start, size and payload SHA-256 remained unchanged.
This verifies the image operation only; the physical write/readback and Pi boot
procedure in [Internal storage](internal-storage.md) remains deferred.

Those checkpoints preceded the final clean v5 pair. The acceptance record at
the top of this page closes the software release work with actual complete
builds, comparisons, signing, archive verification and the master-plan audit.