fds-os/docs/developer/m12-validation.md

M12 production preparation: software evidence

Development reference and historical context. For current operating instructions, use the user manual. Acceptance applies only to the source and artifacts identified in each record.

Implementation ledger · EEPROM · Recovery

M12 software acceptance is complete, including the signed local FDS/OS 0.1.0 release. Physical Pi tests, EEPROM application, display behavior and storage power-loss behavior are deferred. Earlier checkpoints below preserve the development history; the final acceptance identifies the released bytes.

Final local release acceptance

Completed on 2026-09-21. The local release is out/fds-os-0.1.0/README.md. It contains 38 signed manifest entries, versioned images and packages, profile inventories, source and complete build-input archives, guides and reproducibility evidence. It has not been published or installed on hardware.

• Frozen source SHA-256: 8947e57552f5db834fbb6bcce6d79623bf0605072b42b4ccf04d1e4fe7bb9f29.
• Input-lock SHA-256: 6fa630446f91a7a0a2b85324b7a258ea6351978ff8f8a4cf9970f73f8d24fe2c.
• Manifest SHA-256: 0c223b4996258efbe33db780ab0d05019c35e36be291499a9548c27371c451a4.
• Signing public-key fingerprint (decoded 32 bytes): d5c95e8c1a079ed523b859c0ee78b8e849bbf387b3719dcdae1661ae4f7eb267.
• Signing public-key text-file SHA-256: 30d832747e95dca2c7ad3f5a117d2069a2d94e6d3eee2dd385caea688159e677.

Two newly restored trees, out/rebuild-m12-v5-a and out/rebuild-m12-v5-b, ran tools/in-frozen-build make all sequentially with no network access. Both started without FDS build outputs. The complete 27-artifact comparison passed, covering all five installable images, three rootfs tars, four initramfs formats, kernel/DTB, four EEPROM preview files and all nine FDS packages. Logs: out/logs/m12-offline-v5-{a,b}.log; comparison: out/m12-reproducibility-v5.json.

Every one of those artifacts also matches the normal build used by the complete Clap runtime acceptance. The identity proof is out/m12-tested-to-release-v5.json, and out/m12-clap-acceptance-index.json records the 21 acceptance logs and hashes. This establishes exact-byte coverage of the released images without repeating the same VM suite against identical images. Physical tests remain deferred.

The first restored tree additionally passed make smoke-test, make check and make signing-test inside the isolated wrapper. A second full comparison after those checks still passed. Logs are out/logs/m12-v5-{smoke-test,check,signing-test}.log; the comparison is out/m12-reproducibility-v5-after-checks.json. The optional direct host qemu-aarch64 smoke run is reported as skipped when absent; the separate ARM signing and OS runtime suites execute real ARM code.

The assembler verified source identity and rootfs build inputs before signing. Verification against the separately retained public key checked every manifest entry. OpenSSL independently verified the actual release signature. Both actual release archives were extracted with mode-preserving flags; the complete input lock and standalone source identity passed. Evidence: out/m12-release-acceptance-v5.json, with logs in out/logs/m12-v5-*.log and extracted files under out/m12-release-acceptance.mzkfsvev.

The private signing key is outside the checkout, snapshot and release output. Its corresponding trusted public key is /home/felis/.local/share/fds/signing-0.1.0-20260921/release-01.pub. On this workstation, from the checkout:

out/rebuild-m12-v5-a/target/x86_64-unknown-linux-gnu/release/fds-release verify \
  out/fds-os-0.1.0 \
  --key /home/felis/.local/share/fds/signing-0.1.0-20260921/release-01.pub

Image	Bytes	SHA-256
fds-boot-0.1.0.img	536,870,912	1466bbac3f782115bce78b48376ab90a7364416379700e944c529280b7253703
fds-internal-0.1.0.img	1,881,145,344	5197cd5edbbb9f689f12d233371cf0079ac50b36a4d3ca3cfe4c38cd320fc468
fds-recovery-0.1.0.img	491,593,728	4b0c9dad4336535d5b90c35cacd1500528836008be3384bdcd22b078b8a25f5b
fds-system-cli-0.1.0.img	493,879,296	03233a1118471cd8550ef2813fa99d371ef00b23edde5a3ade161d371e36c177
fds-system-development-0.1.0.img	1,989,148,672	9110f966212b59186e60ae7f88b6500fb0909478e6cbbb17735eebc969092c1e
kernel_2712-0.1.0.img	26,751,488	9e811bc55aa8e0b6a7233c6c84978244f2affee7bc808f3913dcd2fab3cd8013

The 67-section ledger records implementation, evidence and physical deferrals for the entire master plan. Main-checkout status documentation was updated after acceptance; the frozen trees and signed archives retain the exact checkpoint documentation used for their source identity. Consequently, their historical “verification in progress” wording predates this record. Runtime code, configuration and image bytes were not changed by this final documentation update. Use the preserved inputs to reproduce the exact release images. The signed RELEASE-NOTES.md also corrects an archived guide's fingerprint terminology: the tool hashes decoded key bytes, not the text file. The standalone release verification guide includes that correction. The original assembler output is retained at out/fds-os-0.1.0-checkpoint/; final packaging adds these signed release notes and updated standalone guidance without changing any image, package, source archive or input archive.

EEPROM configuration and rollback

make eeprom-test passed against the actual pinned 2 MiB Pi 5 EEPROM image. Evidence is in out/m12-eeprom.5b97ph36/; the public run is recorded in out/logs/m12-eeprom-final.log. make check also passed afterward.

The tests cover both production and development settings, exact configuration readback, unchanged firmware payloads, identical repeated outputs, preservation of unrelated conditional settings, exact rollback configuration, existing-output refusal, and rejection of invalid firmware/device/config inputs. The tool never invokes an update/apply command. The default firmware is a preparation input, not a backup of the user's Pi.

See the EEPROM guide for the source commit, hashes, complete profile settings, generated files, and the deferred physical application/rollback steps.

Independent recovery

The recovery implementation has its own rootfs profile, EROFS builder, local maintenance console, read-only cartridge defaults, and explicit DATA check/repair commands. It retains native s6 and the base Dasung controller. No additional Rust crate or target package was needed: repair uses the existing e2fsprogs and kernel loop support while preserving the physical disk reservation.

make recovery, make rootfs-test and the complete make recovery-test passed. The final runtime evidence is out/m12-recovery.rlvmytrz/, linked from out/m12-recovery-latest. Logs are out/logs/m12-recovery-identity-build.log, out/logs/m12-recovery-identity-rootfs.log, and out/logs/m12-recovery-final-acceptance.log.

Artifact	Verified result
Recovery rootfs	out/rootfs-build.D5TCGs/rootfs-aarch64.tar, 499,609,600 bytes; 180 packages and 1,394 AArch64 ELF files
Rootfs SHA-256	0611c0f8f24d04a733401063f9f52c7bc08f6389cb707601df083390fe063ae1
Recovery EROFS	out/recovery-build.9ukiaknh/recovery.erofs, 490,598,400 bytes
EROFS SHA-256	7a219ca803325e4d60c3acf1b4cffe8717e7a7ec4e65377f48c900c131910bcd

The tests use the packaged recovery image and Pi kernel. The only rootfs change for virtual cartridge tests is the observed virtual bay map; no replacement runtime binaries or simulated success responses are inserted.

Verified cases:

• Two independent EROFS constructions from the same tar produce identical bytes.
• Recovery boots with SYSTEM absent and with an unreadable SYSTEM. Its root is read-only, PID 1 is native s6, the maintenance console is root, and Dasung is supervised without needing the physical monitor.
• DATA remains read-only; ENVIRONMENT and Ethernet insertion do not automatically start the desktop or DHCP. An ordinary user cannot invoke recovery repairs.
• A real ext4 inode reference-count error is detected without changing DATA. Wrong confirmations and busy mounts are rejected. Confirmed repair fixes the error, flushes/invalidate caches, and passes another read-only check. The original test file survives, and a separate host e2fsck also passes.
• Failed checks survive daemon restart as quarantine. Confirmation from a previous insertion is rejected. Clean checks preserve every byte of the DATA image.
• Recovery writes a complete known-good SYSTEM image to a disposable virtual USB device, verifies it by readback, inspects it, and excludes it from DATA repair. A separate VM boots that written image to the ordinary UID 1000 FDS console. The source SYSTEM is the previously verified M10 CLI image; this proves the recovery replacement workflow, not a new release-signing claim.
• Killing the daemon during a real checker kills the child and preserves DATA. An explicit retry succeeds. Twenty partition-table rereads and a subsequent service restart preserve the unmounted SAFE state; temporary loops are gone.
• The diagnostic collector completes all 17 commands and verifies its output checksums. Successful cases power off through native s6.

The interruption/rescan case also passed three consecutive focused runs in out/m12-recovery-rescan.rlr9u2aq/, out/m12-recovery-rescan.cvzye8z0/, and out/m12-recovery-rescan.wj0jlfkq/. This caught and fixed a real bug: temporary partition disappearance during a reread previously cleared the eject record. The daemon now retains that record while the same physical disk insertion is present; a new insertion still invalidates it.

The normal CLI rebuild and regression checks also passed: make rootfs PROFILE=cli, make rootfs-test, make system-card PROFILE=cli, make init-test, make console-test, make data-test, make tooling-test, and make check. Evidence is in out/logs/m12-cli-*.log, out/logs/m12-tooling.log, and out/logs/m12-check.log. The normal-image root check emitted FDS_M12_RECOVERY_BOUNDARY, confirming that recovery repair remains unavailable outside the recovery profile, even to root.

That CLI rootfs is out/rootfs-build.HqBtpz/rootfs-aarch64.tar (SHA-256 af132511a6784eced9e52e971f010df6717e3caed751d14017ca9e924196e8b3), and its SYSTEM image is out/system-build.psymtwbr/system.img (SHA-256 5bc69f31d74a4c07596b696a72138cd24327ff4bc467f428717a56ce44c6bd27). Native init passed in out/m2-vm.GyJsNg, console checks in out/m5-vm.h5ygfe1j, and DATA checks in out/m7-vm.nt_tsdfs. Console readiness was 1,853.221941 ms, 32.182153 ms above the earlier M10 CLI baseline of 1,821.039788 ms; this is below the 100 ms investigation threshold. These artifacts precede the internal-settings implementation below. No physical timing or media-durability claim follows from these VM measurements.

Internal NVMe and machine settings

Implementation and current usage are described in Internal storage. make internal-test passed in out/m12-internal.vqr74fcs/, linked from out/m12-internal-latest. The public log is out/logs/m12-internal-final-acceptance.log; rootfs evidence is in out/logs/m12-machine-final-rootfs.log. The test uses packaged CLI and recovery images, the actual Pi kernel, and disposable virtual NVMe devices. No runtime binary or configuration file is injected into the booted OS.

Verified cases:

• Two complete 1,881,145,344-byte internal GPT images are byte-identical, with independent FAT, EROFS, ext4, GPT and payload checks. Their SHA-256 is 0d464ba18b0988a79a75ea54b6d81c5fa01acfea567b3d0a79f9faf2a428dfc6.
• make internal-image publishes the same hash at out/internal-build.xxzvw_3l/internal.img, linked from out/fds-internal.img; its log is out/logs/m12-internal-published.log.
• The actual loader selects one correctly laid-out NVMe and leaves every byte unchanged during read-only loading. The private mount is gone afterward.
• Root recovery installs validated settings atomically, preserves previous bytes, flushes/unmounts storage, and keeps this boot's active mapping unchanged even across a cartridge-daemon restart.
• A separate reboot loads the saved machine name, bay map and hardware catalog. The real virtual keyboard is recognized in its assigned bay with its saved name.
• Boot diagnostics survive reboot and are retrieved byte-for-byte. Non-root writes, path traversal, duplicates, existing retrieval outputs and oversized diagnostic files are rejected. Independent post-write e2fsck passes.
• BOOT and RECOVERY payload hashes remain unchanged after settings/diagnostic writes. Ordinary user data is not placed on internal storage.
• A complete USB lookalike cannot supply settings. Two eligible NVMes boot the normal UID 1000 console with explicit ambiguity/fallback status.
• Invalid JSON, writable settings, symlinks and unclean ext4 leave the disk bytes unchanged, record an error and retain a usable recovery console with image defaults. No automatic filesystem repair or journal replay is performed.

An earlier test used unsupported PCI hotplug on QEMU's root PCI bus; the final ambiguity fixture instead boots the ordinary SYSTEM with both NVMes present. Earlier partial runs remain available, but are not counted as acceptance.

The settings trust checks exposed omitted / metadata in the rootfs archive. EROFS had synthesized a host-owned mode-0777 root directory. The archive now records root:root mode 0755 explicitly, both OS builders reject old archives, and rootfs/VM tests check the result. out/m12-root-metadata-repro-audit.json compares two otherwise-identical recovery builds; only the new . entry differs. The build-specific auxiliary linker cache has been removed.

The tested recovery rootfs is out/rootfs-build.P4WWwz/rootfs-aarch64.tar (SHA-256 9d0e54533399da79616853e8233d2abb8efbc6b2d51772bd86f2669134c957b4), and its EROFS is out/recovery-build.1ukd0xgl/recovery.erofs (SHA-256 1c1930c86960f8e208408f4a0c244e4e5ae876263bab234b25307d4154ef3f91). The updated CLI rootfs is out/rootfs-build.BP1Ylv/rootfs-aarch64.tar (SHA-256 58eb153416972dca1c29a87d7b356638b8b1c08df56231470c18ee50fbad2d29), and SYSTEM is out/system-build.o9d6affu/system.img (SHA-256 cb544eb25f18b6572f78fc245818bd2cbfa18423b44d57b5af251069f2e6498e).

The complete recovery regression also passed against these updated artifacts in out/m12-recovery.idrwxslu/ (out/logs/m12-machine-recovery-regression.log). It repeats the missing/damaged SYSTEM, actual DATA repair, interrupted-checker, partition-reread, full SYSTEM burn/readback, diagnostics and replacement-boot checks described above. This time the source SYSTEM is the updated CLI image listed here, including the settings loader and corrected filesystem-root metadata.

The updated normal CLI has already passed rootfs, SYSTEM construction, native init, console independence and DATA regression checks. Logs are out/logs/m12-machine-cli-*.log; native init evidence is out/m2-vm.5FHPFv, console evidence is out/m5-vm.ifuo9ei6, and DATA evidence is out/m7-vm.6xcpnj97. The measured VM console time was 1,865.948488 ms, 44.908700 ms above the earlier M10 CLI baseline, below the 100 ms review threshold.

Build-host regression

After the host image-tool changes, make bootstrap, make smoke-test, and make check all passed. Logs are out/logs/m12-foundation-bootstrap.log, out/logs/m12-foundation-smoke.log, and out/logs/m12-foundation-check.log. The smoke test cross-built and inspected GNU hello as aarch64 glibc and the Rust smoketest as static aarch64-musl. Optional direct ARM execution was skipped because the workstation lacks qemu-aarch64; the separate ARM VM/runtime suites above still executed the actual OS tools. Upstream tracked Void files remain unchanged.

Development profile

The native GNU C/C++ and Rust toolchains, build systems, Git, GDB, strace and Vim are selected and documented in Development. The expanded image passed make development-test in out/m12-development.r6mrgm0b/. The rootfs is out/rootfs-build.p8DKvd/rootfs-aarch64.tar (2,012,047,360 bytes; 245 packages and 1,731 AArch64 ELF files), and its SYSTEM is out/system-build.jnubdw1v/system.img. Logs are out/logs/m12-development-clock-{build,rootfs,system,acceptance}.log.

The ordinary UID 1000 console compiled and ran C/C++ with CMake/Ninja, C with Meson/Ninja and Make, and a dependency-free Rust project with offline Cargo. GDB stopped at main, reported a backtrace and continued; strace captured the program's write; a local Git commit and headless Vim edit also succeeded. SYSTEM remained read-only and unchanged, optional services stayed off, and native shutdown completed. Desktop/network acceptance then passed in out/m8-vm.4c6_jj4h/, and the complete shutdown/DATA error suite passed in out/m10-vm.q_ewzl4g/; see out/logs/m12-development-{desktop,power}.log.

This test found a real startup issue: the VM's wall clock started at Unix second 1 while CMake's immutable inputs had the source timestamp 1,789,909,701. Ninja consequently tried to regenerate its manifest repeatedly. Native startup now advances only an older clock to the fixed image timestamp, without a time server or waiting. The compiler acceptance verifies that path; a separate make clock-test passed in out/m12-clock.l39g3uhx/, preserving a guest clock set to 2030 before the actual startup code runs. Non-root clock changes are rejected and boot measurements still use Linux CLOCK_BOOTTIME. out/logs/m12-clock-final.log records that check. This is not physical RTC validation and the floor does not supply the accurate current date.

The first corrected-clock run also exposed a test transport mistake: interactive Bash consumed the literal tab in a generated Makefile. Test files now cross the serial console as base64, preserving their exact bytes.

Release signature tool

make signing-test passed in out/m12-signing.md4r2kbo/, recorded in out/logs/m12-signing-public.log. It builds both x86_64 and static-musl ARM executables, checks the RFC 8032 Ed25519 vector, and independently signs/verifies the exact manifest domain with OpenSSL. Host and ARM signatures match OpenSSL's bytes. Wrong and weak keys, altered signature/manifest/artifact bytes, wrong sizes, missing files, symlinks, FIFOs, path traversal, duplicates, unknown fields, oversized manifests, insecure private-key permissions and output overwrites are rejected. These are disposable test keys, not a signed OS release.

The verifier is included in the base CLI package for all profiles. Its guide and dependency rationale are in Release signatures, and the build now collects license notices for all 68 resolved third-party ARM workspace crates, including the vendored libusb notices. make bootstrap, make smoke-test and make check passed after adding the signing dependencies, with logs under out/logs/m12-signing-{bootstrap,smoke,check}.log. The later package-notice and offline-build interface changes are being rebuilt and checked separately.

Repeated console timing checkpoint

After the clock floor, packaged signer/notices and deterministic package epoch changes, one console run measured 2,103.545197 ms. This exceeded the 100 ms investigation threshold against the earlier single M10 sample. Five alternating pairs then compared the preceding internal-settings CLI image with the updated CLI image, using the same kernel and initramfs for every run. Evidence and image hashes are in out/m12-console-pairs.yuoz7unr/comparison.json; all ten serial logs are retained beside it. The execution log is out/logs/m12-console-pairs.log.

Image	Five console-ready samples, ms	Median, ms
Before	1884.195231, 1909.087577, 1960.273690, 2055.400846, 1908.411769	1909.087577
Updated	1938.522887, 1913.188789, 1935.906888, 1958.748315, 1943.363749	1938.522887

The median increased by 29.435310 ms, below the threshold. The initial updated sample was outside the repeated updated range. This is workstation/QEMU evidence including scheduling noise, not physical Pi performance; no artificial delay or boot barrier was added. This checkpoint precedes Rust source-path normalization.

Complete build and release interface checkpoint

make packages now uses the same complete base-package builder as rootfs assembly. It passed in out/logs/m12-packages-public.log, including a fresh cross-build of the Pi kernel. The preceding make bootstrap, make smoke-test and make check passed in out/logs/m12-offline-foundation-{bootstrap,smoke,check}.log. make all now sequences the complete image set. Its first complete run passed in out/logs/m12-all-first.log. Subsequent rootfs and foundation checks passed, as did the complete boot matrix (out/m4-vm.kn9Hku/), native development tools (out/m12-development.hpk59nul/) and internal-storage acceptance (out/m12-internal.8z6nffpa/). The corresponding logs are out/logs/m12-all-{recovery-rootfs,check,boot,development,internal}.log. These results precede source-path normalization in Rust release builds.

The frozen-input, restore, network-isolation, comparison and signed-assembly interfaces are documented in Offline rebuilds. make release-contract-test passed in out/m12-release-contracts.yg33ry5r/ (out/logs/m12-release-contracts.log). Its small fixtures verify rejection of changed input bytes/modes/paths/symlinks, invalid lock metadata, differing or missing artifacts, comparison of one tree with itself, changed input locks, existing output reports/directories and private-key symlinks. Snapshot creation also refuses an active base-package build. These contract checks do not stand in for two actual complete offline builds.

A settings-size regression test also passed: individually valid TOML comments can expand during JSON escaping. The machine configuration now rejects an encoded bundle exceeding its loader limit before installation writes any data. All 15 shared-contract tests passed in out/logs/m12-machine-encoding-tests.log.

Earlier release checkpoints

The user-requested Clap migration is complete: 48 Rust tests, ARM CLI/alias execution, the complete image build, all runtime suites, signing and Dasung checks, and repeated startup measurements passed. All Rust command-line handling now uses the shared locked Clap dependency. The v4 offline candidate was deliberately stopped during kernel compilation to include this change; its second tree was not built and it has no accepted comparison. The accepted v5 pair uses this updated source checkpoint.

make release-contract-test also passed with the actual archive writer and the documented extraction flags in out/logs/m12-release-contracts-v4.log: repeated archives match, and extraction preserves ordinary executable modes, sticky directories, symlinks and the complete frozen lock. Default tar extraction had stripped those permissions in a small diagnostic fixture. The v3 candidate was stopped during kernel compilation before acceptance so that the corrected archive instructions and the completed user-guide audit could be frozen together. It is not a failed runtime test or an accepted release build.

The first real snapshot exposed two restore defects, now corrected: shallow Void bundles need their Git boundary file, and xbps-src's cross repository configuration retains only custom filenames containing local. The dependency that revealed the second issue was already captured; no network fallback was added. out/logs/m12-offline-v1-a.log and out/logs/m12-offline-v2-a.log retain the failed attempts. The resumed v2 tree is a diagnostic probe, not a clean release acceptance run.

The fresh offline v2 compilation matched the normal build byte-for-byte for all seven core FDS binaries, Dasung, the Pi kernel/DTB, and the kernel/Dasung packages. Records are out/m12-offline-v2-rust-comparison.json and out/m12-offline-v2-native-comparison.json. Personal absolute build paths are absent from all eight release executables after path normalization; see out/m12-remap-binaries.json. Bootstrap, smoke-test and checks passed in out/logs/m12-remap-{bootstrap,smoke,check}.log.

The documented larger-disk GPT relocation was checked on a disposable regular file in out/m12-internal-relocation.j29f1rx6/. After extending the image by 512 MiB and relocating its backup GPT, independent partition-table verification passed and every partition's start, size and payload SHA-256 remained unchanged. This verifies the image operation only; the physical write/readback and Pi boot procedure in Internal storage remains deferred.

Those checkpoints preceded the final clean v5 pair. The acceptance record at the top of this page closes the software release work with actual complete builds, comparisons, signing, archive verification and the master-plan audit.