fds-os/docs/developer/reproducible-builds.md

# Frozen inputs and offline rebuilds

Development reference and historical context. For current operating instructions, use the [user manual](../README.md). Acceptance applies only to the source and artifacts identified in each record.


[Documentation index](README.md) · [Build host](build-host.md) · [Release signatures](releases.md)

M12 adds a local input snapshot, fresh restore, network-isolated build, and
complete artifact comparison. **Two fresh network-isolated builds passed with
all 27 artifacts byte-identical.** The [M12 evidence page](m12-validation.md#final-local-release-acceptance)
records the source/input identities, signed local release and actual archive
round-trip. This result applies to the recorded inputs and documented build
host; it does not claim equivalence across arbitrary host environments.

Normal builds still use Void's signed rolling repositories. A frozen build uses
the selected package files and build environment from one explicit snapshot.
It does not silently consult a newer repository if an input is missing.

Current builds keep the upstream reference in `vendor/void-packages/` and use
`.host/void-packages/` for overlays and cached build state. New snapshots and
restores use this separate layout. Restoring an older
snapshot retains its historical layout and saved build scripts; its files and
acceptance are not rewritten. See [the workspace change](void-workspace.md) for
fresh validation of this migration.

## What is frozen

`tools/frozen-inputs create NEW_DIRECTORY` records:

- Project-owned source, documentation, configuration and package overlays, with
  a content identity independent of an FDS Git commit. This works before the
  repository's first commit too; it does not invent a commit identifier.
- The exact pinned Void checkout in a Git bundle, preserving upstream files and
  the shallow-history boundary needed to read its commit timestamp after restore.
- Cargo's locked vendored sources, the selected Rust toolchain and its static
  ARM target. Cargo credentials and user configuration are not copied.
- The project-local Void binary build environment, static XBPS host tools and
  image-tool prefix. Temporary build trees, personal home directories and
  mount points are excluded.
- Cached build dependencies, exact third-party packages selected by each of
  the three rootfs profiles, and kernel/DHCP/hello source archives. Local FDS
  packages are rebuilt from source instead of seeding the output repository.
- Pinned Pi firmware, the VM kernel fixture, and the official EEPROM preview
  inputs. The preview is not a backup of a physical Pi.

`lock.json` records paths, kinds, modes, lengths, hashes and symlink targets.
Added, missing or changed entries fail verification. The snapshot is several
gigabytes. The completed offline diagnostic build occupied about 24 GB and its
snapshot occupied 6.9 GB. Two restored builds need separate space; retained VM
test images add further usage. Check available space before starting and keep
headroom beyond these observations, which are not upper bounds. The workstation still supplies the documented
Arch/Linux prerequisites, including Python, bubblewrap, Git, Make and compression
tools. Their environment is recorded; this is not a replacement for a host OS.

## Prepare the snapshot

Complete the [first-build setup](getting-started.md), then run the complete
artifact build from the repository root:

```sh
make all
```

`make all` runs bootstrap and package construction, builds both normal SYSTEM
profiles and independent recovery, then creates the four initramfs formats,
production Pi boot volume, complete internal disk, signing tools and production
EEPROM preview. It does not write physical media or change EEPROM settings.
Use `make packages` to build only the FDS base packages. Builders perform their
normal format, ABI and rootfs checks; the complete runtime acceptance suite
remains separate.

Runtime test targets select their required profile explicitly: init, boot,
console, cartridge, DATA, media and stress checks use CLI; desktop and power
checks use development; recovery checks use recovery. They no longer depend on
which rootfs happened to be built last. `make rootfs-test` intentionally checks
the most recently exported archive, so use it immediately after a profile build
when validating that specific export.

Fetch all locked crates, including platform-specific sources that Cargo's vendor
operation requires even when they are not built for ARM:

```sh
cargo fetch --locked
./tools/frozen-inputs create "$PWD/out/inputs-0.1.0"
./tools/frozen-inputs verify "$PWD/out/inputs-0.1.0"
```

Use a new output directory. The command will not overwrite a previous snapshot.
The snapshot operation itself is offline; missing selected packages or locked
crate sources fail with an error. The subsequent fresh build establishes whether
the complete dependency set was captured. Keep failed outputs for diagnosis and
retry into a new directory.
Run every build, snapshot and VM workflow sequentially: the normal checkout
shares one Void masterdir. Snapshot creation locks rootfs/base-package assembly,
but a separate standalone kernel or VM command must not be started concurrently.

## Restore into two independent trees

If starting from the signed release's input archive, extract it as the ordinary
build user with `tar --extract --zstd --same-permissions --no-same-owner` and
the desired `--file` and new `--directory` paths. Mode preservation matters:
default extraction can mask writable bits or remove sticky bits and cause the
input lock to reject an otherwise intact archive. Verify the extracted snapshot
before restoring it. The release's README provides the complete commands.

The destination parent directory must already exist, and the destination itself
must not exist. Use paths without whitespace, as required by xbps-src:

```sh
./tools/frozen-inputs restore "$PWD/out/inputs-0.1.0" "$PWD/out/rebuild-a"
./tools/frozen-inputs restore "$PWD/out/inputs-0.1.0" "$PWD/out/rebuild-b"
```

Restoration verifies every locked entry first. Each new tree gets its own Void
masterdir, source/download caches, Cargo directory and Rustup directory. It starts
without an FDS target directory or generated FDS packages/images.

Run the builds sequentially and retain their full logs:

```sh
out/rebuild-a/tools/in-frozen-build make all >out/rebuild-a.log 2>&1
out/rebuild-b/tools/in-frozen-build make all >out/rebuild-b.log 2>&1
./tools/compare-builds out/rebuild-a out/rebuild-b \
  --report out/reproducibility-0.1.0.json
```

The wrapper changes to the restored project directory, uses its private Rust
caches, makes the rest of the workstation read-only, and removes network access
for the build and all child processes. Missing input therefore cannot be hidden
by a download. FDS's xbps-src wrapper keeps dependency resolution enabled and
excludes remote repositories; the custom repository filename includes `local`
so the pinned xbps-src also retains it in its cross sysroot. Rootfs assembly uses the frozen per-profile
repository. Setting `FDS_OFFLINE=1` alone selects offline inputs but does not
provide the wrapper's network isolation.

The comparison checks both rootfs tars, recovery rootfs, all installable images,
all four initramfs formats, Pi kernel/DTB, every FDS package and the EEPROM
configuration/rollback outputs. It reports each mismatch and exits nonzero if
anything differs. Matching files alone do not attest how they were built; retain
the isolated build logs alongside the report.

## Assemble a local signed release

After both complete builds match, create or choose a signing key as described in
[Release signatures](releases.md). Use the assembler in the first restored tree:

```sh
out/rebuild-a/tools/assemble-release \
  --inputs "$PWD/out/inputs-0.1.0" \
  --comparison-build "$PWD/out/rebuild-b" \
  --output-directory "$PWD/out/fds-os-0.1.0" \
  --key "$HOME/fds-signing/release-01.key"
```

The assembler verifies the snapshot again, checks that the first checkout still
matches its source identity, compares the complete pair, and verifies recorded
rootfs build inputs. It creates versioned image names, source and complete input
archives, package inventories, comparison evidence and usage guides. It signs
the manifest and immediately verifies every listed file. The private key is
never copied into the release, and existing output directories are refused.

The result is a **local** signed release with physical validation marked
`deferred`; it is not published or installed automatically. The supplied key
defines its signing identity. Distribute its fingerprint through an independently
trusted channel before someone relies on the bundled public key.

## Determinism and troubleshooting

FDS uses the pinned Void commit's timestamp for package and image timestamps.
Owned Rust binaries are stripped. `make` uses `tools/cargo-build` to map checkout,
Rustup and Cargo registry source paths to stable `/usr/src/fds` paths; otherwise
panic messages can embed the builder's absolute directory even in stripped
binaries. The helper preserves the target's static linking flags, and its flags
participate in Cargo's build cache. Direct `cargo build` remains useful for local
development, but release construction uses the helper. This uses Rust's
[source-path remapping](https://doc.rust-lang.org/rustc/command-line-arguments.html#--remap-path-prefix-remap-source-names-in-output)
and Cargo's [merged configuration arrays](https://doc.rust-lang.org/cargo/reference/config.html).
The kernel has explicit build user/host/time,
and filesystem UUIDs derive from content/configuration identities. Rootfs export
restores target ownership and capabilities instead of preserving the builder's
UID. It removes only the build-specific auxiliary linker cache and fetched XBPS
repository indexes; the installed package database, keys and runtime linker
cache remain available.

If verification reports changed snapshot entries, restore the original snapshot
or create a new one from the intended inputs. Do not edit `lock.json` to hide a
mismatch. If the two builds differ, use the comparison's named artifacts and
each build's rootfs/package manifests to identify the first differing stage.
Archive compression is not a substitute for comparing the uncompressed images.
Physical performance and durability still require the actual Pi and storage.