From f4bc28043acff7c0f5f045fddec35e73e389b728 Mon Sep 17 00:00:00 2001 From: Felis Date: Sun, 27 Sep 2026 00:29:53 +0800 Subject: [PATCH] Support native SD storage and consolidate fds-flash tooling Accept native SD cards for internal settings alongside legacy NVMe, reject USB ancestry and ambiguous disks, and preserve read-only boot loading with explicit recovery writes. Require built-in MMC drivers, validate cached kernel configuration, and provide SD-only, SD-first and USB-first EEPROM profiles. Expose typed create and inspect commands through fds-flash, reuse the existing cartridge creation code, and document the optional e2fsprogs package dependency. Extend storage, EEPROM, flashing and hardware-test coverage and advance the wiki reference to the published SD guide. Validation: rootfs checks for CLI/development, boot matrix, internal storage, EEPROM, flash, workstation, emulator, make check and wiki checks passed. The full internal suite passed on an unchanged rerun after one unexplained VM shutdown stall. Standalone init-test was blocked by its unavailable pinned upstream kernel. Physical Pi checks remain pending. --- Makefile | 3 +- README.md | 2 +- config/eeprom/development.conf | 4 +- config/eeprom/maintenance.conf | 8 ++ config/eeprom/production.conf | 4 +- fds-os.wiki | 2 +- image/build-boot-volume | 2 +- image/build-internal | 6 +- image/pi5/config.txt | 3 +- packages/fds-kernel/files/fds.config | 10 ++ packaging/arch/PKGBUILD | 1 + rust/fds-cli/src/machine.rs | 152 ++++++++++++++++++---- rust/fds-workstation/src/cartridge.rs | 2 +- rust/fds-workstation/src/flash/cli.rs | 166 +++++++++++++++++++++---- rust/fds-workstation/src/flash/mod.rs | 43 +++++-- tests/hardware/session-template.json | 39 +++++- tests/integration/m12-eeprom.py | 5 +- tests/integration/m12-internal.py | 102 ++++++++++++--- tests/integration/workstation-flash.py | 49 +++++++- tools/build-kernel | 17 ++- tools/clean-builds | 2 +- tools/configure-pi-eeprom | 2 +- 22 files changed, 527 insertions(+), 97 deletions(-) create mode 100644 config/eeprom/maintenance.conf diff --git a/Makefile b/Makefile index 884c35d..ce62359 100644 --- a/Makefile +++ b/Makefile @@ -49,7 +49,7 @@ help: 'make recovery Build the independent recovery rootfs and EROFS payload' \ 'make recovery-test Verify SYSTEM-independent recovery and DATA repair in ARM VMs' \ 'make internal-image Assemble FDS_BOOT, FDS_RECOVERY and machine settings as one GPT disk' \ - 'make internal-test Verify virtual NVMe settings, persistence and rejection paths' \ + 'make internal-test Verify virtual SD/NVMe settings, persistence and rejection paths' \ 'make development-test Compile, execute and debug C/C++/Rust in the development SYSTEM' \ 'make clock-test Verify that native startup preserves a newer guest clock' \ 'make signing Build the host release signer and static ARM verifier' \ @@ -226,6 +226,7 @@ internal-image: ./image/build-internal $(if $(MACHINE_CONFIG),--machine-config "$(MACHINE_CONFIG)") 2>&1 | tee out/logs/internal-build.log internal-test: + cargo test --locked --offline --target x86_64-unknown-linux-gnu -p fds-cli python3 tests/integration/m12-internal.py 2>&1 | tee out/logs/internal-checks.log .PHONY: development-test clock-test diff --git a/README.md b/README.md index 1e30ed6..065847b 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ authoritative manual: - [Project overview](http://gitea.home.arpa/felis/fds-os/wiki/Overview) - [Run on real hardware, including HDMI + USB wiring](http://gitea.home.arpa/felis/fds-os/wiki/Hardware-Setup) -- [Flash NVMe and cartridge disks](http://gitea.home.arpa/felis/fds-os/wiki/Flashing) +- [Flash SD and cartridge disks](http://gitea.home.arpa/felis/fds-os/wiki/Flashing) - [Build the OS and start the emulator](http://gitea.home.arpa/felis/fds-os/wiki/Getting-Started) - [Build software cartridges and use workstation tools](http://gitea.home.arpa/felis/fds-os/wiki/Workstation) - [Use cartridges](http://gitea.home.arpa/felis/fds-os/wiki/Cartridges), [DATA](http://gitea.home.arpa/felis/fds-os/wiki/Data) and [the desktop](http://gitea.home.arpa/felis/fds-os/wiki/Desktop) diff --git a/config/eeprom/development.conf b/config/eeprom/development.conf index e5c3f45..5af0787 100644 --- a/config/eeprom/development.conf +++ b/config/eeprom/development.conf @@ -1,6 +1,6 @@ -# Pi 5 development: NVMe first, SD rescue fallback, diagnostic UART enabled. +# Pi 5 development: SD first, USB maintenance fallback, diagnostic UART enabled. [all] -BOOT_ORDER=0xf16 +BOOT_ORDER=0xf41 BOOT_UART=1 NET_INSTALL_ENABLED=0 NET_INSTALL_AT_POWER_ON=0 diff --git a/config/eeprom/maintenance.conf b/config/eeprom/maintenance.conf new file mode 100644 index 0000000..b2d7430 --- /dev/null +++ b/config/eeprom/maintenance.conf @@ -0,0 +1,8 @@ +# Pi 5 maintenance: boot a Raspberry Pi OS USB drive before the FDS microSD. +[all] +BOOT_ORDER=0xf14 +BOOT_UART=1 +NET_INSTALL_ENABLED=0 +NET_INSTALL_AT_POWER_ON=0 +POWER_OFF_ON_HALT=1 +WAIT_FOR_POWER_BUTTON=0 diff --git a/config/eeprom/production.conf b/config/eeprom/production.conf index 5c004a4..d4bcdb2 100644 --- a/config/eeprom/production.conf +++ b/config/eeprom/production.conf @@ -1,6 +1,6 @@ -# Pi 5 production: internal NVMe only, no firmware network-install scanning. +# Pi 5 production: native microSD only, no firmware USB/network-install scanning. [all] -BOOT_ORDER=0xf6 +BOOT_ORDER=0xf1 BOOT_UART=0 NET_INSTALL_ENABLED=0 NET_INSTALL_AT_POWER_ON=0 diff --git a/fds-os.wiki b/fds-os.wiki index ef88914..51edcf8 160000 --- a/fds-os.wiki +++ b/fds-os.wiki @@ -1 +1 @@ -Subproject commit ef88914454b46bf905cc2a37ce1e1f46af0539c6 +Subproject commit 51edcf8714a5c0166b14e975dcb34fa4201841e8 diff --git a/image/build-boot-volume b/image/build-boot-volume index e399d4c..0e08979 100755 --- a/image/build-boot-volume +++ b/image/build-boot-volume @@ -99,4 +99,4 @@ for name, source in [('fds-boot.img', work.name+'/boot.fat'), ('boot-volume', wo target.symlink_to(source) target.replace(project/'out'/name) print(f'PASS: 512 MiB FDS_BOOT FAT32 partition image, all files verified: {work}') -print('SKIP: firmware boot, NVMe and display mode validation require a physical Pi') +print('SKIP: firmware boot, native SD and display mode validation require a physical Pi') diff --git a/image/build-internal b/image/build-internal index 7c172af..3408d44 100755 --- a/image/build-internal +++ b/image/build-internal @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Create a complete internal NVMe GPT image as a new ordinary file.""" +"""Create a complete internal SD/NVMe GPT image as a new ordinary file.""" import argparse import json import os @@ -93,5 +93,5 @@ if args.output_directory is None: link = project / 'out/fds-internal.img.next' link.symlink_to(work.name + '/internal.img') link.replace(project / 'out/fds-internal.img') -print(f'PASS: complete internal NVMe image, three verified GPT payloads: {work}') -print('SKIP: no physical disk was written; Pi firmware/NVMe boot requires hardware') +print(f'PASS: complete internal SD/NVMe image, three verified GPT payloads: {work}') +print('SKIP: no physical disk was written; Pi firmware/SD boot requires hardware') diff --git a/image/pi5/config.txt b/image/pi5/config.txt index f4a23c6..e69a555 100644 --- a/image/pi5/config.txt +++ b/image/pi5/config.txt @@ -1,9 +1,8 @@ -# FDS/OS Raspberry Pi 5: internal NVMe supplies boot files. +# FDS/OS Raspberry Pi 5: native microSD supplies boot files. [pi5] arm_64bit=1 kernel=kernel_2712.img initramfs fds-initramfs.img followkernel -dtparam=pciex1 dtoverlay=vc4-kms-v3d,noaudio camera_auto_detect=0 display_auto_detect=0 diff --git a/packages/fds-kernel/files/fds.config b/packages/fds-kernel/files/fds.config index 7f9d699..57e7bcd 100644 --- a/packages/fds-kernel/files/fds.config +++ b/packages/fds-kernel/files/fds.config @@ -33,6 +33,16 @@ CONFIG_PINCTRL_RP1=y CONFIG_COMMON_CLK_RP1=y CONFIG_BCM2712_IOMMU=y CONFIG_BLK_DEV_NVME=y +# Native Pi SD must be available before modules can be read from SYSTEM. +CONFIG_MMC=y +CONFIG_MMC_BLOCK=y +CONFIG_MMC_SDHCI=y +CONFIG_MMC_SDHCI_PLTFM=y +CONFIG_MMC_SDHCI_IPROC=y +CONFIG_MMC_SDHCI_BRCMSTB=y +CONFIG_MMC_BCM2835=y +# Generic PCI SDHCI exercises native MMC discovery in the ARM VM fixture. +CONFIG_MMC_SDHCI_PCI=y CONFIG_HID=y CONFIG_HID_GENERIC=y CONFIG_USB_HID=y diff --git a/packaging/arch/PKGBUILD b/packaging/arch/PKGBUILD index bb907a1..7094c73 100644 --- a/packaging/arch/PKGBUILD +++ b/packaging/arch/PKGBUILD @@ -11,6 +11,7 @@ makedepends=('rust' 'python' 'git') optdepends=( 'bubblewrap: cartridge creation and inspection in a private namespace' 'erofs-utils: create and inspect cartridge payload filesystems' + 'e2fsprogs: create DATA images with fds-flash' 'qemu-system-aarch64: run FDS in the emulator' 'qemu-img: writable emulator DATA overlays' ) diff --git a/rust/fds-cli/src/machine.rs b/rust/fds-cli/src/machine.rs index e948c62..be1294c 100644 --- a/rust/fds-cli/src/machine.rs +++ b/rust/fds-cli/src/machine.rs @@ -1,4 +1,4 @@ -//! Internal NVMe access is explicit and isolated in this process's mount namespace. +//! Internal SD/NVMe access is explicit and isolated in this process's mount namespace. use fds_common::{ Error, Result, machine::{self, Config}, @@ -58,29 +58,55 @@ fn parent(part: &sysfs::BlockPartition) -> Result { .map(Path::to_path_buf) .ok_or_else(|| Error("Partition has no parent disk".into())) } -fn select() -> Result<(sysfs::BlockPartition, PathBuf, String)> { +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Transport { + Sd, + Nvme, +} +impl Transport { + fn source(self) -> &'static str { + match self { + Self::Sd => "internal_sd", + Self::Nvme => "internal_nvme", + } + } +} +fn transport(disk: &Path) -> Result> { + let buses: Vec<_> = disk + .ancestors() + .filter_map(|path| fs::read_link(path.join("subsystem")).ok()) + .filter_map(|path| path.file_name().map(|s| s.to_owned())) + .collect(); + // A USB card reader is a workstation flashing destination, never internal + // machine storage at runtime. Names such as mmcblk0 are not identities. + if buses.iter().any(|b| b == "usb") { + return Ok(None); + } + if buses.iter().any(|b| b == "mmc") { + let kind = read_text(&disk.join("device/type"), 32)?; + return Ok((kind.trim() == "SD").then_some(Transport::Sd)); + } + if buses.iter().any(|b| b == "nvme") && read_text(&disk.join("removable"), 32)?.trim() == "0" { + return Ok(Some(Transport::Nvme)); + } + Ok(None) +} +fn select() -> Result<(sysfs::BlockPartition, PathBuf, String, Transport)> { let parts = sysfs::partitions(Path::new("/sys"))?; let mut candidates = Vec::new(); for part in parts.iter().filter(|p| p.partition_name == "FDS_INTERNAL") { let disk = parent(part)?; - let buses: Vec<_> = disk - .ancestors() - .filter_map(|path| fs::read_link(path.join("subsystem")).ok()) - .filter_map(|path| path.file_name().map(|s| s.to_owned())) - .collect(); - if !buses.iter().any(|b| b == "nvme") || buses.iter().any(|b| b == "usb") { + let Some(transport) = transport(&disk)? else { continue; - } - if read_text(&disk.join("removable"), 32)?.trim() != "0" { - continue; - } + }; let siblings: Vec<_> = parts .iter() .filter(|p| parent(p).ok().as_ref() == Some(&disk)) .collect(); if siblings.len() != 3 { return Err(Error( - "Internal NVMe must contain exactly FDS_BOOT, FDS_RECOVERY and FDS_INTERNAL".into(), + "Internal storage must contain exactly FDS_BOOT, FDS_RECOVERY and FDS_INTERNAL" + .into(), )); } for (number, label) in [(1, "FDS_BOOT"), (2, "FDS_RECOVERY"), (3, "FDS_INTERNAL")] { @@ -99,22 +125,22 @@ fn select() -> Result<(sysfs::BlockPartition, PathBuf, String)> { .trim() != number.to_string() { - return Err(Error("Invalid internal NVMe partition layout".into())); + return Err(Error("Invalid internal storage partition layout".into())); } } let sequence = read_text(&disk.join("diskseq"), 32)?.trim().to_owned(); sequence .parse::() .map_err(|_| Error("Invalid internal disk sequence".into()))?; - candidates.push((part.clone(), disk, sequence)); + candidates.push((part.clone(), disk, sequence, transport)); } match candidates.len() { 0 => Err(Error( - "No complete internal NVMe layout found; image defaults remain available".into(), + "No complete internal SD/NVMe layout found; image defaults remain available".into(), )), 1 => Ok(candidates.pop().unwrap()), _ => Err(Error( - "Multiple internal NVMe layouts found; refusing to choose one".into(), + "Multiple internal SD/NVMe layouts found; refusing to choose one".into(), )), } } @@ -122,11 +148,12 @@ struct Internal { file: File, disk: PathBuf, sequence: String, + transport: Transport, mounted: bool, } impl Internal { fn open(writable: bool) -> Result { - let (part, disk, sequence) = select()?; + let (part, disk, sequence, transport) = select()?; let mounts = read_text(Path::new("/proc/self/mountinfo"), 4 * 1024 * 1024)?; if mounts.lines().any(|line| { line.split_whitespace().nth(2) == Some(&format!("{}:{}", part.major, part.minor)) @@ -180,6 +207,7 @@ impl Internal { file, disk, sequence, + transport, mounted: false, }; internal.identity()?; @@ -209,7 +237,9 @@ impl Internal { } fn identity(&self) -> Result<()> { if read_text(&self.disk.join("diskseq"), 32)?.trim() != self.sequence { - return Err(Error("Internal NVMe changed during the operation".into())); + return Err(Error( + "Internal storage changed during the operation".into(), + )); } Ok(()) } @@ -309,15 +339,16 @@ pub fn load() -> Result<()> { if Path::new(machine::SNAPSHOT).exists() && Path::new(machine::STATUS).exists() { return Ok(()); } - let attempt = || -> Result<(Config, String)> { + let attempt = || -> Result<(Config, String, Transport)> { let internal = Internal::open(false)?; let config = Config::parse(&machine::trusted_text( Path::new(&format!("{MOUNT}/config/machine.json")), machine::MAX_BUNDLE, )?)?; let sequence = internal.sequence.clone(); + let transport = internal.transport; internal.close(false)?; - Ok((config, sequence)) + Ok((config, sequence, transport)) }; let emulated = emulator_config()?; let (config, status) = if let Some(config) = emulated { @@ -325,8 +356,8 @@ pub fn load() -> Result<()> { (config, status) } else { match attempt() { - Ok((config, sequence)) => { - let status = serde_json::json!({"source":"internal_nvme", "name":config.name, "disk_sequence":sequence, "error":null}); + Ok((config, sequence, transport)) => { + let status = serde_json::json!({"source":transport.source(), "name":config.name, "disk_sequence":sequence, "error":null}); (config, status) } Err(error) => { @@ -410,7 +441,7 @@ fn store(name: &str, input: &Path) -> Result<()> { } atomic(&output, &bytes, 0o600)?; internal.close(true)?; - println!("Saved diagnostics/{name} on internal NVMe; storage is flushed and unmounted."); + println!("Saved diagnostics/{name} on internal storage; storage is flushed and unmounted."); Ok(()) } pub fn run(command: crate::cli::MachineCommand, json: bool) -> Result<()> { @@ -508,3 +539,76 @@ pub fn run(command: crate::cli::MachineCommand, json: bool) -> Result<()> { } Ok(()) } + +#[cfg(test)] +mod transport_tests { + use super::{Transport, transport}; + use std::{fs, os::unix::fs::symlink, path::PathBuf}; + + struct Fixture(PathBuf); + impl Fixture { + fn new() -> Self { + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let root = std::env::temp_dir().join(format!( + "fds-internal-transport-{}-{nonce}", + std::process::id() + )); + fs::create_dir(&root).unwrap(); + Self(root) + } + fn disk(&self, bus: &str, kind: &str, removable: &str, usb: bool) -> PathBuf { + let ancestor = self.0.join(format!("{bus}-{kind}-{removable}-{usb}")); + let card = ancestor.join("card"); + let disk = card.join("block/arbitrary-disk-name"); + fs::create_dir_all(&disk).unwrap(); + symlink(format!("/sys/bus/{bus}"), card.join("subsystem")).unwrap(); + if usb { + symlink("/sys/bus/usb", ancestor.join("subsystem")).unwrap(); + } + symlink(&card, disk.join("device")).unwrap(); + fs::write(card.join("type"), kind).unwrap(); + fs::write(disk.join("removable"), removable).unwrap(); + disk + } + } + impl Drop for Fixture { + fn drop(&mut self) { + fs::remove_dir_all(&self.0).unwrap(); + } + } + + #[test] + fn native_sd_uses_mmc_card_identity_not_removable_flag_or_device_name() { + let fixture = Fixture::new(); + for removable in ["0", "1"] { + let disk = fixture.disk("mmc", "SD", removable, false); + assert_eq!(transport(&disk).unwrap(), Some(Transport::Sd)); + } + assert_eq!(Transport::Sd.source(), "internal_sd"); + let disk = fixture.disk("nvme", "", "0", false); + assert_eq!(transport(&disk).unwrap(), Some(Transport::Nvme)); + assert_eq!(Transport::Nvme.source(), "internal_nvme"); + } + + #[test] + fn usb_lookalikes_other_mmc_devices_and_unknown_transports_are_rejected() { + let fixture = Fixture::new(); + for (bus, kind, removable, usb) in [ + ("mmc", "SD", "0", true), + ("nvme", "", "0", true), + ("mmc", "MMC", "0", false), + ("mmc", "SDIO", "1", false), + ("scsi", "SD", "0", false), + ("nvme", "", "1", false), + ] { + let disk = fixture.disk(bus, kind, removable, usb); + assert_eq!(transport(&disk).unwrap(), None, "{}", disk.display()); + } + let disk = fixture.disk("mmc", "SD", "1", false); + fs::remove_file(disk.join("device/type")).unwrap(); + assert!(transport(&disk).is_err()); + } +} diff --git a/rust/fds-workstation/src/cartridge.rs b/rust/fds-workstation/src/cartridge.rs index d77cffe..eec1e38 100644 --- a/rust/fds-workstation/src/cartridge.rs +++ b/rust/fds-workstation/src/cartridge.rs @@ -211,7 +211,7 @@ pub fn inspect(path: &Path, runner: Option<&Path>) -> Result { let mut info = image::inspect(&file, file.metadata()?.len())?; if info.filesystem != "erofs" { return Err(Error( - "Use fds-burn inspect for DATA geometry; this inspector validates software cartridges" + "Use fds-flash inspect for DATA geometry; this inspector validates software cartridges" .into(), )); } diff --git a/rust/fds-workstation/src/flash/cli.rs b/rust/fds-workstation/src/flash/cli.rs index 6e47a3d..6d67a87 100644 --- a/rust/fds-workstation/src/flash/cli.rs +++ b/rust/fds-workstation/src/flash/cli.rs @@ -1,6 +1,6 @@ -use super::{Prepared, device, prepare}; -use clap::{Parser, Subcommand}; -use fds_common::{Error, Result}; +use super::{Inspection, Prepared, device, inspect, prepare}; +use clap::{Args, Parser, Subcommand}; +use fds_common::{Error, Result, manifest::Class}; use std::{ io::{self, BufRead, IsTerminal, Write}, path::{Path, PathBuf}, @@ -15,8 +15,8 @@ fn digest(value: &str) -> std::result::Result { #[derive(Debug, Parser)] #[command(name = "fds-flash", version = env!("FDS_BUILD_VERSION"), - about = "Flash complete FDS internal and cartridge disk images on Linux", - after_help = "With no flags, choose an image and disk interactively. Physical writes need root. Every write flushes and verifies readback; no partition or filesystem is expanded. Verify downloaded release signatures separately.", + about = "Create, inspect and flash FDS internal and cartridge disk images on Linux", + after_help = "With no flags, choose an image and disk interactively. Physical writes need root. Every write flushes and verifies readback; no partition or filesystem is expanded. Create and inspect work on regular files without a destination disk. Build new PROGRAM images with fds-cartridge. Verify downloaded release signatures separately.", args_conflicts_with_subcommands = true)] pub struct Cli { #[command(subcommand)] @@ -49,6 +49,19 @@ pub struct Cli { #[derive(Debug, Subcommand)] enum Action { + /// Inspect a complete internal or cartridge image and report its SHA-256. + Inspect { + image: PathBuf, + #[arg(long)] + json: bool, + }, + /// Create a cartridge image from a prepared tree containing FDS/CARTRIDGE.TOML. + Create { + #[command(subcommand)] + command: CreateCommand, + #[arg(long, global = true)] + json: bool, + }, /// List physical whole disks, identities and reasons they cannot be flashed. List { #[arg(long)] @@ -56,6 +69,58 @@ enum Action { }, } +#[derive(Debug, Args)] +struct ImageTree { + /// Prepared tree, including its FDS/CARTRIDGE.TOML manifest. + source_directory: PathBuf, + /// New regular image file outside the source tree; never overwritten. + output: PathBuf, +} + +#[derive(Debug, Subcommand)] +enum CreateCommand { + /// Create a writable DATA cartridge with an ext4 payload (requires e2fsprogs). + Data { + #[command(flatten)] + tree: ImageTree, + /// DATA filesystem size in MiB; the complete GPT image is slightly larger. + #[arg(long, default_value_t = 128, value_parser = clap::value_parser!(u64).range(32..=1048576))] + size_mib: u64, + }, + /// Create an ENVIRONMENT descriptor cartridge (requires erofs-utils). + Environment(ImageTree), + /// Wrap a fully prepared FDS SYSTEM root (requires erofs-utils). + System(ImageTree), +} + +fn image_result(status: &str, inspection: &Inspection, json: bool) -> Result<()> { + if json { + println!( + "{}", + serde_json::to_string_pretty(&serde_json::json!({ + "format": 1, "status": status, "inspection": inspection + })) + .map_err(|e| Error(e.to_string()))? + ); + } else { + println!( + "{}: {}\nKind: {}\nImage bytes: {}\nSHA-256: {}", + status.to_uppercase(), + inspection.image_path.display(), + inspection.image.kind, + inspection.image.bytes, + inspection.sha256 + ); + for partition in &inspection.image.partitions { + println!( + " Partition {}: {} ({} bytes)", + partition.number, partition.name, partition.bytes + ); + } + } + Ok(()) +} + fn prompt(text: &str) -> Result { eprint!("{text}"); io::stderr().flush()?; @@ -158,26 +223,41 @@ fn show(prepared: &Prepared) { } pub fn run(cli: Cli) -> Result<()> { - if let Some(Action::List { json }) = cli.command { - let disks = device::list()?; - if json { - println!( - "{}", - serde_json::to_string_pretty(&disks).map_err(|e| Error(e.to_string()))? - ); - } else { - for disk in disks { - println!( - "{}\n target_id: {}\n {}", - describe(&disk.target), - disk.target_id, - disk.blocked - .map(|r| format!("BLOCKED: {r}")) - .unwrap_or_else(|| "Available for explicit selection".into()) - ); - } + match cli.command { + Some(Action::Inspect { image, json }) => { + return image_result("inspected", &inspect(&image)?, json); } - return Ok(()); + Some(Action::Create { command, json }) => { + let (class, tree, size) = match command { + CreateCommand::Data { tree, size_mib } => (Class::Data, tree, Some(size_mib)), + CreateCommand::Environment(tree) => (Class::Environment, tree, None), + CreateCommand::System(tree) => (Class::System, tree, None), + }; + fds_burn::create::create(class, &tree.source_directory, &tree.output, size)?; + return image_result("created", &inspect(&tree.output)?, json); + } + Some(Action::List { json }) => { + let disks = device::list()?; + if json { + println!( + "{}", + serde_json::to_string_pretty(&disks).map_err(|e| Error(e.to_string()))? + ); + } else { + for disk in disks { + println!( + "{}\n target_id: {}\n {}", + describe(&disk.target), + disk.target_id, + disk.blocked + .map(|r| format!("BLOCKED: {r}")) + .unwrap_or_else(|| "Available for explicit selection".into()) + ); + } + } + return Ok(()); + } + None => (), } if !cli.unattended && !cli.dry_run && !io::stdin().is_terminal() { return Err(Error("Interactive flashing requires a terminal; use --dry-run or explicit --unattended options".into())); @@ -279,4 +359,42 @@ mod tests { assert!(Cli::try_parse_from(args).is_ok()); assert!(digest("wrong").is_err()); } + + #[test] + fn image_commands_are_typed_and_do_not_accept_disk_write_options() { + for args in [ + vec!["inspect", "card.img", "--json"], + vec![ + "create", + "data", + "tree", + "card.img", + "--size-mib", + "32", + "--json", + ], + vec!["create", "--json", "environment", "tree", "card.img"], + vec!["create", "system", "tree", "card.img"], + ] { + assert!(Cli::try_parse_from(std::iter::once("fds-flash").chain(args)).is_ok()); + } + for args in [ + vec!["inspect", "card.img", "--device", "/dev/sda"], + vec!["--unattended", "create", "data", "tree", "card.img"], + vec!["create", "program", "tree", "card.img"], + vec!["create", "data", "tree"], + vec!["create", "data", "tree", "card.img", "--size-mib", "31"], + vec!["create", "data", "tree", "card.img", "--size-mib", "bad"], + vec![ + "create", + "environment", + "tree", + "card.img", + "--size-mib", + "32", + ], + ] { + assert!(Cli::try_parse_from(std::iter::once("fds-flash").chain(args)).is_err()); + } + } } diff --git a/rust/fds-workstation/src/flash/mod.rs b/rust/fds-workstation/src/flash/mod.rs index 4b1461e..9dd48f2 100644 --- a/rust/fds-workstation/src/flash/mod.rs +++ b/rust/fds-workstation/src/flash/mod.rs @@ -1,4 +1,4 @@ -//! Guided and unattended installation of complete disk images on Linux. +//! Image inspection and guided/unattended installation of complete FDS disks. pub mod cli; pub mod device; mod image; @@ -25,12 +25,14 @@ pub struct Prepared { source: File, } -pub fn prepare( - image_path: &Path, - device_path: &Path, - file_target: bool, - expected_sha256: Option<&str>, -) -> Result { +#[derive(Serialize)] +pub struct Inspection { + pub image_path: PathBuf, + pub image: image::Geometry, + pub sha256: String, +} + +fn inspect_source(image_path: &Path, expected_sha256: Option<&str>) -> Result<(File, Inspection)> { let image_path = image_path.canonicalize()?; let source = crate::cartridge::open_image(&image_path)?; let geometry = image::inspect(&source, source.metadata()?.len())?; @@ -43,6 +45,33 @@ pub fn prepare( if image::inspect(&source, source.metadata()?.len())? != geometry { return Err(Error("Image geometry changed during inspection".into())); } + Ok(( + source, + Inspection { + image_path, + image: geometry, + sha256, + }, + )) +} + +/// Inspect a complete internal or cartridge image without selecting a target. +pub fn inspect(image_path: &Path) -> Result { + Ok(inspect_source(image_path, None)?.1) +} + +pub fn prepare( + image_path: &Path, + device_path: &Path, + file_target: bool, + expected_sha256: Option<&str>, +) -> Result { + let (source, inspection) = inspect_source(image_path, expected_sha256)?; + let Inspection { + image_path, + image: geometry, + sha256, + } = inspection; let target = device::inspect(device_path, file_target)?; let metadata = std::fs::metadata(target.path())?; if source.metadata()?.dev() == metadata.dev() && source.metadata()?.ino() == metadata.ino() { diff --git a/tests/hardware/session-template.json b/tests/hardware/session-template.json index 2bdbeab..c4eff9e 100644 --- a/tests/hardware/session-template.json +++ b/tests/hardware/session-template.json @@ -17,7 +17,12 @@ "power_source": null, "battery_state": null, "hub_models": [], - "display_model": "Dasung Paperlike 13K" + "display_model": "Dasung Paperlike 13K", + "internal_medium": null, + "sd_card_model": null, + "sd_card_capacity_bytes": null, + "maintenance_usb_model": null, + "boot_order": null }, "bay_map": [ { @@ -197,6 +202,38 @@ "samples": [], "errors": [], "notes": null + }, + { + "name": "sd_usb_system_boot", + "status": "not_run", + "captures": [], + "samples": [], + "errors": [], + "notes": null + }, + { + "name": "sd_recovery_persistence", + "status": "not_run", + "captures": [], + "samples": [], + "errors": [], + "notes": null + }, + { + "name": "maintenance_usb_override", + "status": "not_run", + "captures": [], + "samples": [], + "errors": [], + "notes": null + }, + { + "name": "return_to_fds_sd", + "status": "not_run", + "captures": [], + "samples": [], + "errors": [], + "notes": null } ], "timing": { diff --git a/tests/integration/m12-eeprom.py b/tests/integration/m12-eeprom.py index 08531cf..e1aa18c 100644 --- a/tests/integration/m12-eeprom.py +++ b/tests/integration/m12-eeprom.py @@ -27,11 +27,12 @@ def run(arguments,ok=True): assert (result.returncode==0)==ok,(arguments,result.stdout,result.stderr) return result -for profile in ['production','development']: +for profile, boot_order in [('production','0xf1'),('development','0xf41'),('maintenance','0xf14')]: output=work/profile run(['--profile',profile,'--output-directory',output]) manifest=json.loads((output/'manifest.json').read_text()) assert manifest['hardware_modified'] is False and manifest['custom_inputs_provided'] is False + assert manifest['settings']['BOOT_ORDER'] == boot_order for name,digest in manifest['files'].items():assert sha256(output/name)==digest image=upstream.BootloaderImage(str(output/'configured.bin')) for name,data in protected.items():assert image.get_file(name)==data,(profile,name) @@ -47,7 +48,7 @@ saved.write_text('[all]\nBOOT_ORDER=0xf461\nCUSTOM_BOARD_SETTING=retained\n[gpio output=work/'custom' run(['--current-config',saved,'--output-directory',output]) changed=(output/'configured.conf').read_text() -assert changed.count('BOOT_ORDER=')==1 and 'BOOT_ORDER=0xf6' in changed +assert changed.count('BOOT_ORDER=')==1 and 'BOOT_ORDER=0xf1' in changed assert 'CUSTOM_BOARD_SETTING=retained' in changed and '[gpio8=0]\nOTHER_SETTING=unchanged' in changed rollback=upstream.BootloaderImage(str(output/'rollback.bin')).get_file('bootconf.txt').decode() assert rollback==saved.read_text(),'Rollback lost the original conditional settings' diff --git a/tests/integration/m12-internal.py b/tests/integration/m12-internal.py index 0d8fb11..c9d2231 100644 --- a/tests/integration/m12-internal.py +++ b/tests/integration/m12-internal.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Exercise the packaged machine settings service against disposable virtual NVMe.""" +"""Exercise packaged machine settings against disposable native SD and NVMe.""" import json from pathlib import Path import shlex @@ -41,6 +41,22 @@ def extra(path, node='internal', readonly=False): return ['-drive', f'file={path},if=none,id={node},format=raw' + (',readonly=on' if readonly else ''), '-device', f'nvme,drive={node},serial=FDS-{node.upper()}'] +def sd_extra(path): + return ['-drive', f'file={path},if=none,id=internal_sd,format=raw', + '-device', 'sdhci-pci,id=sdhci', + '-device', 'sd-card,drive=internal_sd'] + +def sd_capacity(path): + # QEMU SD cards require a power-of-two capacity. Sparse extension and GPT + # relocation model flashing the small image onto a larger native card. + with path.open('r+b') as output: + output.truncate(2 * 1024**3) + subprocess.run(['sfdisk', '--relocate', 'gpt-bak-std', str(path)], check=True, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + subprocess.run(['sfdisk', '--verify', str(path)], check=True, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return path + def wait(operation, condition, timeout=60): deadline = time.monotonic() + timeout while True: @@ -128,27 +144,62 @@ with VM(work, 'persistent-settings', empty, extra=[*extra(machine), '-device', c finish(vm) print('PASS: independent reboot loads saved bay/catalog/name settings and retrieves the previous boot record', flush=True) +sd = sd_capacity(copy_image('native-sd')) +original_sd = digest(sd) +with VM(work, 'sd-install-settings', empty, extra=sd_extra(sd)) as vm: + status = enter(vm) + assert status['source'] == 'internal_sd' and status['name'] == 'FP-85', status + assert digest(sd) == original_sd, 'Loading SD settings changed the card' + assert vm.capture('cat /sys/class/block/mmcblk0/device/type') == 'SD' + vm.capture('fds machine export /tmp/sd-machine') + vm.capture("printf 'format=1\\nname=\"FP-85 SD\"\\n' >/tmp/sd-machine/machine.toml") + vm.capture('fds machine validate /tmp/sd-machine && fds machine install /tmp/sd-machine') + assert query(vm, ['machine', 'status'])['name'] == 'FP-85' + vm.capture('fds --json boot-profile >/tmp/boot.json && fds machine store sd-boot.json /tmp/boot.json') + vm.capture('test -z "$(findmnt -nr -o TARGET | grep /run/fds/machine/internal)"') + finish(vm) +unchanged_payloads(sd) +with VM(work, 'sd-persistent-settings', empty, extra=sd_extra(sd)) as vm: + status = enter(vm) + assert status['source'] == 'internal_sd' and status['name'] == 'FP-85 SD', status + vm.capture('fds machine fetch sd-boot.json /tmp/saved.json') + assert json.loads(vm.capture('cat /tmp/saved.json'))['boot_id'] != query(vm, ['boot-profile'])['boot_id'] + (work / 'sd-persistent-status.json').write_text(json.dumps(status, indent=2) + '\n') + finish(vm) + +# SD supplies machine storage while the actual normal SYSTEM remains on USB. +with VM(work, 'sd-usb-system', project / 'out/fds-system-cli.img', system_usb=True, extra=sd_extra(sd)) as vm: + vm.expect(rb'FDS> ') + wait(lambda: vm.capture('test -S /run/fds/control.sock && echo ready || echo pending'), lambda text: text == 'ready') + status = query(vm, ['machine', 'status']) + assert status['source'] == 'internal_sd' and status['name'] == 'FP-85 SD', status + assert vm.capture('findmnt -no FSTYPE /') == 'erofs' + finish(vm) +print('PASS: native MMC SD recovery, read-only settings load, explicit writes, reboot persistence and normal USB SYSTEM handoff', flush=True) + # Independent filesystem inspection after real guest writes. part = layout['partitions'][2] -settings = work / 'after-guest.ext4' -with machine.open('rb') as source, settings.open('wb') as output: - source.seek(part['start'] * 512) - remaining = part['payload_bytes'] - while remaining: - chunk = source.read(min(remaining, 1024 * 1024)) - assert chunk - output.write(chunk) - remaining -= len(chunk) -with (work / 'after-guest-fsck.log').open('wb') as log: - subprocess.run([runner, 'e2fsck', '-f', '-n', str(settings)], check=True, stdout=log, stderr=subprocess.STDOUT) -previous = subprocess.check_output([runner, 'debugfs', '-R', 'cat /config/previous.json', str(settings)], stderr=subprocess.DEVNULL) -assert json.loads(previous)['name'] == 'FP-85' +for source_disk, name in [(machine, 'nvme'), (sd, 'sd')]: + settings = work / (name + '-after-guest.ext4') + with source_disk.open('rb') as source, settings.open('wb') as output: + source.seek(part['start'] * 512) + remaining = part['payload_bytes'] + while remaining: + chunk = source.read(min(remaining, 1024 * 1024)) + assert chunk + output.write(chunk) + remaining -= len(chunk) + with (work / (name + '-after-guest-fsck.log')).open('wb') as log: + subprocess.run([runner, 'e2fsck', '-f', '-n', str(settings)], check=True, stdout=log, stderr=subprocess.STDOUT) + previous = subprocess.check_output([runner, 'debugfs', '-R', 'cat /config/previous.json', str(settings)], stderr=subprocess.DEVNULL) + assert json.loads(previous)['name'] == 'FP-85' +print('PASS: independent ext4 checks after guest SD/NVMe writes and previous-settings retention', flush=True) # Both recovery and settings are present on USB, but only recovery may be selected # by its GPT label. USB cannot supply persistent machine settings. with VM(work, 'usb-lookalike', image, system_usb=True) as vm: status = enter(vm) - assert status['source'] == 'image_defaults' and 'No complete internal NVMe' in status['error'], status + assert status['source'] == 'image_defaults' and 'No complete internal SD/NVMe' in status['error'], status finish(vm) # Boot the ordinary CLI SYSTEM with two eligible NVMes already present. Stage0 @@ -159,10 +210,19 @@ with VM(work, 'ambiguous-internal', project / 'out/fds-system-cli.img', extra=[* assert vm.capture('stat -c %u:%g:%a /') == '0:0:755' wait(lambda: vm.capture('test -S /run/fds/control.sock && echo ready || echo pending'), lambda text: text == 'ready') status = query(vm, ['machine', 'status']) - assert status['source'] == 'image_defaults' and 'Multiple internal NVMe' in status['error'], status + assert status['source'] == 'image_defaults' and 'Multiple internal SD/NVMe' in status['error'], status finish(vm) print('PASS: USB settings spoof and two eligible NVMes fall back with explicit diagnostics', flush=True) +with VM(work, 'ambiguous-sd-nvme', project / 'out/fds-system-cli.img', + extra=[*sd_extra(sd), *extra(image, 'old_nvme', True)]) as vm: + vm.expect(rb'FDS> ') + wait(lambda: vm.capture('test -S /run/fds/control.sock && echo ready || echo pending'), lambda text: text == 'ready') + status = query(vm, ['machine', 'status']) + assert status['source'] == 'image_defaults' and 'Multiple internal SD/NVMe' in status['error'], status + finish(vm) +print('PASS: concurrent eligible SD and NVMe are rejected instead of silently selecting settings', flush=True) + # Corrupt configuration and an unclean ext4 are distinct failure cases. Build # fixtures by changing only the settings partition, retaining the packaged OS. def altered(name, commands): @@ -191,8 +251,16 @@ for name, disk, expected in fixtures: assert digest(disk) == before, 'Failure path wrote internal storage' finish(vm) print('PASS: invalid JSON, writable/symlink settings and unclean ext4 preserve bytes and leave the recovery console usable', flush=True) +dirty_sd = sd_capacity(altered('unclean-sd', ['set_super_value state 0'])) +before = digest(dirty_sd) +with VM(work, 'unclean-sd', empty, extra=sd_extra(dirty_sd)) as vm: + status = enter(vm) + assert status['source'] == 'image_defaults' and 'unclean' in status['error'], status + assert digest(dirty_sd) == before, 'Unclean SD was modified during fallback' + finish(vm) +print('PASS: unclean SD falls back without journal replay or persistent writes', flush=True) link = project / 'out/m12-internal-latest.next' link.symlink_to(work.name) link.replace(project / 'out/m12-internal-latest') print(f'PASS: internal storage software acceptance: {work}', flush=True) -print('SKIP: Pi EEPROM/NVMe boot, real bay calibration, power-loss and flash durability') +print('SKIP: physical Pi EEPROM/SD/NVMe boot, maintenance USB boot, real bay calibration, power-loss and flash durability') diff --git a/tests/integration/workstation-flash.py b/tests/integration/workstation-flash.py index 357bec0..6c1457f 100644 --- a/tests/integration/workstation-flash.py +++ b/tests/integration/workstation-flash.py @@ -87,6 +87,11 @@ def verify(image, disk): for image in [internal, system]: source_hash = digest(image) + inspection = json.loads(invoke(['inspect', image, '--json']).stdout) + assert inspection['status'] == 'inspected' + assert inspection['inspection']['sha256'] == source_hash + assert inspection['inspection']['image']['kind'] == image.stem + assert source_hash in invoke(['inspect', image]).stdout for extra in [0, 512, 4*1024*1024]: disk = target(f'{image.stem}-{extra}.target', image.stat().st_size + extra) before = digest(disk) @@ -108,11 +113,48 @@ for name, partitions in [ ('software', [(label, LINUX_FILESYSTEM, parts[1][2]) for label in ('FDS_METADATA','FDS_PAYLOAD02','FDS_PAYLOAD03')]), ]: image=work/(name+'.img');gpt(image,partitions) + inspection=json.loads(invoke(['inspect',image,'--json']).stdout)['inspection'] + assert inspection['sha256']==digest(image) + assert [p['name'] for p in inspection['image']['partitions']]==[p[0] for p in partitions] disk=target(name+'.target',image.stat().st_size+1024*1024) plan=preview(image,disk) assert json.loads(invoke(unattended(image,disk,plan)).stdout)['status']=='verified' verify(image,disk) +# The consolidated command creates a real DATA filesystem, inspects it, and +# flashes it through the same unattended writer used for existing images. +# e2fsprogs is required for this acceptance check, as for DATA creation itself. +tree=work/'data-tree';(tree/'FDS').mkdir(parents=True) +(tree/'FDS/CARTRIDGE.TOML').write_text('format=1\n[cartridge]\nid="flash.data"\nname="Flash test"\nclass="data"\nversion="1"\n[media]\nwritable=true\n') +(tree/'sample.txt').write_text('Created and flashed with fds-flash\n') +created=work/'created-data.img' +result=json.loads(invoke(['create','data',tree,created,'--size-mib','32','--json']).stdout) +assert result['status']=='created' +inspection=json.loads(invoke(['inspect',created,'--json']).stdout)['inspection'] +assert result['inspection']==inspection and inspection['sha256']==digest(created) +disk=target('created-data.target',created.stat().st_size+1024*1024) +assert json.loads(invoke(unattended(created,disk,preview(created,disk))).stdout)['status']=='verified' +verify(created,disk) +partition=inspection['image']['partitions'][0] +payload=work/'created-data.ext4' +with disk.open('rb') as stream: + stream.seek(partition['start']);payload.write_bytes(stream.read(partition['bytes'])) +readback=subprocess.run(['debugfs','-R','cat /sample.txt',str(payload)],capture_output=True,text=True,check=True) +assert readback.stdout==(tree/'sample.txt').read_text(),readback +before=digest(created) +invoke(['create','data',tree,created,'--size-mib','32'],False) +assert digest(created)==before +for arguments in [ + ['create','data',tree,tree/'inside.img','--size-mib','32'], + ['create','environment',tree,work/'wrong-class.img'], + ['create','system',tree,work/'wrong-system.img'], + ['create','program',tree,work/'legacy-program.img'], + ['create','data',tree,work/'bad-size.img','--size-mib','31'], +]: + invoke(arguments,False) +assert not (tree/'inside.img').exists() +assert not any((work/name).exists() for name in ['wrong-class.img','wrong-system.img','legacy-program.img','bad-size.img']) + # Invalid source geometry and destinations must fail without writing. disk = target('protected.target', internal.stat().st_size) before = digest(disk) @@ -130,7 +172,9 @@ for name, change in [('bad-primary', 512+16), ('bad-backup', internal.stat().st_ with image.open('r+b') as stream: stream.seek(change); byte=stream.read(1); stream.seek(change); stream.write(bytes([byte[0]^1])) invoke(['--image', image, '--device', disk, '--file-target', '--dry-run'], False) + invoke(['inspect', image, '--json'], False) invoke(['--image', parts[0][2], '--device', disk, '--file-target', '--dry-run'], False) +invoke(['inspect', parts[0][2]], False) assert digest(disk) == before # Drive the real terminal workflow, including cancellation and changed inputs @@ -198,10 +242,11 @@ for name in ('fds-internal.img', 'fds-system-cli.img'): print('SKIP: build image not available for read-only inspection:',image) record=dict(status='passed',cli=str(cli),cli_sha256=digest(cli),internal_and_all_cartridge_classes=True,interactive_terminal=True, + image_inspection_without_target=True,data_create_inspect_flash_and_filesystem_readback=True, unattended_bindings=True,exact_larger_and_overlapping_gpt=True,independent_sfdisk=True, partition_payloads_unchanged=True,invalid_sources_and_targets_rejected=True, changed_source_and_target_rejected=True,actual_build_images_inspected=actual_images,physical_disks_written=False) (work/'acceptance.json').write_text(json.dumps(record,indent=2)+'\n') (project/'out/workstation-flash-current.txt').write_text(str(work)+'\n') -print('PASS: interactive/unattended flash, rejection paths, payload readback and relocated GPT:',work) -print('SKIP: physical devices; fixtures are disposable files with inert filesystem signatures') +print('PASS: create/inspect, interactive/unattended flash, rejection paths, payload readback and relocated GPT:',work) +print('SKIP: physical devices; disposable-file fixtures include real DATA and inert signatures for other layouts') diff --git a/tools/build-kernel b/tools/build-kernel index bb3121b..4a715f3 100755 --- a/tools/build-kernel +++ b/tools/build-kernel @@ -32,11 +32,20 @@ XBPS_ARCH=aarch64 xbps-rindex -fa "out/packages/$package" work=$(mktemp -d "$FDS_ROOT/out/kernel-build.XXXXXX") tar -xf "out/packages/$package" -C "$work" [[ -s $work/boot/kernel_2712.img && -s $work/boot/bcm2712-rpi-5-b.dtb ]] || die 'Missing Pi 5 kernel/DTB' -# Preserve the existing mandatory display-protection requirement. +# Reject stale packages missing any boot-critical or display-protection setting. while IFS= read -r setting; do - [[ $setting == CONFIG_*=* || $setting == '# CONFIG_'*' is not set' ]] || continue - grep -Fqx "$setting" "$work/boot/config-fds" || die "Dasung kernel requirement missing: $setting" -done "$work/artifacts.sha256" cp "$work/artifacts.sha256" out/manifests/kernel-artifacts.sha256 cp "$inputs" "$work/build-inputs.sha256" diff --git a/tools/clean-builds b/tools/clean-builds index 05784b4..52d5196 100755 --- a/tools/clean-builds +++ b/tools/clean-builds @@ -17,7 +17,7 @@ import sys PREFIXES = ( 'rootfs-build', 'kernel-build', 'system-build', 'initramfs-build', 'boot-build', 'recovery-build', 'internal-build', - 'eeprom-production', 'eeprom-development', + 'eeprom-production', 'eeprom-development', 'eeprom-maintenance', 'cartridged-source', 'init-source', 'dasung-services', 'dasung-s6', 'rust-notices', 'verify-package', 'dasung-check', 'dasung-s6-run', 'm1-checks', 'm2-vm', 'm4-vm', 'm5-vm', 'm6-vm', 'm7-vm', 'm8-vm', diff --git a/tools/configure-pi-eeprom b/tools/configure-pi-eeprom index 7def3d1..aea7a46 100755 --- a/tools/configure-pi-eeprom +++ b/tools/configure-pi-eeprom @@ -36,7 +36,7 @@ def merge(original,profile): def main(): parser=argparse.ArgumentParser(description=__doc__) - parser.add_argument('--profile',choices=['production','development'],default='production') + parser.add_argument('--profile',choices=['production','development','maintenance'],default='production') parser.add_argument('--base-image',type=Path,help='Regular-file Pi 5 EEPROM image; defaults to the pinned preview firmware') parser.add_argument('--current-config',type=Path,help='Saved board configuration to preserve instead of the image defaults') parser.add_argument('--output-directory',type=Path,help='New directory; existing paths are refused')