fds-os/docs/stress-testing.md

Twelve-bay stress tests

Documentation index · Cartridge configuration · Shutdown

M11 exercises twelve devices together and failures during media writing. The software tests run on the build workstation. The separate physical procedure below is for the assembled Pi; no virtual result certifies its wiring, power budget, USB hubs, storage controllers or display.

Run the virtual tests

Complete the boot-image prerequisites, then run sequentially:

make rootfs PROFILE=cli
make rootfs-test
make system-card PROFILE=cli
make media-image-test
make stress-test

make stress-test compiles two test-only static ARM workloads, boots the actual packaged CLI image and Pi kernel, then runs the media failure suite. To repeat only the latter after those prerequisites:

make media-fault-test

No host disk or physical USB device is passed through. All media are disposable regular files inside a new out/m11-vm.* or out/m11-faults.* directory. QEMU runs inside the existing private network namespace. The SYSTEM root is a protected read-only virtual disk; the twelve additional USB disks stress all twelve bay slots. Earlier boot tests cover a removable USB SYSTEM itself.

The virtual controller explicitly has twelve USB 2 and USB 3 ports. Its live controller identity is discovered before generating the test map. This is not a map to install on the Pi. Physical hub chains and USB 2/3 aliases must be calibrated on the actual machine using the cartridge guide.

The test records:

• Three cold-start pairs with empty and fully populated bays, using the actual ordinary-user console prompt and identical image/controller configuration.
• Twelve correct cartridge IDs and serials, followed by three simultaneous removal/insertion cycles with reversed enumeration and rotated bay positions.
• Eleven direct-I/O readers checking known bytes while DATA writes and flushes 64 MiB. Per-thread intervals must overlap, and QEMU's real block counters must show reads/writes for every disk.
• Console readiness while the cartridge daemon is deliberately held at an event gate with all twelve disks present. No sleep releases that gate.
• Native reboot during sustained DATA writes, independent e2fsck, exact payload verification, another boot with a different device declaration order, and halt.
• Kernel logs checked for unexpected I/O failures, USB resets, filesystem errors and controller failures. Intentional fault tests have separate logs.

Boot samples, shutdown events, bay inventories, I/O counters and kernel logs stay beside the images. A greater-than-100 ms device-load change must receive the recorded review explanation; it must not be hidden by selecting a faster sample. Three emulator samples do not establish a Pi performance claim.

The fault suite injects actual EIO during writing, cache flush and readback. Other cases interrupt a confirmed transfer after at least 64 MiB by cancellation, worker termination or daemon termination. Failed operations must remain failed across restart, the daemon must remain usable, and none may acquire SAFE status. A test-only QEMU bandwidth limit keeps the large transfer observable; production code has no corresponding delay. See QEMU fault injection.

Prepare the physical session

Use twelve labelled test cartridges and a written bay inventory. Include the intended SYSTEM, a backed-up or disposable DATA cartridge, and the PROGRAM, ENVIRONMENT and hardware cartridges you plan to use. Keep a known-working SYSTEM available for rollback. Record the Pi revision, EEPROM version, hub/controller models, cable arrangement, power source, storage serials and image hashes.

Label physical bays 01 through 12. Confirm one device at a time with fds topology and fds bay N, then create the permanent machine map following the cartridge guide. Check both USB 2 and USB 3 modes where supported. A serial identifies the cartridge, while the stable controller/port path identifies its bay; the /dev/sdX name is not a bay label. Do not copy the VM map.

No destructive operation is necessary merely to collect diagnostics. For write stress, use test media whose contents can be replaced. Any fds burn or fds format operation still requires its exact per-operation confirmation.

Capture evidence on the target

Copy capture-hardware to the DATA cartridge. At the FDS console, run it explicitly through Bash; DATA intentionally does not permit direct execution:

bash /data/capture-hardware /data/capture-before

The destination must be a new absolute directory. It is created private to the current user. The script collects FDS status, topology, boot events, kernel logs, USB/block inventories, mounts, network links and package versions. It writes a status table and checksums. It does not change services, mounts, firmware, disks or networking. A command denied to the ordinary user is recorded with its exit status; use an administrative maintenance session for the missing diagnostic if needed. A successful capture is not a hardware PASS.

Choose a new directory for every capture. If DATA itself is under investigation, write to /tmp/capture-before instead and copy the directory to trusted storage before shutdown; /tmp is volatile. Review captured device serials and network identifiers before sharing the files.

Physical acceptance sequence

Copy the physical session template and fill it with observed values. Null means unknown, not zero. Record every run, including failures. The following is an initial repeatable matrix, not evidence that it has already passed:

Scenario	Procedure and required evidence
Cold boot baseline	Run five full cold boots with the intended SYSTEM and no optional media. Capture boot reports and external power-on/display timing.
Fully populated cold boot	Repeat five times with all twelve bays occupied. Compare prompt timing, every physical bay/serial pair and kernel errors against baseline.
Hot insertion	Insert each cartridge individually, then insert several together. Verify all twelve positions and expected activation. Capture before/after inventories.
Safe removal	Stop explicit jobs and use fds eject N for storage. Remove only after SAFE. Verify that reusing the same bay does not retain the old cartridge's identity or SAFE state.
Rotated placement	Move the test cartridges to different bays and repeat. Each bay must follow its physical position regardless of Linux disk names.
Simultaneous I/O	Read known files from every readable cartridge while a managed DATA job writes a known payload. Record hashes, throughput and logs; USB hardware without storage needs its own functional workload.
Reboot under load	Request fds reboot. Verify the ordered shutdown record, boot again, and compare DATA hashes. Run a filesystem check only after that filesystem is unmounted in maintenance/recovery.
Poweroff under load	Request fds poweroff; record software phases and external power/display behavior. A blocked command is not permission to unplug DATA.
Power and USB behavior	Observe the intended supply/battery configuration under simultaneous device load, within each component's ratings. Record undervoltage indications, resets, disconnects and any display recovery failure.

Use external instrumentation for power applied, visible E-Ink readiness and actual power-off. Kernel timestamps cannot measure time before the kernel starts or prove that a flash controller honored a flush. Do not convert a VM measurement into a physical result. Intentional power-loss or surprise-removal durability tests need a separate disposable-data session and an explicit test plan; ordinary SAFE acceptance does not establish power-loss durability.

Pass, fail and follow-up

A physical run fails if a cartridge appears in the wrong bay, a confirmed write or DATA eject reports success after an actual I/O error, a busy DATA shutdown forces poweroff, a kernel/controller error is unexplained, or verified payloads change. Capture logs before restarting a service or rebooting so its evidence is not lost. Preserve both the observed failure and any later successful rerun.

Compare the same image, power source and device population when evaluating a code change. Explain or fix a boot regression over 100 ms. Keep the idle <1 second and normal DATA <1.5 second shutdown targets separate from measured outcomes. The physical report should explicitly list untested devices, profiles and power modes.

No new target dependency is introduced by M11. Test workloads use Rust's standard library; the capture script uses Bash and tools already included in the base image. QEMU, filesystem tools and Python were existing host test dependencies.