update docs
This commit is contained in:
@@ -0,0 +1,154 @@
|
||||
# Twelve-bay stress tests
|
||||
|
||||
Development reference and historical context. For current operating instructions, use the [user manual](../README.md). Acceptance applies only to the source and artifacts identified in each record.
|
||||
|
||||
|
||||
[Documentation index](README.md) · [Cartridge configuration](cartridges.md) · [Shutdown](power.md)
|
||||
|
||||
M11 exercises twelve devices together and failures during media writing. The
|
||||
software tests run on the build workstation. The separate physical procedure
|
||||
below is for the assembled Pi; no virtual result certifies its wiring, power
|
||||
budget, USB hubs, storage controllers or display.
|
||||
|
||||
## Run the virtual tests
|
||||
|
||||
Complete the [boot-image prerequisites](boot.md), then run sequentially:
|
||||
|
||||
```sh
|
||||
make rootfs PROFILE=cli
|
||||
make rootfs-test
|
||||
make system-card PROFILE=cli
|
||||
make media-image-test
|
||||
make stress-test
|
||||
```
|
||||
|
||||
`make stress-test` compiles two test-only static ARM workloads, boots the actual
|
||||
packaged CLI image and Pi kernel, then runs the media failure suite. To repeat
|
||||
only the latter after those prerequisites:
|
||||
|
||||
```sh
|
||||
make media-fault-test
|
||||
```
|
||||
|
||||
No host disk or physical USB device is passed through. All media are disposable
|
||||
regular files inside a new `out/m11-vm.*` or `out/m11-faults.*` directory. QEMU runs
|
||||
inside the existing private network namespace. The SYSTEM root is a protected
|
||||
read-only virtual disk; the twelve additional USB disks stress all twelve bay
|
||||
slots. Earlier boot tests cover a removable USB SYSTEM itself.
|
||||
|
||||
The virtual controller explicitly has twelve USB 2 and USB 3 ports. Its live
|
||||
controller identity is discovered before generating the test map. This is not a
|
||||
map to install on the Pi. Physical hub chains and USB 2/3 aliases must be calibrated
|
||||
on the actual machine using [the cartridge guide](cartridges.md).
|
||||
|
||||
The test records:
|
||||
|
||||
- Three cold-start pairs with empty and fully populated bays, using the actual
|
||||
ordinary-user console prompt and identical image/controller configuration.
|
||||
- Twelve correct cartridge IDs and serials, followed by three simultaneous
|
||||
removal/insertion cycles with reversed enumeration and rotated bay positions.
|
||||
- Eleven direct-I/O readers checking known bytes while DATA writes and flushes
|
||||
64 MiB. Per-thread intervals must overlap, and QEMU's real block counters must
|
||||
show reads/writes for every disk.
|
||||
- Console readiness while the cartridge daemon is deliberately held at an
|
||||
event gate with all twelve disks present. No sleep releases that gate.
|
||||
- Native reboot during sustained DATA writes, independent `e2fsck`, exact payload
|
||||
verification, another boot with a different device declaration order, and halt.
|
||||
- Kernel logs checked for unexpected I/O failures, USB resets, filesystem errors
|
||||
and controller failures. Intentional fault tests have separate logs.
|
||||
|
||||
Boot samples, shutdown events, bay inventories, I/O counters and kernel logs stay
|
||||
beside the images. A greater-than-100 ms device-load change must receive the
|
||||
recorded review explanation; it must not be hidden by selecting a faster sample.
|
||||
Three emulator samples do not establish a Pi performance claim.
|
||||
|
||||
The fault suite injects actual EIO during writing, cache flush and readback. Other
|
||||
cases interrupt a confirmed transfer after at least 64 MiB by cancellation,
|
||||
worker termination or daemon termination. Failed operations must remain failed
|
||||
across restart, the daemon must remain usable, and none may acquire SAFE status.
|
||||
A test-only QEMU bandwidth limit keeps the large transfer observable; production
|
||||
code has no corresponding delay. See [QEMU fault injection](https://www.qemu.org/docs/master/devel/testing/blkdebug.html).
|
||||
|
||||
## Prepare the physical session
|
||||
|
||||
Use twelve labelled test cartridges and a written bay inventory. Include the
|
||||
intended SYSTEM, a backed-up or disposable DATA cartridge, and the PROGRAM,
|
||||
ENVIRONMENT and hardware cartridges you plan to use. Keep a known-working SYSTEM
|
||||
available for rollback. Record the Pi revision, EEPROM version, hub/controller
|
||||
models, cable arrangement, power source, storage serials and image hashes.
|
||||
|
||||
Label physical bays `01` through `12`. Confirm one device at a time with
|
||||
`fds topology` and `fds bay N`, then create the permanent machine map following
|
||||
the cartridge guide. Check both USB 2 and USB 3 modes where supported. A serial
|
||||
identifies the cartridge, while the stable controller/port path identifies its
|
||||
bay; the `/dev/sdX` name is not a bay label. Do not copy the VM map.
|
||||
|
||||
No destructive operation is necessary merely to collect diagnostics. For write
|
||||
stress, use test media whose contents can be replaced. Any `fds burn` or
|
||||
`fds format` operation still requires its exact per-operation confirmation.
|
||||
|
||||
## Capture evidence on the target
|
||||
|
||||
Copy [capture-hardware](../../tools/capture-hardware) to the DATA cartridge. At the
|
||||
FDS console, run it explicitly through Bash; DATA intentionally does not permit
|
||||
direct execution:
|
||||
|
||||
```sh
|
||||
bash /data/capture-hardware /data/capture-before
|
||||
```
|
||||
|
||||
The destination must be a new absolute directory. It is created private to the
|
||||
current user. The script collects FDS status, topology, boot events, kernel logs,
|
||||
USB/block inventories, mounts, network links and package versions. It writes a
|
||||
status table and checksums. It does not change services, mounts, firmware, disks
|
||||
or networking. A command denied to the ordinary user is recorded with its exit
|
||||
status; use an administrative maintenance session for the missing diagnostic if
|
||||
needed. A successful capture is not a hardware PASS.
|
||||
|
||||
Choose a new directory for every capture. If DATA itself is under investigation,
|
||||
write to `/tmp/capture-before` instead and copy the directory to trusted storage
|
||||
before shutdown; `/tmp` is volatile. Review captured device serials and network
|
||||
identifiers before sharing the files.
|
||||
|
||||
## Physical acceptance sequence
|
||||
|
||||
Copy the [physical session template](../../tests/hardware/session-template.json)
|
||||
and fill it with observed values. Null means unknown, not zero.
|
||||
Record every run, including failures. The following is an initial repeatable
|
||||
matrix, not evidence that it has already passed:
|
||||
|
||||
| Scenario | Procedure and required evidence |
|
||||
| --- | --- |
|
||||
| Cold boot baseline | Run five full cold boots with the intended SYSTEM and no optional media. Capture boot reports and external power-on/display timing. |
|
||||
| Fully populated cold boot | Repeat five times with all twelve bays occupied. Compare prompt timing, every physical bay/serial pair and kernel errors against baseline. |
|
||||
| Hot insertion | Insert each cartridge individually, then insert several together. Verify all twelve positions and expected activation. Capture before/after inventories. |
|
||||
| Safe removal | Stop explicit jobs and use `fds eject N` for storage. Remove only after SAFE. Verify that reusing the same bay does not retain the old cartridge's identity or SAFE state. |
|
||||
| Rotated placement | Move the test cartridges to different bays and repeat. Each bay must follow its physical position regardless of Linux disk names. |
|
||||
| Simultaneous I/O | Read known files from every readable cartridge while a managed DATA job writes a known payload. Record hashes, throughput and logs; USB hardware without storage needs its own functional workload. |
|
||||
| Reboot under load | Request `fds reboot`. Verify the ordered shutdown record, boot again, and compare DATA hashes. Run a filesystem check only after that filesystem is unmounted in maintenance/recovery. |
|
||||
| Poweroff under load | Request `fds poweroff`; record software phases and external power/display behavior. A blocked command is not permission to unplug DATA. |
|
||||
| Power and USB behavior | Observe the intended supply/battery configuration under simultaneous device load, within each component's ratings. Record undervoltage indications, resets, disconnects and any display recovery failure. |
|
||||
|
||||
Use external instrumentation for power applied, visible E-Ink readiness and actual
|
||||
power-off. Kernel timestamps cannot measure time before the kernel starts or prove
|
||||
that a flash controller honored a flush. Do not convert a VM measurement into a
|
||||
physical result. Intentional power-loss or surprise-removal durability tests need
|
||||
a separate disposable-data session and an explicit test plan; ordinary SAFE
|
||||
acceptance does not establish power-loss durability.
|
||||
|
||||
## Pass, fail and follow-up
|
||||
|
||||
A physical run fails if a cartridge appears in the wrong bay, a confirmed write
|
||||
or DATA eject reports success after an actual I/O error, a busy DATA shutdown
|
||||
forces poweroff, a kernel/controller error is unexplained, or verified payloads
|
||||
change. Capture logs before restarting a service or rebooting so its evidence is
|
||||
not lost. Preserve both the observed failure and any later successful rerun.
|
||||
|
||||
Compare the same image, power source and device population when evaluating a code
|
||||
change. Explain or fix a boot regression over 100 ms. Keep the idle <1 second and
|
||||
normal DATA <1.5 second shutdown targets separate from measured outcomes. The
|
||||
physical report should explicitly list untested devices, profiles and power modes.
|
||||
|
||||
No new target dependency is introduced by M11. Test workloads use Rust's standard
|
||||
library; the capture script uses Bash and tools already included in the base
|
||||
image. QEMU, filesystem tools and Python were existing host test dependencies.
|
||||
Reference in New Issue
Block a user