155 lines
9.0 KiB
Markdown
155 lines
9.0 KiB
Markdown
# Twelve-bay stress tests
|
|
|
|
Development reference and historical context. For current operating instructions, use the [user manual](../README.md). Acceptance applies only to the source and artifacts identified in each record.
|
|
|
|
|
|
[Documentation index](README.md) · [Cartridge configuration](cartridges.md) · [Shutdown](power.md)
|
|
|
|
M11 exercises twelve devices together and failures during media writing. The
|
|
software tests run on the build workstation. The separate physical procedure
|
|
below is for the assembled Pi; no virtual result certifies its wiring, power
|
|
budget, USB hubs, storage controllers or display.
|
|
|
|
## Run the virtual tests
|
|
|
|
Complete the [boot-image prerequisites](boot.md), then run sequentially:
|
|
|
|
```sh
|
|
make rootfs PROFILE=cli
|
|
make rootfs-test
|
|
make system-card PROFILE=cli
|
|
make media-image-test
|
|
make stress-test
|
|
```
|
|
|
|
`make stress-test` compiles two test-only static ARM workloads, boots the actual
|
|
packaged CLI image and Pi kernel, then runs the media failure suite. To repeat
|
|
only the latter after those prerequisites:
|
|
|
|
```sh
|
|
make media-fault-test
|
|
```
|
|
|
|
No host disk or physical USB device is passed through. All media are disposable
|
|
regular files inside a new `out/m11-vm.*` or `out/m11-faults.*` directory. QEMU runs
|
|
inside the existing private network namespace. The SYSTEM root is a protected
|
|
read-only virtual disk; the twelve additional USB disks stress all twelve bay
|
|
slots. Earlier boot tests cover a removable USB SYSTEM itself.
|
|
|
|
The virtual controller explicitly has twelve USB 2 and USB 3 ports. Its live
|
|
controller identity is discovered before generating the test map. This is not a
|
|
map to install on the Pi. Physical hub chains and USB 2/3 aliases must be calibrated
|
|
on the actual machine using [the cartridge guide](cartridges.md).
|
|
|
|
The test records:
|
|
|
|
- Three cold-start pairs with empty and fully populated bays, using the actual
|
|
ordinary-user console prompt and identical image/controller configuration.
|
|
- Twelve correct cartridge IDs and serials, followed by three simultaneous
|
|
removal/insertion cycles with reversed enumeration and rotated bay positions.
|
|
- Eleven direct-I/O readers checking known bytes while DATA writes and flushes
|
|
64 MiB. Per-thread intervals must overlap, and QEMU's real block counters must
|
|
show reads/writes for every disk.
|
|
- Console readiness while the cartridge daemon is deliberately held at an
|
|
event gate with all twelve disks present. No sleep releases that gate.
|
|
- Native reboot during sustained DATA writes, independent `e2fsck`, exact payload
|
|
verification, another boot with a different device declaration order, and halt.
|
|
- Kernel logs checked for unexpected I/O failures, USB resets, filesystem errors
|
|
and controller failures. Intentional fault tests have separate logs.
|
|
|
|
Boot samples, shutdown events, bay inventories, I/O counters and kernel logs stay
|
|
beside the images. A greater-than-100 ms device-load change must receive the
|
|
recorded review explanation; it must not be hidden by selecting a faster sample.
|
|
Three emulator samples do not establish a Pi performance claim.
|
|
|
|
The fault suite injects actual EIO during writing, cache flush and readback. Other
|
|
cases interrupt a confirmed transfer after at least 64 MiB by cancellation,
|
|
worker termination or daemon termination. Failed operations must remain failed
|
|
across restart, the daemon must remain usable, and none may acquire SAFE status.
|
|
A test-only QEMU bandwidth limit keeps the large transfer observable; production
|
|
code has no corresponding delay. See [QEMU fault injection](https://www.qemu.org/docs/master/devel/testing/blkdebug.html).
|
|
|
|
## Prepare the physical session
|
|
|
|
Use twelve labelled test cartridges and a written bay inventory. Include the
|
|
intended SYSTEM, a backed-up or disposable DATA cartridge, and the PROGRAM,
|
|
ENVIRONMENT and hardware cartridges you plan to use. Keep a known-working SYSTEM
|
|
available for rollback. Record the Pi revision, EEPROM version, hub/controller
|
|
models, cable arrangement, power source, storage serials and image hashes.
|
|
|
|
Label physical bays `01` through `12`. Confirm one device at a time with
|
|
`fds topology` and `fds bay N`, then create the permanent machine map following
|
|
the cartridge guide. Check both USB 2 and USB 3 modes where supported. A serial
|
|
identifies the cartridge, while the stable controller/port path identifies its
|
|
bay; the `/dev/sdX` name is not a bay label. Do not copy the VM map.
|
|
|
|
No destructive operation is necessary merely to collect diagnostics. For write
|
|
stress, use test media whose contents can be replaced. Any `fds burn` or
|
|
`fds format` operation still requires its exact per-operation confirmation.
|
|
|
|
## Capture evidence on the target
|
|
|
|
Copy [capture-hardware](../../tools/capture-hardware) to the DATA cartridge. At the
|
|
FDS console, run it explicitly through Bash; DATA intentionally does not permit
|
|
direct execution:
|
|
|
|
```sh
|
|
bash /data/capture-hardware /data/capture-before
|
|
```
|
|
|
|
The destination must be a new absolute directory. It is created private to the
|
|
current user. The script collects FDS status, topology, boot events, kernel logs,
|
|
USB/block inventories, mounts, network links and package versions. It writes a
|
|
status table and checksums. It does not change services, mounts, firmware, disks
|
|
or networking. A command denied to the ordinary user is recorded with its exit
|
|
status; use an administrative maintenance session for the missing diagnostic if
|
|
needed. A successful capture is not a hardware PASS.
|
|
|
|
Choose a new directory for every capture. If DATA itself is under investigation,
|
|
write to `/tmp/capture-before` instead and copy the directory to trusted storage
|
|
before shutdown; `/tmp` is volatile. Review captured device serials and network
|
|
identifiers before sharing the files.
|
|
|
|
## Physical acceptance sequence
|
|
|
|
Copy the [physical session template](../../tests/hardware/session-template.json)
|
|
and fill it with observed values. Null means unknown, not zero.
|
|
Record every run, including failures. The following is an initial repeatable
|
|
matrix, not evidence that it has already passed:
|
|
|
|
| Scenario | Procedure and required evidence |
|
|
| --- | --- |
|
|
| Cold boot baseline | Run five full cold boots with the intended SYSTEM and no optional media. Capture boot reports and external power-on/display timing. |
|
|
| Fully populated cold boot | Repeat five times with all twelve bays occupied. Compare prompt timing, every physical bay/serial pair and kernel errors against baseline. |
|
|
| Hot insertion | Insert each cartridge individually, then insert several together. Verify all twelve positions and expected activation. Capture before/after inventories. |
|
|
| Safe removal | Stop explicit jobs and use `fds eject N` for storage. Remove only after SAFE. Verify that reusing the same bay does not retain the old cartridge's identity or SAFE state. |
|
|
| Rotated placement | Move the test cartridges to different bays and repeat. Each bay must follow its physical position regardless of Linux disk names. |
|
|
| Simultaneous I/O | Read known files from every readable cartridge while a managed DATA job writes a known payload. Record hashes, throughput and logs; USB hardware without storage needs its own functional workload. |
|
|
| Reboot under load | Request `fds reboot`. Verify the ordered shutdown record, boot again, and compare DATA hashes. Run a filesystem check only after that filesystem is unmounted in maintenance/recovery. |
|
|
| Poweroff under load | Request `fds poweroff`; record software phases and external power/display behavior. A blocked command is not permission to unplug DATA. |
|
|
| Power and USB behavior | Observe the intended supply/battery configuration under simultaneous device load, within each component's ratings. Record undervoltage indications, resets, disconnects and any display recovery failure. |
|
|
|
|
Use external instrumentation for power applied, visible E-Ink readiness and actual
|
|
power-off. Kernel timestamps cannot measure time before the kernel starts or prove
|
|
that a flash controller honored a flush. Do not convert a VM measurement into a
|
|
physical result. Intentional power-loss or surprise-removal durability tests need
|
|
a separate disposable-data session and an explicit test plan; ordinary SAFE
|
|
acceptance does not establish power-loss durability.
|
|
|
|
## Pass, fail and follow-up
|
|
|
|
A physical run fails if a cartridge appears in the wrong bay, a confirmed write
|
|
or DATA eject reports success after an actual I/O error, a busy DATA shutdown
|
|
forces poweroff, a kernel/controller error is unexplained, or verified payloads
|
|
change. Capture logs before restarting a service or rebooting so its evidence is
|
|
not lost. Preserve both the observed failure and any later successful rerun.
|
|
|
|
Compare the same image, power source and device population when evaluating a code
|
|
change. Explain or fix a boot regression over 100 ms. Keep the idle <1 second and
|
|
normal DATA <1.5 second shutdown targets separate from measured outcomes. The
|
|
physical report should explicitly list untested devices, profiles and power modes.
|
|
|
|
No new target dependency is introduced by M11. Test workloads use Rust's standard
|
|
library; the capture script uses Bash and tools already included in the base
|
|
image. QEMU, filesystem tools and Python were existing host test dependencies.
|