mirror of
https://github.com/meshtastic/firmware.git
synced 2026-10-10 15:43:36 -04:00
test(harness): make run-tests.sh drivable by a caller that cannot see the terminal (#11862)
* test(harness): make run-tests.sh drivable by a caller that cannot see the terminal A tool call or a fresh session reads a captured output file, gets interrupted mid-run, and starts cold. Two things in the wrapper tripped that caller: an interrupt left pio's build tree running with no record of it (the next invocation started a second build into the same .pio/build/, or pgrep'd and matched itself); and pio's own "[PASSED]" / "N succeeded" lines made a half-finished output file read as green. run-tests.sh: - Run record in .pio/runtests/current.tsv for the life of a run. A second invocation prints RESULT: BUSY and exits 4 without touching the build directory. Valid while the holder pid OR the recorded process group is alive, so a SIGKILLed wrapper with live scons children reads ORPHANED rather than clear. - --status / --wait / --abort. --abort kills the whole tree by pgid. - pio runs under setsid with its pgid recorded; INT/TERM/HUP kill the tree and record RESULT: ABORTED (exit 5), log kept. - One result() for every verdict: prints to the stdout the script started with (a signal can arrive inside a redirected pio call) and writes .pio/runtests/last-result.tsv with head, args, env, finish time, kept log and a tree fingerprint (HEAD + working-tree diff + untracked files). --status marks the last verdict STALE when the tree has changed since. - Banner naming the final RESULT: line as the only verdict. - Non-Linux host: RESULT: UNSUPPORTED, exit 6, instead of the AMBER code. - A failed build removes .pio/build/<env>/meshtasticd, which run bare would reprint the last good run. - FILTERED lists the not-run count, not 77 suite names. bin/run-tests.cmd forwards into WSL with the exit code passed through, so the same command line works from cmd.exe and PowerShell; no logic is duplicated. test/README.md, copilot-instructions.md and the mirrors document the new codes and the rule. * test(harness): one-suite warm-up, and name the build phase instead of freezing the counter Measured on a full native run: the warm-up, `pio test --without-testing` with no filter, builds AND links every suite - 78 links, 2320 s, 29.7 s each, 39 minutes before the first test ran - and prints a "[PASSED]" line for each program it merely linked. CI never did this; its warm-up is one `platformio run`. The shared src objects are the same whichever suite links them, so the warm-up now links one: the filtered suite when there is one, else test_utf8. The run itself still builds every suite, as it must. The heartbeat counted objects newer than its marker, which sits still through PlatformIO's single-threaded scons dependency scan and through each link - twelve minutes at "430/754 objs, ETA 17m" on that run, which reads as a hung build to a caller who cannot run ps. It now names the phase from the processes in the recorded group: [scons] / [compile] (with the ETA) / [link] / [test], and --status prints the same phase word. * test(harness): define phase_of_run before --status can call it * test(harness): bind --wait to the run it observed; serialize the run-record publish Review findings on #11862, all three valid: - --wait stored a state and then waited for any record to clear, so a run that finished and a second that started between polls would be followed to the second's verdict, and a run that turned ORPHANED mid-wait could print a stale last-result. Each run now has an id (pid-start) in current.tsv and last-result.tsv; --wait captures it and reports only a matching verdict, else ABORTED-without-verdict. - run_state() then the current.tsv write was a check-then-act pair: two invocations in the same instant could both see IDLE. The pair is now one critical section under a short-lived flock, and both records are published by rename so no reader can see a partial file. The record stays the ownership token; the lock only serializes the handoff (a SIGKILLed holder releases flock but not the record, which is why flock alone was rejected). - The FILTERED and AMBER examples in copilot-instructions.md carried a literal suite count, which the same document says never to do. Verified on a live run: --status RUNNING, a concurrent start refused BUSY, a --wait started before --abort reported the aborted run's own verdict with the matching id, exit 5; no build process survived. * fix(waypoints): do not create an empty store file on clear clearAllWaypoints() wrote a two-byte empty store unconditionally, so test_waypoint_expiry left Waypoints_default.wpts behind in every run and the suite has read AMBER (undeclared shared state) since it landed. An existing file - stale or unreadable included - is still rewritten as empty, so a reset after a failed load clears flash as before; a file that is not there is left not there.
This commit is contained in:
7 files changed
+404
-95
No files matched your search
+8
-3
@@ -9,12 +9,17 @@ This directory contains C++ unit tests that run on the host machine via Platform
|
||||
```bash
|
||||
./bin/run-tests.sh # all suites
|
||||
./bin/run-tests.sh -f test_traffic_management # single suite
|
||||
./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt
|
||||
./bin/run-tests.sh -f test_traffic_management --quiet # prints only the RESULT: line - the mode for tool calls
|
||||
./bin/run-tests.sh --status # a run in progress? else the last verdict, marked STALE if the tree changed since
|
||||
./bin/run-tests.sh --wait # attach to the run in progress; exits with its verdict
|
||||
./bin/run-tests.sh --abort # stop it (the whole build tree); its log is kept
|
||||
```
|
||||
|
||||
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED.
|
||||
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in progress; nothing started), 5 = ABORTED, 6 = UNSUPPORTED host.
|
||||
|
||||
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On macOS or Windows, run the suite in a container: `./bin/test-native-docker.sh`.
|
||||
**The `RESULT:` line is the only verdict.** pio prints `[PASSED]` per suite and `N succeeded` per invocation long before the wrapper has decided anything, so a captured output file looks green within the first minute; grade on the final `RESULT:` line and nothing else. One run at a time per checkout: a second invocation is refused with `BUSY` rather than started, because two `pio` jobs share `.pio/build/` and wipe each other's objects. Never look for a run with `pgrep` - ask `--status`. The last verdict and its log live in `.pio/runtests/`; an interrupted run records `ABORTED`, not nothing.
|
||||
|
||||
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On Windows, `bin\run-tests.cmd <same args>` forwards into WSL with the exit code passed through; on macOS, or without WSL, run the suite in a container: `./bin/test-native-docker.sh`.
|
||||
|
||||
**`-f` is not a gate.** A filtered run can pass while a full run fails, because filtering removes the suites that _create_ the state a later suite trips over. Iterate with `-f`; gate on a full run.
|
||||
|
||||
|
||||
Reference in new issue
Block a user