test(harness): make run-tests.sh drivable by a caller that cannot see the terminal (#11862)

* test(harness): make run-tests.sh drivable by a caller that cannot see the terminal

A tool call or a fresh session reads a captured output file, gets interrupted
mid-run, and starts cold. Two things in the wrapper tripped that caller: an
interrupt left pio's build tree running with no record of it (the next
invocation started a second build into the same .pio/build/, or pgrep'd and
matched itself); and pio's own "[PASSED]" / "N succeeded" lines made a
half-finished output file read as green.

run-tests.sh:
- Run record in .pio/runtests/current.tsv for the life of a run. A second
  invocation prints RESULT: BUSY and exits 4 without touching the build
  directory. Valid while the holder pid OR the recorded process group is
  alive, so a SIGKILLed wrapper with live scons children reads ORPHANED
  rather than clear.
- --status / --wait / --abort. --abort kills the whole tree by pgid.
- pio runs under setsid with its pgid recorded; INT/TERM/HUP kill the tree
  and record RESULT: ABORTED (exit 5), log kept.
- One result() for every verdict: prints to the stdout the script started
  with (a signal can arrive inside a redirected pio call) and writes
  .pio/runtests/last-result.tsv with head, args, env, finish time, kept log
  and a tree fingerprint (HEAD + working-tree diff + untracked files).
  --status marks the last verdict STALE when the tree has changed since.
- Banner naming the final RESULT: line as the only verdict.
- Non-Linux host: RESULT: UNSUPPORTED, exit 6, instead of the AMBER code.
- A failed build removes .pio/build/<env>/meshtasticd, which run bare would
  reprint the last good run.
- FILTERED lists the not-run count, not 77 suite names.

bin/run-tests.cmd forwards into WSL with the exit code passed through, so the
same command line works from cmd.exe and PowerShell; no logic is duplicated.

test/README.md, copilot-instructions.md and the mirrors document the new
codes and the rule.

* test(harness): one-suite warm-up, and name the build phase instead of freezing the counter

Measured on a full native run: the warm-up, `pio test --without-testing`
with no filter, builds AND links every suite - 78 links, 2320 s, 29.7 s
each, 39 minutes before the first test ran - and prints a "[PASSED]" line
for each program it merely linked. CI never did this; its warm-up is one
`platformio run`. The shared src objects are the same whichever suite links
them, so the warm-up now links one: the filtered suite when there is one,
else test_utf8. The run itself still builds every suite, as it must.

The heartbeat counted objects newer than its marker, which sits still through
PlatformIO's single-threaded scons dependency scan and through each link -
twelve minutes at "430/754 objs, ETA 17m" on that run, which reads as a hung
build to a caller who cannot run ps. It now names the phase from the
processes in the recorded group: [scons] / [compile] (with the ETA) / [link]
/ [test], and --status prints the same phase word.

* test(harness): define phase_of_run before --status can call it

* test(harness): bind --wait to the run it observed; serialize the run-record publish

Review findings on #11862, all three valid:

- --wait stored a state and then waited for any record to clear, so a run
  that finished and a second that started between polls would be followed to
  the second's verdict, and a run that turned ORPHANED mid-wait could print a
  stale last-result. Each run now has an id (pid-start) in current.tsv and
  last-result.tsv; --wait captures it and reports only a matching verdict,
  else ABORTED-without-verdict.
- run_state() then the current.tsv write was a check-then-act pair: two
  invocations in the same instant could both see IDLE. The pair is now one
  critical section under a short-lived flock, and both records are published
  by rename so no reader can see a partial file. The record stays the
  ownership token; the lock only serializes the handoff (a SIGKILLed holder
  releases flock but not the record, which is why flock alone was rejected).
- The FILTERED and AMBER examples in copilot-instructions.md carried a
  literal suite count, which the same document says never to do.

Verified on a live run: --status RUNNING, a concurrent start refused BUSY, a
--wait started before --abort reported the aborted run's own verdict with the
matching id, exit 5; no build process survived.

* fix(waypoints): do not create an empty store file on clear

clearAllWaypoints() wrote a two-byte empty store unconditionally, so test_waypoint_expiry left
Waypoints_default.wpts behind in every run and the suite has read AMBER (undeclared shared state)
since it landed. An existing file - stale or unreadable included - is still rewritten as empty, so
a reset after a failed load clears flash as before; a file that is not there is left not there.
This commit is contained in:
Tom authored and GitHub committed 2026-09-22 11:10:53 +00:00
1 parent 43479aa4e1
commit 08cd97ea2d
7 files changed
+404 -95

No files matched your search

+8 -3
View File
@@ -9,12 +9,17 @@ This directory contains C++ unit tests that run on the host machine via Platform
```bash
./bin/run-tests.sh # all suites
./bin/run-tests.sh -f test_traffic_management # single suite
./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt
./bin/run-tests.sh -f test_traffic_management --quiet # prints only the RESULT: line - the mode for tool calls
./bin/run-tests.sh --status # a run in progress? else the last verdict, marked STALE if the tree changed since
./bin/run-tests.sh --wait # attach to the run in progress; exits with its verdict
./bin/run-tests.sh --abort # stop it (the whole build tree); its log is kept
```
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED.
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in progress; nothing started), 5 = ABORTED, 6 = UNSUPPORTED host.
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On macOS or Windows, run the suite in a container: `./bin/test-native-docker.sh`.
**The `RESULT:` line is the only verdict.** pio prints `[PASSED]` per suite and `N succeeded` per invocation long before the wrapper has decided anything, so a captured output file looks green within the first minute; grade on the final `RESULT:` line and nothing else. One run at a time per checkout: a second invocation is refused with `BUSY` rather than started, because two `pio` jobs share `.pio/build/` and wipe each other's objects. Never look for a run with `pgrep` - ask `--status`. The last verdict and its log live in `.pio/runtests/`; an interrupted run records `ABORTED`, not nothing.
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On Windows, `bin\run-tests.cmd <same args>` forwards into WSL with the exit code passed through; on macOS, or without WSL, run the suite in a container: `./bin/test-native-docker.sh`.
**`-f` is not a gate.** A filtered run can pass while a full run fails, because filtering removes the suites that _create_ the state a later suite trips over. Iterate with `-f`; gate on a full run.