mirror/firmware
1
0 Fork 0
mirror of https://github.com/meshtastic/firmware.git synced 2026-10-08 22:29:03 -04:00
Code Issues Packages Projects Releases Wiki Activity

test(harness): make run-tests.sh drivable by a caller that cannot see the terminal (#11862)

Browse Source 
* test(harness): make run-tests.sh drivable by a caller that cannot see the terminal

A tool call or a fresh session reads a captured output file, gets interrupted
mid-run, and starts cold. Two things in the wrapper tripped that caller: an
interrupt left pio's build tree running with no record of it (the next
invocation started a second build into the same .pio/build/, or pgrep'd and
matched itself); and pio's own "[PASSED]" / "N succeeded" lines made a
half-finished output file read as green.

run-tests.sh:
- Run record in .pio/runtests/current.tsv for the life of a run. A second
  invocation prints RESULT: BUSY and exits 4 without touching the build
  directory. Valid while the holder pid OR the recorded process group is
  alive, so a SIGKILLed wrapper with live scons children reads ORPHANED
  rather than clear.
- --status / --wait / --abort. --abort kills the whole tree by pgid.
- pio runs under setsid with its pgid recorded; INT/TERM/HUP kill the tree
  and record RESULT: ABORTED (exit 5), log kept.
- One result() for every verdict: prints to the stdout the script started
  with (a signal can arrive inside a redirected pio call) and writes
  .pio/runtests/last-result.tsv with head, args, env, finish time, kept log
  and a tree fingerprint (HEAD + working-tree diff + untracked files).
  --status marks the last verdict STALE when the tree has changed since.
- Banner naming the final RESULT: line as the only verdict.
- Non-Linux host: RESULT: UNSUPPORTED, exit 6, instead of the AMBER code.
- A failed build removes .pio/build/<env>/meshtasticd, which run bare would
  reprint the last good run.
- FILTERED lists the not-run count, not 77 suite names.

bin/run-tests.cmd forwards into WSL with the exit code passed through, so the
same command line works from cmd.exe and PowerShell; no logic is duplicated.

test/README.md, copilot-instructions.md and the mirrors document the new
codes and the rule.

* test(harness): one-suite warm-up, and name the build phase instead of freezing the counter

Measured on a full native run: the warm-up, `pio test --without-testing`
with no filter, builds AND links every suite - 78 links, 2320 s, 29.7 s
each, 39 minutes before the first test ran - and prints a "[PASSED]" line
for each program it merely linked. CI never did this; its warm-up is one
`platformio run`. The shared src objects are the same whichever suite links
them, so the warm-up now links one: the filtered suite when there is one,
else test_utf8. The run itself still builds every suite, as it must.

The heartbeat counted objects newer than its marker, which sits still through
PlatformIO's single-threaded scons dependency scan and through each link -
twelve minutes at "430/754 objs, ETA 17m" on that run, which reads as a hung
build to a caller who cannot run ps. It now names the phase from the
processes in the recorded group: [scons] / [compile] (with the ETA) / [link]
/ [test], and --status prints the same phase word.

* test(harness): define phase_of_run before --status can call it

* test(harness): bind --wait to the run it observed; serialize the run-record publish

Review findings on #11862, all three valid:

- --wait stored a state and then waited for any record to clear, so a run
  that finished and a second that started between polls would be followed to
  the second's verdict, and a run that turned ORPHANED mid-wait could print a
  stale last-result. Each run now has an id (pid-start) in current.tsv and
  last-result.tsv; --wait captures it and reports only a matching verdict,
  else ABORTED-without-verdict.
- run_state() then the current.tsv write was a check-then-act pair: two
  invocations in the same instant could both see IDLE. The pair is now one
  critical section under a short-lived flock, and both records are published
  by rename so no reader can see a partial file. The record stays the
  ownership token; the lock only serializes the handoff (a SIGKILLed holder
  releases flock but not the record, which is why flock alone was rejected).
- The FILTERED and AMBER examples in copilot-instructions.md carried a
  literal suite count, which the same document says never to do.

Verified on a live run: --status RUNNING, a concurrent start refused BUSY, a
--wait started before --abort reported the aborted run's own verdict with the
matching id, exit 5; no build process survived.

* fix(waypoints): do not create an empty store file on clear

clearAllWaypoints() wrote a two-byte empty store unconditionally, so test_waypoint_expiry left
Waypoints_default.wpts behind in every run and the suite has read AMBER (undeclared shared state)
since it landed. An existing file - stale or unreadable included - is still rewritten as empty, so
a reset after a failed load clears flash as before; a file that is not there is left not there.
This commit is contained in:
Tom authored and GitHub committed 2026-09-22 11:10:53 +00:00
1 parent 43479aa4e1
commit 08cd97ea2d
7 files changed
+404 -95
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/copilot-instructions.md
+19 -14
View File
@@ -2,12 +2,12 @@
> **TL;DR**
>
> | | |
> | -------------- | ---------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) |
> | | |
> | -------------- | ------------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) |
>
> **Need this? It's here.**
>
@@ -772,12 +772,15 @@ Unit tests in `test/` directory. The canonical suite count is detected on the fl
Exit codes and verdicts (exact counts will vary; examples below are illustrative):
| Exit | Verdict | Meaning |
| ---- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases |
| 1 | `RED` | At least one failure, build error, or sanitizer fault |
| 2 | `AMBER` | All that ran passed, but something was lost or unexplained: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or a suite left behind shared state it does not declare |
| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run |
| Exit | Verdict | Meaning |
| ---- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases |
| 1 | `RED` | At least one failure, build error, or sanitizer fault |
| 2 | `AMBER` | All that ran passed, but something was lost or unexplained: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or a suite left behind shared state it does not declare |
| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run |
| 4 | `BUSY` | A run is already in progress (this or another session); nothing was started. `--status` to see it, `--wait` to attach, `--abort` to stop it |
| 5 | `ABORTED` | The run was stopped by a signal or `--abort`; its log is kept. `--status` shows it |
| 6 | `UNSUPPORTED` | Not a Linux host; nothing ran. Use WSL (`bin\run-tests.cmd` forwards) or `./bin/test-native-docker.sh` |
Examples - exact counts will vary by suite count and env:
@@ -792,12 +795,14 @@ RESULT: RED 1 failed
RESULT: RED exit-time abort (tests passed; likely sanitizer - see hint above)
# AMBER: a suite silently went missing on a full run
RESULT: AMBER 23/24 suites ran (missing: test_radio) - all that ran passed
RESULT: AMBER N-1/N suites ran (missing: test_radio) - all that ran passed
# FILTERED: single suite run completed cleanly
RESULT: FILTERED 1/24 suites ran (not run: test_admin_radio test_atak …) - filtered: test_serial
RESULT: FILTERED 1/N suites ran (N-1 not run) - filtered: test_serial
```
The script is written to be driven by a caller that cannot see the terminal: the final `RESULT:` line is the only verdict (pio's own `[PASSED]` and `N succeeded` lines precede it and mean nothing on their own); a second invocation while a run is in progress is refused with `BUSY` rather than started; the last verdict is kept in `.pio/runtests/last-result.tsv` with its log, and `./bin/run-tests.sh --status` prints it, marking it **STALE** when the tree has changed since. Never `pgrep` for a run - ask `--status`.
> **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run.
Raw `pio test` (no sanitizers, no verdict logic) - use only when you need to override the env:
AGENTS.md
+6 -6
View File
@@ -2,12 +2,12 @@
> **TL;DR**
>
> | | |
> | -------------- | ---------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) |
> | | |
> | -------------- | ------------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) |
>
> **Need this? It's here.**
>
CLAUDE.md
+6 -6
View File
@@ -2,12 +2,12 @@
> **TL;DR**
>
> | | |
> | -------------- | ---------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` |
> | | |
> | -------------- | ------------------------------------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first |
> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` |
>
> **Need this? It's here.**
>
bin/run-tests.cmd
+12
View File
@@ -0,0 +1,12 @@
@echo off
rem Forwarder, not a port. bin\run-tests.sh is Linux-only by design (its HOST note says why); this
rem runs it inside WSL from this same checkout, passing the arguments and the exit code straight
rem through, so the same command line and the same RESULT: line work from cmd.exe and PowerShell.
rem
rem bin\run-tests.cmd -e native -f test_utf8 --quiet
rem bin\run-tests.cmd --status
rem
rem Needs WSL with a distro that has the PlatformIO venv. A checkout on the Windows drive (/mnt/c)
rem builds several times slower than one on the WSL filesystem (\\wsl$\...); both paths work here.
wsl.exe --cd "%~dp0.." -e ./bin/run-tests.sh %*
exit /b %ERRORLEVEL%
bin/run-tests.sh
+338 -60
View File
@@ -17,8 +17,23 @@
# ./bin/run-tests.sh --keep-state # keep every suite's sandbox, not just the interesting ones
# ./bin/run-tests.sh --shuffle # randomise suite order (seed from HEAD; printed)
# ./bin/run-tests.sh --seed 12345 # replay an exact order (implies --shuffle)
# ./bin/run-tests.sh --status # is a run in progress? if not, what did the last one say?
# ./bin/run-tests.sh --wait # attach to the run in progress; exit with its verdict
# ./bin/run-tests.sh --abort # stop the run in progress (whole build tree), keep its log
# bin\run-tests.cmd <same args> # from a Windows shell: forwards into WSL, exit code passed through
#
# Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED.
# Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in
# progress - never start a second one), 5 = ABORTED (signal or --abort), 6 = UNSUPPORTED host
# (nothing ran; Linux only - WSL and ./bin/test-native-docker.sh are the routes from elsewhere).
#
# FOR A CALLER THAT CANNOT SEE THE TERMINAL (a tool call, a backgrounded job, a fresh session):
# - The verdict is the final "RESULT:" line and nothing else. pio prints "[PASSED]" per suite and
# "N succeeded" per invocation long before the verdict exists; a captured output file looks
# green within the first minute. Use --quiet, which prints only the RESULT line.
# - One run at a time. A second invocation while one is in progress prints RESULT: BUSY and exits
# 4 without touching the build directory. Ask with --status; block with --wait; never pgrep.
# - The last verdict is on disk: .pio/runtests/last-result.tsv (--status prints it when idle),
# with the run's log kept beside it. An interrupted run records RESULT: ABORTED, not nothing.
#
# HOST. This is a Linux tool: bash 4+ (mapfile), GNU coreutils and GNU find (`-printf`, md5sum,
# `-executable`). That is a deliberate choice, not an oversight - the alternative is a second,
@@ -59,7 +74,7 @@
# RESULT: GREEN N/N suites passed
# RESULT: AMBER N/M suites ran (missing: test_radio test_serial) - all that ran passed
# RESULT: AMBER 3 test case(s) ignored
# RESULT: FILTERED 1/N suites ran (not run: …) - filtered: test_utf8
# RESULT: FILTERED 1/N suites ran (N-1 not run) - filtered: test_utf8
# RESULT: RED test attribution failed - suites did not run their own tests
# RESULT: RED test_traffic_management: 1 failed (or: build/crash error)
# RESULT: RED sanitizer fault - SUMMARY: AddressSanitizer: 1272 byte(s) leaked (tests may have
@@ -73,8 +88,10 @@ set -uo pipefail
# mis-hash the sandbox, mis-read a suite list and report a verdict that looks real.
if [[ $(uname -s) != Linux ]]; then
echo "run-tests.sh is Linux-only (bash 4+, GNU coreutils, GNU find); this host is $(uname -s)." >&2
echo "Run the suite in a container instead: ./bin/test-native-docker.sh" >&2
exit 2
echo "Run it under WSL, or in a container: ./bin/test-native-docker.sh" >&2
# Its own code, not AMBER: nothing ran, and a caller must not read this as "passed with caveats".
echo "RESULT: UNSUPPORTED host $(uname -s) - nothing ran; use WSL or ./bin/test-native-docker.sh"
exit 6
fi
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -93,9 +110,15 @@ PASSTHRU=()
# must still forward everything else the user gave (-v, -vvv, ...) - otherwise a shuffled run
# builds with those flags and then runs without them.
EXTRA_ARGS=()
ORIG_ARGS=("$@")
SUBCMD=""
while [[ $# -gt 0 ]]; do
case "$1" in
--status | --wait | --abort)
SUBCMD="${1#--}"
shift
;;
-f)
FILTER="$2"
PASSTHRU+=("-f" "$2")
@@ -134,11 +157,259 @@ while [[ $# -gt 0 ]]; do
esac
done
# --- Run record ----------------------------------------------------------------
# One run at a time, and the last verdict on disk. Two things a caller that cannot see the terminal
# needs: a way to ask "is something running?" that is not `pgrep -f` (which matches the asker), and
# a way to read the last verdict that is not scrollback. Both live in .pio/runtests/.
#
# current.tsv exists for exactly the life of a run: written before anything touches .pio/, removed
# by result(). A lock alone would not do - a SIGKILLed wrapper releases flock while its scons tree
# lives on - so a record is valid while EITHER its holder pid OR its recorded process group is
# alive; the second case is ORPHANED, and only --abort (or the tree finishing) clears it.
RUN_DIR="$ROOT_DIR/.pio/runtests"
RUN_RECORD="$RUN_DIR/current.tsv"
LAST_RESULT="$RUN_DIR/last-result.tsv"
KEPT_LOG="$RUN_DIR/last.log"
mkdir -p "$RUN_DIR"
LOG=""
BUILD_LOG=""
# The verdict prints to the stdout this script STARTED with. A signal can arrive while a pio call
# has stdout redirected into a log file, and the trap runs inside that redirect.
exec 3>&1
RUN_START=$(date +%s)
RUN_ID="$$-$RUN_START" # names this run in current.tsv and last-result.tsv, so --wait cannot follow a later one
HEAD_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo unknown)
# What the verdict was FOR. HEAD alone is not enough - a verdict from before the current edits is
# the most convincing stale green there is - so the fingerprint covers the working tree too:
# tracked changes by content, untracked files by content. Recorded with the verdict; --status
# recomputes and says whether the last verdict still describes this tree.
tree_fingerprint() {
{
git rev-parse HEAD 2>/dev/null
git diff HEAD 2>/dev/null
git ls-files --others --exclude-standard -z 2>/dev/null | xargs -0 -r md5sum 2>/dev/null
} | md5sum | cut -c1-12
}
TREE_FP=$(tree_fingerprint)
# A build that fails leaves the previous binary in place, and run bare it reprints the last good
# run - a green that predates the failure. Remove it; the next build regenerates it.
drop_stale_program() {
local prog=".pio/build/${ENV}/meshtasticd"
if [[ -f $prog ]]; then
rm -f "$prog"
echo " removed $prog: it predates this failure and run bare would reprint the last good run"
fi
}
ARGS_STR="${ORIG_ARGS[*]-}"
tsv_get() { awk -F'\t' -v k="$2" '$1 == k { print $2; exit }' "$1" 2>/dev/null; }
pid_alive() { [[ -n ${1-} ]] && kill -0 "$1" 2>/dev/null; }
pgid_alive() { [[ -n ${1-} ]] && ps -eo pgid= 2>/dev/null | tr -d ' ' | grep -qx "$1"; }
elapsed_since() {
local s=$(($(date +%s) - ${1:-0}))
printf '%dm%02ds' $((s / 60)) $((s % 60))
}
run_state() {
[[ -f $RUN_RECORD ]] || {
echo IDLE
return
}
if pid_alive "$(tsv_get "$RUN_RECORD" pid)"; then
echo RUNNING
elif pgid_alive "$(tsv_get "$RUN_RECORD" pgid)"; then
echo ORPHANED
else
# Holder and tree both gone with the record still here: the wrapper was killed before
# result() ran. Nothing to report for it beyond the kept log; clear the way.
rm -f "$RUN_RECORD"
echo IDLE
fi
}
# The one place a verdict is printed. Records it, keeps the run log beside it, drops the run
# record, exits. Every RESULT: line in this script goes through here so --status can never
# disagree with what the caller saw.
result() {
local code=$1 text=$2 src kept="-"
echo "RESULT: $text" >&3
# Keep whichever log has content: the test log, or during the build phase the build log.
for src in "$LOG" "$BUILD_LOG"; do
if [[ -n $src && -s $src ]] && cp "$src" "$KEPT_LOG" 2>/dev/null; then
kept="$KEPT_LOG"
break
fi
done
printf 'result\tRESULT: %s\ncode\t%s\nrun\t%s\nhead\t%s\ntree\t%s\nargs\t%s\nenv\t%s\nfinished\t%s\nlog\t%s\n' \
"$text" "$code" "$RUN_ID" "$HEAD_SHA" "$TREE_FP" "$ARGS_STR" "$ENV" "$(date +%s)" "$kept" >"$LAST_RESULT.tmp" &&
mv -f "$LAST_RESULT.tmp" "$LAST_RESULT"
rm -f "$RUN_RECORD"
exit "$code"
}
# What the build tree is doing right now, from the processes in the recorded group: PlatformIO
# alternates single-threaded scons dependency scans, parallel compiles and one long link per
# suite, and an object counter freezes through the first and the last of those - which reads as a
# hung build to anyone who cannot run ps.
phase_of_run() {
local pgid comms
pgid=$(tsv_get "$RUN_RECORD" pgid)
[[ -z $pgid ]] && {
echo idle
return
}
comms=" $(ps -eo pgid=,comm= 2>/dev/null | awk -v p="$pgid" '$1 == p { print $2 }' | tr '\n' ' ') "
case $comms in
*" cc1plus "* | *" cc1 "* | *" as "*) echo compile ;;
*" ld "* | *" ld.bfd "* | *" ld.gold "* | *" ld.lld "* | *" mold "* | *" collect2 "*) echo link ;;
*" meshtasticd "* | *" program "*) echo test ;;
*python*) echo scons ;;
*) echo idle ;;
esac
}
status_cmd() {
local st
st=$(run_state)
case $st in
RUNNING | ORPHANED)
local progress
progress=$(tail -n1 "$(tsv_get "$RUN_RECORD" progress)" 2>/dev/null)
echo "STATUS: $st pid=$(tsv_get "$RUN_RECORD" pid) pgid=$(tsv_get "$RUN_RECORD" pgid) since=$(elapsed_since "$(tsv_get "$RUN_RECORD" started)") now=$(phase_of_run) head=$(tsv_get "$RUN_RECORD" head) args=\"$(tsv_get "$RUN_RECORD" args)\""
echo " progress: ${progress:-none yet} (tail -f $(tsv_get "$RUN_RECORD" progress))"
if [[ $st == ORPHANED ]]; then
echo " the wrapper died but its build tree is still running: ./bin/run-tests.sh --abort to stop it, or --wait for it to finish (no verdict will be recorded)"
else
echo " ./bin/run-tests.sh --wait blocks until the verdict; --abort stops it"
fi
;;
IDLE)
if [[ -f $LAST_RESULT ]]; then
local fresh="current"
[[ $(tsv_get "$LAST_RESULT" tree) == "$(tree_fingerprint)" ]] || fresh="STALE - the tree has changed since; this verdict is not about the current code"
echo "STATUS: IDLE - last run ($fresh): $(tsv_get "$LAST_RESULT" result)"
echo " exit $(tsv_get "$LAST_RESULT" code), head $(tsv_get "$LAST_RESULT" head), env $(tsv_get "$LAST_RESULT" env), args \"$(tsv_get "$LAST_RESULT" args)\", finished $(elapsed_since "$(tsv_get "$LAST_RESULT" finished)") ago"
echo " log: $(tsv_get "$LAST_RESULT" log)"
else
echo "STATUS: IDLE - no run recorded"
fi
;;
esac
}
wait_cmd() {
local st observed
st=$(run_state)
[[ $st == IDLE ]] && {
status_cmd
[[ -f $LAST_RESULT ]] && exit "$(tsv_get "$LAST_RESULT" code)"
exit 0
}
# Bound to THIS run: if it ends and another starts between polls, its verdict - or the
# absence of one - is what gets reported, never the newcomer's.
observed=$(tsv_get "$RUN_RECORD" run)
echo "waiting for $st run $observed pgid=$(tsv_get "$RUN_RECORD" pgid)..." >&2
while [[ -f $RUN_RECORD && $(tsv_get "$RUN_RECORD" run) == "$observed" && $(run_state) != IDLE ]]; do sleep 5; done
if [[ -f $LAST_RESULT && $(tsv_get "$LAST_RESULT" run) == "$observed" ]]; then
tsv_get "$LAST_RESULT" result
exit "$(tsv_get "$LAST_RESULT" code)"
fi
echo "RESULT: ABORTED run $observed ended without recording a verdict (its wrapper was gone; log, if kept: $KEPT_LOG)"
exit 5
}
abort_cmd() {
local st pid pgid
st=$(run_state)
[[ $st == IDLE ]] && {
echo "nothing to abort"
status_cmd
exit 0
}
pid=$(tsv_get "$RUN_RECORD" pid)
pgid=$(tsv_get "$RUN_RECORD" pgid)
if [[ $st == RUNNING ]]; then
# The holder's own trap kills the tree and records ABORTED; give it a moment to do so.
kill -TERM "$pid" 2>/dev/null
for _ in 1 2 3 4 5 6 7 8 9 10; do
[[ $(run_state) == IDLE ]] && break
sleep 1
done
fi
if [[ $(run_state) != IDLE ]]; then
[[ -n $pgid ]] && kill -TERM -- "-$pgid" 2>/dev/null
sleep 2
[[ -n $pgid ]] && kill -KILL -- "-$pgid" 2>/dev/null
printf 'result\tRESULT: ABORTED by --abort (wrapper pid %s was already gone)\ncode\t5\nrun\t%s\nhead\t%s\nargs\t%s\nenv\t%s\nfinished\t%s\nlog\t%s\n' \
"$pid" "$(tsv_get "$RUN_RECORD" run)" "$(tsv_get "$RUN_RECORD" head)" "$(tsv_get "$RUN_RECORD" args)" "$(tsv_get "$RUN_RECORD" env)" "$(date +%s)" "$KEPT_LOG" >"$LAST_RESULT.tmp" &&
mv -f "$LAST_RESULT.tmp" "$LAST_RESULT"
rm -f "$RUN_RECORD"
fi
status_cmd
exit 0
}
case $SUBCMD in
status)
status_cmd
exit 0
;;
wait) wait_cmd ;;
abort) abort_cmd ;;
esac
# A run is a run: refuse while one is in progress, whatever env it is for. Two pio jobs share
# .pio/build/ and libdeps and wipe each other's objects, and the state summary below is per run.
#
# The check and the publish are one critical section under a short-lived flock, so two
# invocations in the same instant cannot both see IDLE; the record itself, published atomically
# via rename, stays the ownership token (a SIGKILLed holder releases the lock but not the record).
PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress"
exec 9>"$RUN_DIR/lock"
flock 9
case $(run_state) in
RUNNING | ORPHANED)
flock -u 9
status_cmd
echo "RESULT: BUSY a run is already in progress - never start a second one (--status, --wait, --abort)"
exit 4
;;
esac
printf 'run\t%s\npid\t%s\npgid\t\nstarted\t%s\nargs\t%s\nenv\t%s\nhead\t%s\nprogress\t%s\n' \
"$RUN_ID" "$$" "$RUN_START" "$ARGS_STR" "$ENV" "$HEAD_SHA" "$PROGRESS_FILE" >"$RUN_RECORD.tmp" &&
mv -f "$RUN_RECORD.tmp" "$RUN_RECORD"
flock -u 9
# Every pio invocation goes through here: its own process group (setsid), pgid recorded in the run
# record so a signal to this script - or --abort from another session - takes the whole scons tree
# with it. Called inside pipelines, hence the record file rather than a shell variable.
run_pio() {
setsid "$PIO" "$@" &
local pid=$!
sed -i "s/^pgid\t.*/pgid\t$pid/" "$RUN_RECORD"
wait "$pid"
}
abort_run() {
local pgid
pgid=$(tsv_get "$RUN_RECORD" pgid)
if [[ -n $pgid ]]; then
kill -TERM -- "-$pgid" 2>/dev/null
sleep 2
kill -KILL -- "-$pgid" 2>/dev/null
fi
result 5 "ABORTED after $(elapsed_since "$RUN_START") ($1) - ./bin/run-tests.sh --status for the kept log"
}
trap 'abort_run SIGINT' INT
trap 'abort_run SIGTERM' TERM
trap 'abort_run SIGHUP' HUP
# Locate pio (PATH, then the standard PlatformIO venv).
PIO="$(command -v pio || command -v platformio || echo "$HOME/.platformio/penv/bin/pio")"
if [[ ! -x $PIO ]] && ! command -v "$PIO" >/dev/null 2>&1; then
echo "RESULT: RED pio not found (looked in PATH and ~/.platformio/penv/bin)"
exit 1
result 1 "RED pio not found (looked in PATH and ~/.platformio/penv/bin)"
fi
LOG="$(mktemp -t meshtest.XXXXXX.log)"
@@ -187,7 +458,6 @@ BASELINE_FILE=".pio/build/${ENV}/.runtests-objcount"
# Progress trail file (gitignored build dir). ALWAYS written so a backgrounded/piped run can be
# checked mid-build with `tail -f` - that's the whole point: don't fly blind on a 20-min rebuild.
PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress"
# --- Progress heartbeat ------------------------------------------------------
# Emit ONE status line every few seconds: build = objects (re)compiled this run / cached total +
@@ -195,24 +465,32 @@ PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress"
# check on a backgrounded run); also live-updates the tty when $5=1 (interactive --quiet). Never
# touches $LOG, which is parsed for the verdict, so piped/CI captures stay clean.
progress_monitor() {
local marker="$1" objtotal="$2" testtotal="$3" pfile="$4" totty="$5" start now el done ran eta line
local marker="$1" objtotal="$2" testtotal="$3" pfile="$4" totty="$5" start now el done ran eta line phase
start=$(date +%s)
while :; do
now=$(date +%s)
el=$((now - start))
phase=$(phase_of_run)
if grep -q 'Testing\.\.\.' "$LOG" 2>/dev/null; then
ran=$(grep -cE "${ENV}:test_[a-z0-9_]+ \[(PASSED|FAILED|ERRORED)\]" "$LOG" 2>/dev/null)
line=$(printf '[test] %s/%s suites done - %dm%02ds' "$ran" "$testtotal" $((el / 60)) $((el % 60)))
line=$(printf '[test] %s/%s suites done - %dm%02ds - now: %s' "$ran" "$testtotal" $((el / 60)) $((el % 60)) "$phase")
else
done=$(find ".pio/build/${ENV}" -name '*.o' -newer "$marker" 2>/dev/null | wc -l)
if ((objtotal > 0 && done > 0)); then
eta=$((objtotal > done ? (objtotal - done) * el / done : 0))
line=$(printf '[build] %d/%d objs - %dm%02ds - ETA ~%dm%02ds' \
"$done" "$objtotal" $((el / 60)) $((el % 60)) $((eta / 60)) $((eta % 60)))
else
# done==0 (incremental: nothing to rebuild yet) or no cached baseline - no ETA yet.
line=$(printf '[build] %d objs compiled - %dm%02ds' "$done" $((el / 60)) $((el % 60)))
fi
case $phase in
compile)
if ((objtotal > 0 && done > 0)); then
eta=$((objtotal > done ? (objtotal - done) * el / done : 0))
line=$(printf '[compile] %d/%d objs - %dm%02ds - ETA ~%dm%02ds' \
"$done" "$objtotal" $((el / 60)) $((el % 60)) $((eta / 60)) $((eta % 60)))
else
# done==0 (incremental: nothing to rebuild yet) or no cached baseline - no ETA yet.
line=$(printf '[compile] %d objs so far - %dm%02ds' "$done" $((el / 60)) $((el % 60)))
fi
;;
link) line=$(printf '[link] %d objs compiled, linking - %dm%02ds' "$done" $((el / 60)) $((el % 60))) ;;
scons) line=$(printf '[scons] dependency scan (single-threaded; the counter is expected to sit) - %d objs so far - %dm%02ds' "$done" $((el / 60)) $((el % 60))) ;;
*) line=$(printf '[build] %d objs so far - %dm%02ds - now: %s' "$done" $((el / 60)) $((el % 60)) "$phase") ;;
esac
fi
printf '%s\n' "$line" >>"$pfile" 2>/dev/null # file trail (always)
[[ $totty == 1 ]] && printf '\r\033[K%s' "$line" >/dev/tty 2>/dev/null # live line (human)
@@ -239,6 +517,7 @@ if ! $QUIET; then
echo "Running: $PIO test -e $ENV ${PASSTHRU[*]-} (expecting $EXPECTED_COUNT suites)"
fi
echo "progress: tail -f $PROGRESS_FILE" >&2
echo "RUN pending (pid $$): the verdict is the final RESULT: line only - [PASSED] and \"N succeeded\" lines before it are pio's, not a verdict. status: ./bin/run-tests.sh --status" >&2
if [[ ! -t 1 ]] && ! $QUIET; then
echo "hint: stdout is a pipe - build errors appear at the top of output and may be lost; use --quiet to get just the RESULT line" >&2
fi
@@ -263,22 +542,33 @@ if $SHUFFLE; then
echo "suite order: shuffled with --seed $SEED (${#RUN_ORDER[@]} suites)"
fi
# Warm the shared src objects before running any suite, the way .github/workflows/test_native.yml
# does. Fused build+run makes whichever suite PlatformIO's directory walk reaches first absorb the
# whole src compile and report it as its own duration - that is how a 35s suite once reported 13
# minutes, and it hides the build cost from every timing the summary prints.
# Warm the shared src objects before running any suite. Fused build+run makes whichever suite
# PlatformIO's directory walk reaches first absorb the whole src compile and report it as its own
# duration - that is how a 35s suite once reported 13 minutes, and it hides the build cost from
# every timing the summary prints.
#
# ONE suite, not the whole set: `--without-testing` with no filter builds AND LINKS every suite -
# 78 links, 39 minutes, measured - and prints a full "[PASSED]" table for programs that never ran.
# The shared src objects are the same whichever suite links them, so the filtered suite when there
# is one, else the cheapest to link, warms everything the run below needs.
#
# This is a WARM-UP ONLY: the run below must still build. PlatformIO links every test program to
# the one $BUILD_DIR/$PROGNAME path, so a `--without-building` run executes whichever suite was
# linked last - every suite, under its own name, all PASSED. The warm-up keeps the src compile out
# of the suite timings; the per-suite step is then just one test_main.cpp plus a link.
WARM_SUITE="$FILTER"
if [[ -z $WARM_SUITE ]]; then
WARM_SUITE="test_utf8"
[[ -d test/$WARM_SUITE ]] || WARM_SUITE="${ALL_SUITES[0]}"
fi
BUILD_SECS=0
build_started=$SECONDS
if $QUIET; then
"$PIO" test -e "$ENV" "${PASSTHRU[@]}" --without-testing >"$BUILD_LOG" 2>&1
run_pio test -e "$ENV" "${EXTRA_ARGS[@]}" -f "$WARM_SUITE" --without-testing >"$BUILD_LOG" 2>&1
BUILD_RC=$?
else
"$PIO" test -e "$ENV" "${PASSTHRU[@]}" --without-testing 2>&1 | tee "$BUILD_LOG"
echo "warm-up: building src + linking $WARM_SUITE (a [PASSED] line below means built, not run)"
run_pio test -e "$ENV" "${EXTRA_ARGS[@]}" -f "$WARM_SUITE" --without-testing 2>&1 | tee "$BUILD_LOG"
BUILD_RC=${PIPESTATUS[0]}
fi
BUILD_SECS=$((SECONDS - build_started))
@@ -291,8 +581,8 @@ if ((BUILD_RC != 0)); then
echo "RED - build failed before any suite ran:"
grep -nE 'error:|undefined reference|\[ERRORED\]' "$BUILD_LOG" | head -5 | sed 's/^/ /'
[[ -n $BUILD_FAIL_LOG ]] && echo " -> full build output: $BUILD_FAIL_LOG"
echo "RESULT: RED build failed in ${BUILD_SECS}s (no suites ran)"
exit 1
drop_stale_program
result 1 "RED build failed in ${BUILD_SECS}s (no suites ran)"
fi
if ! $QUIET; then
echo "build: ${BUILD_SECS}s (shared by every suite; suite durations below exclude it)"
@@ -306,22 +596,22 @@ if $SHUFFLE; then
: >"$LOG"
for suite in "${RUN_ORDER[@]}"; do
if $QUIET; then
"$PIO" test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \
run_pio test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \
--junit-output-path "$ATTRIB_DIR/$suite.xml" >>"$LOG" 2>&1
rc=$?
else
"$PIO" test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \
run_pio test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \
--junit-output-path "$ATTRIB_DIR/$suite.xml" 2>&1 | tee -a "$LOG"
rc=${PIPESTATUS[0]}
fi
((rc != 0)) && PIO_RC=$rc
done
elif $QUIET; then
"$PIO" test -e "$ENV" "${PASSTHRU[@]}" \
run_pio test -e "$ENV" "${PASSTHRU[@]}" \
--junit-output-path "$ATTRIB_DIR/all.xml" >"$LOG" 2>&1
PIO_RC=$?
else
"$PIO" test -e "$ENV" "${PASSTHRU[@]}" \
run_pio test -e "$ENV" "${PASSTHRU[@]}" \
--junit-output-path "$ATTRIB_DIR/all.xml" 2>&1 | tee "$LOG"
PIO_RC=${PIPESTATUS[0]}
fi
@@ -443,8 +733,7 @@ verdict_red() {
grep -nE "$SAN_RE" "$LOG" | head -4 | sed 's/^/ /'
echo " -> sanitizer fault: if every test above is PASS, this is an exit-time abort, not a failed assertion."
echo " -> read the full report by running the binary BARE (gdb hides it via ptrace): ./$bin 2>&1 | tail -40"
echo "RESULT: RED sanitizer fault - $(grep -ohE 'SUMMARY: [A-Za-z]+Sanitizer:.*' "$LOG" | tail -1 || echo 'see report above')"
exit 1
result 1 "RED sanitizer fault - $(grep -ohE 'SUMMARY: [A-Za-z]+Sanitizer:.*' "$LOG" | tail -1 || echo 'see report above')"
fi
# A guard in test/TestUtil.cpp aborting on purpose - a listening socket, or force_simradio put
@@ -455,8 +744,7 @@ verdict_red() {
grep -E '^FATAL: ' "$LOG" | head -3 | sed 's/^/ /'
echo " -> a harness guard aborted the suite deliberately. Not a crash and not a sanitizer"
echo " fault; the reason is the FATAL line above, and the suite's sandbox has the full log."
echo "RESULT: RED harness guard - $(grep -m1 -oE '^FATAL: .*' "$LOG")"
exit 1
result 1 "RED harness guard - $(grep -m1 -oE '^FATAL: .*' "$LOG")"
fi
# All tests passed but the process still aborted at EXIT (ERRORED/SIGHUP/SIGABRT) and the
@@ -465,12 +753,13 @@ verdict_red() {
if grep -qE "$PASS_RE" "$LOG" && grep -qE '\[ERRORED\]|SIGHUP|SIGABRT' "$LOG" && ! grep -qE ':FAIL\b|\[FAILED\]' "$LOG"; then
echo " -> all tests passed but the process aborted at EXIT - likely an ASan/LSan fault whose report"
echo " the runner swallowed (commonly shown as SIGHUP). Run the binary BARE to see it: ./$bin 2>&1 | tail -40"
echo "RESULT: RED exit-time abort (tests passed; likely sanitizer - see hint above)"
exit 1
result 1 "RED exit-time abort (tests passed; likely sanitizer - see hint above)"
fi
echo "RESULT: RED $(grep -oE '[0-9]+ failed' "$LOG" | tail -1 || echo 'build/crash error')"
exit 1
# A per-suite compile error ([ERRORED]) leaves the previous suite's binary behind, same as a
# failed warm-up build.
grep -qE '\[ERRORED\]|error:|undefined reference' "$LOG" && drop_stale_program
result 1 "RED $(grep -oE '[0-9]+ failed' "$LOG" | tail -1 || echo 'build/crash error')"
}
# RED: pio non-zero, any failure marker, or no positive summary at all (build died early).
@@ -482,8 +771,8 @@ if ! grep -qE "$PASS_RE" "$LOG"; then
# This path never runs verdict_red, and if the build died before any suite started there is no
# per-suite sandbox either - so without preserving here, "see log" points at nothing.
preserve_run_log
echo "RESULT: RED no success summary found (build error / no tests ran?)"
exit 1
drop_stale_program
result 1 "RED no success summary found (build error / no tests ran?)"
fi
# Verdict-line suffix. The suite count itself is derived from the test_* directories on the fly
@@ -518,8 +807,7 @@ if ((ATTRIB_RC != 0)); then
echo ""
echo "$ATTRIB_OUT" | sed 's/^/ /'
preserve_run_log
echo "RESULT: RED test attribution failed - suites did not run their own tests $(verdict_suffix)"
exit 1
result 1 "RED test attribution failed - suites did not run their own tests $(verdict_suffix)"
fi
$QUIET || echo "$ATTRIB_OUT" | tail -1
@@ -583,8 +871,7 @@ if [[ $IGNORED_COUNT -gt 0 ]]; then
echo ""
echo "$IGNORE_DETAIL"
echo ""
echo "RESULT: AMBER ${IGNORED_COUNT} test case(s) ignored $(verdict_suffix)"
exit 2
result 2 "AMBER ${IGNORED_COUNT} test case(s) ignored $(verdict_suffix)"
fi
# AMBER: full run only - a canonical suite neither ran NOR was explicitly skipped (silently missing).
@@ -595,8 +882,7 @@ if [[ -z $FILTER && $ACCOUNTED_COUNT -lt $EXPECTED_COUNT ]]; then
printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || missing+=("$s")
done
echo ""
echo "RESULT: AMBER ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (missing: ${missing[*]}) - all that ran passed $(verdict_suffix)"
exit 2
result 2 "AMBER ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (missing: ${missing[*]}) - all that ran passed $(verdict_suffix)"
fi
# AMBER: a suite mutated shared state it does not declare. Per-suite isolation means this is no
@@ -609,8 +895,7 @@ if ((${#DIRTY_SUITES[@]} > 0)); then
echo ""
echo " -> declare these in test/state-manifest.tsv with a reason, or stop the write."
echo " -> ./bin/run-tests.sh --write-manifest prints the entries to paste."
echo "RESULT: AMBER ${#DIRTY_SUITES[@]} suite(s) left undeclared shared state $(verdict_suffix)"
exit 2
result 2 "AMBER ${#DIRTY_SUITES[@]} suite(s) left undeclared shared state $(verdict_suffix)"
fi
# AMBER: a suite spent its LOG_ERROR budget, or came in under a declared floor. Over budget buries a
@@ -624,8 +909,7 @@ if ((${#ERROR_BUDGET_SUITES[@]} > 0)); then
echo ""
echo " -> over: demote the log line if the condition is expected, or declare errors=<max> in"
echo " test/state-manifest.tsv with a reason. Under: check the suite still exercises the path."
echo "RESULT: AMBER ${#ERROR_BUDGET_SUITES[@]} suite(s) outside their error budget $(verdict_suffix)"
exit 2
result 2 "AMBER ${#ERROR_BUDGET_SUITES[@]} suite(s) outside their error budget $(verdict_suffix)"
fi
# AMBER: a suite was still running after PlatformIO reported it. A bare UNITY_END() ends the
@@ -640,21 +924,15 @@ if ((${#SURVIVOR_SUITES[@]} > 0)); then
echo ""
echo " -> end every setup() branch with exit(UNITY_END()), not a bare UNITY_END()."
echo " -> ./bin/lint-unity-exit.sh test/**/*.cpp finds the sites; see test/README.md."
echo "RESULT: AMBER ${#SURVIVOR_SUITES[@]} suite(s) still running after the suite finished $(verdict_suffix)"
exit 2
result 2 "AMBER ${#SURVIVOR_SUITES[@]} suite(s) still running after the suite finished $(verdict_suffix)"
fi
# FILTERED: a -f run completed cleanly. Suites outside the filter were intentionally not run;
# this is not a quality signal and is distinct from suites that went missing unexpectedly.
# this is not a quality signal and is distinct from suites that went missing unexpectedly. The
# not-run set is by construction "everything outside the filter", so it is a count, not a list.
if [[ -n $FILTER ]]; then
not_run=()
for s in "${ALL_SUITES[@]}"; do
printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || not_run+=("$s")
done
echo "RESULT: FILTERED ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (not run: ${not_run[*]}) - filtered: $FILTER $(verdict_suffix)"
exit 3
result 3 "FILTERED ${RAN_COUNT}/${EXPECTED_COUNT} suites ran ($((EXPECTED_COUNT - RAN_COUNT - ${#SKIPPED_SUITES[@]})) not run) - filtered: $FILTER $(verdict_suffix)"
fi
# GREEN: all canonical suites ran, all passed, no ignored test cases, nothing undeclared left behind.
echo "RESULT: GREEN ${RAN_COUNT}/${EXPECTED_COUNT} suites passed, all CLEAN $(verdict_suffix)"
exit 0
result 0 "GREEN ${RAN_COUNT}/${EXPECTED_COUNT} suites passed, all CLEAN $(verdict_suffix)"
src/WaypointStore.cpp
+15 -6
View File
@@ -359,15 +359,24 @@ void WaypointStore::clearAllWaypoints()
std::deque<StoredWaypoint>().swap(waypoints);
#if ENABLE_WAYPOINT_PERSISTENCE && defined(FSCom)
SafeFile f(WAYPOINT_STORE_FILENAME, false);
// Rewrite an existing store (stale or unreadable included) as empty; never create one just to
// say so. Checked before SafeFile, which takes spiLock itself.
bool onFlash;
{
concurrency::LockGuard guard(spiLock);
const uint8_t version = WAYPOINT_STORE_VERSION;
const uint8_t count = 0;
f.write(&version, 1);
f.write(&count, 1);
onFlash = FSCom.exists(WAYPOINT_STORE_FILENAME);
}
if (onFlash) {
SafeFile f(WAYPOINT_STORE_FILENAME, false);
{
concurrency::LockGuard guard(spiLock);
const uint8_t version = WAYPOINT_STORE_VERSION;
const uint8_t count = 0;
f.write(&version, 1);
f.write(&count, 1);
}
f.close();
}
f.close();
#endif
#if ENABLE_WAYPOINT_PERSISTENCE
test/README.md
+8 -3
View File
@@ -9,12 +9,17 @@ This directory contains C++ unit tests that run on the host machine via Platform
```bash
./bin/run-tests.sh # all suites
./bin/run-tests.sh -f test_traffic_management # single suite
./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt
./bin/run-tests.sh -f test_traffic_management --quiet # prints only the RESULT: line - the mode for tool calls
./bin/run-tests.sh --status # a run in progress? else the last verdict, marked STALE if the tree changed since
./bin/run-tests.sh --wait # attach to the run in progress; exits with its verdict
./bin/run-tests.sh --abort # stop it (the whole build tree); its log is kept
```
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED.
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in progress; nothing started), 5 = ABORTED, 6 = UNSUPPORTED host.
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On macOS or Windows, run the suite in a container: `./bin/test-native-docker.sh`.
**The `RESULT:` line is the only verdict.** pio prints `[PASSED]` per suite and `N succeeded` per invocation long before the wrapper has decided anything, so a captured output file looks green within the first minute; grade on the final `RESULT:` line and nothing else. One run at a time per checkout: a second invocation is refused with `BUSY` rather than started, because two `pio` jobs share `.pio/build/` and wipe each other's objects. Never look for a run with `pgrep` - ask `--status`. The last verdict and its log live in `.pio/runtests/`; an interrupted run records `ABORTED`, not nothing.
**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On Windows, `bin\run-tests.cmd <same args>` forwards into WSL with the exit code passed through; on macOS, or without WSL, run the suite in a container: `./bin/test-native-docker.sh`.
**`-f` is not a gate.** A filtered run can pass while a full run fails, because filtering removes the suites that _create_ the state a later suite trips over. Iterate with `-f`; gate on a full run.