diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ad2043eb44..5f5fe04484 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,12 +2,12 @@ > **TL;DR** > -> | | | -> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | -> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | -> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | -> | Format | `trunk fmt` | -> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) | +> | | | +> | -------------- | ------------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) | > > **Need this? It's here.** > @@ -772,12 +772,15 @@ Unit tests in `test/` directory. The canonical suite count is detected on the fl Exit codes and verdicts (exact counts will vary; examples below are illustrative): -| Exit | Verdict | Meaning | -| ---- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases | -| 1 | `RED` | At least one failure, build error, or sanitizer fault | -| 2 | `AMBER` | All that ran passed, but something was lost or unexplained: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or a suite left behind shared state it does not declare | -| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run | +| Exit | Verdict | Meaning | +| ---- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases | +| 1 | `RED` | At least one failure, build error, or sanitizer fault | +| 2 | `AMBER` | All that ran passed, but something was lost or unexplained: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or a suite left behind shared state it does not declare | +| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run | +| 4 | `BUSY` | A run is already in progress (this or another session); nothing was started. `--status` to see it, `--wait` to attach, `--abort` to stop it | +| 5 | `ABORTED` | The run was stopped by a signal or `--abort`; its log is kept. `--status` shows it | +| 6 | `UNSUPPORTED` | Not a Linux host; nothing ran. Use WSL (`bin\run-tests.cmd` forwards) or `./bin/test-native-docker.sh` | Examples - exact counts will vary by suite count and env: @@ -792,12 +795,14 @@ RESULT: RED 1 failed RESULT: RED exit-time abort (tests passed; likely sanitizer - see hint above) # AMBER: a suite silently went missing on a full run -RESULT: AMBER 23/24 suites ran (missing: test_radio) - all that ran passed +RESULT: AMBER N-1/N suites ran (missing: test_radio) - all that ran passed # FILTERED: single suite run completed cleanly -RESULT: FILTERED 1/24 suites ran (not run: test_admin_radio test_atak …) - filtered: test_serial +RESULT: FILTERED 1/N suites ran (N-1 not run) - filtered: test_serial ``` +The script is written to be driven by a caller that cannot see the terminal: the final `RESULT:` line is the only verdict (pio's own `[PASSED]` and `N succeeded` lines precede it and mean nothing on their own); a second invocation while a run is in progress is refused with `BUSY` rather than started; the last verdict is kept in `.pio/runtests/last-result.tsv` with its log, and `./bin/run-tests.sh --status` prints it, marking it **STALE** when the tree has changed since. Never `pgrep` for a run - ask `--status`. + > **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run. Raw `pio test` (no sanitizers, no verdict logic) - use only when you need to override the env: diff --git a/AGENTS.md b/AGENTS.md index ec315c9dae..a8fed6b5ae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,12 +2,12 @@ > **TL;DR** > -> | | | -> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | -> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | -> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | -> | Format | `trunk fmt` | -> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) | +> | | | +> | -------------- | ------------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) | > > **Need this? It's here.** > diff --git a/CLAUDE.md b/CLAUDE.md index 8869be9803..b83685065b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,12 +2,12 @@ > **TL;DR** > -> | | | -> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | -> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | -> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | -> | Format | `trunk fmt` | -> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` | +> | | | +> | -------------- | ------------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED · 4 BUSY · 5 ABORTED · 6 UNSUPPORTED); `--status` first | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` | > > **Need this? It's here.** > diff --git a/bin/run-tests.cmd b/bin/run-tests.cmd new file mode 100644 index 0000000000..ef8e0ae512 --- /dev/null +++ b/bin/run-tests.cmd @@ -0,0 +1,12 @@ +@echo off +rem Forwarder, not a port. bin\run-tests.sh is Linux-only by design (its HOST note says why); this +rem runs it inside WSL from this same checkout, passing the arguments and the exit code straight +rem through, so the same command line and the same RESULT: line work from cmd.exe and PowerShell. +rem +rem bin\run-tests.cmd -e native -f test_utf8 --quiet +rem bin\run-tests.cmd --status +rem +rem Needs WSL with a distro that has the PlatformIO venv. A checkout on the Windows drive (/mnt/c) +rem builds several times slower than one on the WSL filesystem (\\wsl$\...); both paths work here. +wsl.exe --cd "%~dp0.." -e ./bin/run-tests.sh %* +exit /b %ERRORLEVEL% diff --git a/bin/run-tests.sh b/bin/run-tests.sh index f454d5c8fd..b39af5aaee 100755 --- a/bin/run-tests.sh +++ b/bin/run-tests.sh @@ -17,8 +17,23 @@ # ./bin/run-tests.sh --keep-state # keep every suite's sandbox, not just the interesting ones # ./bin/run-tests.sh --shuffle # randomise suite order (seed from HEAD; printed) # ./bin/run-tests.sh --seed 12345 # replay an exact order (implies --shuffle) +# ./bin/run-tests.sh --status # is a run in progress? if not, what did the last one say? +# ./bin/run-tests.sh --wait # attach to the run in progress; exit with its verdict +# ./bin/run-tests.sh --abort # stop the run in progress (whole build tree), keep its log +# bin\run-tests.cmd # from a Windows shell: forwards into WSL, exit code passed through # -# Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED. +# Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in +# progress - never start a second one), 5 = ABORTED (signal or --abort), 6 = UNSUPPORTED host +# (nothing ran; Linux only - WSL and ./bin/test-native-docker.sh are the routes from elsewhere). +# +# FOR A CALLER THAT CANNOT SEE THE TERMINAL (a tool call, a backgrounded job, a fresh session): +# - The verdict is the final "RESULT:" line and nothing else. pio prints "[PASSED]" per suite and +# "N succeeded" per invocation long before the verdict exists; a captured output file looks +# green within the first minute. Use --quiet, which prints only the RESULT line. +# - One run at a time. A second invocation while one is in progress prints RESULT: BUSY and exits +# 4 without touching the build directory. Ask with --status; block with --wait; never pgrep. +# - The last verdict is on disk: .pio/runtests/last-result.tsv (--status prints it when idle), +# with the run's log kept beside it. An interrupted run records RESULT: ABORTED, not nothing. # # HOST. This is a Linux tool: bash 4+ (mapfile), GNU coreutils and GNU find (`-printf`, md5sum, # `-executable`). That is a deliberate choice, not an oversight - the alternative is a second, @@ -59,7 +74,7 @@ # RESULT: GREEN N/N suites passed # RESULT: AMBER N/M suites ran (missing: test_radio test_serial) - all that ran passed # RESULT: AMBER 3 test case(s) ignored -# RESULT: FILTERED 1/N suites ran (not run: …) - filtered: test_utf8 +# RESULT: FILTERED 1/N suites ran (N-1 not run) - filtered: test_utf8 # RESULT: RED test attribution failed - suites did not run their own tests # RESULT: RED test_traffic_management: 1 failed (or: build/crash error) # RESULT: RED sanitizer fault - SUMMARY: AddressSanitizer: 1272 byte(s) leaked (tests may have @@ -73,8 +88,10 @@ set -uo pipefail # mis-hash the sandbox, mis-read a suite list and report a verdict that looks real. if [[ $(uname -s) != Linux ]]; then echo "run-tests.sh is Linux-only (bash 4+, GNU coreutils, GNU find); this host is $(uname -s)." >&2 - echo "Run the suite in a container instead: ./bin/test-native-docker.sh" >&2 - exit 2 + echo "Run it under WSL, or in a container: ./bin/test-native-docker.sh" >&2 + # Its own code, not AMBER: nothing ran, and a caller must not read this as "passed with caveats". + echo "RESULT: UNSUPPORTED host $(uname -s) - nothing ran; use WSL or ./bin/test-native-docker.sh" + exit 6 fi SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -93,9 +110,15 @@ PASSTHRU=() # must still forward everything else the user gave (-v, -vvv, ...) - otherwise a shuffled run # builds with those flags and then runs without them. EXTRA_ARGS=() +ORIG_ARGS=("$@") +SUBCMD="" while [[ $# -gt 0 ]]; do case "$1" in + --status | --wait | --abort) + SUBCMD="${1#--}" + shift + ;; -f) FILTER="$2" PASSTHRU+=("-f" "$2") @@ -134,11 +157,259 @@ while [[ $# -gt 0 ]]; do esac done +# --- Run record ---------------------------------------------------------------- +# One run at a time, and the last verdict on disk. Two things a caller that cannot see the terminal +# needs: a way to ask "is something running?" that is not `pgrep -f` (which matches the asker), and +# a way to read the last verdict that is not scrollback. Both live in .pio/runtests/. +# +# current.tsv exists for exactly the life of a run: written before anything touches .pio/, removed +# by result(). A lock alone would not do - a SIGKILLed wrapper releases flock while its scons tree +# lives on - so a record is valid while EITHER its holder pid OR its recorded process group is +# alive; the second case is ORPHANED, and only --abort (or the tree finishing) clears it. +RUN_DIR="$ROOT_DIR/.pio/runtests" +RUN_RECORD="$RUN_DIR/current.tsv" +LAST_RESULT="$RUN_DIR/last-result.tsv" +KEPT_LOG="$RUN_DIR/last.log" +mkdir -p "$RUN_DIR" +LOG="" +BUILD_LOG="" +# The verdict prints to the stdout this script STARTED with. A signal can arrive while a pio call +# has stdout redirected into a log file, and the trap runs inside that redirect. +exec 3>&1 +RUN_START=$(date +%s) +RUN_ID="$$-$RUN_START" # names this run in current.tsv and last-result.tsv, so --wait cannot follow a later one +HEAD_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo unknown) + +# What the verdict was FOR. HEAD alone is not enough - a verdict from before the current edits is +# the most convincing stale green there is - so the fingerprint covers the working tree too: +# tracked changes by content, untracked files by content. Recorded with the verdict; --status +# recomputes and says whether the last verdict still describes this tree. +tree_fingerprint() { + { + git rev-parse HEAD 2>/dev/null + git diff HEAD 2>/dev/null + git ls-files --others --exclude-standard -z 2>/dev/null | xargs -0 -r md5sum 2>/dev/null + } | md5sum | cut -c1-12 +} +TREE_FP=$(tree_fingerprint) + +# A build that fails leaves the previous binary in place, and run bare it reprints the last good +# run - a green that predates the failure. Remove it; the next build regenerates it. +drop_stale_program() { + local prog=".pio/build/${ENV}/meshtasticd" + if [[ -f $prog ]]; then + rm -f "$prog" + echo " removed $prog: it predates this failure and run bare would reprint the last good run" + fi +} +ARGS_STR="${ORIG_ARGS[*]-}" + +tsv_get() { awk -F'\t' -v k="$2" '$1 == k { print $2; exit }' "$1" 2>/dev/null; } +pid_alive() { [[ -n ${1-} ]] && kill -0 "$1" 2>/dev/null; } +pgid_alive() { [[ -n ${1-} ]] && ps -eo pgid= 2>/dev/null | tr -d ' ' | grep -qx "$1"; } +elapsed_since() { + local s=$(($(date +%s) - ${1:-0})) + printf '%dm%02ds' $((s / 60)) $((s % 60)) +} + +run_state() { + [[ -f $RUN_RECORD ]] || { + echo IDLE + return + } + if pid_alive "$(tsv_get "$RUN_RECORD" pid)"; then + echo RUNNING + elif pgid_alive "$(tsv_get "$RUN_RECORD" pgid)"; then + echo ORPHANED + else + # Holder and tree both gone with the record still here: the wrapper was killed before + # result() ran. Nothing to report for it beyond the kept log; clear the way. + rm -f "$RUN_RECORD" + echo IDLE + fi +} + +# The one place a verdict is printed. Records it, keeps the run log beside it, drops the run +# record, exits. Every RESULT: line in this script goes through here so --status can never +# disagree with what the caller saw. +result() { + local code=$1 text=$2 src kept="-" + echo "RESULT: $text" >&3 + # Keep whichever log has content: the test log, or during the build phase the build log. + for src in "$LOG" "$BUILD_LOG"; do + if [[ -n $src && -s $src ]] && cp "$src" "$KEPT_LOG" 2>/dev/null; then + kept="$KEPT_LOG" + break + fi + done + printf 'result\tRESULT: %s\ncode\t%s\nrun\t%s\nhead\t%s\ntree\t%s\nargs\t%s\nenv\t%s\nfinished\t%s\nlog\t%s\n' \ + "$text" "$code" "$RUN_ID" "$HEAD_SHA" "$TREE_FP" "$ARGS_STR" "$ENV" "$(date +%s)" "$kept" >"$LAST_RESULT.tmp" && + mv -f "$LAST_RESULT.tmp" "$LAST_RESULT" + rm -f "$RUN_RECORD" + exit "$code" +} + +# What the build tree is doing right now, from the processes in the recorded group: PlatformIO +# alternates single-threaded scons dependency scans, parallel compiles and one long link per +# suite, and an object counter freezes through the first and the last of those - which reads as a +# hung build to anyone who cannot run ps. +phase_of_run() { + local pgid comms + pgid=$(tsv_get "$RUN_RECORD" pgid) + [[ -z $pgid ]] && { + echo idle + return + } + comms=" $(ps -eo pgid=,comm= 2>/dev/null | awk -v p="$pgid" '$1 == p { print $2 }' | tr '\n' ' ') " + case $comms in + *" cc1plus "* | *" cc1 "* | *" as "*) echo compile ;; + *" ld "* | *" ld.bfd "* | *" ld.gold "* | *" ld.lld "* | *" mold "* | *" collect2 "*) echo link ;; + *" meshtasticd "* | *" program "*) echo test ;; + *python*) echo scons ;; + *) echo idle ;; + esac +} + +status_cmd() { + local st + st=$(run_state) + case $st in + RUNNING | ORPHANED) + local progress + progress=$(tail -n1 "$(tsv_get "$RUN_RECORD" progress)" 2>/dev/null) + echo "STATUS: $st pid=$(tsv_get "$RUN_RECORD" pid) pgid=$(tsv_get "$RUN_RECORD" pgid) since=$(elapsed_since "$(tsv_get "$RUN_RECORD" started)") now=$(phase_of_run) head=$(tsv_get "$RUN_RECORD" head) args=\"$(tsv_get "$RUN_RECORD" args)\"" + echo " progress: ${progress:-none yet} (tail -f $(tsv_get "$RUN_RECORD" progress))" + if [[ $st == ORPHANED ]]; then + echo " the wrapper died but its build tree is still running: ./bin/run-tests.sh --abort to stop it, or --wait for it to finish (no verdict will be recorded)" + else + echo " ./bin/run-tests.sh --wait blocks until the verdict; --abort stops it" + fi + ;; + IDLE) + if [[ -f $LAST_RESULT ]]; then + local fresh="current" + [[ $(tsv_get "$LAST_RESULT" tree) == "$(tree_fingerprint)" ]] || fresh="STALE - the tree has changed since; this verdict is not about the current code" + echo "STATUS: IDLE - last run ($fresh): $(tsv_get "$LAST_RESULT" result)" + echo " exit $(tsv_get "$LAST_RESULT" code), head $(tsv_get "$LAST_RESULT" head), env $(tsv_get "$LAST_RESULT" env), args \"$(tsv_get "$LAST_RESULT" args)\", finished $(elapsed_since "$(tsv_get "$LAST_RESULT" finished)") ago" + echo " log: $(tsv_get "$LAST_RESULT" log)" + else + echo "STATUS: IDLE - no run recorded" + fi + ;; + esac +} + +wait_cmd() { + local st observed + st=$(run_state) + [[ $st == IDLE ]] && { + status_cmd + [[ -f $LAST_RESULT ]] && exit "$(tsv_get "$LAST_RESULT" code)" + exit 0 + } + # Bound to THIS run: if it ends and another starts between polls, its verdict - or the + # absence of one - is what gets reported, never the newcomer's. + observed=$(tsv_get "$RUN_RECORD" run) + echo "waiting for $st run $observed pgid=$(tsv_get "$RUN_RECORD" pgid)..." >&2 + while [[ -f $RUN_RECORD && $(tsv_get "$RUN_RECORD" run) == "$observed" && $(run_state) != IDLE ]]; do sleep 5; done + if [[ -f $LAST_RESULT && $(tsv_get "$LAST_RESULT" run) == "$observed" ]]; then + tsv_get "$LAST_RESULT" result + exit "$(tsv_get "$LAST_RESULT" code)" + fi + echo "RESULT: ABORTED run $observed ended without recording a verdict (its wrapper was gone; log, if kept: $KEPT_LOG)" + exit 5 +} + +abort_cmd() { + local st pid pgid + st=$(run_state) + [[ $st == IDLE ]] && { + echo "nothing to abort" + status_cmd + exit 0 + } + pid=$(tsv_get "$RUN_RECORD" pid) + pgid=$(tsv_get "$RUN_RECORD" pgid) + if [[ $st == RUNNING ]]; then + # The holder's own trap kills the tree and records ABORTED; give it a moment to do so. + kill -TERM "$pid" 2>/dev/null + for _ in 1 2 3 4 5 6 7 8 9 10; do + [[ $(run_state) == IDLE ]] && break + sleep 1 + done + fi + if [[ $(run_state) != IDLE ]]; then + [[ -n $pgid ]] && kill -TERM -- "-$pgid" 2>/dev/null + sleep 2 + [[ -n $pgid ]] && kill -KILL -- "-$pgid" 2>/dev/null + printf 'result\tRESULT: ABORTED by --abort (wrapper pid %s was already gone)\ncode\t5\nrun\t%s\nhead\t%s\nargs\t%s\nenv\t%s\nfinished\t%s\nlog\t%s\n' \ + "$pid" "$(tsv_get "$RUN_RECORD" run)" "$(tsv_get "$RUN_RECORD" head)" "$(tsv_get "$RUN_RECORD" args)" "$(tsv_get "$RUN_RECORD" env)" "$(date +%s)" "$KEPT_LOG" >"$LAST_RESULT.tmp" && + mv -f "$LAST_RESULT.tmp" "$LAST_RESULT" + rm -f "$RUN_RECORD" + fi + status_cmd + exit 0 +} + +case $SUBCMD in +status) + status_cmd + exit 0 + ;; +wait) wait_cmd ;; +abort) abort_cmd ;; +esac + +# A run is a run: refuse while one is in progress, whatever env it is for. Two pio jobs share +# .pio/build/ and libdeps and wipe each other's objects, and the state summary below is per run. +# +# The check and the publish are one critical section under a short-lived flock, so two +# invocations in the same instant cannot both see IDLE; the record itself, published atomically +# via rename, stays the ownership token (a SIGKILLed holder releases the lock but not the record). +PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress" +exec 9>"$RUN_DIR/lock" +flock 9 +case $(run_state) in +RUNNING | ORPHANED) + flock -u 9 + status_cmd + echo "RESULT: BUSY a run is already in progress - never start a second one (--status, --wait, --abort)" + exit 4 + ;; +esac +printf 'run\t%s\npid\t%s\npgid\t\nstarted\t%s\nargs\t%s\nenv\t%s\nhead\t%s\nprogress\t%s\n' \ + "$RUN_ID" "$$" "$RUN_START" "$ARGS_STR" "$ENV" "$HEAD_SHA" "$PROGRESS_FILE" >"$RUN_RECORD.tmp" && + mv -f "$RUN_RECORD.tmp" "$RUN_RECORD" +flock -u 9 + +# Every pio invocation goes through here: its own process group (setsid), pgid recorded in the run +# record so a signal to this script - or --abort from another session - takes the whole scons tree +# with it. Called inside pipelines, hence the record file rather than a shell variable. +run_pio() { + setsid "$PIO" "$@" & + local pid=$! + sed -i "s/^pgid\t.*/pgid\t$pid/" "$RUN_RECORD" + wait "$pid" +} + +abort_run() { + local pgid + pgid=$(tsv_get "$RUN_RECORD" pgid) + if [[ -n $pgid ]]; then + kill -TERM -- "-$pgid" 2>/dev/null + sleep 2 + kill -KILL -- "-$pgid" 2>/dev/null + fi + result 5 "ABORTED after $(elapsed_since "$RUN_START") ($1) - ./bin/run-tests.sh --status for the kept log" +} +trap 'abort_run SIGINT' INT +trap 'abort_run SIGTERM' TERM +trap 'abort_run SIGHUP' HUP + # Locate pio (PATH, then the standard PlatformIO venv). PIO="$(command -v pio || command -v platformio || echo "$HOME/.platformio/penv/bin/pio")" if [[ ! -x $PIO ]] && ! command -v "$PIO" >/dev/null 2>&1; then - echo "RESULT: RED pio not found (looked in PATH and ~/.platformio/penv/bin)" - exit 1 + result 1 "RED pio not found (looked in PATH and ~/.platformio/penv/bin)" fi LOG="$(mktemp -t meshtest.XXXXXX.log)" @@ -187,7 +458,6 @@ BASELINE_FILE=".pio/build/${ENV}/.runtests-objcount" # Progress trail file (gitignored build dir). ALWAYS written so a backgrounded/piped run can be # checked mid-build with `tail -f` - that's the whole point: don't fly blind on a 20-min rebuild. -PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress" # --- Progress heartbeat ------------------------------------------------------ # Emit ONE status line every few seconds: build = objects (re)compiled this run / cached total + @@ -195,24 +465,32 @@ PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress" # check on a backgrounded run); also live-updates the tty when $5=1 (interactive --quiet). Never # touches $LOG, which is parsed for the verdict, so piped/CI captures stay clean. progress_monitor() { - local marker="$1" objtotal="$2" testtotal="$3" pfile="$4" totty="$5" start now el done ran eta line + local marker="$1" objtotal="$2" testtotal="$3" pfile="$4" totty="$5" start now el done ran eta line phase start=$(date +%s) while :; do now=$(date +%s) el=$((now - start)) + phase=$(phase_of_run) if grep -q 'Testing\.\.\.' "$LOG" 2>/dev/null; then ran=$(grep -cE "${ENV}:test_[a-z0-9_]+ \[(PASSED|FAILED|ERRORED)\]" "$LOG" 2>/dev/null) - line=$(printf '[test] %s/%s suites done - %dm%02ds' "$ran" "$testtotal" $((el / 60)) $((el % 60))) + line=$(printf '[test] %s/%s suites done - %dm%02ds - now: %s' "$ran" "$testtotal" $((el / 60)) $((el % 60)) "$phase") else done=$(find ".pio/build/${ENV}" -name '*.o' -newer "$marker" 2>/dev/null | wc -l) - if ((objtotal > 0 && done > 0)); then - eta=$((objtotal > done ? (objtotal - done) * el / done : 0)) - line=$(printf '[build] %d/%d objs - %dm%02ds - ETA ~%dm%02ds' \ - "$done" "$objtotal" $((el / 60)) $((el % 60)) $((eta / 60)) $((eta % 60))) - else - # done==0 (incremental: nothing to rebuild yet) or no cached baseline - no ETA yet. - line=$(printf '[build] %d objs compiled - %dm%02ds' "$done" $((el / 60)) $((el % 60))) - fi + case $phase in + compile) + if ((objtotal > 0 && done > 0)); then + eta=$((objtotal > done ? (objtotal - done) * el / done : 0)) + line=$(printf '[compile] %d/%d objs - %dm%02ds - ETA ~%dm%02ds' \ + "$done" "$objtotal" $((el / 60)) $((el % 60)) $((eta / 60)) $((eta % 60))) + else + # done==0 (incremental: nothing to rebuild yet) or no cached baseline - no ETA yet. + line=$(printf '[compile] %d objs so far - %dm%02ds' "$done" $((el / 60)) $((el % 60))) + fi + ;; + link) line=$(printf '[link] %d objs compiled, linking - %dm%02ds' "$done" $((el / 60)) $((el % 60))) ;; + scons) line=$(printf '[scons] dependency scan (single-threaded; the counter is expected to sit) - %d objs so far - %dm%02ds' "$done" $((el / 60)) $((el % 60))) ;; + *) line=$(printf '[build] %d objs so far - %dm%02ds - now: %s' "$done" $((el / 60)) $((el % 60)) "$phase") ;; + esac fi printf '%s\n' "$line" >>"$pfile" 2>/dev/null # file trail (always) [[ $totty == 1 ]] && printf '\r\033[K%s' "$line" >/dev/tty 2>/dev/null # live line (human) @@ -239,6 +517,7 @@ if ! $QUIET; then echo "Running: $PIO test -e $ENV ${PASSTHRU[*]-} (expecting $EXPECTED_COUNT suites)" fi echo "progress: tail -f $PROGRESS_FILE" >&2 +echo "RUN pending (pid $$): the verdict is the final RESULT: line only - [PASSED] and \"N succeeded\" lines before it are pio's, not a verdict. status: ./bin/run-tests.sh --status" >&2 if [[ ! -t 1 ]] && ! $QUIET; then echo "hint: stdout is a pipe - build errors appear at the top of output and may be lost; use --quiet to get just the RESULT line" >&2 fi @@ -263,22 +542,33 @@ if $SHUFFLE; then echo "suite order: shuffled with --seed $SEED (${#RUN_ORDER[@]} suites)" fi -# Warm the shared src objects before running any suite, the way .github/workflows/test_native.yml -# does. Fused build+run makes whichever suite PlatformIO's directory walk reaches first absorb the -# whole src compile and report it as its own duration - that is how a 35s suite once reported 13 -# minutes, and it hides the build cost from every timing the summary prints. +# Warm the shared src objects before running any suite. Fused build+run makes whichever suite +# PlatformIO's directory walk reaches first absorb the whole src compile and report it as its own +# duration - that is how a 35s suite once reported 13 minutes, and it hides the build cost from +# every timing the summary prints. +# +# ONE suite, not the whole set: `--without-testing` with no filter builds AND LINKS every suite - +# 78 links, 39 minutes, measured - and prints a full "[PASSED]" table for programs that never ran. +# The shared src objects are the same whichever suite links them, so the filtered suite when there +# is one, else the cheapest to link, warms everything the run below needs. # # This is a WARM-UP ONLY: the run below must still build. PlatformIO links every test program to # the one $BUILD_DIR/$PROGNAME path, so a `--without-building` run executes whichever suite was # linked last - every suite, under its own name, all PASSED. The warm-up keeps the src compile out # of the suite timings; the per-suite step is then just one test_main.cpp plus a link. +WARM_SUITE="$FILTER" +if [[ -z $WARM_SUITE ]]; then + WARM_SUITE="test_utf8" + [[ -d test/$WARM_SUITE ]] || WARM_SUITE="${ALL_SUITES[0]}" +fi BUILD_SECS=0 build_started=$SECONDS if $QUIET; then - "$PIO" test -e "$ENV" "${PASSTHRU[@]}" --without-testing >"$BUILD_LOG" 2>&1 + run_pio test -e "$ENV" "${EXTRA_ARGS[@]}" -f "$WARM_SUITE" --without-testing >"$BUILD_LOG" 2>&1 BUILD_RC=$? else - "$PIO" test -e "$ENV" "${PASSTHRU[@]}" --without-testing 2>&1 | tee "$BUILD_LOG" + echo "warm-up: building src + linking $WARM_SUITE (a [PASSED] line below means built, not run)" + run_pio test -e "$ENV" "${EXTRA_ARGS[@]}" -f "$WARM_SUITE" --without-testing 2>&1 | tee "$BUILD_LOG" BUILD_RC=${PIPESTATUS[0]} fi BUILD_SECS=$((SECONDS - build_started)) @@ -291,8 +581,8 @@ if ((BUILD_RC != 0)); then echo "RED - build failed before any suite ran:" grep -nE 'error:|undefined reference|\[ERRORED\]' "$BUILD_LOG" | head -5 | sed 's/^/ /' [[ -n $BUILD_FAIL_LOG ]] && echo " -> full build output: $BUILD_FAIL_LOG" - echo "RESULT: RED build failed in ${BUILD_SECS}s (no suites ran)" - exit 1 + drop_stale_program + result 1 "RED build failed in ${BUILD_SECS}s (no suites ran)" fi if ! $QUIET; then echo "build: ${BUILD_SECS}s (shared by every suite; suite durations below exclude it)" @@ -306,22 +596,22 @@ if $SHUFFLE; then : >"$LOG" for suite in "${RUN_ORDER[@]}"; do if $QUIET; then - "$PIO" test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \ + run_pio test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \ --junit-output-path "$ATTRIB_DIR/$suite.xml" >>"$LOG" 2>&1 rc=$? else - "$PIO" test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \ + run_pio test -e "$ENV" -f "$suite" "${EXTRA_ARGS[@]}" \ --junit-output-path "$ATTRIB_DIR/$suite.xml" 2>&1 | tee -a "$LOG" rc=${PIPESTATUS[0]} fi ((rc != 0)) && PIO_RC=$rc done elif $QUIET; then - "$PIO" test -e "$ENV" "${PASSTHRU[@]}" \ + run_pio test -e "$ENV" "${PASSTHRU[@]}" \ --junit-output-path "$ATTRIB_DIR/all.xml" >"$LOG" 2>&1 PIO_RC=$? else - "$PIO" test -e "$ENV" "${PASSTHRU[@]}" \ + run_pio test -e "$ENV" "${PASSTHRU[@]}" \ --junit-output-path "$ATTRIB_DIR/all.xml" 2>&1 | tee "$LOG" PIO_RC=${PIPESTATUS[0]} fi @@ -443,8 +733,7 @@ verdict_red() { grep -nE "$SAN_RE" "$LOG" | head -4 | sed 's/^/ /' echo " -> sanitizer fault: if every test above is PASS, this is an exit-time abort, not a failed assertion." echo " -> read the full report by running the binary BARE (gdb hides it via ptrace): ./$bin 2>&1 | tail -40" - echo "RESULT: RED sanitizer fault - $(grep -ohE 'SUMMARY: [A-Za-z]+Sanitizer:.*' "$LOG" | tail -1 || echo 'see report above')" - exit 1 + result 1 "RED sanitizer fault - $(grep -ohE 'SUMMARY: [A-Za-z]+Sanitizer:.*' "$LOG" | tail -1 || echo 'see report above')" fi # A guard in test/TestUtil.cpp aborting on purpose - a listening socket, or force_simradio put @@ -455,8 +744,7 @@ verdict_red() { grep -E '^FATAL: ' "$LOG" | head -3 | sed 's/^/ /' echo " -> a harness guard aborted the suite deliberately. Not a crash and not a sanitizer" echo " fault; the reason is the FATAL line above, and the suite's sandbox has the full log." - echo "RESULT: RED harness guard - $(grep -m1 -oE '^FATAL: .*' "$LOG")" - exit 1 + result 1 "RED harness guard - $(grep -m1 -oE '^FATAL: .*' "$LOG")" fi # All tests passed but the process still aborted at EXIT (ERRORED/SIGHUP/SIGABRT) and the @@ -465,12 +753,13 @@ verdict_red() { if grep -qE "$PASS_RE" "$LOG" && grep -qE '\[ERRORED\]|SIGHUP|SIGABRT' "$LOG" && ! grep -qE ':FAIL\b|\[FAILED\]' "$LOG"; then echo " -> all tests passed but the process aborted at EXIT - likely an ASan/LSan fault whose report" echo " the runner swallowed (commonly shown as SIGHUP). Run the binary BARE to see it: ./$bin 2>&1 | tail -40" - echo "RESULT: RED exit-time abort (tests passed; likely sanitizer - see hint above)" - exit 1 + result 1 "RED exit-time abort (tests passed; likely sanitizer - see hint above)" fi - echo "RESULT: RED $(grep -oE '[0-9]+ failed' "$LOG" | tail -1 || echo 'build/crash error')" - exit 1 + # A per-suite compile error ([ERRORED]) leaves the previous suite's binary behind, same as a + # failed warm-up build. + grep -qE '\[ERRORED\]|error:|undefined reference' "$LOG" && drop_stale_program + result 1 "RED $(grep -oE '[0-9]+ failed' "$LOG" | tail -1 || echo 'build/crash error')" } # RED: pio non-zero, any failure marker, or no positive summary at all (build died early). @@ -482,8 +771,8 @@ if ! grep -qE "$PASS_RE" "$LOG"; then # This path never runs verdict_red, and if the build died before any suite started there is no # per-suite sandbox either - so without preserving here, "see log" points at nothing. preserve_run_log - echo "RESULT: RED no success summary found (build error / no tests ran?)" - exit 1 + drop_stale_program + result 1 "RED no success summary found (build error / no tests ran?)" fi # Verdict-line suffix. The suite count itself is derived from the test_* directories on the fly @@ -518,8 +807,7 @@ if ((ATTRIB_RC != 0)); then echo "" echo "$ATTRIB_OUT" | sed 's/^/ /' preserve_run_log - echo "RESULT: RED test attribution failed - suites did not run their own tests $(verdict_suffix)" - exit 1 + result 1 "RED test attribution failed - suites did not run their own tests $(verdict_suffix)" fi $QUIET || echo "$ATTRIB_OUT" | tail -1 @@ -583,8 +871,7 @@ if [[ $IGNORED_COUNT -gt 0 ]]; then echo "" echo "$IGNORE_DETAIL" echo "" - echo "RESULT: AMBER ${IGNORED_COUNT} test case(s) ignored $(verdict_suffix)" - exit 2 + result 2 "AMBER ${IGNORED_COUNT} test case(s) ignored $(verdict_suffix)" fi # AMBER: full run only - a canonical suite neither ran NOR was explicitly skipped (silently missing). @@ -595,8 +882,7 @@ if [[ -z $FILTER && $ACCOUNTED_COUNT -lt $EXPECTED_COUNT ]]; then printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || missing+=("$s") done echo "" - echo "RESULT: AMBER ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (missing: ${missing[*]}) - all that ran passed $(verdict_suffix)" - exit 2 + result 2 "AMBER ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (missing: ${missing[*]}) - all that ran passed $(verdict_suffix)" fi # AMBER: a suite mutated shared state it does not declare. Per-suite isolation means this is no @@ -609,8 +895,7 @@ if ((${#DIRTY_SUITES[@]} > 0)); then echo "" echo " -> declare these in test/state-manifest.tsv with a reason, or stop the write." echo " -> ./bin/run-tests.sh --write-manifest prints the entries to paste." - echo "RESULT: AMBER ${#DIRTY_SUITES[@]} suite(s) left undeclared shared state $(verdict_suffix)" - exit 2 + result 2 "AMBER ${#DIRTY_SUITES[@]} suite(s) left undeclared shared state $(verdict_suffix)" fi # AMBER: a suite spent its LOG_ERROR budget, or came in under a declared floor. Over budget buries a @@ -624,8 +909,7 @@ if ((${#ERROR_BUDGET_SUITES[@]} > 0)); then echo "" echo " -> over: demote the log line if the condition is expected, or declare errors= in" echo " test/state-manifest.tsv with a reason. Under: check the suite still exercises the path." - echo "RESULT: AMBER ${#ERROR_BUDGET_SUITES[@]} suite(s) outside their error budget $(verdict_suffix)" - exit 2 + result 2 "AMBER ${#ERROR_BUDGET_SUITES[@]} suite(s) outside their error budget $(verdict_suffix)" fi # AMBER: a suite was still running after PlatformIO reported it. A bare UNITY_END() ends the @@ -640,21 +924,15 @@ if ((${#SURVIVOR_SUITES[@]} > 0)); then echo "" echo " -> end every setup() branch with exit(UNITY_END()), not a bare UNITY_END()." echo " -> ./bin/lint-unity-exit.sh test/**/*.cpp finds the sites; see test/README.md." - echo "RESULT: AMBER ${#SURVIVOR_SUITES[@]} suite(s) still running after the suite finished $(verdict_suffix)" - exit 2 + result 2 "AMBER ${#SURVIVOR_SUITES[@]} suite(s) still running after the suite finished $(verdict_suffix)" fi # FILTERED: a -f run completed cleanly. Suites outside the filter were intentionally not run; -# this is not a quality signal and is distinct from suites that went missing unexpectedly. +# this is not a quality signal and is distinct from suites that went missing unexpectedly. The +# not-run set is by construction "everything outside the filter", so it is a count, not a list. if [[ -n $FILTER ]]; then - not_run=() - for s in "${ALL_SUITES[@]}"; do - printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || not_run+=("$s") - done - echo "RESULT: FILTERED ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (not run: ${not_run[*]}) - filtered: $FILTER $(verdict_suffix)" - exit 3 + result 3 "FILTERED ${RAN_COUNT}/${EXPECTED_COUNT} suites ran ($((EXPECTED_COUNT - RAN_COUNT - ${#SKIPPED_SUITES[@]})) not run) - filtered: $FILTER $(verdict_suffix)" fi # GREEN: all canonical suites ran, all passed, no ignored test cases, nothing undeclared left behind. -echo "RESULT: GREEN ${RAN_COUNT}/${EXPECTED_COUNT} suites passed, all CLEAN $(verdict_suffix)" -exit 0 +result 0 "GREEN ${RAN_COUNT}/${EXPECTED_COUNT} suites passed, all CLEAN $(verdict_suffix)" diff --git a/src/WaypointStore.cpp b/src/WaypointStore.cpp index b5a0208740..d92f96d225 100644 --- a/src/WaypointStore.cpp +++ b/src/WaypointStore.cpp @@ -359,15 +359,24 @@ void WaypointStore::clearAllWaypoints() std::deque().swap(waypoints); #if ENABLE_WAYPOINT_PERSISTENCE && defined(FSCom) - SafeFile f(WAYPOINT_STORE_FILENAME, false); + // Rewrite an existing store (stale or unreadable included) as empty; never create one just to + // say so. Checked before SafeFile, which takes spiLock itself. + bool onFlash; { concurrency::LockGuard guard(spiLock); - const uint8_t version = WAYPOINT_STORE_VERSION; - const uint8_t count = 0; - f.write(&version, 1); - f.write(&count, 1); + onFlash = FSCom.exists(WAYPOINT_STORE_FILENAME); + } + if (onFlash) { + SafeFile f(WAYPOINT_STORE_FILENAME, false); + { + concurrency::LockGuard guard(spiLock); + const uint8_t version = WAYPOINT_STORE_VERSION; + const uint8_t count = 0; + f.write(&version, 1); + f.write(&count, 1); + } + f.close(); } - f.close(); #endif #if ENABLE_WAYPOINT_PERSISTENCE diff --git a/test/README.md b/test/README.md index ceb8192061..1f163cceed 100644 --- a/test/README.md +++ b/test/README.md @@ -9,12 +9,17 @@ This directory contains C++ unit tests that run on the host machine via Platform ```bash ./bin/run-tests.sh # all suites ./bin/run-tests.sh -f test_traffic_management # single suite -./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt +./bin/run-tests.sh -f test_traffic_management --quiet # prints only the RESULT: line - the mode for tool calls +./bin/run-tests.sh --status # a run in progress? else the last verdict, marked STALE if the tree changed since +./bin/run-tests.sh --wait # attach to the run in progress; exits with its verdict +./bin/run-tests.sh --abort # stop it (the whole build tree); its log is kept ``` -Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED. +Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED, 4 = BUSY (a run is already in progress; nothing started), 5 = ABORTED, 6 = UNSUPPORTED host. -**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On macOS or Windows, run the suite in a container: `./bin/test-native-docker.sh`. +**The `RESULT:` line is the only verdict.** pio prints `[PASSED]` per suite and `N succeeded` per invocation long before the wrapper has decided anything, so a captured output file looks green within the first minute; grade on the final `RESULT:` line and nothing else. One run at a time per checkout: a second invocation is refused with `BUSY` rather than started, because two `pio` jobs share `.pio/build/` and wipe each other's objects. Never look for a run with `pgrep` - ask `--status`. The last verdict and its log live in `.pio/runtests/`; an interrupted run records `ABORTED`, not nothing. + +**The harness is Linux-only, by choice.** `bin/run-tests.sh` and the per-suite isolation it drives need bash 4+ and GNU coreutils/find (`find -printf`, `md5sum`), and the script refuses to start anywhere else rather than degrade quietly - a shared-state check that silently mis-hashes a sandbox still prints a verdict, and that verdict would be worthless. The `native-macos` PlatformIO env is a **build** target for `meshtasticd`, not a test host; the isolation wrapper is registered for `env:native` and `env:coverage` only. On Windows, `bin\run-tests.cmd ` forwards into WSL with the exit code passed through; on macOS, or without WSL, run the suite in a container: `./bin/test-native-docker.sh`. **`-f` is not a gate.** A filtered run can pass while a full run fails, because filtering removes the suites that _create_ the state a later suite trips over. Iterate with `-f`; gate on a full run.