Add USB camera and uhubctl support for new test suite. Also included some bug fixes (#10204)

* Add USB camera and uhubctl support for new test suite. Also added some bug fixes * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Refactor test messages for clarity and consistency in regex tests --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-25 01:12:45 -04:00 · 2026-04-19 06:51:41 -05:00
parent 6b15571e14
commit de23e5199d
48 changed files with 3486 additions and 63 deletions
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -474,7 +474,7 @@ The repo registers the server via `.mcp.json` at the repo root — Claude Code p

 **One MCP call per port at a time.** `SerialInterface` holds an exclusive OS-level lock on the serial port for its lifetime. If a `serial_*` session is open on `/dev/cu.usbmodem101`, calling `device_info` on the same port will fail fast pointing at the active session. Sequence calls: open → read/mutate → close, then next device. Never parallelize tool calls on the same port.

-### MCP tool surface (~32 tools)
+### MCP tool surface (43 tools)

 Grouped by purpose. Full argument shapes in `mcp-server/README.md`; a few high-value signatures are called out here.

@@ -482,11 +482,13 @@ Grouped by purpose. Full argument shapes in `mcp-server/README.md`; a few high-v
 - **Build & flash**: `build`, `clean`, `pio_flash`, `erase_and_flash` (ESP32 only), `update_flash` (ESP32 OTA), `touch_1200bps`
 - **Serial sessions** (long-running, 10k-line ring buffer): `serial_open`, `serial_read`, `serial_list`, `serial_close`
 - **Device reads**: `device_info`, `list_nodes`
- **Device writes** (all require `confirm=True`): `set_owner`, `get_config`, `set_config`, `get_channel_url`, `set_channel_url`, `send_text`, `reboot`, `shutdown`, `factory_reset`, `set_debug_log_api`
+- **Device writes**: `set_owner`, `get_config`, `set_config`, `get_channel_url`, `set_channel_url`, `send_text`, `send_input_event` (inject a button/key press via the firmware's InputBroker), `set_debug_log_api`; destructive/power-state writes require `confirm=True`: `reboot`, `shutdown`, `factory_reset`
 - **userPrefs admin** (build-time constants, not runtime config): `userprefs_get`, `userprefs_set`, `userprefs_reset`, `userprefs_manifest`, `userprefs_testing_profile`
 - **Vendor escape hatches**: `esptool_chip_info`, `esptool_erase_flash`, `esptool_raw`, `nrfutil_dfu`, `nrfutil_raw`, `picotool_info`, `picotool_load`, `picotool_raw`
+- **USB power control** (via `uhubctl`, per-port PPPS toggle): `uhubctl_list` (read-only), `uhubctl_power(action='on'|'off', confirm=True)`, `uhubctl_cycle(delay_s, confirm=True)`. Target by raw `(location, port)` or by `role` (`"nrf52"`, `"esp32s3"`); role lookup checks `MESHTASTIC_UHUBCTL_LOCATION_<ROLE>` + `_PORT_<ROLE>` env vars first, falls back to VID auto-detection.
+- **Observability** (UI tier + operator ad-hoc): `capture_screen(role, ocr=True)` — grabs a USB-webcam frame of the device OLED and optionally OCRs it. Requires `mcp-server[ui]` extras (`opencv-python-headless`, `easyocr`) and `MESHTASTIC_UI_CAMERA_DEVICE_<ROLE>` env var; falls through to a 1×1 black PNG `NullBackend` when unconfigured.

-`confirm=True` is a tool-level gate on top of whatever permission prompt your MCP host shows. **Don't bypass it** by asking the host to auto-approve — it exists specifically because MCP hosts sometimes remember "always allow this tool" and that's dangerous for `factory_reset` and `erase_and_flash`.
+`confirm=True` is a tool-level gate on top of whatever permission prompt your MCP host shows. **Don't bypass it** by asking the host to auto-approve — it exists specifically because MCP hosts sometimes remember "always allow this tool" and that's dangerous for `factory_reset`, `erase_and_flash`, `uhubctl_power(action='off')`, and `uhubctl_cycle`.

 ### Hardware test suite (`mcp-server/run-tests.sh`)

@@ -494,14 +496,16 @@ The wrapper auto-detects connected devices (VID → role map: `0x239A` → `nrf5

 Suite tiers (collected + run in this order via `pytest_collection_modifyitems`):

-1. `tests/unit/` — pure Python (boards parse, pio wrapper, userPrefs parse, testing profile). No hardware.
+1. `tests/unit/` — pure Python (boards parse, pio wrapper, userPrefs parse, testing profile, uhubctl parser). No hardware.
 2. `tests/test_00_bake.py` — flashes each detected device with current `userPrefs.jsonc` merged with the session's test profile. Has its own skip-if-already-baked check comparing region + primary channel to the session profile; skips cheaply on warm devices.
-3. `tests/mesh/` — multi-device mesh: bidirectional send, broadcast delivery, direct-with-ACK, mesh formation within 60s. Parametrized `[nrf52->esp32s3]` and `[esp32s3->nrf52]`.
+3. `tests/mesh/` — multi-device mesh: bidirectional send, broadcast delivery, direct-with-ACK, mesh formation within 60s. Parametrized `[nrf52->esp32s3]` and `[esp32s3->nrf52]`. Includes `test_peer_offline_recovery` which uses uhubctl to physically power off one peer mid-conversation (requires uhubctl; skips without).
 4. `tests/telemetry/` — `DEVICE_METRICS_APP` broadcast timing.
 5. `tests/monitor/` — boot-log panic check.
-6. `tests/fleet/` — PSK seed session isolation.
-7. `tests/admin/` — channel URL roundtrip, owner persistence across reboot.
-8. `tests/provisioning/` — region + modem + slot bake, admin key presence, `UNSET` region blocks TX, userPrefs survive factory reset.
+6. `tests/recovery/` — `uhubctl` power-cycle round-trip + NVS persistence across hard reset. Requires `uhubctl` installed and a PPPS-capable hub; entire tier auto-skips otherwise.
+7. `tests/ui/` — input-broker-driven screen navigation with camera + OCR evidence.
+8. `tests/fleet/` — PSK seed session isolation.
+9. `tests/admin/` — channel URL roundtrip, owner persistence across reboot.
+10. `tests/provisioning/` — region + modem + slot bake, admin key presence, `UNSET` region blocks TX, userPrefs survive factory reset.

 Invocation patterns:

@@ -586,15 +590,19 @@ If you're modifying `StreamAPI`, `PhoneAPI`, `NodeInfoModule`, or `userPrefs` fl

 ### Recovery playbooks

-| Symptom                                                    | First check                                                   | Fix                                                                                                                                                                        |
-| ---------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `userPrefs.jsonc` dirty after test run                     | `git status --porcelain userPrefs.jsonc`                      | If non-empty, re-run `./mcp-server/run-tests.sh` once — the pre-flight self-heal restores from sidecar. If still dirty, `git checkout userPrefs.jsonc`.                    |
-| Port busy / wedged CP2102 on macOS                         | `lsof /dev/cu.usbserial-0001`                                 | Kill the holder. USB replug if the kernel still reports busy. Often a stale `pio device monitor` or zombie `meshtastic_mcp` process.                                       |
-| nRF52 appears unresponsive                                 | `list_devices` shows VID `0x239A` but `device_info` times out | `touch_1200bps(port=...)` drops it into the DFU bootloader → `pio_flash` re-installs.                                                                                      |
-| Multiple MCP server processes                              | `ps aux \| grep meshtastic_mcp` shows >1                      | Kill all but the one your MCP host spawned. Zombies hold ports and break tests.                                                                                            |
-| Mesh formation fails, one side sees peer but other doesn't | `/diagnose` (or `list_nodes` on both sides)                   | Asymmetric NodeInfo. `test_direct_with_ack` has a heal path; `/repro` it a few times. If persistent, both devices' clocks may be out of sync with their NodeInfo cooldown. |
-| "role not present on hub" in skip reasons                  | `list_devices`                                                | Expected if a device is unplugged. Reconnect before re-running the tier.                                                                                                   |
-| Tests fail only on first attempt then pass on rerun        | —                                                             | State leak from a prior session. Run with `--force-bake` to reset to a known state.                                                                                        |
+| Symptom                                                                           | First check                                                   | Fix                                                                                                                                                                                                                              |
+| --------------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `userPrefs.jsonc` dirty after test run                                            | `git status --porcelain userPrefs.jsonc`                      | If non-empty, re-run `./mcp-server/run-tests.sh` once — the pre-flight self-heal restores from sidecar. If still dirty, `git checkout userPrefs.jsonc`.                                                                          |
+| Port busy / wedged CP2102 on macOS                                                | `lsof /dev/cu.usbserial-0001`                                 | Kill the holder. USB replug if the kernel still reports busy. Often a stale `pio device monitor` or zombie `meshtastic_mcp` process.                                                                                             |
+| nRF52 appears unresponsive                                                        | `list_devices` shows VID `0x239A` but `device_info` times out | `touch_1200bps(port=...)` drops it into the DFU bootloader → `pio_flash` re-installs.                                                                                                                                            |
+| Device fully wedged (Guru Meditation, frozen CDC, no DFU)                         | `list_devices` shows the VID but every admin call times out   | `uhubctl_cycle(role="nrf52", confirm=True)` hard-power-cycles the port via USB hub PPPS. `baked_single`'s auto-recovery hook does this once automatically if uhubctl is installed. Falls back to physical replug if no PPPS hub. |
+| Multiple MCP server processes                                                     | `ps aux \| grep meshtastic_mcp` shows >1                      | Kill all but the one your MCP host spawned. Zombies hold ports and break tests.                                                                                                                                                  |
+| Mesh formation fails, one side sees peer but other doesn't                        | `/diagnose` (or `list_nodes` on both sides)                   | Asymmetric NodeInfo. `test_direct_with_ack` has a heal path; `/repro` it a few times. If persistent, both devices' clocks may be out of sync with their NodeInfo cooldown.                                                       |
+| "role not present on hub" in skip reasons                                         | `list_devices`                                                | Expected if a device is unplugged. Reconnect before re-running the tier.                                                                                                                                                         |
+| Entire `tests/recovery/` tier skipped                                             | `command -v uhubctl`                                          | Expected if `uhubctl` isn't on PATH. Install via `brew install uhubctl` (macOS) or `apt install uhubctl` (Debian/Ubuntu). Also skips if no hub advertises PPPS.                                                                  |
+| Entire `tests/ui/` tier skipped ("firmware not baked with USERPREFS_UI_TEST_LOG") | reportlog.jsonl for the skip reason                           | Re-run with `--force-bake` so the UI-log macro gets compiled into the fresh firmware. First run after the Round-3 landing always re-bakes.                                                                                       |
+| `tests/ui/` runs but captures are all 1×1 black PNGs                              | `MESHTASTIC_UI_CAMERA_DEVICE_ESP32S3`                         | Env var not set → `NullBackend`. Point a USB webcam at the heltec-v3 OLED and set the device index; `.venv/bin/python -c "import cv2; [print(i, cv2.VideoCapture(i).read()[0]) for i in range(5)]"` discovers it.                |
+| Tests fail only on first attempt then pass on rerun                               | —                                                             | State leak from a prior session. Run with `--force-bake` to reset to a known state.                                                                                                                                              |

 ### Never do these without asking

--- a/.github/prompts/mcp-diagnose.prompt.md
+++ b/.github/prompts/mcp-diagnose.prompt.md
@@ -26,7 +26,12 @@ This prompt assumes the meshtastic MCP server is registered with your VS Code Co
   - `get_config(section="lora", port=<p>)` → region, preset, channel_num, tx_power, hop_limit
   - If anything looks off (can't connect, `num_nodes` wrong, missing `firmware_version`), open a short firmware-log window: `serial_open(port=<p>, env=<inferred>)`, wait 3 seconds, `serial_read(session_id, max_lines=100)`, `serial_close(session_id)`. Infer env from VID (0x239a → `rak4631`, 0x303a/0x10c4 → `heltec-v3`) unless an `MESHTASTIC_MCP_ENV_<ROLE>` env var overrides it.

-4. **Render per-device report** as a compact block:
+4. **Hub health** (call once, not per-device): `uhubctl_list()` — enumerates every USB hub the host sees. Cross-reference each Meshtastic device's VID to find which hub + port it's on. Flag in the report if:
+   - No hub advertises `ppps=true` → `tests/recovery/` can't run; hard-recovery via `uhubctl_cycle` isn't available.
+   - A Meshtastic device is on a non-PPPS hub → note it; moving to a PPPS hub unlocks auto-recovery.
+   - `uhubctl_list` raises `ConfigError: uhubctl not found` → report as "uhubctl not installed"; don't treat as a device fault.
+
+5. **Render per-device report** as a compact block:

   ```text
   [nrf52 @ /dev/cu.usbmodem1101]      fw=2.7.23.bce2825, hw=RAK4631
@@ -35,20 +40,22 @@ This prompt assumes the meshtastic MCP server is registered with your VS Code Co
     tx_power    : 30 dBm, hop_limit=3
     peers       : 1 (esp32s3 0x433c2428, pubkey ✓, SNR 6.0 / RSSI -24 dBm)
     primary ch  : McpTest
+     hub         : 1-1.3 port 2 (PPPS, uhubctl-controllable)
     firmware    : no panics in last 3s
   ```

-   Flag abnormalities inline with `⚠︎ <short reason>` — missing pubkey on a known peer, region UNSET, mismatched channel name, etc.
+   Flag abnormalities inline with `⚠︎ <short reason>` — missing pubkey on a known peer, region UNSET, mismatched channel name, device on non-PPPS hub, etc.

-5. **Cross-device correlation** (when >1 device selected):
+6. **Cross-device correlation** (when >1 device selected):
   - Do both see each other in `nodesByNum`?
   - Do `region`, `channel_num`, `modem_preset` match across devices?
   - Do the primary channel names match? (Different name → different PSK → no decode.)

-6. **Suggest next steps only for recognizable failure modes**, never speculatively:
+7. **Suggest next steps only for recognizable failure modes**, never speculatively:
   - Stale PKI one-way → "`/mcp-test tests/mesh/test_direct_with_ack.py` — the test's retry+nodeinfo-ping heals this."
   - Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`."
-   - Device unreachable → refer operator to the touch_1200bps + CP2102-wedged-driver notes in `run-tests.sh`.
+   - Device unreachable, DFU reachable → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds and the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`.
+   - CP2102-wedged-driver on macOS → see `run-tests.sh` notes.

 ## Hard constraints

--- a/.github/prompts/mcp-repro.prompt.md
+++ b/.github/prompts/mcp-repro.prompt.md
@@ -46,7 +46,8 @@ Equivalent of `.claude/commands/repro.md`. Use when the operator says "that one
   - **LoRa airtime collision** — pass rate improves with fewer concurrent transmitters. Suggest a `time.sleep` gap or retry bump in the test body.
   - **PKI key staleness** — first attempt fails, subsequent ones pass; existing retry-loop pattern in `test_direct_with_ack.py` is the fix.
   - **NodeInfo cooldown** — `Skip send NodeInfo since we sent it <600s ago` in fail-only logs; needs a `broadcast_nodeinfo_ping()` warmup.
-   - **Hardware-specific** — one direction consistently fails, firmware versions differ, CP2102 driver wedged, etc.
+   - **Hardware-specific** — one direction consistently fails, firmware versions differ, CP2102 driver wedged, etc. For a device wedged past `touch_1200bps`, recommend `uhubctl_cycle(role=..., confirm=True)` to hard-power-cycle its hub port (requires `uhubctl` installed).
+   - **Device went dark mid-run** — fails from some iteration onward and never recovers; firmware log stops arriving. Almost always a Guru crash with frozen CDC. Recommend `uhubctl_cycle` before the next iteration; escalate to replug if that also fails.
   - **Unknown** — say so. Don't invent a root cause.

 7. **Report back** with:
--- a/.github/prompts/mcp-test.prompt.md
+++ b/.github/prompts/mcp-test.prompt.md
@@ -21,19 +21,25 @@ Equivalent of the Claude Code `/test` slash command in `.claude/commands/test.md

 2. **Read the pre-flight header** (first few lines of wrapper output). The `detected hub :` line lists role → port → env mappings. If it reads `(none)`, the wrapper narrowed to `tests/unit` only — call that out explicitly so the operator knows hardware tiers were skipped.

-3. **On pass**: one-line summary like `N passed, M skipped in <duration>`. Don't enumerate test names. DO mention any non-placeholder SKIPs (things like "role not present on hub") because they indicate missing hardware or setup issues.
+3. **On pass**: one-line summary like `N passed, M skipped in <duration>`. Don't enumerate test names. DO mention any non-placeholder SKIPs and name the cause:
+   - `"role not present on hub"` → device unplugged; operator should reconnect.
+   - `"firmware not baked with USERPREFS_UI_TEST_LOG"` → tests/ui skipped; the UI-log compile macro isn't in the baked firmware. Suggest `--force-bake`.
+   - `"uhubctl not installed"` → tests/recovery + `test_peer_offline_recovery` skipped. Suggest `brew install uhubctl` / `apt install uhubctl`.
+   - `"no PPPS-capable hubs detected"` → tests/recovery skipped because the attached hub doesn't support per-port power switching; won't run on that setup.
+   - `"opencv-python-headless is not installed"` → tests/ui auto-deselected by `run-tests.sh`. Suggest `pip install -e 'mcp-server/.[ui]'`.

 4. **On failure**: open `mcp-server/tests/report.html` (pytest-html output, self-contained) and extract the `Meshtastic debug` section for each failed test. That section includes a firmware log stream (last 200 lines) and device state dump. For each failure, summarise:
   - test name
   - one-line assertion message
   - the specific firmware log lines that explain why (look for `PKI_UNKNOWN_PUBKEY`, `Skip send NodeInfo`, `Error=`, `Guru Meditation`, `assertion failed`, `No suitable channel`)
+   - for UI-tier failures also check `mcp-server/tests/ui_captures/<session>/<test>/transcript.md` (per-step frame + OCR)

 5. **Classify each failure** as one of:
   - **Transient flake** — LoRa collision, first-attempt NAK with self-heal pattern, timing-sensitive assertion. Suggest `/mcp-repro <test-id>` to confirm.
-   - **Environmental** — device unreachable, port busy, CP2102 driver wedged on macOS. Suggest specific recovery (USB replug, `touch_1200bps`, `git status userPrefs.jsonc`).
+   - **Environmental** — device unreachable, port busy, CP2102 driver wedged on macOS. Suggest recovery in escalation order: (a) replug USB, (b) `touch_1200bps` + `pio_flash` for nRF52 DFU, (c) `uhubctl_cycle(role=..., confirm=True)` for a device wedged past DFU (needs `uhubctl` installed; `baked_single` does this once automatically when available). Also check `git status userPrefs.jsonc`.
   - **Regression** — same assertion fails repeatedly on re-runs, firmware log shows novel errors. Identify the firmware module likely responsible.

-6. **Do NOT run destructive recovery automatically**. If a failure looks like it needs a reflash, factory*reset, or replug — \_describe the steps* and let the operator decide. Never burn airtime or flash cycles without approval.
+6. **Do NOT run destructive recovery automatically**. If a failure looks like it needs a reflash, factory*reset, `uhubctl_cycle`, or replug — \_describe the steps* and let the operator decide. Never burn airtime or flash cycles without approval.

 ## Arguments convention