mirror of
https://github.com/meshtastic/firmware.git
synced 2026-09-19 03:38:21 -04:00
ee48094ea8309858cd579100b8aaf14560bcc4b2
187
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Clive BlackledgeandClaude Opus 5
|
0271be9369 |
fix(SafeFile): remove a stale .tmp before opening it for write (#11428)
* fix(SafeFile): remove a stale .tmp before opening it for write SafeFile writes to <filename>.tmp, verifies it by readback, then renames it over the real file. openFile() never removed a pre-existing .tmp - an unfinished FIXME - and FILE_O_WRITE appends rather than truncates on Adafruit_LittleFS (nRF52) and STM32 LittleFS. So a .tmp left behind by a reset in the window between close() and renameFile() is appended to on the next save. The readback hash covers only the bytes just written, so it mismatches, close() returns false, and the tmp is left behind again - the failure latches and every subsequent save of that file fails. Today saveProto() discards close()'s result, so this is silent and permanent. Guard the remove with exists(): a bare remove() of a missing file logs on Portduino. The same guarded pattern is already used for this exact append trap in xmodem.cpp. Note the FIXME's commented-out body named the wrong path - it removed 'filename', the real file, not 'filenameTmp' - so it would have destroyed the good copy had it ever been enabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(SafeFile): cover the stale-.tmp path, and trim the fix comment Adds test_safefile, the first coverage of SafeFile's write-tmp / verify-by-readback / rename-over path that every saveProto() caller goes through. Five cases pin the contract the fix restores: whatever the backend does on open, a completed save leaves the real file holding exactly the bytes written and nothing else, on both the fullAtomic and the !fullAtomic construction, with no .tmp left behind. These tests cannot go red on the native host, and that gap cannot be closed here. FILE_O_WRITE is an append-then-seek-to-end open only on Adafruit_LittleFS (nRF52) and on the in-repo STM32 port (STM32_LittleFS_File.cpp: LFS_O_RDWR | LFS_O_CREAT followed by lfs_file_seek to LFS_SEEK_END). On Portduino FILE_O_WRITE is the string "w" (FSCommon.h:13), which reaches fopen() and truncates. Reverting the source fix and re-running leaves all five green, verified rather than assumed. test_write_open_truncates _on_this_host asserts that premise out loud, so if the host ever gains the append behaviour the suite starts discriminating instead of quietly agreeing. Why the original FIXME stayed commented out, since that is the real history here. It read "if (fullAtomic) FSCom.remove(filename)" and named the real file, not the tmp. Running it would delete the last good copy before the replacement had been written and verified, which is precisely the guarantee fullAtomic exists to provide. Disabling it was correct. The fix under test removes filenameTmp instead, which is the file that actually carries the stale bytes, and is safe to drop at any point because nothing has been promised about it yet. Scoping the remove to fullAtomic would be wrong for the same reason. Both paths open the same filenameTmp with the same FILE_O_WRITE; fullAtomic only decides whether the real file is nuked up front to free space. The !fullAtomic path is the space-constrained one, so it is if anything the more likely to be interrupted mid-write and inherit a stale tmp. Test 2 pins that. On the cost of the added exists(). Every saveProto() already ends in SafeFile::close(), which calls testReadback(): it reopens the tmp and reads the whole proto back one byte at a time through f2.read() to XOR a verification hash, then renames. So the per-save cost is already an open, a full write, a close, a full byte-wise reread, and a rename. One exists() is a single path lookup with no erase, no program and no data read, and on the common path there is no remove() at all. Next to the readback loop it is noise. Happy to put a number on it if wanted. Scoping it to fullAtomic would also not do what it looks like it does. SafeFile's constructor defaults fullAtomic to false (SafeFile.h:28), and of the saveProto call sites only saveDeviceStateToDisk passes true. Config, moduleconfig, channels, nodedatabase and backup all take the default, so scoping would leave the stale tmp live on almost every save path, including the space-constrained one most likely to be interrupted mid-write. Also trims the fix's comment to two lines per AGENTS.md, and drops the stale-tmp removal log from LOG_WARN to LOG_DEBUG: an interrupted write is recoverable and self-healing, so it does not warrant a warning on every boot after one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(SafeFile): fail the write when a stale tmp cannot be removed openFile() ignored the result of FSCom.remove(). If the removal failed on an append-on-write backend, the open that follows appended to the stale bytes, and the readback hash is an 8 bit XOR over the whole tmp, so polluted content has a real chance of verifying and being renamed over the good file. It now logs and returns an invalid File. SafeFile::write() already no-ops on !f and close() already returns false, so the caller sees the save fail rather than silently getting a corrupt one. This is the only checked FSCom.remove() in the tree; the other call sites are all best-effort cleanups where failure does not compromise anything. Also gates test_write_open_truncates_on_this_host to ARCH_PORTDUINO. It asserts that this host truncates on FILE_O_WRITE, which is false by design on the Adafruit_LittleFS and STM32 backends the fix exists for, so running the suite there would fail on a premise that is only meant to describe the test host. Both reported by CodeRabbit on #11428. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tadayoshi MIURA
|
ac330e6a6b |
fix(radio): MeshBeacon heap leak and runtime packet payload size check (#11573)
* Fix for MeshBeacon packet leakage * fix: add runtime payload size check against radiobuffer * review fix for PR#11573: clear target radio settings before MeshBeacon packet release * add unit test for radio buffer capacity check, removing related assert for the test * review fix for PR#11573: add explicit verifaction against rejected packets |
||
|
Tom
|
bc035bb812 |
feat(lora): state a pinned userPrefs preset as the unset region's intent (#11507)
* feat(lora): state a pinned userPrefs preset as the unset region's intent A vendor build can pin USERPREFS_LORACONFIG_MODEM_PRESET while leaving the region unset, so a fresh flash comes up as region UNSET plus a deliberate preset. Stock installs come up as region UNSET plus the LONG_FAST placeholder, and nothing in FromRadio told the two apart - so clients treat every unset-region node as factory-fresh and replace its preset with the region default as soon as the user picks a region. A mesh pinned to SHORT_TURBO loses every new node to LONG_FAST or LONG_TURBO, silently. getRegionPresetMap() now emits an UNSET entry when, and only when, the build pins a preset, stating that preset as both the group's sole entry and its default. Stock builds are unchanged on the wire: no UNSET entry, which clients already read as unconstrained. This is intent, not enforcement. supportsPreset() still accepts any known preset while the region is unset (#11496) and the radio is held silent either way, so the device continues to honour whatever the user or an admin sets. Costs one group slot and one region slot on pinned builds only (6->7 of 8, 34->35 of 38); exhaustion is logged and degrades to the existing unconstrained behaviour. * Trim comments to the project's one-to-two-line limit |
||
|
Clive BlackledgeandClaude Opus 5
|
389559bddb |
fix(NodeDB): re-derive my_node_num when ensurePkiKeys() mints the identity keypair (#11426)
* fix(pki): re-derive NodeNum when setting a region mints the identity key
A node's mesh address is derived from its identity key:
my_node_num == crc32Buffer(config.security.public_key.bytes, 32)
NodeDB::createNewIdentity() is what establishes that, and NodeDB::
generateCryptoKeyPair() is the only thing that called it.
CryptoEngine::ensurePkiKeys() generates or re-derives the keypair and writes
security.public_key, security.private_key and user.public_key - but never
re-derives my_node_num. Boot-time keygen is suppressed while the LoRa region is
UNSET (generateCryptoKeyPair()'s regionBlocksKeygen guard), so on a fresh device
my_node_num is still the MAC-derived value from pickNewNodeNum(). The user then
sets the region - the stock onboarding flow - ensurePkiKeys() mints a key, and
the invariant is broken.
The node then signs its broadcasts (Router.cpp signs when !pki_encrypted &&
(owner.is_licensed || isBroadcast(p->to))). Every receiver runs
verifyFirstContactNodeInfo, fails crc32Buffer(user.public_key) != p->from, and
drops the NodeInfo. The node's identity beacons are invisible to the mesh.
Nothing reboots to repair it: AdminModule sets requiresReboot = false for LoRa
changes ("All LoRa radio changes apply live via configChanged observer") and
MenuHandler ends at service->reloadConfig(changes).
Four call sites reached ensurePkiKeys():
1. AdminModule set_config LORA, region first set (phone app - the common path)
2. MenuHandler applyLoraRegion (on-device region picker)
3. InkHUD MenuApplet applyLoRaRegion (schedules a reboot, so it
self-healed at next boot)
4. portduino wasm wasm_set_region
The reference implementation was already in the tree: the *licensed* branch of
call site 1, thirteen lines below the broken unlicensed one, calls
nodeDB->generateCryptoKeyPair() (which reaches createNewIdentity()) and widens
the persisted mask with SEGMENT_DEVICESTATE | SEGMENT_NODEDATABASE.
Rather than repeat that at four call sites, the key-mint is routed through one
chokepoint that owns both halves of the identity: NodeDB::ensurePkiIdentity()
calls crypto->ensurePkiKeys() and then createNewIdentity(). It lives in NodeDB
because createNewIdentity() operates on the devicestate/node-DB globals, which
CryptoEngine deliberately does not touch - ensurePkiKeys() takes the security
config and user by reference precisely so it stays free of that dependency, and
it is unit-tested against a standalone CryptoEngine.
ensurePkiIdentity() returns true only when my_node_num actually moved
(createNewIdentity() early-returns when the key is unchanged, so a repeat region
change does not disturb the self entry or force a needless flash write). Callers
use that to widen their save mask; my_node_num lives in devicestate and the self
row moves in the node DB, so both segments must be persisted or the fix would
revert at the next boot. SEGMENT_CONFIG, which carries the key itself, is
already unconditional on all four paths.
The InkHUD reboot is left as-is. It is now redundant for this invariant, but it
covers the rest of that menu's behaviour and a redundant reboot is not a bug.
Adds test_handleSetConfig_persistsUnlicensedFirstRegionIdentity, the unlicensed
twin of the existing licensed test, asserting both the segment mask and
my_node_num == crc32(public_key).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style(NodeDB): trim identity-recovery comments and guard the WASM nodeDB deref
Two review asks, no behaviour change on any built target.
Copilot flagged the unguarded nodeDB deref in the WASM region setter; it is the
only ensurePkiIdentity() call site that did not check the pointer first.
The rest is comment length. AGENTS.md:83 caps code comments at two lines, and the
identity-recovery comments across the four call sites plus the NodeDB.h doc block
ran to four and six lines. The rationale they carried is in the commit messages
and the PR body, which is where AGENTS.md says it belongs.
The PR's own fix in AdminModule.cpp is deliberately untouched.
* fix(NodeDB): keep the identity move authoritative when the self record cannot be created
createNewIdentity() removes the old node entry and assigns myNodeInfo.my_node_num
before it tries to create the row for the new number. If getOrCreateMeshNode()
came back null it returned false, so the first-region callers left
SEGMENT_DEVICESTATE and SEGMENT_NODEDATABASE out of the save mask.
The number had already moved in RAM at that point, and the freshly minted key
goes to flash under SEGMENT_CONFIG regardless. The next boot therefore reloads
the old number alongside the new key, which is exactly the
crc32(public_key) != my_node_num break this path exists to prevent, reached
through the error branch instead of the happy one.
Rolling the number back is not an option either, since the key has already been
replaced by the time this runs. So the move is now reported as the fact it is and
the missing self record is logged separately; getOrCreateMeshNode() will recreate
that row on the next contact. Reachable when the self record is absent and the
table is full of protected nodes.
Reported by CodeRabbit on #11426.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
90a6dec3f3 |
fix(NodeDB): require a full 32-byte key when demoting to the warm tier (#11431)
* fix(NodeDB): require a full 32-byte key when demoting to the warm tier meshtastic_User.public_key is a wire `bytes` field with max_size 32, so any size in 0..32 decodes off the air, and nothing validates it on ingress: NodeInfoModule hands the decoded User straight to NodeDB::updateUser, whose PKI gates are all `== 32` and so fall through for a partial key, and TypeConversions::CopyUserToNodeInfoLite then stores it with the short size. demoteOldestHotNodesToWarm() admitted that partial key into the warm tier on a `size > 0` gate. WarmNodeEntry has no length field - it distinguishes "has a key" from "no key" purely by all-zero - so N real bytes plus 32-N zeros become indistinguishable from a genuine key. copyPublicKeyAuthoritative() then hands that fabricated key back with size = 32 and reports it AUTHORITATIVE, and re-admission writes size = 32 into the hot store. From then on updateUser's key pin permanently rejects the node's real NodeInfo, and DMs to it are encrypted to a key nobody holds. Require a full 32-byte key, so a partial one is absorbed as "no key" (nullptr) rather than as a truncated one. WarmNodeStore::place() already treats a null key as keyless and clears the slot's stale key when repurposing it. This aligns the site with its two siblings, which both already gate on `size == 32` (the purge path in cleanupMeshDB and the runtime eviction in getOrCreateMeshNode). The ingress gap - updateUser accepting a 1..31-byte key at all - is a separate, larger change and is left for its own review. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(NodeDB): shorten warm-demotion comment to two lines Repo guideline (AGENTS.md): keep code comments to one or two lines. Retains the non-obvious invariant - warm entries have no key length field - and drops the restated detail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(NodeDB): cover short-key demotion into the warm tier A warm record stores 32 raw key bytes with no length field, so a partial hot-store key is indistinguishable from a real one once demoted. The public_key.size == 32 gate in demoteOldestHotNodesToWarm() is what keeps a truncated key from being laundered into a full-looking warm key, but nothing exercised it. test_migration_dropsShortKeyOnDemotion overflows the hot store with one node carrying a 31-byte key and asserts it lands as a keyless placeholder while a genuine 32-byte key still survives. push() grows a keySize parameter to seed the partial key, and clearWarm() gives the test an empty warm tier, which it needs because the warm store outlives setUp() and a prior run's warm.dat. Verified to discriminate: with the size gate reverted to size > 0 the new test fails on "a 31-byte key must not be demoted as if it were a full key", and passes again once restored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(NodeDB): assert the keyless placeholder carries last_heard The test only proved a warm metadata row survived the demotion, not that the placeholder does the job the nullptr is there for, which is preserving last_heard when the key is dropped. Asserting the value needed the seeds fixing first. Warm entries pack role, protected category and the xeddsa flag into the low 7 bits of last_heard (WARM_TIME_MASK is 0xFFFFFF80), so warm time has 128 second granularity and the old seeds of 1, 2, 3 all quantised to 0. They are now multiples of 128, which keeps the demotion ordering identical and makes the values survive the round trip. Real last_heard is epoch seconds, so this is closer to production than the old counter was. Reads the entry through WarmNodeStore::take() rather than getOrCreateMeshNode(), which does not restore last_heard from the warm tier and would have been asserting a path that does not exist. Reported by CodeRabbit on #11431. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> |
||
|
Andrew YongandThomas Göttgens
|
93d15a5368 |
Add AS3935 lightning sensor support (#10931)
* Add AS3935 lightning sensor support Implements meshtastic/firmware#10774: an AS3935Sensor (TelemetrySensor subclass) that reports lightning_strike_count_1h and lightning_distance_km on the normal environment telemetry interval, like a rain gauge - strikes are counted over a fixed rolling ~1h window and read non-destructively, so replying to a peer's telemetry request in between broadcasts can't silently drop counted strikes. The AS3935's IRQ pin (opt-in per board via AS3935_IRQ) is polled with a plain digitalRead() in runOnce(), deliberately not attachInterrupt(): the IRQ line is a level that stays asserted until its interrupt register is read, so polling can't miss an event regardless of timing, matching the SparkFun library's own reference examples. An interrupt would also buy nothing here even setting that aside - classification requires an I2C read (readInterruptReg(), which itself calls delay(2) per the datasheet's settle-time requirement), and blocking I2C/delay() calls aren't safe from ISR context on any of this codebase's target platforms, so the ISR could only ever set a flag for later draining - no less work than just polling the pin directly on the next tick. A genuine lightning classification also requests an immediate out-of-cycle send via a new EnvironmentTelemetryModule:: requestImmediateSend() hook. There's no fixed debounce on the request itself - EnvironmentTelemetryModule's existing airtime/duty-cycle gate already paces every send, so it sends as often as airtime allows rather than an arbitrary fixed rate. The request does expire after 5 minutes unfulfilled, so it can't fire an arbitrarily stale broadcast if airtime was blocked for a long stretch. The AS3935's I2C addresses (0x01-0x03) fall inside the range this codebase's I2C scanner otherwise skips as reserved, so detection is a small dedicated probe gated behind AS3935_IRQ and respecting the caller's address filter, rather than a change to the general scan loop. Presence is confirmed via a register write/readback round-trip rather than a fixed expected value, since the AS3935 has no WHOAMI register and a power-on-reset-only check can't survive a warm reboot that doesn't power-cycle the sensor (initDevice() permanently rewrites that register on first configuration). Generated files under src/mesh/generated/ are intentionally excluded from this commit - they're regenerated from the protobufs submodule by update_protobufs.yml, and hand edits get overwritten and conflict once the companion protobufs PR merges and the submodule pointer updates. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Yong <me@ndoo.sg> * fix(as3935): calibration and telemetry logging initDevice() never called the library's calibrateOsc(). The AS3935's internal oscillators are calibrated against the antenna's resonance, which the AFE/watchdog/spike-rejection thresholds depend on; without it, only a directly-driven IRQ pin (bypassing detection entirely) reacted during testing. The sensor could already have a historical detection event latching the IRQ pin high before our initialization. Added an explicit drain read after the IRQ pin is configured, so the sensor doesn't start out stuck asserting IRQ. EnvironmentTelemetryModule::sendTelemetry() logs every other environment metric category on send but was missing lightning; added a matching log line. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Yong <me@ndoo.sg> * Support AS3935 without an IRQ line, make the antenna trim configurable Detection no longer requires AS3935_IRQ. The probe is gated like the other environmental sensors, so an I2C-only breakout is found on any board. Where AS3935_IRQ is defined the pin still gates the I2C read, otherwise runOnce() polls the interrupt register, which latches until read. Antenna tuning capacitance moves to AdminMessage.sensor_config.as3935_config, persisted to /prefs/as3935.dat and defaulting to 96pF. The chip does not retain it across power loss. Disturbers are masked in the chip, since runOnce() now polls every second. The lightning telemetry log is guarded so nodes without the sensor no longer log it on every send. Requires meshtastic/protobufs#981. * Revert protobufs pointer to the develop baseline The submodule bump conflicts on merge and the generated headers come from an out of band CI job, so the pointer moves with that job rather than in this branch. * Report lightning strikes over a true rolling hour strikeCountWindow was zeroed on a fixed interval, so lightning_strike_count_1h reported strikes since the last reset rather than over the preceding hour. RollingCounter is a fixed memory sliding window: one counter per bucket, nothing stored per event, so a storm cannot grow it. The ring holds one bucket more than the window needs so none is recycled while part of it is still inside, and the oldest bucket contributes only the fraction still in range. Both are needed to hold the span at exactly the window length rather than letting it drift by a bucket either way. Expiry is exact to one bucket rather than to the event, which is below the 5 minute floor on mesh telemetry sends. The distance expires with the last strike in the window instead of on the interval reset. Covered by test/test_rolling_counter. * Widen the RollingCounter edge weighting to 64 bit counts * inWindow is a 32 bit product, so a bucket holding more than 2^32 / BucketMs events wraps. At a 5 minute width that is about 14k: a bucket of 50000 reported 11367 instead of 40000 once it reached the window edge. Below the threshold nothing changes, so lightning was unaffected, but the helper is meant to be reused by counters with far higher rates. test_large_burst_at_window_edge covers it. The existing burst test sampled only inside the window, where the bucket is whole and never weighted. * Trim RollingCounter comments to the house limit --------- Signed-off-by: Andrew Yong <me@ndoo.sg> Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com> |
||
|
Ben MeadorsandClaude Fable 5
|
a5fc95f774 |
fix(mesh): coerce coordinate traffic to the position channel on event builds (#11545)
Under USERPREFS_BLOCK_POSITION_ON_EVENT_CHANNEL every coordinate packet a client aimed at the event channel was rejected with the "Location sharing is disabled on this channel" notification - including the phone's own location feed. Both apps hand a GPS-less node its fix as a POSITION_APP packet addressed to the node itself on channel 0; that packet never leaves the device (Router::sendLocal delivers it locally) but resolved to the event channel and was dropped before PositionModule saw it. Result: the toast on every location tick, and nodes without a GPS never learned a position to share on their private channel. Position traffic now converges on the position channel - findPositionChannel(), the first channel with non-zero on-wire precision, which is never the event channel: - From-us-to-us coordinate packets are exempt from the event block. - Local coordinate sends aimed at the event channel (phone share-location, request-position, waypoints, any module/UI originator) are moved onto the position channel in Router::sendLocal and PhoneAPI instead of rejected. The client notification is only sent when no channel carries positions at all. - A position request DM'd to us on the event channel is answered on the position channel at that channel's precision (request_id preserved, same reply throttle); the requester's coordinates are still not stored, forwarded, relayed or published. want_response from the bitfield is merged before the event-channel decode short-circuit so such requests are seen. - PositionModule::sendOurPosition, positionUnchangedSinceLastSend and MeshService::trySendPosition use the shared helper instead of three copies of the same walk. Non-event builds are unaffected: the coercion compiles out and the helper matches the previous walk. Tests: coverage-event-policy (test_event_channel_phone_api, test_event_channel_router, test_position_precision, test_mqtt, test_nexthop_routing) and the same suites with the policy off. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ixitxachitl
|
8fe246e250 |
fix(mesh): relay foreign packets whose channel hash collides with a local channel (#11544)
* fix(mesh): relay foreign packets whose channel hash collides with a local channel
The channel hash is one byte, so a foreign channel's name/PSK can fold to the
same hash as a local channel (~1/256 per local channel held). Since
|
||
|
Ben MeadorsandClaude Opus 5
|
83fd62b756 |
test(native): add 14 suites for routing, persistence, parsing and identity gaps (#11515)
* test(native): add 14 suites for routing, persistence, parsing and identity gaps Coverage audit of the native test tree; adds the highest-value untested logic as 11 new suites and extends 3 existing ones (200 test functions). New: test_stream_framing, test_nodedb_boot_recovery, test_nodedb_legacy_migration, test_nodedb_v25_roundtrip, test_nodedb_identity_hygiene, test_channel_keys, test_reliable_ack_matrix, test_hop_start_policy, test_routing_response_hops, test_phone_api_config_dump, test_observer. Extended: test_rtc, test_mqtt, test_xmodem. Two source changes the audit produced: - StreamAPI::handleRecStream copied stream->read()'s `cInt < 0` EOF check into the buffer-fed path, where there is no EOF sentinel; with signed char any byte >= 0x80 (START1 is 0x94) aborted the parse. Read the byte as uint8_t directly. Latent on develop (no callers), pinned by test_stream_framing. - Extract the post-decode pre-hop predicate from Router::handleReceived into shouldSkipHandleForPostDecodeHop() (NodeDB.h) so test_hop_start_policy drives the exact expression the router calls. No behavior change. test/state-manifest.tsv declares the suites that construct a NodeDB. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): address review - harden observer dispatch, trim comments Review follow-ups on the coverage-audit suites: - Observable::notifyObservers() erased list nodes while holding an iterator into them, so an observer that unobserves itself from onNotify corrupted the dispatch. Today the only self-detacher (PhoneAPI::onNotify -> checkConnectionTimeout -> close -> unobserve) survives solely because it returns -1 and aborts the chain before the increment; that unwritten contract is now gone. Removal during a dispatch nulls the entry and the outermost notify sweeps afterwards, which keeps self-detach, next-detach and destruction-during-notify all safe without an allocation. Hoisting the next iterator instead would have inverted the hazard and broken the existing next-detach case. Two regression tests added. - Correct the documented caller of shouldSkipHandleForPostDecodeHop: the call is in Router::dispatchReceived, not handleReceived. - Cast hop fields to unsigned at the %u call site in test_hop_start_policy. - Trim the new suites' file headers to the one-or-two-line rule in AGENTS.md. - Rename eight test functions whose names were exactly `test_` + 35 chars: that is the shape of a Lob API key, so trufflehog flagged them as secrets and failed the Trunk CI check. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): revert the observer dispatch change, keep the contract test Backs out the notifyObservers() deferred-removal hardening from the previous commit. It was reviewer-driven scope creep: nothing in the coverage audit needed it, no test required it, and it changes dispatch semantics in a header with ~76 observe() call sites on native verification alone. The hazard it addressed is not reachable today. The only observer that unobserves itself from onNotify is PhoneAPI (onNotify -> checkConnectionTimeout -> close -> unobserve), and it returns -1, which aborts the chain before the iterator is advanced past the erased node. test_self_detach_with_abort_during_notify stays: it passes against the unmodified dispatch and pins that the -1 is load-bearing, so a later cleanup that "simplifies" it away goes red. The unsafe variant (self-detach returning 0) is documented in a comment rather than tested, since asserting it would be asserting UB. * fix(serial): recover the frame behind a stray framing marker A byte that failed the START2 check was discarded rather than re-tested as a possible START1, so 0x94 0x94 0xc3 ... lost the real frame: one corrupted byte on a noisy UART silently dropped the frame behind it. Re-test the byte in place instead. Applied to both copies of the receive state machine. readStream() is the one that matters in the field - it is the serial path every phone client uses - while handleRecStream() still has no callers on develop. Strictly widens what the parser accepts; no frame that parsed before parses differently. test_stream_framing covers it on both receive paths, plus a run of stray markers and a START1-then-unrelated-byte resync. This was originally documented as a known gap in the framing suite. Fixing it instead was NomDeTom's call on review: a passing test asserting the bad behavior is what makes it hard to change later, and it is the same defect shape as the signedness fix three functions away. Also: use Throttle::deadlinePassed() in test_reliable_ack_matrix rather than a bare millis() compare, matching the house deadline rule. * test(native): cover the stray-marker resync on the buffer path too The stray-marker fix went into both copies of the receive state machine, but only test_stray_start1_before_frame_still_delivers drove both. The repeated- marker and unrelated-byte cases drove readStream() alone, so a regression in handleRecStream() would have gone unnoticed by two of the three. Verified load-bearing: reverting only the handleRecStream() half of the fix turns test_repeated_stray_start1_before_frame_still_delivers red on the new assertion. test_start1_then_unrelated_byte_resyncs stays green under that mutation by design - its failing byte is 0x00, where both branches reset to 0 - and covers the other half of the ternary. Also drops the stale header on test_stray_start1_before_frame_still_delivers, which still described the gap as pinned-as-is after the fix landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(native): make the hop-start truth table assert the rows it prints test_truth_table_summary was six TEST_MESSAGE lines and no assertion, so it reported as a case that could not fail - the anti-pattern #11517 names in its unfinished assertion-presence lint, and the one exception to NomDeTom's "no RUN_TEST without an assertion" pass over this PR. The printed row and the checked expectation now come from one struct, so the summary cannot narrate a table the predicates no longer implement. It also covers the consequence columns the per-row tests do not assert together: classifyHopStart, shouldDropPacketForPreHop and shouldSkipHandleForPostDecodeHop for the same packet, with the expectations gated on MESHTASTIC_PREHOP_DROP. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Thomas Göttgensandcoderabbitai[bot]
|
ef2be877a5 |
Stop breaking TestUtil.cpp on Windows, dangit! (#11529)
* Stop breaking TestUtil.cpp on Windows, dangit! * Update test/TestUtil.cpp Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Tom
|
bca7c0b480 |
Tom fiddles with the test suite - again (#11517)
* test: make every suite run its own binary, and fail the run when it does not PlatformIO links every native test program to the one $BUILD_DIR/$PROGNAME path and attributes Unity output by text alone, never checking that the source file a case came from belongs to the suite it thinks it ran. Both harnesses had been split into a build pass (--without-testing) and a run pass (--without-building), and for a non-embedded platform the run pass never relinks - so all 57 suites executed whichever suite was linked last, each reporting PASSED under its own name. Introduced for CI in |
||
|
James Rich
|
51eadb77d4 |
fix(NodeDB): reset a persisted event firmware_edition on vanilla builds (#11504)
* fix(NodeDB): reset a persisted event firmware_edition on vanilla builds myNodeInfo lives in devicestate, which survives a firmware reinstall, and the boot-time edition stamp was compiled out entirely on builds without USERPREFS_FIRMWARE_EDITION. A device flashed from an event build back to vanilla therefore kept reporting the event edition forever, and clients kept its branding until a factory reset. Stamp VANILLA in the else branch so the running build is always the source of truth. * Stamp the edition before the boot save decision, and assert the on-disk value Review follow-up: the stamp sat after the devicestate CRC compare, so an edition-only change stayed RAM-only and the persisted event edition survived on disk. Move it next to the other running-build-wins fixups (device_id, min_app_version), which run inside the CRC window, and extend the test to read device.proto back so the persisted value is asserted too. |
||
|
Ben Meadors
|
f57ee0bd71 |
fix(mesh): restore the implicit ACK for our own overheard PKI DMs (#11502)
* fix(mesh): restore the implicit ACK for our own overheard PKI DMs A DM we originate is PKI-encrypted to the recipient, so when we overhear it being rebroadcast we cannot decrypt it. perhapsHandleReceived() classifies it DECODE_OPAQUE and returns before shouldFilterReceived() runs, which is where the implicit ACK for our own transmission is generated. The client therefore never receives the ROUTING_APP ack it renders as "Delivered to mesh" for a DM, and the message sits in "sending" until it either succeeds outright or times out as max retransmissions. The ACK only needs the packet header (from/id), not the decoded payload, so split it out of shouldFilterReceived() into perhapsGenerateImplicitAckForOwnOverheard() and also call it from the opaque short-circuit for packets that are from us. Behavior on the decodable path is unchanged. Broadcasts on a PSK channel decode normally and always reached the generator, which is why channel messages were unaffected and only DMs showed the symptom. * test: rename implicit-ack tests to avoid a trufflehog false positive The camelCase identifiers tripped trunk's trufflehog/Lob secret detector. * test: shorten one test name past trunk's Lob secret-detector pattern trufflehog's Lob rule matches test_ followed by exactly 35 word characters, which both new test names happened to hit. Unrelated to the fix. |
||
|
Ben Meadors
|
905482ccce |
fix(serial): don't sleep forever with pending PhoneAPI output on UART consoles (#11500)
* fix(serial): don't sleep forever with pending PhoneAPI output on UART consoles Since #11164 bounded the stream drain, a config dump can end a dispatch with output still queued. On UART-console ESP32 boards runOnce() then returns INT32_MAX with no RX pending, and neither rxInt() nor onNowHasData() fires for the remaining output, so the download wedges mid nodeinfo stream until the client happens to send a byte. Add StreamAPI::hasPendingOutput() (transport-retained frame or queued PhoneAPI data) and have SerialConsole::runOnce() short-poll (<=25ms) while it holds instead of sleeping INT32_MAX. The #11164 write budget is unchanged; idle sleep behavior with a drained queue is unchanged. The retained-frame probe also covers the ESP32-S2 USB-CDC branch, which takes the same INT32_MAX path. * test(serial): restore scratch NodeDB via tearDown, trim comments to house style A failed TEST_ASSERT longjmps out of a Unity test without running destructors, so RAII cannot restore the swapped nodeDB pointer; install the scratch NodeDB explicitly and restore/delete it in tearDown(), which runs after every test outcome. Also shorten the new comments to the two-line house limit. |
||
|
Thomas Göttgens
|
a661fd8cd4 |
fixes #11466 (#11487)
* fixes #11466 * Keep locally-addressed routing feedback out of the phone echo filter allocForSending stamps ACK/NAK packets with from == our nodenum and sendLocal defaults to RX_SRC_RADIO, so the loopback gate never applies. Filtering on isFromUs alone dropped implicit rebroadcast ACKs, duty-cycle and NO_INTERFACE NAKs, and PhoneAPI rate-limit errors on their way to the client. Add coverage through the real RoutingModule, which the mocked one used by the rest of the suite cannot exercise, and correct the test seam comment. * Clean up the temporary RoutingModule in tearDown() A failed Unity assertion longjmps out of the test, so the in-test delete never ran and the module stayed registered in MeshModule::modules for every later test. Track it at file scope, as realNeighborInfoModule already is. |
||
|
Thomas Göttgens
|
34680833b8 |
fix(test): make the native-windows test suite build and run (#11482)
* fix(test): make the native-windows test suite build and run pio test -e native-windows failed every suite at the build stage. Five independent causes, all Windows-only: - TestUtil.cpp called lstat(), which MinGW-w64 does not provide. The state-checkpoint walk added in #11322 is fenced with ARCH_PORTDUINO, which native-windows also satisfies, so all 53 suites failed to compile. Route it through a stat() shim on _WIN32. - test_default, test_http_content_handler, test_meshpacket_serializer and test_serial define no setUp/tearDown and relied on the weak defaults PlatformIO emits in unity_config.c. GCC lowers a weak definition on PE-COFF to a weak external, leaving the symbol undefined, so it does not satisfy unity.c's reference and the link fails. Define them explicitly, as the other 49 suites already do. - test_mqtt included <arpa/inet.h>, absent on MinGW, for htonl(). Use winsock2.h there. - test_gps_update_scheduling uses TEST_ASSERT_DOUBLE_WITHIN. Unity omits double support unless UNITY_INCLUDE_DOUBLE is defined, so the assertion compiled to an unconditional failure. Define it for the env. - test_getfiles_rejects_overlong_path is excluded on _WIN32. Overrunning the 228-byte file_name needs at least 229 bytes below the portduino root, and that root is already ~34 bytes, so every qualifying path passes the 260-byte MAX_PATH: the nested mkdir() fails, the file is never created, and getFiles() has nothing to drop. No component layout satisfies both limits. Each of the seven suites that failed on Windows was verified individually after the change. test_fscommon_getfiles still fails in a full run, for a cause outside this change: rmDir() does not remove directories on Windows, so empty dirs left by an earlier run survive setUp() and make getFiles() report a depth truncation. That is a pre-existing FSCommon bug, reported separately. No Linux or macOS behaviour changes: every guard is _WIN32-only except UNITY_INCLUDE_DOUBLE, which is scoped to env:native-windows. * fix(test): define UNITY_INCLUDE_DOUBLE for every native env The flag was scoped to env:native-windows, but the gap is not Windows-specific. Verified on Debian with gcc against the Linux env's own Unity 2.6.1 and PlatformIO's generated native unity_config: UNITY_INCLUDE_DOUBLE : NOT defined UNITY_EXCLUDE_DOUBLE : defined test_double_within:FAIL: Unity Double Precision Disabled UNITY_INCLUDE_DOUBLE appears nowhere in the repo, the ini files, the workflow, or PlatformIO's unity runner, which adds only UNITY_INCLUDE_CONFIG_H. So TEST_ASSERT_DOUBLE_* is an always-failing stub on Linux and macOS too, not only on Windows. Moved to portduino_base.build_flags_common, which every native env resolves: native, native-tft, native-fb, native-tft-debug, coverage, coverage-event-policy, native-macos, native-windows and native-wasm. This does change Linux and macOS: TEST_ASSERT_DOUBLE_* becomes a real comparison instead of a stub. test_gps_update_scheduling is the only suite using those macros and its arithmetic is integer-based and bit-identical across platforms, so it should pass wherever it runs. Note it currently reports PASSED on CI in 0.03s while emitting no Unity output at all, so those assertions appear never to execute there; that is tracked separately and is not addressed here. |
||
|
Tom
|
a00675e00c | unset can have what it likes (#11496) | ||
|
Benjamin FaershteinandBen Meadors
|
a400143090 |
fix: improve acknowledged unicast retry reliability (#11320)
* Improve acknowledged unicast retry reliability * Fix merged next-hop routing tests --------- Co-authored-by: Ben Meadors <benmmeadors@gmail.com> |
||
|
Ben Meadors
|
b565a07a83 |
Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 (#11381)
* Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 BSEC2 cost ~37-39 KB flash and ~4-5 KB static RAM on ~190 of ~240 build targets, linked whether or not a BME680 was attached, and was a no-source proprietary archive inside GPLv3 release binaries. The firmware consumed exactly one BSEC-exclusive output: the IAQ value. - New BME680IaqEstimator: clean-room log-domain baseline tracker (humidity-compensated gas resistance vs a rise-fast/decay-slow ceiling, 0-500 scale matching the existing UI bands), pure math, unit-tested on native (test_bme680_iaq, 15 tests incl. a deep-sleep reboot simulation). Warm-up/burn-in progress persists to /prefs/bme680.dat via SafeFile so one-sample-per-wake SENSOR nodes converge across reboots; stale /prefs/bsec.dat is removed once. - BME680Sensor: single-path rewrite on Adafruit_BME680 with async once-per-minute sampling (~20x lower heater duty than BSEC LP mode), a hard 2-minute publish-freshness bound (a dead sensor stops reporting instead of freezing its last reading on the wire), and suppression of bogus gas_resistance=0 points from heater-unstable cycles. - platformio.ini: environmental_extra_common/_extra/_no_bsec collapsed into one section; Bosch BSEC2 + BME68x deps deleted; per-variant BSEC link-path hacks and the TEMPORARY promicro lib_ignore removed. nrf52_promicro_diy_tcxo regains BME680 support at 36 KB clear of the warm-store cap; rak4631 lands at 75 KB clear. - EnvironmentTelemetry: iaq rendering gates on has_iaq (a genuine IAQ of 0 now displays); stale BSEC comments rewritten. - rak4631 size budgets tightened (113000->108000 RAM, 786000->746000 flash) to lock in the reclaimed headroom. - bin/bme680_iaq_replay.cpp: host-side replay harness for tuning the estimator against captured BSEC traces (mean abs error + band agreement), no reflashing needed. Measured (develop -> this branch): rak4631 -38.8 KB flash / -4.9 KB RAM; heltec-v3 -36.4 KB / -4.0 KB; tlora-v2-1-1_6 +1.3 KB (its IAQ approximation had been dead code since #9663 due to an inverted isfinite check and now actually runs). Note: gas_resistance stays kOhm on the wire for fleet compatibility; the proto comment claiming MOhm gets a separate meshtastic/protobufs docs PR. * Address CodeRabbit review feedback - Use Throttle::isWithinTimespanMs for all elapsed-time predicates in BME680Sensor per coding guidelines (deadline math for the async reading completion stays raw, as it targets an absolute timestamp) - Make the state file name members static constexpr - Replay tool: cast uint16_t before %u (default argument promotion), report malformed input lines instead of silently skipping, and fail non-zero on stream read errors * Address CodeRabbit nitpicks - Replace the local clampf helper with std::clamp (meshUtils.h's clamp drags in Arduino.h, which would break the estimator's standalone host build that the replay harness depends on) - Trim the replay tool's file header to a two-line summary; the full build, capture, and tuning workflow moves to docs/bme680_iaq_replay.md |
||
|
Ben MeadorsandClaude Opus 5
|
230da77642 |
fix(time): convert the millis() rollover sites #11291's CI guard cannot see (#11483)
* MeshPacketQueue: fix millis() rollover in the late-packet drop test replaceLowerPriorityPacket() read `backPacket->tx_after < now`, with `now` taken from millis() on the line above. tx_after is an absolute deadline, so that comparison inverts while the deadline sits on the far side of the 32-bit wrap: a queued late packet reads as not-yet-due for the rest of the wrap window, or every late packet reads as droppable at once. The same statement ordered two deadlines against each other with `backPacket->tx_after > p->tx_after`, which has the same problem. #11291 swept every site where millis() sits next to the comparison operator, and its CI guard matches that shape. Stashing the clock in a local first is the same bug written so the guard cannot see it. Both tests now subtract before comparing: the due test through Throttle::deadlinePassedAt(), and the ordering through the elapsed-since-now form already used in AdminModule's oldest-slot scan. The snapshot comes from Time::getMillis() so the deadlines and the test read one clock, per the convention deadlinePassedAt() documents. The `dt` the log line reports is now derived from the same elapsed value rather than recomputed. Behaviour is otherwise unchanged, save the boundary: deadlinePassedAt() is inclusive, so a deadline landing exactly on `now` reads as due rather than one millisecond early. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * RadioLibInterface: don't widen a uint32_t deadline delta into a 64-bit long TRANSMIT_DELAY_COMPLETED tested whether the front packet was still waiting with long delay_remaining = txp->tx_after ? txp->tx_after - millis() : 0; if (delay_remaining > 0) ... The subtraction is uint32_t. Where long is 32-bit - every embedded target - an already-due deadline lands negative and the packet transmits, which is why this has never been visible on device. Where long is 64-bit (portduino, and the native test build) the same value zero-extends to ~4.29e9, reads as positive, and the packet is rescheduled 49.7 days out. It stays parked until some later notifyLater() with overwrite happens to reset the timer. That is not an edge case. notifyLater() schedules through setIntervalFromNow(), so the thread wakes at or after the deadline; being a millisecond past due is the ordinary path through this branch. Ask Throttle instead. deadlinePassedAt() is the unsigned half-range test, so there is no signed conversion to get wrong at any width, and the remaining delay handed to notifyLater() is computed from the same snapshot. On 32-bit the behaviour is identical, including at the boundary: a deadline equal to now transmitted before and still does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ExpressLRSFiveWay: convert the two remaining raw window checks to Throttle runOnce() dismissed the alert frame with `now > alertingSinceMs + 2000` and chose its poll rate with `now < keyDownStart + 20000`, both against a millis() snapshot in a local. Same rollover inversion as any other naive compare, and invisible to the millis-deadline-check guard because millis() is not adjacent to the operator. update() in the same file was already on Throttle. hasElapsed()/isWithinTimespanMs() with the stored event give the full ~49.7 day range and need no snapshot. Sentinels are unchanged in meaning: `alerting` is the armed flag for alertingSinceMs and is tested first, and keyDownStart == 0 reads as "recent" for the first 20s of uptime exactly as `now < 0 + 20000` did - a poll rate either way. The arm sites move to Time::getMillis() so the writes land on the clock Throttle reads, which also puts them within reach of Time::setTestMillis(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * GPSUpdateScheduling: record whether a search is running, don't infer it elapsedSearchMs() answered "am I searching?" by ordering two raw millis() stamps: searchStartedMs > searchEndedMs. Whichever stamp lands on the far side of the 32-bit wrap reads as the larger one, so the answer inverts once per wrap cycle, in both directions: - a search that started before the wrap and ended after it keeps reading as "searching". elapsedSearchMs() then grows without bound and searchedTooLong() aborts a search that is not running. - a search that started after the wrap, following one that ended before it, reads as "idle". elapsedSearchMs() returns 0, so an unproductive search is never aborted and the receiver stays powered until it locks. Both self-heal at the next informSearching(), which bounds the damage to one GPS cycle - but the ordering test cannot be made wrap-correct, because the two stamps carry no information about which wrap they belong to. It does not need to be. Whether a search is in progress is a fact the three inform*() calls already have in hand; the ordering was only ever standing in for it. Add the flag and set it there. elapsedSearchMs() keeps its unsigned subtraction, which was always the correct part. The file's clock reads move to Time::getMillis() so the suite can drive them across the wrap. Behaviour-preserving in production - Time::getMillis() is millis() unless a test injects a clock. test_gps_update_scheduling/ gains seven cases: the idle/searching/ended states, elapsed exactness across the wrap, both inversion directions above, and reset(). The two wrap cases fail on the old predicate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * MessageStore: date boot-relative messages in uptime seconds A message received before the wall clock is trustworthy is stamped boot-relative and healed by upgradeBootRelativeTimestamps() once the RTC arrives. Both the stamp and the "same boot?" test were millis() / 1000, which wraps every 49.7 days: a stamp taken before the wrap reads as newer than `bootNow` afterwards, so `m.timestamp <= bootNow` declines to heal it and the message shows "???" until it ages out. MessageRenderer's own copy of the test falls the same way and prints invalidTime. Neither produces a wrong time - the guard is what fails safe - but Time::getUptimeSecs() landed in #11291 for exactly this, and does not wrap for 136 years. Both sites take it, which makes the comparison exact rather than merely fail-safe. While here, the autosave tick had its own hand-rolled deadline helper - `reachedMs(now, target)` as `(int32_t)(now - target) >= 0`. Wrap-correct, but a competing idiom for what Throttle::isWithinTimespanMs() already answers, and the signed cast is the form #11291 replaced everywhere else. Deleted; the stamps read Time::getMillis() so the whole path is on one clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * WebServer: drop the hand-rolled millis() wrap branch getAdaptiveInterval() special-cased the wrap by hand: if (currentTime >= lastActivityTime) timeSinceActivity = currentTime - lastActivityTime; else timeSinceActivity = (UINT32_MAX - lastActivityTime) + currentTime + 1; Those two expressions are the same number - unsigned subtraction already computes the difference modulo 2^32 - so this is not a bug, just eight lines reimplementing what Throttle does. It also reads like a site that has thought about the wrap and settled it, which makes it a bad example to copy. Two isWithinTimespanMs() calls against the stored activity stamp, matching ethApiServer's shape for the same adaptive-interval decision. The stamps move to Time::getMillis() so the writes and the reads share a clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * MeshPacketQueue: only order elapsed times once both deadlines have passed The late-packet eviction I rewrote compared how long ago each deadline passed: backElapsed < (uint32_t)(now - p->tx_after) That is only an ordering when both deadlines are in the past. An incoming packet whose tx_after is still in the future subtracts to a near-2^32 elapsed, which reads as the most overdue packet in the queue rather than the least - so a full queue would drop the overdue packet it was about to transmit in favour of one that is not ready yet. The comparison it replaced, `backPacket->tx_after > p->tx_after`, got this right away from the wrap; I lost it in the conversion. Classify before ordering: p->tx_after must be unset, or passed, before its elapsed time means anything. Two expired deadlines still order by which is further overdue, which is what the branch is for. Caught by CodeRabbit on #11483. test/test_meshpacket_queue/ pins the branch: the future-dated arrival that started this, both directions of the both-expired ordering, the undelayed arrival, and all of it again with the deadlines and `now` on opposite sides of the wrap. maxLen is 1 so the suite reaches the branch without dragging in CompareMeshPacketFunc and a NodeDB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ExpressLRSFiveWay: treat "no key pressed yet" as no activity keyDownStart is 0 until the first press of a boot, and the fast-poll window read that as a press at time zero: 100ms polling for the first 20s of uptime with no activity at all, re-triggering once per millis() wrap. The arithmetic this replaced (`now < keyDownStart + 20000`) did the same, so it is not a regression - but the sentinel is exactly what the conventions say to test before the elapsed comparison, and "has there been recent key activity" has an honest answer here. 250ms is the documented floor for not missing presses, so an idle node simply starts there and moves to 100ms on the first press. Also trims the wrap-cases comment in test_gps_update_scheduling to the two-line house limit. Both from CodeRabbit review on #11483. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Thomas Göttgens
|
3b608b8fc5 |
fix(mesh): keep ROUTING_APP responses when toPhoneQueue is full (#11480)
* fix(mesh): keep ROUTING_APP responses when toPhoneQueue is full #2918 narrowed the overflow policy to evict the oldest entry only for TEXT_MESSAGE_APP and RANGE_TEST_APP, dropping every other portnum. A dropped ROUTING_APP response leaves the phone with no delivery confirmation for a message it sent. Add ROUTING_APP to the eviction list and pin the policy in test/test_tophone_queue. Fixes #11439 * fix(mesh): gate the queue-overflow portnum check on the decoded variant decoded.portnum aliases encrypted.size in the payload union, so an encrypted packet could be read as a privileged portnum by its ciphertext length. Restore config.device.rebroadcast_mode in the test teardown. * test: rename a test to avoid a trufflehog false positive test_text_still_admitted_when_queue_full is "test_" followed by exactly 35 characters, which matches the Lob API key shape and fails trunk check. |
||
|
Tom
|
f5314148c2 |
Serialise AirTime behind a lock, and stop handing out its buckets (#11362)
* Copy airtime reports into a caller buffer instead of exposing the array
airtimeReport() returned a pointer into the rotating bucket arrays, so the
caller held a handle to state that logAirtime() and every accessor mutate
underneath it. Copy into a caller-supplied buffer instead, and report failure
for a null buffer, a count past the log depth, or an unknown report type.
ContentHandler owns its buffer and hoists getPeriodsToLog() out of the three
calls that repeated it.
* Cover the AirTime report API and log-dispatch contract
Half of AirTime's surface had no tests: which store each report type feeds,
what airtimeReport() does when misused, how the first sync seeds itself, and
whether calling several entry points in one interval compounds the rotation.
Eighteen tests, asserted through the public API rather than the public bucket
arrays - those arrays are meant to become private, and a test that reads them
would have to be rewritten rather than pinning a contract.
Two of them state a convention that was never written down: the report arrays
are shift-ordered with slot 0 newest, and slot 0 covers only the time since the
last rotation. channelUtilization and utilizationTX use the opposite convention
- a modular ring indexed by uptime phase - and reading one as if it were the
other is a defect that has already happened once.
* Characterise AirTime window decay, TX gates, and sleep behaviour
Thirty-three tests in three kinds. Invariants must hold forever; boundaries pin
off-by-ones a refactor would move; five characterisations encode today's wrong
numbers, each tagged with the phase that will flip it.
Readings are asserted against an event-log oracle - airtime physically on air
inside (now - window, now], computed from a list of completed packets - rather
than against hand-worked constants, so a test states "this matches the
definition" instead of "this looked right when I wrote it".
The characterisations, all measured rather than assumed:
- the window covers (N-1)p + phase but divides by Np, so a steady 10% load
reads 8.33% right after a bucket boundary -> phase 5
- the same load sweeps across bucket phase instead of holding -> phase 5
- the hour window carries the same defect, 10x smaller -> phase 5
- a packet longer than its bucket is credited whole to the bucket
it completed in, so a saturated LONG_SLOW channel reads >100% -> phase 4b
- getSilentMinutes() reads a modular ring as if the index were an
age, so identical airtime gives different answers by phase -> phase 6
Two tests needed correcting during the write, both my expectations rather than
the code: a six-bucket ring sheds whole buckets, so a 30s gap drops three of
five survivors and not "half"; and the oracle sees 59 completions in a 60s
window, not 60, because the one on the lower edge is outside it.
Not written: the planned RX_LOG/RX_ALL_LOG disjointness test. That is a
property of the two radio drivers, which choose one or the other per packet -
it is not observable from AirTime, which records what it is told. The
AirTime-side half is already covered by the routing tests.
* Drop write-only and undefined AirTime members
None of this was reachable:
air_period_tx / air_period_rx file-scope mirrors of airtimes.periodTX/RX,
accumulated, rotated and memset in lockstep
with them but never read out or serialised.
Orphaned when #2552 re-pointed the writes at
bare globals instead of deleting them.
lastUtilPeriod, lastUtilPeriodTX written on every sync, read nowhere
airtimes.lastPeriodIndex written on every rotation, read nowhere
currentPeriodIndex() computes (secs / 3600) % 8 - a modular-ring
index for the one array that is shift-ordered
rather than a ring. Its only two uses were the
dead field above and a log line. It is the
fossil of the same confusion that makes
getSilentMinutes() wrong.
UtilizationPercentTX() declared, never defined
free logAirtime()/airtimeReport() declared, never defined; the latter still
carried the array-returning signature the
previous commit removed, so it actively misled
Also fixes the rotation log line, which read currentPeriodIndex() from inside
the loop although the index is advanced before it - on a multi-hour wake it
printed the same final value once per rotation. It now reports which of the
crossed hours is being rotated.
airtimeRotatePeriod() is kept: it has no caller in the tree either, but unlike
the above it is a defined public method, so out-of-tree callers are plausible.
Measured, not estimated: sizeof(AirTime) 464 -> 456 B, plus 64 B of globals, so
-72 B of static RAM. Padding accounts for the difference from the 66 B the plan
predicted by counting declared bytes.
The whole point of writing the tests first: the suite is green here with zero
test changes.
* Document what the AirTime figures measure and how they are stored
Comments only, but four of the things they replace were false.
The header's example analytics claimed RX_ALL_LOG was "all received lora
packets" and offered "RX_ALL_LOG - RX_LOG = other lora radios". Both radio
drivers pick exactly one of the two per packet, so they are disjoint: RX_ALL_LOG
is airtime we could not parse, the subtraction can go negative, and the total is
TX + RX + RX_ALL. Replaced with the actual contract - four inputs, eight
outputs, the window each spans, and the fact that the three thresholds are
hard-coded members rather than the settings they look like.
Names the two storage conventions on their declarations, because mixing them up
is what makes getSilentMinutes() wrong: channelUtilization and utilizationTX are
modular rings indexed by uptime phase, where the oldest bucket is (current + 1)
% N; airtimes.period* is shift-ordered with slot 0 newest, where the index IS an
age and slot 0 is a partial hour.
Defines the measurement as wall time rather than awake time, and says why: a
sleeping node still hears traffic, and per-node redefinition would make two
broadcast readings incomparable. Records that the 60s figure is published to the
mesh at >= 1h cadence, so what other nodes see is a snapshot - at LONG_FAST and
1% occupancy it reads exactly 0 in about 44% of reports - and that the contention
window it feeds moves in 20-percentage-point steps, so small errors never reach
the backoff.
Finally, states that rotation happens on access rather than on the scheduler
tick, names the test that enforces it, and leaves a TODO pointing at the plan
phases that fix the characterised accuracy defects.
* Serialise AirTime behind a lock proven by a private token
Two mechanisms solving different halves. A lock-free inner core (Windows) holds
all state and all logic; it has no lock and no way to reach one, so nesting is
impossible by construction. A private Held token takes the lock in its own
constructor and is the only thing that can be passed where a core method demands
one, so the lock cannot be forgotten either.
The rule is now uniform with no exceptions to remember: every public method
takes the lock once and delegates. In particular isTxAllowed*() lock like
everything else - before the split they could not, because they called the
public accessors and the lock is not recursive. That asymmetry was the foot-gun
the previous design documented in prose and hoped nobody would trip.
getPeriodsToLog()/getSecondsPerPeriod() still take no lock; they return
compile-time constants and touch no state.
channelUtilization[] and utilizationTX[] were public, so the lock was bypassable
at compile time. They move into the private core. Four test sites reached in;
all four now use logAirtime() plus the virtual clock, and no new test seam was
needed. Nothing in src/ was affected.
The re-entry assert is guarded on PIO_UNIT_TESTING, so it exists in test builds
only. The design sketched #ifdef DEBUG, but nothing in this tree defines DEBUG
or NDEBUG, so either spelling ships the assert to every board - and
nrf52_promicro_diy_tcxo has ~128 bytes of headroom under its 0xEA000 warm-store
cap, which the assert's strings and abort path overrun. It would have worked on
hardware, since the check runs in Held's owner initialiser and so precedes the
blocking take; the objection is that abort()ing a live mesh node is a poor trade
for a bug never seen in the field. Native tests are where it earns its keep
anyway: Portduino compiles Lock::lock() to an empty body, so a nested take there
succeeds silently and nothing else would notice.
Also comments out ScopedBusyAirTime in test_traffic_management. It is inert
twice over: the module holds no reference to airTime at all since hop exhaustion
was shelved, and the fixture never worked anyway - writing the buckets on a
fresh AirTime is undone by the first accessor call, which takes the firstTime
branch and memsets them. It reported 0%, not the 100% it claimed. Left in place,
commented, with both reasons recorded.
Cost on the tightest board in the tree, nrf52_promicro_diy_tcxo: the six phases
together add 96 bytes of flash, leaving it 32 bytes clear of the warm-store
guard. RAM is 72 bytes lower from the dead-state removal. Suite green at 47/47,
with test_airtime unedited apart from the added nesting test.
* Count rotations with the loop variable, not a separate tally
LOG_DEBUG compiles to nothing under DEBUG_MUTE, so the counter's only read
disappeared with it and the tally became write-only. It does not warn today -
this build has -Wunused-but-set-variable on, and it fires for other locals, but
not for one that is only initialised and never read - so it was latent rather
than broken: a stricter flag or -Werror would have failed muted builds only.
Using the loop variable removes the class of problem, since the loop condition
reads it, and drops the elapsedAirtimePeriods-- mutation as a side benefit.
Same iteration count, same output.
Found by compiling nrf52_promicro_diy_tcxo with -D DEBUG_MUTE, which is worth
recording for its own sake: muting logs takes that image from 802 784 to
673 416 bytes, 98.5% to 82.6% of flash. Logging is 16% of the largest nrf52
image, and its 32 bytes of warm-store headroom are a logging-verbosity question
rather than a code-size one.
* Tighten the comments added by this branch
Comment-only: with comments stripped, all five files are byte-identical to the
previous commit.
Removed the references to the planning notes. Those documents are working
material and will go stale; the code should not depend on them. The five
CHARACTERISATION tags now describe the defect they pin and stop there, and the
accuracy TODO names the four defects and points at the tests instead of a plan
file.
Also removed, as noise rather than information:
- comparisons against pre-#11291 behaviour, which nobody reading this needs
- a comment describing the lock restructure as future work, written before it
landed
- speculation ("plausible", "worth pinning so a future...")
- an aside arguing with an arithmetic slip made while writing the test
Kept the mechanical facts that are slow to re-derive: the two storage orderings
and which array uses which, RX_LOG/RX_ALL_LOG disjointness, the locking rule and
the addSpanned() constraint that protects it, why the re-entry assert is
test-only, and the concrete numbers - (N-1)p + phase, 14 164 ms, the 20 pp
contention-window steps.
Net 16 comment lines out of src/, 33 out of test/.
* Gate the AirTime re-entry check on the host, not on testing
PIO_UNIT_TESTING is injected by PlatformIO purely on BUILD_TYPE, with no
platform check, so it is defined on an on-target `pio test` run too. The
check arms before the lock is taken - a nested take blocks forever, so a
later check would never run - which under preemption false-positives on
legitimate contention and races on its own write.
Derive AIRTIME_REENTRY_CHECK once from PIO_UNIT_TESTING && !HAS_FREE_RTOS
and use it at all three sites. Had the three conditions ever diverged, an
on-target test build would fail to compile on a member the header no
longer declares.
* Log AirTime outside the lock it serialises
DEBUG_PORT.log() blocks on a UART write, and `lock` is a plain binary
semaphore with no priority inheritance, so holding it across a log call
lets the main thread stall the radio thread in getTxDelayMsec().
Move logAirtime()'s LOG_DEBUG into the shell, after the Held scope
closes; the shell already has both arguments, so nothing has to be
passed back out of the core. isTxAllowed{ChannelUtil,AirUtil} read into
a local under the lock and warn after it. The log bodies are braced
because LOG_DEBUG compiles away under DEBUG_MUTE and a bare `if (x) ;`
trips -Wempty-body.
Fold the two doubled index calls into `+=` while touching the lines.
* Give each airtime report its own buffer
handleReport() reused one array across the three airtimeReport() calls
and ignored the bool. A failed report would have left the previous
type's data in place and emitted it under the next type's key. Build
each through a lambda whose buffer is zeroed per call, so a failure
emits zeros.
Unreachable today - the count is always PERIODS_TO_LOG and the type is
always valid - but the old shape only read as correct by accident.
* Drop a stray semicolon from the inert-guard comment
* Address external review: name the race, tighten the claims and the tests
The header sold the lock as mechanism without naming a second thread, which
invites the reasonable objection that this is a cooperative OSThread codebase.
There is a real race and it is nRF52-only: NRF52Bluetooth registers its ToRadio
write callback with defer == false, so a phone's packet runs handleToRadio ->
sendToMesh -> Router::send on the Bluefruit BLE task, reading
utilizationTXPercent() and getSilentMinutes() while loopTask may be inside
logAirtime(). ESP32 hands BLE work to the main task and does not have it.
Three claims in the header were wrong or overstated:
- "nesting is impossible by construction" - Windows is a nested class with an
enclosing class's access rights, and `extern AirTime *airTime` is in the
same header, so airTime->anyPublicMethod() from inside it is well-formed
and would hang. Nothing does it; the assert is the backstop. Say that
instead, because the comment below instructs contributors to add helpers
to Windows on the strength of the guarantee.
- "every public method takes the lock exactly once" - two constant accessors
take none and isTxAllowedAirUtil() takes it zero or one times. State the
exceptions where the invariant is stated, not only at the definitions.
- "both radio drivers pick exactly one per packet" - five drop paths log
neither. At most one. Recorded against plan4 rather than fixed here: it
changes a telemetry value.
getPeriodsToLog()/getSecondsPerPeriod() become static constexpr, which removes
them from the locking claim structurally and lets ContentHandler size its
buffer and its count from one constant.
Tests:
- C14's saturated AirTime is installed by a helper and restored in tearDown.
Unity's TEST_ABORT() is longjmp and does not run destructors of automatic
objects, so the scoped guard it replaces would leave airTime dangling into
an abandoned frame on any assertion failure - and the same commit that
added it removed the tearDown reset that did cover that.
- test_getSilentMinutes_counts_minutes_until_enough_ages_out asserted only
`mins <= 60`, which neither return path can violate. The answer is 59.
- test_backwards_uptime_degrades_safely stepped 600s -> 60s, which leaves
elapsedAirtimePeriods at 0, so it never reached the hourly-report branch
its own comment describes. Step by the wrap instead and assert the exact
figures.
- test_airtime leaked EU_868 out of the duty-cycle case into every later one,
and the reentry test's isTxAllowedAirUtil() coverage depended on it.
Restore the region in tearDown and set it explicitly where it is wanted.
- Rename that test to what it can actually check: no single method takes the
lock twice. The calls are sequential, so it cannot catch two methods
nesting.
* trunk: suppress trufflehog/Lob false positives in test_airtime
* Address CodeRabbit review: the rotate trace, the cap warn, the backoff
Four findings from the CodeRabbit pass. Two were introduced by this branch,
one is a real inconsistency it inherited, one is a naming slip.
The rotate trace was the one that mattered. "Log AirTime outside the lock it
serialises" moved the per-packet lines and the two TX-gate warnings out to the
shell, but missed LOG_DEBUG("Rotate airtimes, crossed hour %u") because it does
not sit in the shell at all: it is inside Windows::syncNow(), the lock-free
core, which by construction only ever runs under Held. Nothing at that line
looks like a lock, which is why it survived.
The exposure is smaller than the review suggests - runOnce() syncs at 1 Hz, so
in steady state this is one line an hour, and the PERIODS_TO_LOG - 1 burst
needs an hour of light sleep with no intervening sync - but a UART write under
a plain binary semaphore with no priority inheritance is exactly what the
comment above logAirtime() says this code does not do. syncNow() now
accumulates crossings in rotationsPendingLog and runOnce() drains it inside the
Held scope, then logs after release. Any caller can cross an hour; only that
thread reports it, so a crossing raised elsewhere is traced at most one tick
late. The `if (rotations > 0)` guard keeps the drained value read under
DEBUG_MUTE, where LOG_DEBUG expands to nothing - the write-only tally that
"Count rotations with the loop variable" removed.
addFromContact()'s favorite fallback stamped silently when the protected cap
refused it. The stamp is new on this branch; the two sibling refusals (ignore,
verify) both emit PROTECTED_CAP_WARN_FMT, so the operator lost the only signal
that the cap was hit on the one path that has a fallback.
lfs_assert() mixed clocks: Throttle read Time::getMillis(), the remainder was
computed from a second, bare millis(). The review's stated failure mode - a
native test overriding the clock - cannot happen, since the hook is behind
PIO_UNIT_TESTING and this file is nRF52-only. The real defect is the second
read: a tick landing on the 20-minute boundary between the check and the
subtraction underflows the remainder into delay(~50 days), on a device that has
just found its flash corrupt. One read, clamped, and preFSBegin() stores from
the same clock.
The eviction test is renamed to
test_eviction_prefersCurrentBootStampOverPost2038Epoch. The finding is right
that it was snake_case, but the suggested testEvictionPrefers... does not match
this file either, which is test_<area>_<camelCase> throughout.
Not taken, both pre-existing and out of scope for a rollover branch:
- t5s3_epaper's touchResumeAtMs/suppressFromMs read an active suppression as
inactive if the wake lands in the 1 ms where millis() is 0. Consequence is
one skipped 150 ms touch-settle window per 49.7-day wrap.
- NRF52Bluetooth::onPairingPasskey() busy-waits 30 s in a BLE callback. Worth
saying plainly that this branch makes it more visible: the old
`millis() < start_time + 30000` overflowed at the wrap and cut the wait
short, so the correct Throttle form is what lets it run the full 30 s.
Reworking it into an OSThread is its own change.
Native suite GREEN, 48/48, 672 cases.
|
||
|
Tom
|
6745995442 |
docs: move the firmware design docs to the documentation site (#11488)
The five documents under docs/ were written in this repo while their features
were developed. Four of them describe shipped, upstream behaviour and belong on
meshtastic.org, where users and client authors will look for them:
traffic_management_module.md -> configuration/module/traffic-management
+ development/reference/traffic-management-internals
node_info_stores.md -> development/reference/node-info-stores
mesh_beacon_module.md -> configuration/module/mesh-beacon
+ development/reference/mesh-beacon-internals
+ development/device/mesh-beacon-client-interface
lora_region_preset_compatibility_client_spec.md
-> development/device/region-preset-compatibility
Each is split by audience: settings pages carry the config surface in user
terms, reference pages carry firmware mechanism, and the device pages carry the
protocol a client app speaks. The region-preset spec always said it should
graduate out of this repo once its protobuf landed upstream, which it has
(FromRadio.region_presets, field 19).
nexthop-routing-reliability.md is not documentation - it is a working document
with a mitigation plan, a "files to modify" list and commit sequencing. Its
mitigations shipped in #10745, so the plan is history and the analysis is
superseded; it is dropped rather than published.
Comments that cited the deleted files now point at the published pages, and the
NextHop test header cites #10745 instead of the deleted plan.
|
||
|
|
fdb644e0b7 |
Fix millis() rollover in deadline, interval, and timestamp handling (#11291)
* Add native test coverage for the UptimeClock monotonic seam
src/UptimeClock.{h,cpp} shipped without a dedicated test suite. Port the six
tests from the monotonic-time branch (test/test_time), retargeted to the
renamed header.
The wrap test crosses 0xFFFFFFFF via advanceTestMillis() rather than a second
setTestMillis(): setTestMillis() sets clockSourceChanged, which makes
getMillis64() rebase its accumulator and swallow the wrap.
* NextHopRouter: fix 49.7-day millis() rollover in retransmission timing
Resolves the "FIXME, handle 51 day rolloever here!!!" in
NextHopRouter::doRetransmissions() by switching the retransmission-due
comparison from plain unsigned <= to a signed-difference cast.
The previous p.nextTxMsec <= now comparison silently breaks across the
~49.7 day millis() wraparound: pending retransmissions either stall
for the remainder of the wrap window, or all fire simultaneously at
the rollover boundary. Long-running router/infrastructure nodes do hit
this in practice.
The replacement (int32_t)(p.nextTxMsec - now) <= 0 is the standard
Arduino/embedded idiom for rollover-safe deadline checks and behaves
identically to the original for any non-wrap timing.
* Address Copilot review: use unsigned half-range for rollover-safe retransmit check
Review feedback from @Copilot on PR #10227: casting a uint32_t
subtraction to int32_t is implementation-defined in C++ when the
unsigned value exceeds INT32_MAX (even though it works on typical
two's-complement targets).
Switch to the fully well-defined unsigned half-range form:
nextTxMsec is in the past-or-equal iff (now - nextTxMsec) has not
wrapped past 2^31 ms. Future offsets < 2^31 ms wrap into the top
half and read as 'not yet'.
Same semantics as the signed-cast version on every two's-complement
platform we care about, but portable to any conforming C++ impl.
* Use monotonic time for airtime windows
* Document monotonic airtime windows
* Fix test_packet_signing sentinel that #10227's rollover fix inverts
test_C3_invalid_repeated_packet_cannot_ack_or_change_retry_state parked a
pending packet at nextTxMsec = UINT32_MAX to mean "never retransmit", then
asserted that a rejected repeated packet leaves the retry state untouched.
NextHopRouter::doRetransmissions() now tests whether a retransmit is due with
an unsigned half-range compare, (uint32_t)(now - nextTxMsec) < 0x80000000u,
so that retransmission timing survives the ~49.7 day millis() wrap. Under it
now - 0xFFFFFFFF == now + 1, a small positive delta, so UINT32_MAX reads as
~1ms in the past: the retransmit fires and rewrites nextTxMsec, and the test
failed with "Expected 4294967295 Was 6247".
Use a representable future time instead. Production is unaffected either way -
nextTxMsec is only ever written as millis() + d, and UINT32_MAX came from the
test harness alone - so the sentinel is what needs to go, not the comparison.
Special-casing UINT32_MAX in the retransmit path would keep a value that reads
as "expired" under any wrap-correct compare.
The value is held in a local because millis() advances across
runPipelineIngress(), so recomputing it at the assertion would compare against
a different number.
Reported upstream on meshtastic/firmware#10227, whose branch predates this test.
* Make Throttle time-injectable and add hasElapsed()
Throttle backs ~94 call sites, which makes it the highest-leverage place in
the tree to put the clock seam: reading Time::getMillis() instead of millis()
in its three call sites turns all of them into time-injectable code at once,
without touching any of them. The 32-bit millis() wrap is not otherwise
reachable from a native test.
The read is behaviour-preserving - Time::getMillis() returns millis() unless a
test injects a clock - and the full native suite passes with it live.
Also add hasElapsed(), the complement of isWithinTimespanMs(), because 51 of
the 94 call sites are spelled !isWithinTimespanMs and read poorly. Its
boundary is inclusive (>=) since isWithinTimespanMs uses <; both are
documented. It deliberately does not treat lastExecutionMs == 0 as "never
run": call sites pair that test with the interval check themselves, and
absorbing a sentinel into the one helper every module depends on is exactly
the value-overloading hazard being removed elsewhere.
Migrating the existing !isWithinTimespanMs sites is cosmetic and deliberately
left out of this commit.
test/test_throttle/ covers window semantics, both boundaries, the complement
identity, execute()'s first-run and throttled paths, and - the point of the
exercise - a window opened before the wrap closing correctly after it,
including at the 24h interval that is the longest in the tree.
* Stop disarmed deadline sentinels reaching the comparison
Two deadline variables encoded "inactive" as a magic value that only reads as
"never" because the comparison against it is a naive millis() compare. Under
any rollover-correct comparison both invert to "expired ~49 days ago", so they
have to be untangled before those comparisons can be fixed.
Power::reboot() set rebootAtMsec = -1 on platforms with no reboot
implementation, intending "never fire". Every reader already treats 0 as the
disarm value - powerCommandsCheck() tests `if (rebootAtMsec && ...)`, and
AdminModule writes 0 to cancel - so -1 was both wrong and unnecessary. Use 0.
Left as UINT32_MAX it would reboot-loop the moment the comparison is corrected.
ExternalNotificationModule's nag window compared against nagCycleCutoff, which
holds UINT32_MAX once stopped and 1 at boot. isNagging is the real armed flag,
so test it first and short-circuit: a disarmed cutoff can no longer reach the
arithmetic, while an idle module still takes the same sleep path that the
boot-time value of 1 was relying on.
Note this fixes the sentinel only. The comparison itself is still a naive
`nagCycleCutoff < millis()` and remains on the list to convert.
* Fix millis() rollover in every deadline and interval comparison
Roughly 20 sites compared against millis() directly - `millis() > deadline`,
`deadline < millis()`, `last + interval < millis()`. All of them break for
about 24 days after the 32-bit millis() wrap: depending on which side of the
wrap each value sits, the action either stalls for weeks or fires immediately
and repeatedly. The longest affected interval is the 12 hour NTP renewal, a
~50x margin against the wrap, so none of these needed the range - only the
correct comparison.
Add Throttle::deadlinePassed(deadlineMs) for sites that store an absolute
deadline they cannot re-express as "interval since an event". It uses the same
unsigned half-range test as NextHopRouter::doRetransmissions() rather than
introducing a competing signed-cast idiom, and unlike the signed cast it is
defined for every input. Sites that do store an event use the existing
isWithinTimespanMs / hasElapsed. Nothing gained new state.
Because both helpers read Time::getMillis(), every converted site is now
reachable from a native test that drives the clock across the wrap; the
comparison itself is covered directly in test/test_throttle/.
Sentinel handling is the reason this could not be a mechanical rewrite. The
disarm convention is not uniform: 0 means "inactive" for rebootAtMsec,
shutdownAtMsec, alertBannerUntil, fixHoldEnds, suppressUntilMs and
touchResumeBlockUntilMs; 0 means "due now" for ntp_renew, which is forced to 0
at link-up; UINT32_MAX means "inactive" for nagCycleCutoff; and
alertBannerUntil == 0 in isOverlayBannerShowing() means "show indefinitely".
Every inactive marker is arithmetically far in the past, so a correct
comparison fires on it - each site tests its sentinel before the arithmetic,
and keeps the meaning it had.
Two sites carried a second bug found on the way:
BME680Sensor tested (stateUpdateCounter * STATE_SAVE_PERIOD) < millis(). With
a 6 hour period and a uint16_t counter that product overflows uint32_t after
about 198 saves, independently of the millis() wrap. It now measures the
interval since the last save.
EInkDynamicDisplay had `if (previousRunMs > millis()) return;` as a millis()
overflow guard, which skipped rate limiting entirely for the whole post-wrap
period - the bug it meant to prevent. Every check below it already goes
through Throttle, so the guard is removed rather than fixed.
MotionSensor's calibration countdown is converted to a signed delta rather
than deadlinePassed, because it needs the remaining magnitude and not a
boolean; that matches the already-correct check in the same file.
* Remove getMillis64() and use Throttle for the NodeInfo reply window
getMillis64() had exactly one caller and no callers in tests. It also carried
obligations that made it the wrong shape for this firmware: a wrap accumulator
in mutable statics, which is not ISR-safe, and which must be polled at least
once every ~49.7 days or it silently misses a wrap and returns a time ~49 days
short.
Its one caller only wanted to know whether a 12 hour suppression window had
elapsed - which Throttle answers correctly across the wrap without any
accumulator. NodeInfoModule now stores Time::getMillis() in lastNodeInfoSeen
and tests the window with Throttle::isWithinTimespanMs, so the map holds
milliseconds rather than seconds derived from a 64-bit read.
USERPREFS_NODEINFO_REPLY_SUPPRESS_SECS is user-overridable and now feeds a
multiply by 1000, so a static_assert rejects any value too large to express in
milliseconds instead of letting it wrap.
clockSourceChanged goes too. It existed solely to rebase getMillis64()'s
accumulator when a test swapped clock sources, and it made the wrap untestable
through the injection API: setTestMillis() set the flag, so a wrap crossed by
two setTestMillis() calls was swallowed. With the accumulator gone the flag has
nothing to rebase, and the injection API is a plain settable clock.
The three getMillis64 tests are dropped as they no longer describe anything.
One test replaces them, pinning that advanceTestMillis() wraps past
0xFFFFFFFF rather than saturating, since the Throttle wrap tests rely on it.
Also fix eviction in pruneLastNodeInfoCache(): it picked the entry with the
smallest stored stamp, which is the wrong victim once some stamps sit on the
far side of the wrap. It now evicts the largest elapsed time.
* Add CI guard and docs rule against naive millis() comparisons
Fixing the existing sites does not stop the next one being added. The
millis-deadline-check job rejects millis() placed directly next to a comparison
operator, in either order, anywhere in src/. It lives in test_native.yml
alongside suite-count-check, which sets the precedent for a repo-hygiene guard
that CI enforces and bin/run-tests.sh does not.
The correct idioms all subtract before comparing, so none of them match the
pattern. Line comments are stripped first, so documentation is free to name the
broken form - as the guard's own comment and the coding conventions both do.
Writing the check before finishing the sweep turned out to be worth it: it
found roughly 14 sites that a by-hand audit of deadline variables had missed,
including two extra nagCycleCutoff compares, both boot-screen timeouts, and a
6 hour sensor save interval that was also overflowing a uint32_t multiply.
.github/millis-deadline-allowlist.txt covers the cases that are genuinely not
deadline tests. Both current entries are uptime thresholds - "has the device
been up N ms" - with no stored deadline and no event to measure from: a 30s
button holdoff against phantom shutdown from floating pins, and a 10s window
for the OEM boot logo. Each re-crosses its threshold once per wrap, which is
harmless for boot-holdoff logic and not worth new state to avoid. Entries are
keyed on file plus exact source text, without line numbers, so an edit above an
entry does not silently invalidate it.
Locally the guard reports 19 matches before the sweep and 2 after, both
allowlisted.
The Throttle bullet in the coding conventions is rewritten from "prefer
Throttle for rate limiting" to "never compare against millis() directly", lists
all four helpers with when to use which, names the CI guard, and documents the
sentinel hazard with the rebootAtMsec = -1 case that would have become a reboot
loop. Mirrored into AGENTS.md; CLAUDE.md gets a pointer row.
* Trim rollover comments to what the code needs
The comments added with the millis() rollover fixes carried too much of the
investigation that produced them: how many sites were found, which document
recorded them, what the old code used to do. That belongs in the commit history,
not in the source, and some of it was already stale - Power::reboot() still
described the check it disarms as "a naive millis() > deadline" when that
comparison had been fixed in the same series.
What stays is the non-obvious part at each site: which sentinel value the
variable overloads and what it means there, since that differs between call
sites and is what a correct comparison gets wrong. 0 means "not scheduled" for
rebootAtMsec, "renew now" for ntp_renew, and "show indefinitely" in
isOverlayBannerShowing().
Exposition is kept where it earns its place: the Throttle helpers, the uptime
clock's note on why there is no 64-bit variant, and the tests. The Throttle
docs lose only the site count and the "longest interval in the firmware"
statistic, both of which would age badly; the range trade-off between the two
forms is what a caller actually needs.
Comments only - no code changed, verified by diff.
* possible fixes
* Address review feedback on the rollover fixes
- BME680Sensor: checkpoint lastStateSaveMs after a successful write instead of
at the interval test. The first save (IAQ accuracy >= 2) left it at 0, timing
the next save from boot, and stamping before the write deferred the retry a
full period when the write failed. Reads Time::getMillis(), the same clock
Throttle compares against.
- Throttle: add deadlinePassedAt(now, deadline) for loops that snapshot the
clock once and test many deadlines; deadlinePassed() now delegates to it.
NextHopRouter::doRetransmissions() uses it, replacing the inline half-range
compare adopted from #10227 (nightjoker7) - same arithmetic, credited at the
call site - and takes its snapshot from Time::getMillis() so setNextTx()
deadlines and the due test cannot diverge under an injected test clock.
- test_native.yml: set -euo pipefail in the millis-deadline guard, matching the
sibling suite-count job. Without -e a partially failed scan could report "no
violations" from truncated output.
- test_packet_signing: build the not-due deadline from Time::getMillis() rather
than millis(), so the test and the router read one clock.
- test_throttle: cover deadlinePassedAt(), and correct a wrapped-value comment
(0xFFFFFF00 + 400 is 0x00000090, not 0x00000094).
Two review comments were declined: the AirTime mutex (every airTime-> caller
runs in the single cooperative loop, WebServerThread included) and the
MotionSensor 0-sentinel countdown (the calibration frame is only installed
while a window is open).
clod helped out here
* Correct the described failure window of a naive millis() compare
The comments and agent docs said a bare `millis() > deadline` "breaks for ~24
days after the wrap". That figure belongs to the fix, not the bug: it is the
half-range limit of deadlinePassed(), which reads deadlines more than 2^31 ms
ahead as already passed, and the range over which a UINT32_MAX sentinel reads
as passed.
The naive compare's actual failure is an inversion lasting only while the
deadline sits on the far side of the wrap, so it is bounded by the interval:
the action fires immediately and loses its wait, or blocks for about the wait
it should have performed - days for the nRF52 flash-corruption backoff,
one skipped cycle for a seconds-long retransmit timer.
Comments and docs only; the ~24.8 day statements that correctly describe
deadlinePassed()'s own range are left as they were.
clod helped out here
* Restore a monotonic uptime clock and consolidate the wrap counters
Time::getMillisMonotonic() is the getMillis64() shape - a 32-bit wrap
counter carried across reads - promoted to the shared timebase, with
Time::getUptimeSecs() as the derived whole-seconds view. This deliberately
reverses the earlier removal of getMillis64(), and the distinction matters:
removal was right for a lazily-read accumulator with one rare caller, where
a 49.7-day gap between reads silently swallowed a wrap. Here every read is
the poll and AirTime::runOnce() guarantees one per second; the missed-wrap
contract is pinned by a test rather than left as a footnote.
Three private wrap counters collapse into it:
- AirTime::syncNow() takes its seconds from Time::getUptimeSecs() and drops
its lastSyncMsec checkpoint; window rotation is unchanged.
- DeviceTelemetryModule loses refreshUptime()/uptimeWrapCount/uptimeLastMs;
uptime_seconds comes from Time::getUptimeSecs(), which also removes the
0.296s-per-wrap truncation of (0xFFFFFFFF / 1000) * wraps. Its two
interval checks move to Throttle::hasElapsed().
- HostMetricsModule's copies of those members were never read (its uptime
comes from /proc/uptime) - deleted.
Not ISR-safe (unguarded mutable carry): ISRs keep using getMillis(), which
stays a pure read. Audited: no interrupt-context file reads getTime(),
getValidTime(), or the new accessors.
test/native-suite-count 44 -> 45: the bump for test_uptime_clock was lost
in a branch history rewrite, leaving every later value off by one -
run-tests.sh reports AMBER and CI's suite-count-check fails on the current
push until this correction.
* Anchor the wall clock in monotonic milliseconds
getTime() computed elapsed-since-time-set as a 32-bit millis() delta, so a
node that took time once and stayed up past 49.7 days reported a wall clock
one full cycle in the past - and last_heard, rx_time, message and position
stamps all inherited it. The anchor is now the 64-bit monotonic count
(timeStartMsec -> timeStartMs64) and the elapsed term is computed in 64-bit,
so the wall clock is exact at any uptime.
All six anchor writers follow: the five hardware-RTC read branches and
perhapsSetRTC(), which keeps a truncated 32-bit copy of the same instant for
its Throttle-checked rate-limit stamps. The test seams anchor the same way.
Two native regression tests drive getTime() across the wrap through the
Time seam - one anchored before the wrap and read after it, one anchored
after a counted wrap - with the test epoch derived from BUILD_EPOCH so the
plausibility window cannot rot as the build date advances.
* Stamp the rx_time placeholder in monotonic uptime seconds
computeRxTimeStamp() stamped Time::getMillis() when the clock was untrusted,
and reconcilePendingRxTimes() back-calculated with a 32-bit millis() delta -
correct within one wrap, but a placeholder older than 49.7 days aliased to a
small elapsed value and reconciled to a plausible-but-wrong recent epoch:
the exact failure has_rx_time exists to prevent, reachable by an ordinary
unattended router whose phone connects two months in.
The placeholder is now Time::getUptimeSecs(). Both stamps come off the
monotonic counter, so the elapsed term is exact at any age and the aliasing
window is gone outright rather than widened. If elapsed somehow exceeds the
epoch itself, the packet stays un-dated (absent, never wrong) instead of
clamping to a pre-1970 value. Defence in depth: a placeholder that leaks
needs ~50 years of uptime to cross MIN_PLAUSIBLE_EPOCH, where milliseconds
took 18.3 days.
The stream-API reconciliation tests keep their scenarios with the placeholder
unit switched, and ScopedTimeFixture resets the monotonic carry so uptime
seconds are deterministic per case.
* Date nodes heard before the clock arrives, without polluting last_heard
A node first heard while the wall clock was untrusted got no last_heard at
all, and nothing backfilled it once time arrived - the phone showed "Last
heard: unknown" for a node it had just announced. The arrival instant now
waits in a RAM-only sidecar (NodeNum -> uptime seconds, 32 slots,
reuse-oldest - the RouteHealth shape) and is converted to a real epoch on
the clock-becoming-trusted transition, beside the existing rx_time
reconciliation. last_heard itself never holds anything but a real epoch or
0: it persists to flash and the warm tier, where an uptime-relative value
would be meaningless after reboot.
The sidecar's write sites are updateFrom()'s no-trusted-clock path (the
rx_time placeholder already carries the arrival instant, so this is a store,
not a second clock read) and addFromContact's anti-eviction stamps, which
previously wrote a bare getTime() - boot-relative seconds on a clockless
node, the exact value lastHeardIsWallClock() exists to catch. Eviction
ranking honours the stamps: heard-this-boot outranks every stored epoch,
ordered among themselves, so a stamped contact is not the first victim.
PhoneAPI re-reads last_heard at nodeinfo send time: a record prefetched
before the clock became trusted can carry 0 while the store has since been
backfilled, and re-reading at the pop makes handshake ordering (time-set vs
node-list download) irrelevant. Backfill never moves last_heard backwards
and skips the pathological elapsed-exceeds-epoch case. A node evicted to
the warm tier before time arrives is still absorbed with last_heard 0 -
same as before, bounded to the untrusted window.
* Update the agent docs for the monotonic timebase
The conventions bullet asserted there is deliberately no 64-bit millis; the
monotonic uptime clock restored for timestamps changes that contract. State
the split explicitly: Throttle for deadlines and intervals (no carry state),
Time::getMillisMonotonic()/getUptimeSecs() for timestamps, polled by
construction and not ISR-safe.
* Publish the monotonic wrap carry from a single writer
getMillisMonotonic() was a read-modify-write on two unguarded statics, and it
is reached off the main loop: the nRF52 Bluefruit task via
onFromRadioAuthorize() -> PhoneAPI::getFromRadio -> getValidTime(), and the
portduino civetweb workers via the same path. Two readers interleaving inside
the wrap window could each increment the carry, putting every uptime and
wall-clock reading 2^32 ms ahead for the rest of the boot - a permanent ~49.7
day jump in rx_time, last_heard and ClientNotification.time.
Readers no longer write. serviceMonotonic() publishes a snapshot behind a
seqlock and is the only writer; a reader adds its own unsigned elapsed time to
that snapshot, which is exact across the wrap, so it never inspects the
boundary and cannot miscount it. The main loop publishes every iteration, so
the once-per-49.7-days obligation now has the whole window of margin instead of
resting on an instruction-wide race.
AirTime was the guaranteed poller and is now a pure reader, so the two airtime
wrap tests step the clock the way loop() does. The test clock itself is atomic
so a suite can drive it from one thread while others read.
* Re-arm the GPS ephemeris hold when none is in force
The rollover sweep guarded the hold re-arm with `fixHoldEnds != 0 &&`, which
reads like the sentinel rule but inverts this site. The comparison it replaced,
`(fixHoldEnds + GPS_THREAD_INTERVAL) < millis()`, was always true when nothing
was armed - that was the point, since 0 means "not holding" and so is a reason
to arm. With the guard, a publish that cleared the hold without sleeping (the
`shouldPublish && !tooLong && !holdExpired` path, which does not call down())
left hasValidLocation set and prev_fixQual non-zero, so no disjunct held:
nothing re-armed, nothing published, and the receiver stayed powered at the
200ms poll until searchedTooLong() fired.
State the question positively instead. fixHoldInForce() is the only place the
sentinel is interpreted, and both of runOnce()'s decisions derive from it - the
asymmetry is now visible rather than implied, since arming does not require a
prior hold but expiring does. Its `!= 0` test is not redundant with the
arithmetic: deadlinePassed() is an unsigned half-range test, so past 2^31 ms of
uptime the sentinel reads as a deadline ~24.9 days in the future.
Kept beside its caller rather than in a header; the native test build compiles
GPS.cpp, so the suite declares the prototypes.
Also converts the getACK() wait to isWithinTimespanMs(start, interval): it has
both the start instant and the interval in hand, which gives the full 49.7-day
range instead of 24.8 days ahead, and takes its anchor from Time::getMillis()
so the wait is injectable.
* Date the NodeInfo reply window in uptime seconds
The 12h reply-suppression stamp regressed from wrap-immune 64-bit seconds to
raw 32-bit milliseconds, and pruneLastNodeInfoCache() evicts only by node count
and DB membership - never by age. A stable mesh under the node cap therefore
keeps every stamp indefinitely, and once uptime passes 49.7 days an old one
aliases back into the window: `now - stamp` computes as ~0 and a legitimate
NodeInfo request goes unanswered for up to 12h. It self-heals and repeats once
per wrap cycle.
Store Time::getUptimeSecs() instead, which does not wrap for 136 years, and
drop the millisecond conversion the previous shape needed. Entries past the
window are now evicted too: they can only ever decide "don't suppress".
N8-N11 cover the window from both sides, and N10 pins the regression - it needs
a full 2^32 ms of uptime to elapse, not merely a crossing of the boundary,
because that is when a millisecond stamp reads as "answered this instant".
tearDown() now restores the injected clock and C14's region and TX bucket. A
failing assertion aborts the test body, so restoring at the end of it leaked
that state into every later case.
* Update the agent docs for the single-writer clock and sentinel direction
Two rules the preceding three commits changed.
The monotonic clock is no longer maintained by whoever happens to read it:
serviceMonotonic() is the only writer, readers are pure, and calling it from
anywhere but the main loop reintroduces the double-count.
The sentinel guidance gained the half it was missing. It named UINT32_MAX as a
sentinel while prescribing an idiom that only covers 0, and it assumed the
sentinel always means "suppress" - at the GPS fix-hold site it meant "fire",
which is how that regression passed review looking like the rule.
* Name the fix-hold expiry predicate and arm it from the injected clock
holdJustExpired() gives the second reading of the fixHoldEnds sentinel a
name beside the first, so both are pinned by test/test_gps_fix_hold/ and
neither can be respelled at the call site. The old inline form could not
be tested: written as a literal, its guard folds at compile time and the
assertion asserts nothing.
The arm site used bare millis() while the evaluation reads the Throttle
clock; same value in production, but it kept that write out of reach of
Time::setTestMillis(). Remap a deadline that lands on 0, which would
otherwise read as no hold at all.
* Share the extend formula between the clock's reader and writer
getMillisMonotonic() and serviceMonotonic() carried byte-identical wrap
arithmetic. A one-sided edit to either would drift the published carry
from what readers report, so keep one copy.
* Trim the NodeInfo dedup comment to the house limit
* todo note for potential future imrpovments
* fix some simple deadlines
* Trim the hold-expiry test comment to the house limit
* Fix non-blocking uptime publication and pre-clock recency edges (#29)
* fix(time): avoid blocking monotonic readers
* test(time): make paused-publisher check deterministic
* fix(time): address review portability gaps
* Init the eviction sentinel to the newest possible recency
EvictionRecency{} is {0, false}, which evictionRecencyOlder() ranks as older than
every candidate: without the oldestIndex/oldestBoringIndex guards nothing would
ever be selected and a full node DB would stop evicting entirely.
Init to the genuine maximum instead, so the sentinel is correct on its own. The
index guards stay: two independent reasons the scan is right beats one.
* Keep the deadline-guard check name branch protection matches
The guard was widened to cover Time::getMillis() and unqualified getMillis(),
and renamed to suit. Upstream branch protection matches required checks by name,
so a rename means the old name never reports and merges block on a check that
will never arrive.
Widen the guard, keep the name; the descriptive text carries the broader scope.
* Correct native-suite-count to 47 after the develop merge
Upstream #11293 added test_nmea_wpl and took develop's count to 43; this branch
had independently reached 46. Merging develop resolved the counter textually,
keeping 46, while the directory set became the union of both sides at 47.
The suite-count CI gate fails on the mismatch, and it gates the native test jobs,
so the tests themselves were being skipped.
* test(uptime): make the wrap fall where the comment says it does
The concurrent-reader case started at 0xFFFFF000, leaving 0x1000 to the wrap, so
the 0x800 advance annotated "cross the wrap" fell short and the wrap actually
happened during the following 60s advance.
Start at 0xFFFFF800 instead, so the first advance lands exactly on the wrap while
the readers are running and the second is the ordinary time after it - the shape
both comments already described. Total elapsed is unchanged, so the closing
assertion still holds.
* Respond to human comments
* Did I ever tell you about the time I went to Shelbyville? I wore an onion on my belt, which was the style at the time.
* Convert the I2S nag deadline develop dragged in
The HAS_I2S_SPEAKER_NRF52 RTTTL block arrived from develop with a raw
nagCycleCutoff >= millis(), which the deadline guard rejects. Use the same
Throttle::deadlinePassed() form as the two sibling paths in this function.
* Arm the LittleFS format guard with a flag, not a zero timestamp
preFSBegin() runs in the first millisecond of boot, so millis() can legitimately
return 0 there. Both readers of last_format_ms treated 0 as "nothing formatted
this boot", which would skip the repeat-corruption escalation and let a dead
flash reformat-loop instead of reporting FLASH_CORRUPTION_UNRECOVERABLE.
* Note the single-thread contract on AirTime
* Note the AirTime locking TODO, and tighten the thread note
The two constant getters are not constrained, and getSilentMinutes() reads the
buckets without rotating them, so "the accessors mutate" was not accurate.
* trunk: ignore trufflehog false positives on millis-wrap test constants
test_throttle and test_uptime_clock pin dense clusters of hex boundary
constants (0xFFFFFF00u and neighbors) to exercise 32-bit millis()
rollover. trufflehog's Lob detector stitches nearby hex literals into
one candidate string, and the result happens to match a Lob API key
shape - not a secret, just test fixtures.
Same pattern already used for the gitleaks/nodedb-fixture false
positive in this file.
---------
Co-authored-by: nightjoker7 <mattdeering7@gmail.com>
Co-authored-by: Clive Blackledge <clive@ansible.org>
Co-authored-by: Benjamin Faershtein <119711889+RCGV1@users.noreply.github.com>
Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
|
||
|
|
54d6ce833e |
gps: avoid pow() in GPS_HARDSLEEP threshold heuristic (#11179)
* gps: avoid pow() in GPS_HARDSLEEP threshold heuristic GPS::down() used pow(seconds, 1.22) to pick between GPS_SOFTSLEEP and GPS_HARDSLEEP - a curve fit the surrounding comment already describes as "not particularly accurate". On flash-constrained builds where this was the only pow() call site (e.g. wio-e5), it single-handedly pulled in the full double-precision libm pow/rem_pio2 chain for a heuristic threshold decision. Replaces it with gpsHardsleepThresholdMs(), a piecewise-linear lookup over the same curve, sampled at 16 points and verified to track the original formula within ~0.5% for inputs >=10s and ~1.6% for 5-10s (worse only in relative terms below 5s, where the absolute difference is at most a couple of seconds - negligible against update intervals measured in tens of seconds to hours). Signed-off-by: Andrew Yong <me@ndoo.sg> Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> * gps: trim comments to repo convention (1-2 lines) Addresses a CodeRabbit nitpick: the explanatory comments in GPSUpdateScheduling.cpp and test_gps_update_scheduling/test_main.cpp had grown into multi-line blocks with provenance detail that belongs in the commit message, not inline. Trims each to 1-2 lines, keeping only the essential rationale/bounds. Signed-off-by: Andrew Yong <me@ndoo.sg> Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> * Refactor main function to setup and loop for tests Signed-off-by: Thomas Göttgens <tgoettgens@gmail.com> * gps: extend the hardsleep threshold table below 5s and tighten its tests The 0s-to-5s chord read 42% high at 1s, 22% at 2s and 12% at 3s, against the ~1.6% the comment claimed. Adding 1s, 2s and 3s sample points brings the worst error below 10s to 1.60% at 7s. Above 10s it is 0.55% at 728s, unchanged. Tests: sample off-breakpoint values only, including both worst-error inputs (7s and 728s). Replace the 3000ms absolute floor, which made the 1s assertion unfalsifiable given a true value of 2750ms, with 2% and 0.75% bounds. Add breakpoint-exactness and clamp-boundary coverage. --------- Signed-off-by: Andrew Yong <me@ndoo.sg> Signed-off-by: Thomas Göttgens <tgoettgens@gmail.com> Co-authored-by: Austin <vidplace7@gmail.com> Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com> |
||
|
Andrew Yong
|
2f6906974e | gps: replace GeoCoord::latLongToMeter's spherical trig with equirectangular approximation (#11184) | ||
|
Jonathan BennettandClaude Fable 5
|
af56a11f00 |
Replace native-suite-count file with dynamic test discovery (#11413)
* Derive the native suite count on the fly instead of registering it in a file test/native-suite-count was a manually-maintained register of the test_* directory count, reconciled against the actual directories by bin/run-tests.sh (as an AMBER verdict) and by a dedicated suite-count-check CI job. The reconciliation only ever guarded the file itself: the check that matters - suites that actually ran vs. the test_* directories on disk - already derives its expected count from a directory walk, so the file added a bookkeeping step to every suite addition/removal without adding signal. Remove the file and everything that existed to keep it honest: - bin/run-tests.sh: drop the canonical-count file read, the count-mismatch AMBER verdict, and the [canonical: x/y] suffix; the verdict lines already carry ran/expected from the directory walk. The shuffle seed suffix stays. - test_native.yml: delete the suite-count-check job and its needs: edges. - Docs (copilot-instructions.md, AGENTS.md, test/README.md) and the test-script comments now describe the count as derived from test/test_* at run time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Add suite-shrinkage-check: fail a PR that silently loses a test_* suite With test/native-suite-count gone, nothing in CI noticed the suite set shrinking: platformio test discovers and runs whatever test_* directories exist, and bin/run-tests.sh derives its expected count from the same walk, so a suite directory lost in a bad rebase or an overzealous cleanup just means fewer suites run - every remaining check stays green. Restore that tripwire git-aware instead of file-based: on pull_request runs, compare the test_* directory list at the PR's merge base against the PR result. A vanished suite fails the job unless its name appears in the PR title, PR body, or a commit message in the PR's range - a deliberate removal satisfies that by stating what it removes; an accidental loss cannot. Other events skip: they have no natural base, and PRs are where accidents arrive. No job depends on this one (a skipped job would skip its dependents). Incidentally: test/ currently holds 47 test_* directories while the deleted count file said 46 - the manual register had already drifted, which is exactly the bookkeeping failure mode this replaces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Re-pad the verdict table after shortening the AMBER row Shrinking the AMBER cell left the table's column padding inconsistent, which trunk (prettier + markdownlint MD060) rejects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Jonathan Bennett
|
d765bd99ca |
Fix the all-zero MAC address for own node (#11409)
* Fix the all-zero MAC address for own node * Increment native suite count from 46 to 47 * Fix condition for copying MAC address in PhoneAPI |
||
|
|
546b9d9e40 |
Block coordinate traffic on configured event channels (#11045)
* Block coordinate traffic on configured event channels Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Suppress event coordinates in reliable relay paths Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Reject blocked phone coordinates before rate limiting Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Prevent event coordinates from reaching MQTT Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Add event coordinate policy preference Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Test event coordinate policy in native CI Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Make event policy test tolerate a full NodeDB Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Test Router event coordinate enforcement Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Test PhoneAPI event coordinate retry handling Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Test reliable event coordinate suppression Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Test MQTT event coordinate suppression Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * Run event policy behavioral suites in native CI Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * tests: address CodeRabbit review feedback - test_event_channel_phone_api: complete the setUp/tearDown save-restore pair. GlobalState now carries cryptLock and myNodeInfo; setUp() nulls cryptLock before constructing MockRouter (Router's ctor asserts it is unset), and tearDown() restores both so the suite leaves no global mutated. Not reachable today - the globals start null in this binary - but the pair was asymmetric. - Replace the strcpy calls this branch added on Channel.settings.name (char[12]) with the bounded form the rest of the test tree already uses, strncpy(dst, src, sizeof(dst) - 1). Covers the flagged site in test_nexthop_routing plus the six equivalents in test_event_channel_phone_api, test_mqtt and test_position_precision, which trip the same ast-grep dangerous-buffer-functions-cpp rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Jonathan BennettandClaude
|
b8dee13d0b |
fix(BaseUI): never render banner font tags ([S]/[M]/[L]) as literal text (#11392)
The nRF52 BLE pairing banner sends "Bluetooth\nPIN\n[M]<pin>" where [M] is a medium-font marker for the PIN line. The renderer only honored font tags for lines covered by the parsed-line cache, so any path that draws a banner line from the raw message - a stored-without-reparse banner, or a draw racing the parse from the BLE task - showed a literal "[M]" ahead of the pairing PIN. Extract the per-line text/font decision into resolveBannerLine(): parsed (tag-stripped) line when the cache covers it, otherwise strip a leading font tag from the raw line on the fly and honor the font it names. Picker content (e.g. node names) is deliberately exempt - it can be user data and must never be tag-interpreted. Also measure line widths on the text actually rendered (they were measured on the raw, un-stripped string, skewing box width and centering), and check p[1] for NUL before reading p[2] in the tag probe, which could read one byte past the end of a line that ends in '['. Add a native test suite (test_banner_font_tags) covering tag parsing, the BLE pairing message, the cache-miss fallback, and the picker exemption. Claude-Session: https://claude.ai/code/session_01CFMdkE6d8fY28Ax3hFVDMU Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Tom
|
de6b23190a |
Test suite rebuild (#11322)
* docs(nodedb): make the native node cap unambiguous The native node cap was stated in four places that disagreed, and the disagreement already caused a wrong diagnosis: a saturated 200-node database looked arithmetically impossible because the cap had been read as 248, computed from a header that does not apply on this platform. The real value is 198. On portduino MAX_NUM_NODES is not a compile-time constant at all - the variant defines it as `portduino_config.MaxNodes`, resolved at runtime, default 200 and settable per host with `General: MaxNodes`. variant.h is reached before mesh-pb-constants.h, so that header's ARCH_PORTDUINO branch never fires and its plausible-looking 250 is dead code. - #error-guard the dead branch rather than leave a wrong number where people grep. The guard found a real defect: seven translation units reach mesh-pb-constants.h without configuration.h (SerialConsole.cpp, StreamAPI.cpp, PacketAPI.cpp, ServerAPI.cpp, PiWebServer.cpp, ServiceEnvelope.cpp, MeshtasticOTA.cpp, and test/TestUtil.cpp), so each was compiling with a different MAX_NUM_NODES - and therefore a different PACKETHISTORY_MAX - than the rest of the build. Each now includes configuration.h first. It cannot be included from mesh-pb-constants.h itself: that reaches SerialConsole.h through DebugConfiguration.h and closes a cycle. - Name the bare 250 in getMaxNodesAllocatedSize() NODEDB_MIGRATION_LOAD_CEILING. It is a decode allowance for files written by larger-cap firmware, not a cap, and it read like one. - Fix docs/node_info_stores.md, which named the wrong source and a "10-250" range that is wrong for native, and the copilot-instructions tunables line that said "portduino 250". * test(harness): give each suite its own scratch HOME and report leftovers Native suites shared one directory. Every suite that constructs a NodeDB loads and saves ~/.portduino/default/prefs/ - nodes.proto, config.proto, channels.proto, module.proto, device.proto, warm.dat, transmit_history.dat - and nothing cleared it, so state leaked suite -> suite within a run and run -> every run after it. A test run could also rewrite a real meshtasticd node database on the same machine. Per-run isolation does not fix this: the leak is generated inside a single run, so the boundary has to be per suite. bin/pio-test-isolate.sh runs each suite in its own scratch $HOME, registered as test_testing_command for env:native and env:coverage so a bare `pio test` and CI get the same boundary, not just bin/run-tests.sh. It runs the binary unchanged and exits with its exit code, so PlatformIO's pass/fail is untouched. Overriding HOME here rather than around `pio` also sidesteps the blocker that a bare HOME= breaks pio's own ~/.platformio/penv/bin/pio lookup. Leftovers are reported as a second axis, PASS/FAIL x CLEAN/DIRTY, because an unintended write has no matching assertion by definition - nobody writes TEST_ASSERT for a save they do not know is happening. The harness asserts it from outside, so it applies to every suite without the author opting in. - Only the *set of changed paths* is asserted, never contents. Hashes answer the boolean "did this change?" and nothing more; content baselines over protobuf bytes would churn on every NodeInfoLite field added, which is how snapshot suites become noise. - Deliberate writes are declared in test/state-manifest.tsv - one central file, suite / flags / mandatory reason. run-tests.sh prints the opt-out count on every run. - Granularity follows the state flag, so the two ship together: per-test by default (TestUtil redefines RUN_TEST to checkpoint after each test, naming the exact test that dirtied things), suite boundary for state=per-suite, where carrying state across test cases is the declared behaviour. - A declared write that does NOT happen is reported as MISSING, not folded into DIRTY. It catches silently broken persistence; a warning for now, since some are conditional. - Graded AMBER, not RED. With isolation in place DIRTY means "undeclared", not "dangerous", and a check that lands red on day one gets switched off. Guard the guard, both halves: state_assert_empty() refuses to run a suite against a sandbox that is not empty (otherwise the after-diff measures against the wrong baseline and reports CLEAN while meaning nothing), and bin/test-state-check.sh drives the real wrapper with fixtures asserting CLEAN / CLEAN / DIRTY / MISSING plus both directions of the empty assertion. A checker that silently matches everything would otherwise pass forever. --write-manifest proposes entries for a human to paste and justify; it never applies them, and neither does CI. * test(harness): stop reporting Unity's exit code as a signal A native suite ends in exit(UNITY_END()), and UNITY_END() returns the failure count. PlatformIO's native runner reads that non-zero exit code as a POSIX signal number, so four failures print "Program received signal SIGILL", five print "SIGTRAP", and the suite is classified [ERRORED] rather than [FAILED]. There is no crash. The signal name tracks the failure count and nothing else - it moved SIGILL -> SIGTRAP when a diagnostic probe added a fifth failure - and it cost hours of hunting a memory bug that did not exist, on an env (native) that carries no sanitizer at all. It also explains the phantom extra test case in the totals: the runner adds a synthetic entry for the signal it thinks it saw. run-tests.sh now says so inline whenever a signal line appears, and the three agent-facing docs say it too. * test(admin): isolate NodeDB and globals per test setUp() did `if (!nodeDB) nodeDB = new NodeDB();` and never deleted it, so 83 of the 85 tests shared one never-reset database and never restored config, owner, devicestate or channelFile. The fixture that does restore them was opt-in and armed by exactly two tests. The setUp comment claiming the rest "set their own config/region state and are unaffected" was not true - the admin handlers under test write all four globals. Route every test through the fixture instead: setUp saves the globals and installs a fresh NodeDB, tearDown restores and deletes it. The two tests that armed it themselves no longer need to. All 85 pass, so nothing was silently relying on the shared state. It costs about 7% of the suite's runtime (a NodeDB construction is a loadFromDisk plus, with a region set, key generation) - worth paying to write the phase 3 tests against a clean fixture rather than 83 tests' residue. Also cap the per-test attribution in the run summary at five entries; the full list stays in the suite's sandbox. * test(fs): cover the bounded file-manifest walk getFiles() runs on every phone sync via STATE_SEND_FILEMANIFEST, and nothing asserted any of its bounding behaviour. It does execute unasserted from test_stream_api's handshakes, but the cap, the depth limit, the wasLimited paths, overlong-path rejection and capacity release were all unguarded. Eight tests, all describing what the code does today: today's code is already correct here, since #10778 landed the by-reference collectFiles(), the 64-entry cap, the strlcpy bounds and the swap-idiom release. They pass on arrival, which is the point - this is the baseline a later change has to leave alone. Two things they do not cover, and cannot: - Moving reserve() outside the __cpp_exceptions guard. Exceptions are on natively, so the #else branch is not compiled. The suite's job there is to prove that change alters nothing observable. - The file.name() null guard. No in-tree backend returns null; the guard is defensive. The manifest-release test pins the swap idiom rather than calling PhoneAPI's releaseFilesManifest(), which is file-local. It asserts capacity() == 0, not just size() == 0 - a size-only check passes on clear(), which is the bug #7924 shipped. Suite count 43 -> 44, recounted against the directories rather than copied. * test(admin): assert node-DB metadata saves skip the radio reload set_favorite_node, set_ignored_node and toggle_muted_node each persist a NodeInfoLite bit and nothing else. MeshService::reloadConfig() gates its region re-derivation and configChanged notification on saveWhat & (SEGMENT_CONFIG | SEGMENT_CHANNELS), so a SEGMENT_NODEDATABASE-only save already skips the live radio reconfigure. Pure characterization - all three pass on develop. Worth pinning because that reconfigure is the path implicated in the WisMesh Tag favourite-node crash, and develop asserts nothing about it: widening the saveWhat mask or reordering the check would currently go unnoticed. Ported from the config-save series along with ConfigChangedCounter (an Observer<void *> counting configChanged notifications, the only externally visible signal that the reload branch was taken) and TEST_NODE_NUM. They join the existing suite, so no suite-count change. * refactor(menu): extract the mute toggle into a named function The node menu's mute action was inline in a banner-callback lambda, and that lambda only ever runs via screen->showOverlayBanner() - which is why nothing in MenuHandler.cpp was reachable from a test. Lift the `selected == Mute` branch into menuHandler::toggleNodeMuted(uint32_t) and call it from the lambda. Behaviour-neutral by construction: same statements, same order, same bare saveToDisk(). The null check moves into the function, so the call site no longer needs its own lookup. Verified by the native build and suite; the byte-identical-image check on a headroom-constrained nRF52 board was not run locally - CI's firmware-size comment covers it. Three tests come with it, all describing today's behaviour: - the bit flips both ways and no configChanged fires (develop never calls reloadConfig on this path); - an unknown node is a no-op rather than a write; - and the segment mask. Flipping one NodeInfoLite bit currently rewrites all five segments via bare saveToDisk(). That is asserted deliberately, with the comment naming it as characterization of a known defect: a pending fix narrows it to SEGMENT_NODEDATABASE, and when it lands this assertion is expected to change, which makes the improvement visible in the diff instead of silent. saveToDisk() is not virtual, so the mask is observed through its effect - remove the five prefs files, toggle, and see which reappear. * docs(test): make every suite count a pointer to the canonical one test/native-suite-count is the registered total and is machine-checked against test/test_* on every full run and by the suite-count-check CI job. Every other statement of the count is a copy that drifts: copilot-instructions said 12, AGENTS.md said 19, and the real number is 44. Replace both literals with a pointer to the file, say explicitly that no document should state the count as a literal, and reframe the two suite listings as descriptions rather than inventories - they carry per-suite information the count does not, so they stay, but nothing should infer completeness from their length. Register the new FS suite in both. * test(harness): randomise suite order, reproducibly Landed last, deliberately. Randomising an order-dependent suite set does not find bugs so much as convert a silent pass into intermittent red, and the first instinct is to revert the randomisation rather than fix the coupling. Phases 1-2 removed the coupling; this keeps it removed. Both runners previously hid order dependence behind a fixed order that happened to differ between them, and neither order was chosen: CI's area rules put admin first, PlatformIO's local discovery is reverse alphabetical and put it last. CI was green by accident. - bin/run-tests.sh --shuffle / --seed <n>. The seed defaults to HEAD's short SHA: one order per commit, so a red is replayable and attributable to the diff instead of flaky, while the project keeps exploring orders. Printed at the start and carried into the RESULT line, so a verdict is replayable from that line alone; the full order is printed on failure, because for an order-dependent failure the order is the diagnostic. - The shuffle is a Fisher-Yates over a MINSTD generator rather than awk's rand(), whose sequence differs between gawk and mawk. A seed that does not reproduce the same order on another machine is not a seed. - Shuffling needs one `pio test -f <suite>` invocation per suite - PlatformIO orders by its own os.walk() over test/ and filters only select - which measures at about 4.7s per suite of extra startup. - CI shuffles its area order, seeded from GITHUB_SHA and printed with the command to replay it locally. Intra-area order stays PlatformIO's; controlling it there would mean per-suite invocations, which is a cost worth deciding separately. Also records the 16 measured entries in test/state-manifest.tsv, each with its reason, taken from a full run's --write-manifest output rather than guessed. * test(default): cover the region-throttle interval overload getConfiguredOrDefaultMsScaled(configured, default, nodes, TrafficType) is the overload every telemetry and position module actually calls, and nothing referenced TrafficType anywhere under test/. All four of its behaviours were unguarded: the no-region guard, the throttle <= 1 short-circuit, the multiply, and the 64-bit overflow clamp. The throttles are real, not hypothetical - EU_866 carries PROFILE_LITE, which sets both positionThrottle and telemetryThrottle to 10, so a change here moves broadcast spacing in that region by an order of magnitude. Each test pins numOnlineNodes at the congestion threshold and uses ROUTER, which never congestion-scales, so the coefficient is 1 and the throttle is the only variable. The overflow case needs a base above INT32_MAX/10, hence three days rather than one. * ci(test): keep pull-request suite order fixed, seed the rest Shuffling the area order on every run - including pull_request - would turn a contributor's PR red for an ordering they did not choose, which is how a randomisation gets reverted instead of the coupling being fixed. That is the exact dynamic the ordering work was sequenced last to avoid, and the previous commit walked straight into it. - pull_request keeps the fixed declared area order. - push and schedule shuffle, seeded from the commit SHA: deterministic per commit, printed, attributable, and never blocking someone else's PR. - A suite_order_seed input on workflow_call and workflow_dispatch overrides both, so a specific failing order can be replayed anywhere, including on a PR. The run log prints which mode it took, the resulting order, and the local command to replay it. * ci(test): satisfy CKV_GHA_7 and yamllint on the seed input The seed is reachable through workflow_call, which callers can pass programmatically. The workflow_dispatch copy tripped checkov's "workflow_dispatch inputs MUST be empty" rule, and suppressing it was not worth it: replaying a specific order is a local operation, and the run log already prints the exact bin/run-tests.sh command to do it. * style(menu): apply the node-ID format convention RadioInterface.cpp documents the rule: 0x%08x in logs, !%08x in user-facing display. MenuHandler held every remaining exception - seven logs printing bare %08X, and two display labels doing the same. Repo-wide there are now no bare %08X node IDs left in log calls. * ci(test): pass workflow inputs through env, not shell interpolation suite_order_seed and github.event_name were spliced into the run: script as ${{ }} text, so a value carrying shell metacharacters would execute as code on the runner rather than being read as data. semgrep (run-shell-injection) and zizmor (template-injection) both flag it. Both now arrive as environment variables and are read as "$VAR". * refactor(test): share the seeded shuffle between the harness and CI bin/run-tests.sh and test_native.yml each carried a byte-identical copy of the MINSTD Fisher-Yates awk. The workflow prints "replay locally: ./bin/run-tests.sh --shuffle --seed $seed" after a shuffled CI run, and that instruction is only true while the two agree - drift would be announced by a replay quietly reproducing a different order than the one that failed. Extract shuffle_suites() to bin/lib/shuffle.sh and source it from both. Permutations verified identical across seeds before and after the move. * fix(test): correct the shared-state MISSING check and summary join Three defects in the new harness: state_classify() matched declarations two different ways - state_path_declared() for "undeclared", a hand-rolled regex for "missing". Interpolating an entry into an ERE also let a metacharacter in a manifest name match a file that is not the declared one. Both directions now go through the one helper. `paste -sd'; '` does not join with "; ": with -s, paste cycles through a multi-character delimiter one character per join, so paths rendered as "a;b c;d e". Replaced with an awk join. test-state-check.sh ran on after a failed cd instead of stopping (SC2164). ./bin/test-state-check.sh: 6/6 fixtures pass, MISSING included. * fix(portduino): bound General.MaxNodes MaxNodes was validated only for <= 0. Any positive value, including a typo'd or pasted-in one, propagates to MAX_NUM_NODES and scales both the node DB and the nodes.proto decode ceiling - failing at boot with no obvious cause. The ceiling is a sanity bound, not a capability limit; raise it if a host genuinely needs more. * docs(nodedb): reconcile the capacity tables The property matrix omitted the ESP32-S3 100-node flash tier that the platform table above it lists, and neither mentioned that the WASM build overrides MaxNodes to 80 in wasm_config_apply(). * fix(nodedb): make mesh-pb-constants.h self-sufficient on portduino The ARCH_PORTDUINO #error assumed it was unreachable in a normal build. It is not: the vendored device-ui sources include this header without configuration.h, which broke both native-tft docker builds. Include configuration.h here instead, ahead of every compile-time default - variant.h overrides MAX_RX_TOPHONE as well as MAX_NUM_NODES, so placing it lower in the file just moves the divergence to a redefinition. The #error stays as a backstop for the case where that include genuinely stops providing the cap. Verified with the native env's own flags: a TU including only this header now compiles, normal-order use of both macros compiles, and NodeDB.cpp compiles. * fix(portduino): raise the MaxNodes ceiling to 16000 Marked artificial: nothing in the node DB fails at 16001. 16000 sits just under the 16384 (128 x 128) population where HopScalingModule saturates its sampling denominator and starts dropping nodes, so a host inside the bound still gets meaningful hop recommendations. * lint(trunk): advise on node IDs logged as bare %08x RadioInterface.cpp documents the convention - 0x%08x in logs, !%08x in display - but nothing enforced it, which is how the MenuHandler cluster drifted. 22 call sites in PacketHistory, NodeInfoModule and PositionModule are still off it. A trunk linter rather than a CI grep job, because trunk checks changed files: new violations get flagged without a 22-site cleanup landing in an unrelated PR. Modelled on the existing too-many-defined definition. Scoped to values it can tell are IDs - an ID-shaped argument (->num, .from, getNodeNum) or message text naming one. A 32-bit hex that is not an ID is out of scope, so the CRC32 logs in ethOTA.cpp are correctly ignored. Emits "note", trunk's only non-blocking level: "warning" and "info" both exit non-zero and would gate CI, which is not what a log-format nit deserves. The pre-existing sites are line-scoped in the allowlist, so a new bad call in those same files is still caught. * lint(trunk): stop exempting the known node-id-format sites The seeded allowlist made the rule green by declaring the backlog acceptable. Empty it instead, so the 22 pre-existing sites are reported and get cleaned up by whoever next edits those files. Costs nothing to do: the rule emits "note", so these are non-blocking either way. The allowlist stays for its real purpose - a value the linter misreads as an ID. * style: log node and packet IDs as 0x%08x Clears the 22 sites the node-id-format linter reports, so the rule starts from zero rather than from a backlog nobody can see - trunk suppresses pre-existing findings by default, so left alone these would not have surfaced on edit the way an empty allowlist implies. Format strings only; no argument or control flow changes. The !%08x user-facing display forms are deliberately untouched - that is the other half of the same convention. * test(harness): build once up front, so suite timings mean something run-tests.sh fused build and run in a single pio invocation, so whichever suite PlatformIO's directory walk reached first absorbed the entire src compile and reported it as its own duration. On a real run that made a 0.03s suite report 13m21s, and hid the build cost from every other number in the summary. Do what .github/workflows/test_native.yml already does: one --without-testing build pass, then run with --without-building. Measured on a full 44-suite run - the build is now a single reported figure and 968 test cases execute in 1.9s, with no suite above 0.084s. Build output goes to its own log rather than $LOG: the outcome regexes match "error:" and "[ERRORED]", so a compiler diagnostic sharing that file would read as a test failure. Both red paths now keep the log they quote from. $LOG and the build log are mktemps the EXIT trap removes, so the three grepped lines were previously all anyone ever saw - and the cause is usually further up than the first [FAILED]. * test(harness): keep the run log on every red path bin/pio-test-isolate.sh already keeps a failing or DIRTY suite's sandbox and log under .pio/test-state/<suite>/. What was missing is the cross-suite view: $LOG is a mktemp the EXIT trap deletes, so run-tests.sh quoted three grepped lines from a file that no longer existed by the time anyone looked. Preserve it as .pio/build/<env>/test-failure.log from both red paths - including "no success summary found", which said "see log" while preserving nothing, and which is exactly the case where the build died before any suite ran and so left no per-suite sandbox either. Cleared at the start of every run, so a green run cannot leave a red one's log lying around looking current. * fix(test): report the real failure count on a shuffled red A shuffled run is one `pio test` invocation per suite, all appending to the same log, so the log carries one PlatformIO "N test cases:" summary per suite. verdict_red() took `tail -1`, which reports whatever the LAST suite did: a failure in suite 3 printed a "0 failed" summary from suite 44 directly under "RED - failures detected:". Sum the summaries instead. A single summary line - every unshuffled run - is passed through verbatim, so the familiar output is byte-identical. The patterns are passed to the awk helper as strings rather than /regex/ literals: awk evaluates a regex literal in argument position as `$0 ~ /re/`, so the callee would receive 0 or 1 and silently sum garbage. * fix(test): do not emit an empty suite name for an empty shuffle `printf '%s\n' "$@"` with no arguments still writes one empty line, and both callers read shuffle_suites through mapfile, so an empty suite list arrived as a single suite named "". Return before the printf when there is nothing to shuffle. * test(harness): state and enforce the Linux host requirement The native harness is a Linux tool: bash 4+ (mapfile), GNU coreutils and GNU find (-printf, md5sum, -executable). Most of that predates this branch - mapfile and both find predicates are already on develop - but none of it was written down, so the requirement was there to be discovered rather than read. Refuse to start on a non-Linux uname instead of degrading. On a BSD userland this would not fail cleanly: it would mis-hash the sandbox and mis-read the suite list, and still print a verdict. A state check that silently measures the wrong thing is worse than one that declines to run. Carrying a per-host fallback was the alternative, and it buys a second code path that nothing in CI exercises. bin/test-native-docker.sh already exists for macOS and non-Linux hosts, and the native-macos PlatformIO env is a build target for meshtasticd, not a test host - the isolation wrapper is registered for env:native and env:coverage only. Documented in the script header, test/README.md, and both agent docs. * fix(test): terminate every suite with exit(UNITY_END()) Two sites across two suites ended on a bare UNITY_END(). That ends the reporting, not the suite: setup() returns, the runtime goes on calling loop(), and the process runs forever. PlatformIO does not notice - it reports a suite from its Unity output, not from process exit - so the suite passes, the run goes green, and the binary stays resident. Thirteen of them had accumulated on one dev box, the oldest 19 hours old. The costs are quiet by construction: - the per-suite sandbox is deleted underneath a live process, so its CLEAN/DIRTY verdict describes what the suite had written when the harness stopped looking, not what it left behind; - .gcda coverage and LeakSanitizer's report both flush from atexit handlers, so a suite that never exits contributes no coverage and gets no leak check; - each survivor pins its own deleted 94 MB binary, which du cannot see. One of the two is the #else of an architecture guard, which is the easiest one to get wrong - it looks like there is nothing to clean up. test_mqtt has a correct exit(UNITY_END()) in its live branch, so a "does this file call exit() anywhere" check passes the file whole. test_serial had two more. develop's serial-config validation rework restructured that suite - the architecture guard is gone and both remaining branches now exit correctly - so this commit no longer has anything to change there; bin/lint-unity-exit.sh, added later on this branch, is what keeps it that way. test/README.md gets a section on it, since the skeleton showing the right shape had not stopped this happening. * test(harness): detect and reap suites that outlive their run A suite that never exits was invisible: PlatformIO reports a suite from its Unity output, so the run stayed green while the binary kept running. Two checks, because they fail differently. Runtime, in bin/pio-test-isolate.sh: the sandbox $HOME is mktemp-unique per suite, so any process still holding it is a survivor of that suite. Matching on the environment rather than a remembered PID identifies one whatever its parentage - a fork, a grandchild, a process already reparented to init - none of which a $! comparison catches. Reaped before the after-fingerprint is taken, so that fingerprint measures a tree nobody is still writing to, and so a run cannot leave processes accumulating on the host. Recorded as a sixth summary column and graded AMBER: the tests did pass, but the CLEAN verdict and the coverage were measured under a false assumption. Author-time, as bin/lint-unity-exit.sh, wired into trunk at "note" like node-id-format: every UNITY_END() must be wrapped in exit(). The rule is per occurrence, and that is the point - a file-level "calls exit() somewhere" check passes test_serial and test_mqtt, which have a correct one in their live branch and a bare one in the #else. Running it over the tree turned up test_mqtt, which the file-level pass had missed. It allows `int rc = UNITY_END(); ...; exit(rc)`, used by test_packet_signing to restore globals between the summary and the exit. That is where the rule gives ground: capturing and never exiting would leak and is not flagged. Flagging a correct idiom would push someone to "fix" working code. bin/test-state-check.sh gains a survivor fixture, asserting the wrapper both reports and reaps - a detector that only reports leaves the host accumulating processes, which is half the harm. 8/8. * fix(lint): make the unity-exit scanner statement-aware The rule judged one physical line at a time, which reports two kinds of correct code as bare: /* a comment that happens to mention UNITY_END() */ <- interior lines were never stripped exit( UNITY_END()); <- exit( and the macro never met On a probe of both, two of three findings were wrong. This is a note-level rule whose whole job is advice, and bin/lint-node-id-format.sh already says why that matters: a false positive costs more than a miss. One that cries wolf gets ignored, and the real finding goes with it. Carry /* ... */ state across lines and accumulate logical statements before testing, with a 12-line cap so one unclosed call cannot swallow the rest of the file - the same structure lint-node-id-format.sh uses, so the two custom linters in bin/ work alike rather than each having its own idea. Verified both directions: the develop-era sources still produce the same four findings, the fixed tree produces none, and a probe covering block-comment interiors, wrapped exit(), line comments, return UNITY_END() and capture-then- exit reports only the genuinely bare calls - including a complete block comment followed by real bare code on the same line, which the state machine has to keep live. Reported by CodeRabbit on #11322. * fix(lint): tokenise instead of pattern-matching, and self-test it Second round of review findings on the same scanner, all confirmed by direct test before changing anything. Six defects, one root cause: layered regexes cannot tokenise C++. False positives (correct code reported): - UNITY_END() inside a string literal read as code False negatives (real leaks missed): - a string containing "/*" opened comment state and swallowed later lines - greedy .* removed everything between two block comments on one line, taking a bare call with it - myexit(UNITY_END()) matched the exit() exemption as a substring - x == UNITY_END() and total += UNITY_END() matched the assignment exemption Replaced with a character-level scan carrying comment state, and token-bounded exemptions: exit must be a whole identifier, and the capture form must be a plain `=`. Raw string literals are still not modelled - there are none under test/, and delimiter tracking for a case that does not occur would be untested code guarding untested code, so it is documented rather than guessed at. Also drops the `return UNITY_END()` exemption. It only terminates from main(), there is no main() under test/, and from a helper it just returns a count. bin/test-lint-unity-exit.sh pins all fifteen cases, every false positive and false negative found in review among them. The rule has been wrong twice in a way that looked fine by inspection; it needed a self-test more than it needed another careful reading. Two further findings in the same review: - bin/run-tests.sh dropped PASSTHRU in shuffled mode, so `--shuffle -vvv` built verbosely and then ran quietly. The shuffled loop now forwards EXTRA_ARGS, which is PASSTHRU minus the -f pair it supplies per suite. - bin/run-tests.sh did not guard `cd "$ROOT_DIR"`. And one that did not reproduce: the survivor fixture's glob does find the pid file (verified with the lookup instrumented - the earlier failure was an artifact of running the script from /tmp, where SCRIPT_DIR cannot resolve). The assertion was still weak, because an empty pid took the "not running" branch and passed vacuously. It now fails if the pid was never recorded, and finds the file by search rather than assuming a directory depth. Reported by CodeRabbit on #11322. * fix(lint): report each UNITY_END occurrence at its own location The self-test only asked "did the linter say anything", so it could not have caught a wrong line, a wrong column, or a missing second finding. Fixtures now assert the exact diagnostics as line:col, and the first run of that assertion found two real problems. The caret pointed at the wrong occurrence. For `exit(UNITY_END()); UNITY_END();` the verdict was right but the column was 17 - the wrapped call - because the scanner stripped terminating forms out of the whole statement and then reported the first occurrence it had seen. Two bare calls on one line reported once. Judged per occurrence now, by looking back through whitespace at what wraps it, so both the count and the caret are right. That also needed a position map from strip_noncode(): removing a comment or collapsing a literal shifts every later column, and counting occurrences in the raw line does not recover it either - TEST_MESSAGE("... UNITY_END() ..."); UNITY_END(); has two occurrences in the raw text and one in the code. Four of the expected columns I wrote by hand were also wrong, off by one. The linter was right in every case; the assertions were not. They are computed from the fixture text now rather than pasted from output, because a baseline accepted from the tool it is testing asserts nothing. 17 fixtures, including the two-on-one-line case from review and its mirror. Reported by CodeRabbit on #11322. |
||
|
Benjamin Faershtein
|
7d54c13ec0 | fix(crypto): hash full size_t inputs (#11359) | ||
|
Ethac.chenandThomas Göttgens
|
08722da3d9 |
fix(lr2021): live LF↔HF reconfigure via full begin() (#11279)
* fix(lr2021): full begin() on live LF/HF reconfigure.Avoid RadioLib -706 / assert when live-switching Sub-GHz ↔ LORA_24. * fix(lr2021): align band-hop begin() with init() robustnessMirror. RF-switch GPIOs, SPI/TCXO retries, and log CRC/RX-gain errors. * chore: trunk fmt LR20x0Interface bandHop line wrap * fix(lr2021): harden live band reconfigure (companion #1) * fix(lr2021): reject invalid freq in band-hop path select Require requestedMHz > 0 in isLr20x0BandHop, expose lr20x0ReconfigurePathfor FullBegin vs incremental selection, and extend native radio tests. --------- Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com> |
||
|
|
03e6b80989 |
Serial config validation (#11339)
* fix(serial): validate serial module config on every platform
AdminModule guarded the serial config validation by architecture but not the
assignment beneath it:
#if ARCH_ESP32 || ARCH_NRF52 || ARCH_RP2040
if (!SerialModule::isValidConfig(...)) return false;
disableBluetooth();
#endif
moduleConfig.serial = c.payload_variant.serial;
So on every other platform an admin "set module config: serial" stored a config
the firmware rejects on ESP32. override_console_serial_port combined with
DEFAULT, SIMPLE, TEXTMSG or PROTO is accepted and persisted today.
Two families are affected, for different reasons:
- portduino/meshtasticd, where the validation did not exist at all:
isValidConfig was a static member of SerialModule, and that class is inside
the same architecture guard, so `nm` finds no such symbol in the native
object.
- STM32WL (rak3172, wio-e5, CDEBYTE_E77-MBL, russell), where it existed and
was never called: the class guard includes ARCH_STM32WL and the AdminModule
call site did not.
Validation is pure config logic with no serial hardware behind it, so it moves
out of the class and out of the guard as a free serialConfigIsValid(). Its only
external references - clientNotificationPool, service, getValidTime - are
already unguarded elsewhere, so it links on every target. AdminModule's include
of SerialModule.h is unguarded for the same reason; the class itself stays
guarded inside the header. Only disableBluetooth() remains architecture-specific.
This changes what meshtasticd and the STM32WL targets accept: a host relying on
the unvalidated path (override_console_serial_port with a mode other than NMEA,
CalTopo or MS_CONFIG) is now rejected, as it already is on ESP32.
test/test_serial has asserted nothing since it was added in
|
||
|
Matias Denda
|
1dde97f807 |
Fix stack buffer overflow in aes_ccm_encr for partial blocks (#11347)
* Fix out-of-bounds write in aes_ccm_encr for partial blocks aes_ccm_encr() writes a full 16-byte AES block to the output before XOR-ing with the input, so a trailing partial block writes up to 15 bytes past the length the caller asked for. Every caller in the tree passes a buffer with enough slack, so nothing misbehaves today, but the decrypt path clears it by only a few bytes. Encrypt into a temporary block and XOR out of it, matching what aes_ccm_encr_auth() and aes_ccm_decr_auth() already do in this same file. The ciphertext is unchanged. * Add regression test for the CCM partial-block write and drop stale workarounds The guard bytes past the caller's buffer catch the overflow without relying on a sanitizer, so the test is meaningful in the native environment too. encryptCurve25519() no longer needs to write extraNonce before aes_ccm_ae(): the call stays inside numBytes now, so the copy after it is the only one required. The comment warning about the 15-byte overshoot no longer describes the code. |
||
|
TomandClaude Opus 4.8
|
e26b6bfc5a |
TMM & Warmstore key source naming (#11119)
* one becomes two * warmstore clarify * Address PR review: key-provenance terminology consistency - Log line now says "not key-proven" (gate is XEdDSA OR manual, not just signer) - Rename markKeySignerProvenForTest -> markKeyXeddsaSignedForTest (sets only the XEdDSA bit) - Docs + test comments: "signer bit" -> "XEdDSA-signed bit" clod helped too * Rename signer-proven -> key-proven for broadened provenance predicate Address PR #11119 review: the copyPublicKey()/copyUser() out-parameter and the cache-path replay gate now report entry->keyProven() (XEdDSA-signed OR manually verified), so the "signerProven" name and "signer-proven" comments were misleading. Rename the public out-param to keyProven, the local cachedKeySignerProven to cachedKeyProven, and update coupled callers, log strings, docs headings, and comments to say "key-proven". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * nitpicks --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Thomas GöttgensandAndrew Yong
|
6367132919 |
Fix the NMEA checksum offset and harden the buffer writes around it (#11293)
* Checksum NMEA sentences from the $ delimiter The PositionLite printWPL() format begins with a CRLF, so the fixed start offset of 1 folded the newline and the $ into the checksum and every sentence went out with a wrong value. Locate the $ instead and stop at the terminator or a \*. * Clamp truncated writes and harden the remaining fixed buffers snprintf returns the length it would have written, so a truncated NMEA sentence made buf + len point past the buffer and bufsz - len underflow into a huge size for the checksum append. Clamp after each write. Also pulls in the rest of #11236: the two remaining Dropzone sprintf calls, the dead strcpy in mt_sprintf that wrote one byte past a zero-size allocation for an empty format, and the 10-byte errcode buffer that INT32_MIN overflows. Co-Authored-By: Andrew Yong <me@ndoo.sg> * Bail out on a zero-sized buffer and cast err for %ld snprintf writes nothing at all when bufsz is 0, not even a terminator, so the checksum helper would run strchr over whatever the buffer already held. Return before touching it. int32_t is not long on every target, so cast before formatting with %ld. Co-Authored-By: Andrew Yong <me@ndoo.sg> * Add NMEA sentence regression tests Covers checksum computation from the $ delimiter for both printWPL overloads and printGGA, zero-sized buffers, and truncated buffers down to one byte. Co-Authored-By: Andrew Yong <me@ndoo.sg> * Tighten checksum parsing and pin the WPL fixture checksum Require exactly two hex digits followed by the sentence terminator, and assert both WPL overloads against a known checksum instead of comparing them to each other. * Bump native suite count to 43 --------- Co-authored-by: Andrew Yong <me@ndoo.sg> |
||
|
TomandClaude Opus 5
|
21e3a583bd |
Yaml check for Meshtasticd (#11224)
* feat(portduino): add `meshtasticd --check` config validator Users hand-writing files in /etc/meshtasticd/config.d/ get no feedback when a key is misplaced, misspelled or duplicated: meshtasticd silently ignores what it does not read, so a broken config looks identical to a working one. Add a --check mode that loads the configuration exactly as startup does, then reports what it found and exits: - Duplicate keys, via the yaml-cpp Parser/EventHandler stream. The Node API cannot see them because the map is already collapsed by the time it exists, and yaml-cpp keeps the FIRST occurrence, so a later override is discarded. - Unknown or misnested keys, against a schema mirroring what loadConfig() reads, with a hint naming the section a stray key actually belongs to. - rfswitch_table validation: unrecognised pins, mode rows whose length does not match the pin list, values that are not HIGH/LOW, and unknown modes. - Cross-file overlap: every .yaml in the config directory merges into one portduino_config, so the file loaded LAST wins, the opposite of the within-file rule. Those files are read in filesystem order, not alphabetical. - A warning when more than one file defines a Lora section: spidev, spiSpeed, gpiochip, DIO2_AS_RF_SWITCH, DIO3_TCXO_VOLTAGE and USB_PID/VID/Serialnum are assigned unconditionally with a default every time one is seen, so any of them not repeated in the last file loaded is silently reset. - The resolved gpiochip/line for each pin, since a line that exists on the wrong chip is claimed successfully and then silently does nothing. Exits non-zero when errors were found so it can also gate CI over bin/config.d/**, keeping one implementation rather than a second schema. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(portduino): flag pins that resolve to -1 in --check A pin key whose value will not convert to a number falls back to RADIOLIB_NC (-1) while still being marked enabled, and initGPIOPin() then trips an assertion inside LinuxGPIOPin rather than failing cleanly. YAML indentation makes this easy to hit by accident: a stray line under "CS: 8" folds into the value as a multi-line scalar, so the file parses, the daemon crashes with a stack trace from a library file, and --check reported "Configuration looks good" while printing "pin -1" two lines above. Report it as an error naming the likely cause instead. Also correct a comment claiming unparseable config.d files are skipped silently. They are not: loadConfig() prints "*** Exception ..." with the line and column. It is the discarded return value, not the diagnostic, that makes the file's absence from the merged config easy to miss. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(portduino): cover `meshtasticd --check` with fixtures and a fuzz suite Adds the tests the config validator was missing, and the checks and fixes that writing them turned up. The theme throughout is configuration that the YAML parser accepts but that does not mean what it looks like it means. Tests ----- bin/test-config-check.sh - 57 assertions driving a built meshtasticd against test/fixtures/portduino-config (50 fixtures plus two config.d trees). A shell test rather than a Unity suite because both behaviours under test are properties of the process: --check is judged by its exit status and printed report, and the "a normal run rejects a bad config" path ends in exit() inside portduinoSetup(), neither of which is reachable from a suite that links one translation unit. Every fixture carries a comment header naming its planted fault and the expected finding, so it can be read on its own. Coverage: * a clean config for each of the ten radio module families (RF95, sx1262, sx1268, LLCC68, sx1280, lr1110, lr1120, lr1121, sim, auto), asserted both findings-free and resolving to that module, so a silent fallback to sim cannot pass * LR11xx rfswitch tables: unrecognised pins, rows longer and shorter than the pin count, levels that are not exactly HIGH, a missing pins list, more than five pins, a scalar table, unknown MODE_ keys, a MODE_ row stranded one level out, and a legal partial table * the PA gain table in both accepted shapes, entries outside the uint16 range it is stored in, and more than the 22 points that are kept * values of the wrong type, split by consequence: the two settings read with no fallback stop meshtasticd starting, everything else is silently replaced by its default * out-of-range and unit mistakes: TCXO voltage written in millivolts, ports outside their usable range, an over-long StatusMessage * MAC sources: both keys set at once, a malformed address, an interface that does not exist * structural faults: duplicate keys, non-mapping and unknown sections, a key left at the top level, a sequence at the document root, an empty file, unreadable pins, unparseable YAML * cross-file behaviour over a config.d directory, including the switch tables that do not override each other * five configs run WITHOUT --check, each of which must still be refused, so check mode cannot quietly make the normal path permissive test/test_fuzz_config - adversarial fuzzing of the checker itself, the "the tool meant to diagnose your config crashes on it" failure mode. Scope is deliberately narrow: yaml-cpp does the parsing and is fuzzed upstream, so what is exercised here is our code above the parse, above all the duplicate-key detector, which is the one hand-rolled piece and walks the raw parser event stream with its own stack. Groups: the checked-in fixtures as a seed corpus, 3000 byte mutations of them (flips, truncation, insertion, splicing, deletion), and structural torture (nesting to 4096 in flow and block style, duplicate keys at depth, anchors, aliases and merge keys, 64KB keys, 256KB scalars, multi-document files). A fourth group of random bytes is present but disabled behind FUZZ_CONFIG_RANDOM_BYTES: it was half the runtime for the least return, since uniform noise is rejected on the first token. The contract is crash-freedom and termination under AddressSanitizer, not any particular finding. CI runs the shell test in the existing native simulator job; the fuzz suite is picked up by the existing ^test_fuzz_ area rule. native-suite-count 40 -> 41. The fixtures are exempt from trunk in .trunk/trunk.yaml, since prettier rejects the duplicate keys and bad indentation that are the point of them. Checker fixes found while writing the tests ------------------------------------------- --check reported a clean exit 0 on configs meshtasticd then refuses to boot, the worst failure a diagnostic tool can have. Four hard exits inside loadConfig() killed the report before it printed: an unparseable file, an unknown Lora.Module, MACAddress and MACAddressSource both set, and HUB75 on a build without it. All are now reported as findings, and all are still refused on a normal run. New validation: Lora.Module against the accepted spellings, which are matched exactly and inconsistently cased, with a suggestion when only case differs; a per-key value type table covering ~85 keys, tested by asking yaml-cpp to perform the same conversion loadConfig() will so it cannot drift; the PA gain table; DIO3_TCXO_VOLTAGE, which is in volts and multiplied by 1000, so the millivolt value everything else uses silently asks for 1800V; APIPort and Webserver.Port ranges; MaxNodes; StatusMessage truncation; MAC address and source; and an unreadable ConfigDirectory. Also fixes a crash: a ConfigDirectory that cannot be read threw an uncaught filesystem_error from directory_iterator and aborted meshtasticd with SIGABRT, taking --check down with it. It now fails cleanly. Two smaller ones: cppcheck's uselessCallsSubstr on the ancestor walk, which was failing every check job; and the duplicate-key detector's stack pop, which was unguarded and relied on yaml-cpp emitting balanced events. Switch tables are the one place "the file loaded last wins" is false. The loader only ever writes HIGH and never writes LOW back, so a HIGH from an earlier file survives a later file that clears it and the radio drives the OR of every table loaded. Confirmed with --output-yaml. Reported as an error for now; the loader itself is left alone, as that changes RF behaviour. * fix(portduino): report CH341 pins as adapter indexes, not gpiochip lines --check printed "Resolved GPIO lines (what meshtasticd will try to claim)" for every config, listing a gpiochip and line for each Lora pin and advising they be confirmed against gpiodetect and gpioinfo. For spidev: ch341 every part of that is false. portduinoSetup() skips initGPIOPin() for every Lora pin when spidev is ch341 and hands the raw numbers to Ch341Hal, so nothing is claimed from a gpiochip -- and on Windows and macOS, where a USB adapter is the only way to attach a radio, there is no gpiochip, gpiodetect or gpioinfo to check against in the first place. The checker had no ch341 coverage at all: not one fixture used it, so the whole USB-SPI path went unexercised. The summary now splits on the transport. A ch341 device gets its pins listed as adapter indexes with the gpiod advice dropped, and a gpiochip or line mapping written alongside it is reported: those are read, stored, and never used. Also: "RF switch table: not set" read as a gap on an SX126x, where there is nothing to set. setRfSwitchTable() is only ever called for an LR11xx, so absence is now "not needed for this module" everywhere else, and "not resolved yet" for auto, which has no module to judge against. Fixtures: usb-ch341.yaml (clean, the meshstick shape) and ch341-gpiochip.yaml. CI fix ------ test-native was RED on "config.d overrides are reported", which wanted 2 warnings and got 1. The fixture's two config.d files name different modules, so which one wins -- and whether the LR11xx-without-a-switch-table warning fires -- depends on the order the filesystem returns them in. That is the very thing the fixture exists to demonstrate, so the count is no longer asserted; the report's own order caveat is asserted instead. Review fixes ------------ The unreadable-ConfigDirectory diagnostic was the one new print in PortduinoGlue.cpp not gated behind !configCheck, so it landed ahead of the report header and broke the clean output the rest of the change is careful to keep. Docs: rfswitch-valid.yaml carries seven modes, not eight, and empty-file.yaml is comments-only rather than zero bytes. * style(portduino): trim --check comment blocks and reconcile suite count Condense the multi-paragraph comment blocks in the --check validator to the one-to-two-line convention, and bump test/native-suite-count to 42 for the test_fuzz_config suite added here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Faershtein
|
9a37250438 |
fix(router): clear failed reliable send retries (#11267)
* fix(router): clear failed reliable send retries * test(router): cover failed interface enqueue * fix(router): retain retries after duty cycle limits |
||
|
Tom
|
2024bb8384 |
Arrival time fix perhaps (#11274)
* Add explicit presence for MeshPacket.rx_time (arrival time) rx_time is now proto3 optional with a has_rx_time presence bit, matching the rx_rssi treatment. A node with no GPS and no phone connected yet has no time source at all, so a bare 0 was indistinguishable from a genuine 1970-01-01 reading; downstream consumers (replay packets, JSON serialization) now check has_rx_time instead of the value. * Dedupe rx_time stamping into a shared helper; trim a debug log string Extract the repeated haveTime/rx_time/has_rx_time stamp logic (5 call sites across Router.cpp, MeshBeaconModule.cpp, MeshService.cpp) into Router::computeRxTimeStamp()/stampRxTime(). Also shorten the new RTC.cpp LOG_DEBUG string. Saves 48 bytes of flash on rak4631 (measured), no behavior change. * Fix has_rx_rssi presence carried unconditionally through StoreForward replay preparePayload() set has_rx_rssi = true unconditionally on replay, regardless of whether the packet's rx_rssi at store time was a genuine measurement (e.g. MQTT-relayed packets carry no real RSSI). Store the presence bit alongside rx_rssi in PacketHistoryStruct and restore it on replay instead. Flagged by Copilot on #11271 (same root cause the has_rx_time explicit presence work fixes) but never addressed before that PR merged. * Trim comment blocks to the repo's 1-2 line guideline .github/copilot-instructions.md:338 caps code comments at 1-2 lines; several blocks added across the rx_time explicit-presence work ran well past that. Also consolidates Time.cpp's file-level doc comment into Time.h, where the rest of the Time:: API contract already lives. No behavior change. * Add rx_time explicit-presence test coverage - test_meshpacket_serializer: has_rx_time=false fixture plus tests asserting JsonSerialize/JsonSerializeEncrypted emit 0 rather than leaking the millis() placeholder, alongside the has_rx_time=true baseline. - test_stream_api: two tests driving a real PhoneAPI handshake (want_config_id through STATE_SEND_PACKETS) that simulate a phone time-giving transaction arriving before vs. after a queued packet is drained - covering both the reconciled and the ships-with-placeholder-absent paths of MeshService::reconcilePendingRxTimes(). * Fix three correctness issues flagged in review - Time.h: drop the reserved-identifier include guard (_MT_TIME_H); pragma once already covers it, matching convention elsewhere (e.g. RTC.h). - Time.cpp: rebase getMillis64()'s wrap accumulator when the test seam swaps clock sources, so a real<->injected clock jump isn't miscounted as a genuine 32-bit wrap. - NodeInfoModule: the 12h reply-suppression window is a local dedup duration, not a wall-clock reading - switch it to Time::getMillis64() so RTC-quality jumps and replayed packets' stale rx_time can't perturb it. - StoreForwardModule: has_rx_time was derived from *current* RTC quality at replay time rather than stored at capture time, so a history entry saved while time-blind could be misreported as a valid epoch once the clock later improved. Persist the presence bit in PacketHistoryStruct instead. * tryfix CI * post review fixes * more test fixes |
||
|
Benjamin Faershtein
|
0fef83d434 |
Add configurable event mode hop limit (#11275)
* feat: resolve event mode hop limit * feat: bake event mode hop limit * fix: honor event mode hop cap in routing * docs: expose event mode hop limit preference * fix: enforce event hop defaults across routing * docs: clarify event hop override behavior * refactor: simplify event mode hop preference * fix: cap equal event hop limit |
||
|
Tom
|
2c57a17124 |
Phantom node fix perhaps (#11271)
* trying to fix phantom nodes * fix comment spam * oops - missed one |
||
|
Ben Meadors
|
a8623a60c5 |
Stream our own position to the phone/UI while mesh position sharing is opt-in (#11270)
* Position: stream our own position to the phone/UI while mesh sharing is opt-in Position broadcasts became opt-in in 2.8 (#10929): with every public channel at position_precision 0, sendOurPosition() finds no eligible channel and returns without queueing anything, so the connected phone or on-device UI never sees the node's own GPS fix ("GPS looks dead" on standalone MUI devices even though the receiver has a lock). Mirror device telemetry's local delivery: once a minute, when the toPhone queue is idle, stream our own position to the connected client at full precision. The packet is handed straight to sendToPhone() and never touches the mesh, so the per-channel opt-in and the public-channel precision clamp still govern everything on the air. Also stop logging "Send pos ... to mesh" before the channel scan has found an eligible channel; when sharing is disabled everywhere the skip is now logged explicitly instead of pretending a send happened. The cadence gate is a pure static (shouldSendPositionToPhone) alongside the existing broadcast-policy helpers, with unit tests covering the first-send, cadence, gating, and millis() rollover cases. * Review: only advance phone cadence on a queued packet; drop the ms==0 sentinel sendOurPositionToPhone() now reports whether a packet actually reached the phone queue, and runOnce() stamps the cadence only on success, so a guard or allocation failure retries on the next tick instead of waiting out a minute. The never-sent state is a dedicated hasSentPositionToPhone flag rather than lastPhoneSendMs == 0, so a send stamped exactly at millis() == 0 still holds the cadence. New regression test covers that case; existing cases updated to the explicit flag. * Review: align the rollover test fixture with its documented elapsed times lastSent now sits exactly 30,000 ms before the uint32 wrap, so the two cases are precisely 70,000 ms (sends) and 40,000 ms (held) - the previous comments claimed 70s/20s against actual elapsed values of 70,001/40,001 ms. |
||
|
Ben Meadors
|
ae16c052d5 |
Time out an abandoned admin edit transaction (#11254)
begin_edit_settings sets a bool that only the matching commit ever clears. If the commit never arrives -- the client dropped its link partway through a bulk config import, or went away entirely -- the transaction stays open indefinitely, and every later config write from any client is applied to RAM, acknowledged, and then never saved. The writes look successful and are visible in get_config, but the node reverts all of them at the next boot, and only a reboot clears it. Give the transaction a one minute idle timeout. Each deferred write restarts the clock, so the window bounds the gap between writes rather than the length of the edit; a bulk import sends them milliseconds apart. The next admin message after it lapses retires the transaction, persisting the segments it had deferred and flushing the warnings it was holding. The check runs after the auth gates and before the switch, so every case sees consistent state and the recovery's flash write happens in the main loop rather than a disconnect callback. It saves rather than rolls back because there is nothing to roll back to: each write already took effect in RAM and was acknowledged, so the transaction only ever deferred the save. That also makes an early expiry cheap -- the worst case for a client that really was still going is that its remaining writes are saved individually instead of batched. Closing on disconnect instead was tempting, but the reporter's captures show iOS dropping and reconnecting within half a second several times per session, which would abort imports that currently survive the blip. Also drops the file-scope hasOpenEditTransaction, shadowed by the class member at every use site and referenced nowhere else in the tree. Reported in #11245 |
||
|
|
1e982fa78c |
Sign plaintext packets in licensed mode (#10969)
* feat: sign licensed plaintext packets Preserve and publish identity keys in licensed mode so existing XEdDSA signatures can authenticate plaintext traffic. Sign licensed broadcasts and direct messages when they fit, while keeping PKI encryption disabled and normal routing behavior unchanged. * fix(security): persist licensed channel sanitation * fix(security): close licensed migration lifecycle gaps * fix(security): address licensed signing review feedback * test: restore signing globals from Unity teardown * fix(baseui): allow confirmed ham region selection * fix(baseui): guard region picker validation * style: trunk fmt --------- Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: Austin <vidplace7@gmail.com> |
||
|
|
45cb8e7500 |
feat: preserve normal radio profiles across event firmware (#11110)
* feat: isolate event radio profiles * fix: preserve identity on degraded config boot * fix: guard event profile storage * style: align event profile storage names * fix: discard event profile on normal boot * refactor: simplify event profile cleanup * fix: limit inactive profile migration to event firmware * fix(event): make event-profile capacity check portable across filesystems The USERPREFS_EVENT_MODE storage preflight called FSCom.totalBytes() and FSCom.usedBytes(), which only exist on ESP32's LittleFS wrapper. Every other backend failed to compile once event mode was enabled: error: 'class InternalFileSystem' has no member named 'totalBytes' (nRF52) error: no member named 'totalBytes' in 'fs::FS' (Portduino) This was not caught earlier because the event block is behind #if USERPREFS_EVENT_MODE, and the existing unit tests only cover the pure helpers, which compile identically either way. Add fsTotalBytes()/fsUsedBytes() to FSCommon and implement them per backend: littlefs v1 traversal for nRF52 (Adafruit_LittleFS) and STM32 (STM32_LittleFS), FSInfo for RP2040, statvfs for Portduino, and the native methods for ESP32 and nRF54L15. The nRF52/STM32 path reports "full" if the traversal errors so the capacity check fails safe. Verified with USERPREFS_EVENT_MODE=1: native-macos test_event_profile_storage passes 5/5, and heltec-v3, rak4631, rak11310 and rak3172 all build clean. * fix(event): use std::filesystem for Portduino capacity, bump native-suite-count Two CI fixes: - native-windows has no <sys/statvfs.h>, so the Portduino branch of fsTotalBytes()/fsUsedBytes() broke the Windows build. Switch to std::filesystem::space(), which is already used under ARCH_PORTDUINO in HostMetrics.cpp and works on both POSIX and MinGW. Errors report "full" so the capacity check still fails safe. - This PR adds test/test_event_profile_storage, taking test/ from 40 to 41 suite directories, which trips the native-suite-count reconciliation check. * fix(event): scope boot-write deferral to the radio profile, address review Review feedback: - Copilot: saveProto()'s boot-write deferral applied to every file, so loadFromDisk()'s recovery writes (e.g. restoring owner fields into devicestate) were silently dropped and never retried, because the ctor CRC baselines are computed after loadFromDisk(). Deferral now applies only to the radio-profile files, via isRadioProfileFile(). - jp-bennett: eventConfigFromStandard() wrapped a struct copy plus one field assignment in a header helper with its own unit test. Inlined at its single call site and removed; the behaviour is covered end-to-end by hardware validation instead (RAK4631 normal -> event -> normal preserves NodeNum and public key while swapping LoRa, and restores the original channel byte-identically). - jp-bennett: the active-backup encrypted-storage migration ran for normal builds too, which changed non-event behaviour and contradicted the PR's stated event-only scope. Scoped to USERPREFS_EVENT_MODE; the adjacent block already covers the inactive standard backup in event builds. New tests (all verified to fail under mutation, not vacuous): - test_event_paths_never_collide_with_standard_or_shared_files: the core safety property. A path-table slip would make event firmware overwrite the user's real config, channels or backup, or capture a shared file like devicestate/nodedb. Nothing asserted this before. - test_only_radio_profile_files_defer_boot_writes: guards the deferral fix above so re-widening it fails loudly. - test_event_reservation_fits_smallest_supported_filesystem: the reservation is compile-time but must fit filesystems as small as 14 KiB (STM32WL) and 28 KiB (nRF52840). Protobuf growth pushing it past those would silently stop event profiles persisting - the exact failure mode confirmed by fault injection on a RAK4631. Verified with USERPREFS_EVENT_MODE both on and off: 7/7 tests pass, and rak4631, heltec-v3, rak11310 and rak3172 all build clean. * fix: preserve event profile storage recovery --------- Co-authored-by: Jonathan Bennett <jbennett@incomsystems.biz> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: Benjamin Faershtein <benjaminfaershtein@Benjamins-MacBook-Pro-2.local> |
||
|
James RichandClaude Fable 5
|
8e104a909b |
fix(admin): persist TAK module config (team color / member role) (#11216)
Setting TAK team/role ACKed and rebooted but stored nothing: handleSetModuleConfig had no tak case, saveToDisk never set has_tak, and handleGetModuleConfig had no TAK_CONFIG case. Add all three, plus a native suite sweeping every ModuleConfig submessage through set -> save -> load -> get and a TAK value-fidelity suite. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Thomas Göttgens
|
b46c8c9f80 |
Router: defer nested local sends to avoid handleReceived re-entrancy (#11191)
Prevents an nRF52 task-stack overflow on config save. Replaces draft #11155. |
||
|
Ben Meadors
|
01d1873bfc |
Deliver locally-generated replies addressed to us to the phone (#11185)
* Deliver locally-generated replies addressed to us to the phone Config get/set from the phone times out on every device: the client sends an admin request, the node handles it, and the response is silently dropped before it reaches the phone queue. #10967 changed Router::sendLocal's isToUs branch from enqueueReceivedMessage() to handleReceived(p, src), so a local packet keeps its RxSource instead of being relabeled RX_SRC_RADIO by the queue round-trip. That is the right call for the new policy gates, but module replies go out through MeshService::sendToMesh() with the default RX_SRC_LOCAL, and a reply to a phone-originated request is addressed to our own node (setReplyTo resolves from == 0 to ourNodeNum). Those replies now re-enter callModules as RX_SRC_LOCAL, where the loopback gate skips every module whose loopbackOk is false - including RoutingModule, whose promiscuous sniff is the only path that moves a received packet into toPhoneQueue. The reply is released, never sent. Requests still work, because the phone's own packets arrive as RX_SRC_USER and pass the gate, so a set_config is applied and only its acknowledgement is lost. That is why a client can connect and download config but times out on every config screen and every setter. Deliver the phone's copy from sendToMesh() instead: for a local packet addressed to us, the loopback gate is doing its job in keeping the packet away from module re-dispatch, and the phone copy is exactly what is missing. Setting loopbackOk on RoutingModule would instead echo every locally-generated broadcast back to the phone, and relabeling replies RX_SRC_RADIO would undo the origin separation #10967 added. Also stop reporting ERRNO_SHOULD_RELEASE (35) to the phone in the QueueStatus for these packets. It means "caller frees", not a send failure, and the same hunk changed it from the 0 the phone used to see. * Address review: trim comments, assert the QueueStatus count Condense the added comments to the one-or-two-line house style; the rationale lives in the commit message and PR. The reply test drained QueueStatus records in a while loop, which would have passed just as happily on an empty queue. Count them and require both the request's and the reply's. |
||
|
Benjamin Faershtein
|
8acb7fec73 | test: cover coerced packet signature policy |