mirror of
https://github.com/meshtastic/firmware.git
synced 2026-09-16 00:10:11 -04:00
Node-Bridging
209
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
6fa3df0af3 | node bridge | ||
|
|
3022a37768 |
Add the BLE-GATT mesh-peer transport: a phone connects to the node as a mesh peer
The SIG-Mesh "GATT proxy" role: the node runs a GATT server on a private service, phones connect to it as centrals and exchange whole mesh frames over a write/notify characteristic, without a phone-API session. Point-to-point where LoRa and the advertisement transport are one-to-many; a broadcast is N notifies. Two halves: - BLEGattMeshHandler (platform-neutral, built into the native suite): fragment framing shared byte-for-byte with the node-kmp client, bounded reassembly (per-peer and per-in-flight caps, expiry), the ingress guards the UDP and advertisement transports already apply (validate-before-relay, drop senderless / self-claimed / impossible-hop packets, strip local-only metadata), a per-peer TX ring, and no-echo-back-to-the-arrival-peer via a small (from,id)->peer table. - ESP32BLEGattMesh (NimBLE): its own connectable advertising set on instance 2 carrying the service UUID, per-connection notifies, an MTU-derived chunk size, and a GAP handler chained ahead of the Arduino wrapper's so the server's connection/MTU/subscription bookkeeping covers these links too. A mesh-peer disconnect is gated out of the PhoneAPI session teardown. Registry-gated on the new BLE_GATT_PEER protocol flag, so it carries outbound packets only while a phone is being served as a peer. The sdkconfig bump that the second connection needs (CONFIG_BT_NIMBLE_MAX_CONNECTIONS=2, CONFIG_BT_CTRL_BLE_MAX_ACT=6) lands here with the service rather than ahead of it. Protobufs pointer bumped for the TRANSPORT_BLE_GATT and BLE_GATT_PEER enums; generated headers regenerated. Proven: native suite 1418/1418 (26 new cases for this transport); heltec-v3 (ESP32-S3) builds and, with WiFi off, brings the service up and advertises on instance 2 with no OOM / NimBLE 519 / crash at the 2-connection config. Not yet proven: a phone connecting as a mesh peer and a frame crossing device to phone (bench-gated); ESP32-C3 and nRF52 are unbuilt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
d8ea498012 |
Move the MQTT egress tap onto the transport registry
Add a pre-encode fan-out point to MeshTransportBase so a transport can act on the decoded packet (with its now-encrypted copy and the channel index) before the decoded copy is released, alongside the existing post-encode point that UDP and BLE use. A transport opts into exactly one point via its constructor, and the two point at separate registries so they never cross. MQTT registers at the pre-encode point through a thin MQTTTransport adapter, so Router::send() no longer names MQTT directly. Its moduleConfig.mqtt.enabled and isFromUs gate stays at the call site (same point, inside the decoded-tag block) and every MQTT-side gate - via_mqtt loop-prevention, per-channel uplink, DontMqttMeBro, range-test suppression, PKI-vs-channel encryption choice - stays inside MQTT::onSend() verbatim. The old `&& mqtt` null check moves into the adapter's hook. Because the adapter is invisible to the post-encode fan-out, MQTT still fires only for our own originations and never for the relayed / already-encrypted broadcast traffic. LoRa iface->send() and the post-encode UDP/BLE path are untouched. No behaviour change. test_transport_registry gains three cases pinning the new invariant: pre-encode transports receive the encrypted packet, decoded copy and channel index in registration order; the pre- and post-encode points are disjoint; and an empty pre-encode registry is a safe no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZfYRoJFcYiqUxWQKTLUQR |
||
|
|
0091277731 |
Route the non-LoRa egress taps through a transport registry
Router::send fanned outgoing packets to UDP multicast and BLE mesh through two hardcoded, guard-wrapped taps sitting just after encryption. Replace them with MeshTransportBase, a thin registry modeled on MeshModule: each transport self-registers in its constructor, and callTransports() hands the encrypted packet to every enabled transport in registration order. A new broadcast transport now plugs in without editing the funnel. Behavior is unchanged. UdpMulticastHandler and BLEMeshHandler already exposed bool onSend(const meshtastic_MeshPacket*), so they become overrides with no body edits; the per-tap enabled_protocols check moves verbatim into each transport's isEnabled(), and the old udpHandler/bleMeshHandler null checks are subsumed by "an unconstructed transport never registers". Registration order (UDP then BLE) matches the former tap order. Unlike MeshModule there is no CONTINUE/STOP contract: these are parallel media, so one transport accepting a packet never suppresses another - callTransports ignores the return. MQTT stays a hardcoded tap: it fires earlier, inside the decoded-tag block before p_decoded is released, and needs the decoded copy plus the channel index - a pre-encryption hook point this registry does not yet model. LoRa's iface->send is untouched and remains the mandatory path. test_transport_registry pins the invariant the rewrite could break: every enabled transport is called in registration order, a true return does not suppress a later transport, disabled transports are skipped, and an empty registry is a safe no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KZfYRoJFcYiqUxWQKTLUQR |
||
|
|
890f12ec27 |
Add native tests for the BLE mesh transport
Thirteen cases over the half of the transport that has no BLE in it: building an advertisement, the ingress guards, the TX ring and the readiness pump. Full suite 1385/1385. Testable because BLEMeshHandler has no platform BLE dependency - ESP32BLEMesh and NRF52BLEMesh do - so native now compiles it via HAS_BLE_MESH=1 on native_base. Nothing runs there: main() instantiates neither platform subclass on portduino, so bleMeshHandler stays null. Adds one seam for it. deliverToRouter called router->enqueueReceivedMessage directly, which meant the ingress guards could only be tested by standing up a live Router; it now goes through a virtual enqueueReceived() that the test overrides to observe what survives. Production always takes the default. The cases are the bugs this transport actually had, or the ones its guards exist to stop: a relayed packet must not be refused (refusing it capped the mesh at one hop), onSend must queue rather than transmit (advertising inline stalled the router for the length of every burst), a spoofed from=0 and an out-of-range hop count must be dropped, claimed PKI authentication must be stripped rather than believed off the wire, and our own advertisement heard by our own scanner must not loop back in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
14eaa5587d |
Honor mute when waking the screen for a received message (#11688)
* fix(ui): honor mute when waking the screen for a received message TextMessageModule fired powerFSM.trigger(EVENT_RECEIVED_MSG) for every text packet, gated only by shouldWakeOnReceivedMessage(), which checks external notification, device role and battery level but never the mute flags. A muted channel therefore suppressed the banner and still lit the screen. MessageRenderer::handleNewMessage() only computed mute for MessageType::BROADCAST, so a DM from a muted node produced a banner and a wake. Add isMutedForPacket() in Channels: a DM addressed to us reads the sender's NodeInfoLite mute bit, every other packet reads the mute bit of the channel it arrived on. This is the predicate ExternalNotificationModule already applied to the buzzer, vibra and LED outputs, hoisted so all three call sites share it. Bell and alert messages still break through mute on both paths, unchanged. No protobuf or config change: ChannelSettings.module_settings.is_muted and the NodeInfoLite mute bit already exist and are already settable from the device menu and via AdminMessage.toggle_muted_node. Closes #11674 * fix(ui): let an alert break through mute on the screen wake path In COLOR display mode TextMessageModule skips handleNewMessage(), so powerFSM.trigger(EVENT_RECEIVED_MSG) is the only wake an alert gets. Gating it on mute alone dropped that wake for a bell on a muted channel. Add MeshService::isAlertPayload(): an ASCII BEL in the payload while at least one alert_bell_* output is enabled. The wake gate is now "not muted, or an alert". MessageRenderer uses the same predicate instead of its own inline bell scan, which also lifts that scan's arbitrary 100 byte cap. Rename three test cases. Their names carried exactly 35 characters after the test_ prefix, which matches the Lob API key format and tripped trufflehog in the trunk check gate. |
||
|
|
7afd270f39 |
Gut beacon send-as-node and consolidate TX onto broadcast_targets (#11646)
* Gut beacon send-as-node and consolidate TX onto broadcast_targets Two MeshBeaconConfig changes, both against fields that never reached a tagged release, so there is no migration for existing nodes. broadcast_send_as_node let a client name a node ID to send beacons AS, rewriting the packet's `from`. Firmware never applied it - the assignment was commented out, so `from` was always the local node and the field was a settable, persisted no-op. It was also unsound as designed: rewriting `from` forges no signature, it only makes isFromUs() false, so perhapsEncode() skips XEdDSA signing and receivers get an unsigned packet attributed to another node. broadcast_on_channel / broadcast_on_region / broadcast_on_preset were a second way to name a beacon destination alongside broadcast_targets, chosen silently on whether broadcast_targets was empty. The comments claimed the two were equivalent; they were not. An inline ChannelSettings carries name and PSK, so broadcast_on_channel could transmit on a channel absent from the node's channel table, which channel_index cannot express. That is dropped deliberately - the channel must exist on the node. Empty broadcast_targets now synthesises one target on the running preset and region over the primary channel, matching what the scalar path produced when left unset, so an otherwise unconfigured node still beacons. The USERPREFS_MESH_BEACON_ON_* keys go with the fields. A preconfigured build that still defines one now fails at compile time with a pointer to the USERPREFS_MESH_BEACON_TARGET_0_* equivalents, rather than silently losing its beacon channel. The replacement names a channel-table slot, so such a build must also provision that channel. MeshBeaconConfig shrinks 324 -> 240 bytes and ModuleConfig 328 -> 244, against the 512-byte MAX_TO_FROM_RADIO_SIZE ceiling that FromRadio sits 2 bytes under. The protobufs submodule points at a branch carrying both proto changes; it needs re-pointing to master once meshtastic/protobufs#1047 and #1048 merge. * Point protobufs submodule at master now that the beacon protos are merged meshtastic/protobufs#1047 and #1048 are in master, so drop the temporary beacon-proto-integration pin. MeshBeaconConfig stays 240 bytes and ModuleConfig 244, unchanged from the integration branch. The bump also picks up master's unrelated additions: the MESHNOLOGY_W12 and MESHPAGER_X2 hardware models, and a ground-speed unit correction in Position. |
||
|
|
7e9525ad83 |
feat(baseui): default US to LongTurbo on first region selection (#11637)
Selecting US in the BaseUI region chooser now installs LongTurbo instead of LongFast, but only for out-of-box setup: the outgoing region must be UNSET, so a later switch to US leaves whatever preset the node is running alone. Scoped to the menu on purpose. The US entry in regions[] keeps LongFast as its default preset, so preset repair, admin/phone writes and every other route onto US are unchanged. A build pinning USERPREFS_LORACONFIG_MODEM_PRESET, a preset already moved off the install default, or use_preset=false all outrank it. The decision is lifted into menuHandler::presetForRegionSelection() so it is reachable without a Screen, following toggleNodeMuted(). |
||
|
|
63f0f1edd0 |
fix(nodedb): clear the whole LocalModuleConfig when installing defaults (#11627)
installDefaultModuleConfig() memset sizeof(meshtastic_ModuleConfig) - the
368-byte union-backed wire oneof - over `moduleConfig`, which is a
meshtastic_LocalModuleConfig: 1092 bytes with every submessage inlined. The
function assigns only the fields it cares about and relies on that memset to
zero the rest, so every byte past offset 368 that it never assigns kept its
previous value across what is supposed to be a full reset.
installDefaultConfig() directly above already used the correct
sizeof(meshtastic_LocalConfig); only the module variant was wrong.
statusmessage is the field this shows up on. It sits at offset 609 and is
never assigned by the defaults installer, so it survives both routes into
installDefaultModuleConfig():
- moduleConfig.version < DEVICESTATE_MIN_VER -> "old, discard". The decode
succeeded, so the complete old config is in RAM and its statusmessage
survives the discard verbatim.
- loadProto() failure -> whatever a partial decode wrote there survives
(loadProto itself clears correctly, using the caller's objSize).
node_status is char[80]. When the surviving bytes carry no NUL, nanopb
refuses the field ("unterminated string"), pb_encode_to_bytes() returns 0 and
PhoneAPI::getFromRadio() returns 0. config_state has already advanced, so the
frame is never retried - and 0 is the client's end-of-data sentinel, so the
rest of the config dump goes with it and the client never receives
StatusMessageConfig.
traffic_management is not affected: installDefaultModuleConfig() calls
installTrafficManagementDefaults(), which reassigns the whole submessage and
its has_ flag regardless of the memset size.
Also add has_traffic_management to the has_* list in saveToDiskNoRetry() for
consistency - it was the only module config missing from it.
|
||
|
|
9fbc176e91 |
Extend userPrefs coverage to the whole channel table and the missing config fields (#11624)
* Extend userPrefs coverage to the whole channel table and the missing config fields initDefaultChannel() handled only indices 0-2, so USERPREFS_CHANNELS_TO_WRITE above 3 produced live secondary channels carrying the public default PSK; it now covers all eight slots, with bin/platformio-custom.py completing every field of a configured index so indices 0-2 stay byte-identical. Adds USERPREFS_CHANNEL_<n>_IS_MUTED, USERPREFS_CONFIG_DEVICE_REBROADCAST_MODE, USERPREFS_CONFIG_DEVICE_NODE_INFO_BROADCAST_SECS, USERPREFS_CONFIG_LORA_CONFIG_OK_TO_MQTT, USERPREFS_CONFIG_SECURITY_IS_MANAGED and USERPREFS_CANNED_MESSAGES, applied after installRoleDefaults() and validated the way AdminModule validates a set-config. Adds test_userprefs_channels, covering the configured table under coverage-channel-table and the stock defaults under every other env. * Address review: hex channel count, PSK width assert, canned-message termination USERPREFS_CHANNELS_TO_WRITE now parses 0x-prefixed hex, matching the format userPrefs.jsonc documents, without int(x, 0)'s rejection of a leading-zero decimal such as "03". A static_assert rejects a USERPREFS_CHANNEL_<n>_PSK literal wider than psk.bytes, which memcpy would otherwise write over the fields after it. The USERPREFS_CANNED_MESSAGES copy keeps strncpy's zero-padding and terminates explicitly, rather than shortening the length, which would have left the last byte unwritten. |
||
|
|
122ec0e9f4 |
Revert "feat(baseui): default US to LongTurbo on first region selection"
This reverts commit
|
||
|
|
dbba2b3f6c |
feat(baseui): default US to LongTurbo on first region selection
Selecting US in the BaseUI region chooser now installs LongTurbo instead of LongFast, but only for out-of-box setup: the outgoing region must be UNSET, so a later switch to US leaves whatever preset the node is running alone. Scoped to the menu on purpose. The US entry in regions[] keeps LongFast as its default preset, so preset repair, admin/phone writes and every other route onto US are unchanged. A build pinning USERPREFS_LORACONFIG_MODEM_PRESET, a preset already moved off the install default, or use_preset=false all outrank it. The decision is lifted into menuHandler::presetForRegionSelection() so it is reachable without a Screen, following toggleNodeMuted(). |
||
|
|
9a59e9088d |
fix(test): restore the sendAckNak overrides broken by #10767 (#11626)
#10767 added a relaySource parameter to the RoutingModule::sendAckNak virtual, but the five test mocks that derive from RoutingModule still declared the six-parameter signature with `override`. Nothing overrides the new virtual, so all five suites fail to compile and the native test job has been red on develop since the merge: test/test_reliable_ack_matrix/test_main.cpp:167:10: error: 'void MockRoutingModule::sendAckNak(meshtastic_Routing_Error, NodeNum, PacketId, ChannelIndex, uint8_t, bool)' marked 'override', but does not override Widen the five mocks to the new signature. Also carry has_rx_rssi with rx_rssi in allocAckNak(). rx_rssi has explicit presence, so copying only the value left has_rx_rssi false and nanopb dropped the field at encode time - the phone never saw the relayer's RSSI that #10767 set out to deliver. Cover both: test_reliable_ack_matrix asserts the overheard rebroadcast is handed through as the relay source on the decodable path and the opaque #11502 ingress path, and that no other ACK/NAK claims a relayer; test_mesh_module drives a real RoutingModule and asserts the relay fields, has_rx_rssi included, survive all the way to the phone. |
||
|
|
a8934a16d4 |
Route waypoint expiry through waypointIsActive instead of a raw getTime compare (#11621)
* Route waypoint expiry through waypointIsActive instead of a raw getTime compare * Let isExpired own the zero-clock policy for purgeExpired too * Resolve the clock in isExpired when a packet carries no valid rx_time |
||
|
|
a89f1920e1 |
fix(traffic): don't re-stamp dropped duplicate positions, which slid the dedup window indefinitely (#11620)
* fix(traffic): don't re-stamp dropped duplicate positions, which slid the dedup window indefinitely * test(traffic): trim the regression test comment and derive its counts |
||
|
|
514b476189 |
feat(admin): append the optional ham long_name to the call sign (#11612)
* feat(admin): append the optional ham long_name to the call sign HamParameters gained a long_name field (meshtastic/protobufs#941) that handleSetHamMode never read, so a client that sent one still ended up with a node named after the bare call sign. Join it behind the call sign with the "//" separator hams already use on the air: call_sign "N0CALL" plus long_name "Attic Heltec" becomes "N0CALL//Attic Heltec". An absent long_name keeps the previous call-sign-only name, which is what the on-device region picker still sends. Being cosmetic, long_name stays out of the whitespace-only rejection that guards call_sign and short_name: a blank one is dropped rather than costing the operator the whole licensing request over a stray space, which that path would report only as a LOG_WARN and so would be invisible from the app. The composed name is finished with clampLongName() rather than a bare sanitizeUtf8(), matching handleSetOwner and NodeDB: the proto caps the parts at 7 + 2 + 14 bytes, inside the 24-byte local budget, and clampLongName is the backstop if either cap moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(admin): enhance handleSetHamMode to return status for request validation --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
576a1bb008 |
Fix trackball dropping short presses and losing the click when tilted (#11599)
* Fix trackball dropping short presses and losing the click when tilted * Do not let a direction counter overwrite an emitted press event * Accept the first press interrupt when the clock still reads zero * Classify a press released before the first poll by its latched time |
||
|
|
cd6ac90f7e |
Add waypoint & geofence support with notifications for BaseUI and InkHUD (#10920)
* Implement GeofenceModule for waypoint crossing notifications and integrate with existing modules * Waypoint Applet Initial Support on InkHUD * undo tile change * Update screen when Waypoint shows or dissapears * Merge branch 'develop' into waypoint-geofence * Geofence on InkHUD * Update MapTile.h * Update WaypointStore.cpp * Notifications * remove GF from waypoint screen * Prevent Focus from closing the notifiaction banner * Trunk fix * cleanup * undo merge conflix mistake * Waypoint screen on BaseUI * Focus preserve fix * UI bugs * Allow Inkhud to remove waypoint * Respect Locked Waypoints * Trunk fix * Update WaypointStore.cpp * Use 8-digit hex formatting for waypoint IDs. 0x%x was inconsistent with the repo's own convention (0x%08x for 32-bit IDs, used elsewhere in this file). Fixed here and in two other spots I found with the same issue (WaypointModule.cpp, GeofenceModule.cpp). * Update ExternalNotificationModule.cpp * Reject invalid surrogate codepoints in waypoint icon rendering * Update WaypointModule.cpp * Update WaypointStore.cpp * Update WaypointStore.cpp * Update WaypointStore.cpp * trunk fix * fix warnings * power.h rename to Power.h * Update Power.h * Fix executable bit on bin/lint-ifdef-complexity.sh Lost during a prior merge from develop (Windows checkout doesn't preserve file mode), causing "execve failed: Permission denied" in the Trunk Check Runner CI job. develop has this file at 100755; restoring that here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update README.md * Clean up waypoint and geofence integration * Minimize waypoint and geofence implementation * removed unnecessary gating * Geofence alert * trunk fix * Update test_main.cpp * Update WaypointStore.cpp --------- Co-authored-by: Ben Meadors <benmmeadors@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
7e11bde8c8 |
fix(beacon): repair the MeshBeacon radio switch/restore regression from #11573 (#11596)
* fix(radio): put the beacon restore back inside completeSending's if (p) Reverts the RadioLibInterface and RadioInterface changes from #11573 ( |
||
|
|
98c88d7e19 |
fix(position): halve the stationary/fixed-position broadcast floor to 6h (#11606)
The 12h floor introduced with traffic management was too aggressive: a fixed_position or stationary node goes quiet for half a day after its boot broadcast, so anything that missed that one packet - a node that joined later, or one that restarted - shows it with no position until the next refresh. Drop the floor to 6h, and drop the traffic-management identical-position dedup window from 11h to 5h with it. The two are a pair: the dedup window was deliberately sized just under the broadcast floor so a stationary node's periodic refresh clears its neighbours' window instead of being dropped as a duplicate. Leaving it at 11h would have made the extra broadcast pure airtime - aired, then discarded by every receiver - so the mesh would still have seen a 12h refresh. Role caps are unchanged and still bind: tracker 1h, lost-and-found 15m. Both remain shorter than the new 5h default, so those exceptions apply exactly as before. Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
56ce743f75 | Show waypoints sent with no expiry and stop expiring on an unset clock (#11600) | ||
|
|
8a15d9258f |
fix(test): unbreak test_radio under ASan (#11589)
* test_radio: prove the rejected packet was released via pool accounting, not pointer identity * test-state.sh: silence the shell's own open failure when scanning /proc for survivors * Trim the comments added with the test_radio and test-state fixes |
||
|
|
0271be9369 |
fix(SafeFile): remove a stale .tmp before opening it for write (#11428)
* fix(SafeFile): remove a stale .tmp before opening it for write SafeFile writes to <filename>.tmp, verifies it by readback, then renames it over the real file. openFile() never removed a pre-existing .tmp - an unfinished FIXME - and FILE_O_WRITE appends rather than truncates on Adafruit_LittleFS (nRF52) and STM32 LittleFS. So a .tmp left behind by a reset in the window between close() and renameFile() is appended to on the next save. The readback hash covers only the bytes just written, so it mismatches, close() returns false, and the tmp is left behind again - the failure latches and every subsequent save of that file fails. Today saveProto() discards close()'s result, so this is silent and permanent. Guard the remove with exists(): a bare remove() of a missing file logs on Portduino. The same guarded pattern is already used for this exact append trap in xmodem.cpp. Note the FIXME's commented-out body named the wrong path - it removed 'filename', the real file, not 'filenameTmp' - so it would have destroyed the good copy had it ever been enabled. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(SafeFile): cover the stale-.tmp path, and trim the fix comment Adds test_safefile, the first coverage of SafeFile's write-tmp / verify-by-readback / rename-over path that every saveProto() caller goes through. Five cases pin the contract the fix restores: whatever the backend does on open, a completed save leaves the real file holding exactly the bytes written and nothing else, on both the fullAtomic and the !fullAtomic construction, with no .tmp left behind. These tests cannot go red on the native host, and that gap cannot be closed here. FILE_O_WRITE is an append-then-seek-to-end open only on Adafruit_LittleFS (nRF52) and on the in-repo STM32 port (STM32_LittleFS_File.cpp: LFS_O_RDWR | LFS_O_CREAT followed by lfs_file_seek to LFS_SEEK_END). On Portduino FILE_O_WRITE is the string "w" (FSCommon.h:13), which reaches fopen() and truncates. Reverting the source fix and re-running leaves all five green, verified rather than assumed. test_write_open_truncates _on_this_host asserts that premise out loud, so if the host ever gains the append behaviour the suite starts discriminating instead of quietly agreeing. Why the original FIXME stayed commented out, since that is the real history here. It read "if (fullAtomic) FSCom.remove(filename)" and named the real file, not the tmp. Running it would delete the last good copy before the replacement had been written and verified, which is precisely the guarantee fullAtomic exists to provide. Disabling it was correct. The fix under test removes filenameTmp instead, which is the file that actually carries the stale bytes, and is safe to drop at any point because nothing has been promised about it yet. Scoping the remove to fullAtomic would be wrong for the same reason. Both paths open the same filenameTmp with the same FILE_O_WRITE; fullAtomic only decides whether the real file is nuked up front to free space. The !fullAtomic path is the space-constrained one, so it is if anything the more likely to be interrupted mid-write and inherit a stale tmp. Test 2 pins that. On the cost of the added exists(). Every saveProto() already ends in SafeFile::close(), which calls testReadback(): it reopens the tmp and reads the whole proto back one byte at a time through f2.read() to XOR a verification hash, then renames. So the per-save cost is already an open, a full write, a close, a full byte-wise reread, and a rename. One exists() is a single path lookup with no erase, no program and no data read, and on the common path there is no remove() at all. Next to the readback loop it is noise. Happy to put a number on it if wanted. Scoping it to fullAtomic would also not do what it looks like it does. SafeFile's constructor defaults fullAtomic to false (SafeFile.h:28), and of the saveProto call sites only saveDeviceStateToDisk passes true. Config, moduleconfig, channels, nodedatabase and backup all take the default, so scoping would leave the stale tmp live on almost every save path, including the space-constrained one most likely to be interrupted mid-write. Also trims the fix's comment to two lines per AGENTS.md, and drops the stale-tmp removal log from LOG_WARN to LOG_DEBUG: an interrupted write is recoverable and self-healing, so it does not warrant a warning on every boot after one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(SafeFile): fail the write when a stale tmp cannot be removed openFile() ignored the result of FSCom.remove(). If the removal failed on an append-on-write backend, the open that follows appended to the stale bytes, and the readback hash is an 8 bit XOR over the whole tmp, so polluted content has a real chance of verifying and being renamed over the good file. It now logs and returns an invalid File. SafeFile::write() already no-ops on !f and close() already returns false, so the caller sees the save fail rather than silently getting a corrupt one. This is the only checked FSCom.remove() in the tree; the other call sites are all best-effort cleanups where failure does not compromise anything. Also gates test_write_open_truncates_on_this_host to ARCH_PORTDUINO. It asserts that this host truncates on FILE_O_WRITE, which is false by design on the Adafruit_LittleFS and STM32 backends the fix exists for, so running the suite there would fail on a premise that is only meant to describe the test host. Both reported by CodeRabbit on #11428. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
ac330e6a6b |
fix(radio): MeshBeacon heap leak and runtime packet payload size check (#11573)
* Fix for MeshBeacon packet leakage * fix: add runtime payload size check against radiobuffer * review fix for PR#11573: clear target radio settings before MeshBeacon packet release * add unit test for radio buffer capacity check, removing related assert for the test * review fix for PR#11573: add explicit verifaction against rejected packets |
||
|
|
bc035bb812 |
feat(lora): state a pinned userPrefs preset as the unset region's intent (#11507)
* feat(lora): state a pinned userPrefs preset as the unset region's intent A vendor build can pin USERPREFS_LORACONFIG_MODEM_PRESET while leaving the region unset, so a fresh flash comes up as region UNSET plus a deliberate preset. Stock installs come up as region UNSET plus the LONG_FAST placeholder, and nothing in FromRadio told the two apart - so clients treat every unset-region node as factory-fresh and replace its preset with the region default as soon as the user picks a region. A mesh pinned to SHORT_TURBO loses every new node to LONG_FAST or LONG_TURBO, silently. getRegionPresetMap() now emits an UNSET entry when, and only when, the build pins a preset, stating that preset as both the group's sole entry and its default. Stock builds are unchanged on the wire: no UNSET entry, which clients already read as unconstrained. This is intent, not enforcement. supportsPreset() still accepts any known preset while the region is unset (#11496) and the radio is held silent either way, so the device continues to honour whatever the user or an admin sets. Costs one group slot and one region slot on pinned builds only (6->7 of 8, 34->35 of 38); exhaustion is logged and degrades to the existing unconstrained behaviour. * Trim comments to the project's one-to-two-line limit |
||
|
|
389559bddb |
fix(NodeDB): re-derive my_node_num when ensurePkiKeys() mints the identity keypair (#11426)
* fix(pki): re-derive NodeNum when setting a region mints the identity key
A node's mesh address is derived from its identity key:
my_node_num == crc32Buffer(config.security.public_key.bytes, 32)
NodeDB::createNewIdentity() is what establishes that, and NodeDB::
generateCryptoKeyPair() is the only thing that called it.
CryptoEngine::ensurePkiKeys() generates or re-derives the keypair and writes
security.public_key, security.private_key and user.public_key - but never
re-derives my_node_num. Boot-time keygen is suppressed while the LoRa region is
UNSET (generateCryptoKeyPair()'s regionBlocksKeygen guard), so on a fresh device
my_node_num is still the MAC-derived value from pickNewNodeNum(). The user then
sets the region - the stock onboarding flow - ensurePkiKeys() mints a key, and
the invariant is broken.
The node then signs its broadcasts (Router.cpp signs when !pki_encrypted &&
(owner.is_licensed || isBroadcast(p->to))). Every receiver runs
verifyFirstContactNodeInfo, fails crc32Buffer(user.public_key) != p->from, and
drops the NodeInfo. The node's identity beacons are invisible to the mesh.
Nothing reboots to repair it: AdminModule sets requiresReboot = false for LoRa
changes ("All LoRa radio changes apply live via configChanged observer") and
MenuHandler ends at service->reloadConfig(changes).
Four call sites reached ensurePkiKeys():
1. AdminModule set_config LORA, region first set (phone app - the common path)
2. MenuHandler applyLoraRegion (on-device region picker)
3. InkHUD MenuApplet applyLoRaRegion (schedules a reboot, so it
self-healed at next boot)
4. portduino wasm wasm_set_region
The reference implementation was already in the tree: the *licensed* branch of
call site 1, thirteen lines below the broken unlicensed one, calls
nodeDB->generateCryptoKeyPair() (which reaches createNewIdentity()) and widens
the persisted mask with SEGMENT_DEVICESTATE | SEGMENT_NODEDATABASE.
Rather than repeat that at four call sites, the key-mint is routed through one
chokepoint that owns both halves of the identity: NodeDB::ensurePkiIdentity()
calls crypto->ensurePkiKeys() and then createNewIdentity(). It lives in NodeDB
because createNewIdentity() operates on the devicestate/node-DB globals, which
CryptoEngine deliberately does not touch - ensurePkiKeys() takes the security
config and user by reference precisely so it stays free of that dependency, and
it is unit-tested against a standalone CryptoEngine.
ensurePkiIdentity() returns true only when my_node_num actually moved
(createNewIdentity() early-returns when the key is unchanged, so a repeat region
change does not disturb the self entry or force a needless flash write). Callers
use that to widen their save mask; my_node_num lives in devicestate and the self
row moves in the node DB, so both segments must be persisted or the fix would
revert at the next boot. SEGMENT_CONFIG, which carries the key itself, is
already unconditional on all four paths.
The InkHUD reboot is left as-is. It is now redundant for this invariant, but it
covers the rest of that menu's behaviour and a redundant reboot is not a bug.
Adds test_handleSetConfig_persistsUnlicensedFirstRegionIdentity, the unlicensed
twin of the existing licensed test, asserting both the segment mask and
my_node_num == crc32(public_key).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* style(NodeDB): trim identity-recovery comments and guard the WASM nodeDB deref
Two review asks, no behaviour change on any built target.
Copilot flagged the unguarded nodeDB deref in the WASM region setter; it is the
only ensurePkiIdentity() call site that did not check the pointer first.
The rest is comment length. AGENTS.md:83 caps code comments at two lines, and the
identity-recovery comments across the four call sites plus the NodeDB.h doc block
ran to four and six lines. The rationale they carried is in the commit messages
and the PR body, which is where AGENTS.md says it belongs.
The PR's own fix in AdminModule.cpp is deliberately untouched.
* fix(NodeDB): keep the identity move authoritative when the self record cannot be created
createNewIdentity() removes the old node entry and assigns myNodeInfo.my_node_num
before it tries to create the row for the new number. If getOrCreateMeshNode()
came back null it returned false, so the first-region callers left
SEGMENT_DEVICESTATE and SEGMENT_NODEDATABASE out of the save mask.
The number had already moved in RAM at that point, and the freshly minted key
goes to flash under SEGMENT_CONFIG regardless. The next boot therefore reloads
the old number alongside the new key, which is exactly the
crc32(public_key) != my_node_num break this path exists to prevent, reached
through the error branch instead of the happy one.
Rolling the number back is not an option either, since the key has already been
replaced by the time this runs. So the move is now reported as the fact it is and
the missing self record is logged separately; getOrCreateMeshNode() will recreate
that row on the next contact. Reachable when the self record is absent and the
table is full of protected nodes.
Reported by CodeRabbit on #11426.
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
90a6dec3f3 |
fix(NodeDB): require a full 32-byte key when demoting to the warm tier (#11431)
* fix(NodeDB): require a full 32-byte key when demoting to the warm tier meshtastic_User.public_key is a wire `bytes` field with max_size 32, so any size in 0..32 decodes off the air, and nothing validates it on ingress: NodeInfoModule hands the decoded User straight to NodeDB::updateUser, whose PKI gates are all `== 32` and so fall through for a partial key, and TypeConversions::CopyUserToNodeInfoLite then stores it with the short size. demoteOldestHotNodesToWarm() admitted that partial key into the warm tier on a `size > 0` gate. WarmNodeEntry has no length field - it distinguishes "has a key" from "no key" purely by all-zero - so N real bytes plus 32-N zeros become indistinguishable from a genuine key. copyPublicKeyAuthoritative() then hands that fabricated key back with size = 32 and reports it AUTHORITATIVE, and re-admission writes size = 32 into the hot store. From then on updateUser's key pin permanently rejects the node's real NodeInfo, and DMs to it are encrypted to a key nobody holds. Require a full 32-byte key, so a partial one is absorbed as "no key" (nullptr) rather than as a truncated one. WarmNodeStore::place() already treats a null key as keyless and clears the slot's stale key when repurposing it. This aligns the site with its two siblings, which both already gate on `size == 32` (the purge path in cleanupMeshDB and the runtime eviction in getOrCreateMeshNode). The ingress gap - updateUser accepting a 1..31-byte key at all - is a separate, larger change and is left for its own review. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(NodeDB): shorten warm-demotion comment to two lines Repo guideline (AGENTS.md): keep code comments to one or two lines. Retains the non-obvious invariant - warm entries have no key length field - and drops the restated detail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(NodeDB): cover short-key demotion into the warm tier A warm record stores 32 raw key bytes with no length field, so a partial hot-store key is indistinguishable from a real one once demoted. The public_key.size == 32 gate in demoteOldestHotNodesToWarm() is what keeps a truncated key from being laundered into a full-looking warm key, but nothing exercised it. test_migration_dropsShortKeyOnDemotion overflows the hot store with one node carrying a 31-byte key and asserts it lands as a keyless placeholder while a genuine 32-byte key still survives. push() grows a keySize parameter to seed the partial key, and clearWarm() gives the test an empty warm tier, which it needs because the warm store outlives setUp() and a prior run's warm.dat. Verified to discriminate: with the size gate reverted to size > 0 the new test fails on "a 31-byte key must not be demoted as if it were a full key", and passes again once restored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(NodeDB): assert the keyless placeholder carries last_heard The test only proved a warm metadata row survived the demotion, not that the placeholder does the job the nullptr is there for, which is preserving last_heard when the key is dropped. Asserting the value needed the seeds fixing first. Warm entries pack role, protected category and the xeddsa flag into the low 7 bits of last_heard (WARM_TIME_MASK is 0xFFFFFF80), so warm time has 128 second granularity and the old seeds of 1, 2, 3 all quantised to 0. They are now multiples of 128, which keeps the demotion ordering identical and makes the values survive the round trip. Real last_heard is epoch seconds, so this is closer to production than the old counter was. Reads the entry through WarmNodeStore::take() rather than getOrCreateMeshNode(), which does not restore last_heard from the warm tier and would have been asserting a path that does not exist. Reported by CodeRabbit on #11431. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com> |
||
|
|
93d15a5368 |
Add AS3935 lightning sensor support (#10931)
* Add AS3935 lightning sensor support Implements meshtastic/firmware#10774: an AS3935Sensor (TelemetrySensor subclass) that reports lightning_strike_count_1h and lightning_distance_km on the normal environment telemetry interval, like a rain gauge - strikes are counted over a fixed rolling ~1h window and read non-destructively, so replying to a peer's telemetry request in between broadcasts can't silently drop counted strikes. The AS3935's IRQ pin (opt-in per board via AS3935_IRQ) is polled with a plain digitalRead() in runOnce(), deliberately not attachInterrupt(): the IRQ line is a level that stays asserted until its interrupt register is read, so polling can't miss an event regardless of timing, matching the SparkFun library's own reference examples. An interrupt would also buy nothing here even setting that aside - classification requires an I2C read (readInterruptReg(), which itself calls delay(2) per the datasheet's settle-time requirement), and blocking I2C/delay() calls aren't safe from ISR context on any of this codebase's target platforms, so the ISR could only ever set a flag for later draining - no less work than just polling the pin directly on the next tick. A genuine lightning classification also requests an immediate out-of-cycle send via a new EnvironmentTelemetryModule:: requestImmediateSend() hook. There's no fixed debounce on the request itself - EnvironmentTelemetryModule's existing airtime/duty-cycle gate already paces every send, so it sends as often as airtime allows rather than an arbitrary fixed rate. The request does expire after 5 minutes unfulfilled, so it can't fire an arbitrarily stale broadcast if airtime was blocked for a long stretch. The AS3935's I2C addresses (0x01-0x03) fall inside the range this codebase's I2C scanner otherwise skips as reserved, so detection is a small dedicated probe gated behind AS3935_IRQ and respecting the caller's address filter, rather than a change to the general scan loop. Presence is confirmed via a register write/readback round-trip rather than a fixed expected value, since the AS3935 has no WHOAMI register and a power-on-reset-only check can't survive a warm reboot that doesn't power-cycle the sensor (initDevice() permanently rewrites that register on first configuration). Generated files under src/mesh/generated/ are intentionally excluded from this commit - they're regenerated from the protobufs submodule by update_protobufs.yml, and hand edits get overwritten and conflict once the companion protobufs PR merges and the submodule pointer updates. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Yong <me@ndoo.sg> * fix(as3935): calibration and telemetry logging initDevice() never called the library's calibrateOsc(). The AS3935's internal oscillators are calibrated against the antenna's resonance, which the AFE/watchdog/spike-rejection thresholds depend on; without it, only a directly-driven IRQ pin (bypassing detection entirely) reacted during testing. The sensor could already have a historical detection event latching the IRQ pin high before our initialization. Added an explicit drain read after the IRQ pin is configured, so the sensor doesn't start out stuck asserting IRQ. EnvironmentTelemetryModule::sendTelemetry() logs every other environment metric category on send but was missing lightning; added a matching log line. Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Andrew Yong <me@ndoo.sg> * Support AS3935 without an IRQ line, make the antenna trim configurable Detection no longer requires AS3935_IRQ. The probe is gated like the other environmental sensors, so an I2C-only breakout is found on any board. Where AS3935_IRQ is defined the pin still gates the I2C read, otherwise runOnce() polls the interrupt register, which latches until read. Antenna tuning capacitance moves to AdminMessage.sensor_config.as3935_config, persisted to /prefs/as3935.dat and defaulting to 96pF. The chip does not retain it across power loss. Disturbers are masked in the chip, since runOnce() now polls every second. The lightning telemetry log is guarded so nodes without the sensor no longer log it on every send. Requires meshtastic/protobufs#981. * Revert protobufs pointer to the develop baseline The submodule bump conflicts on merge and the generated headers come from an out of band CI job, so the pointer moves with that job rather than in this branch. * Report lightning strikes over a true rolling hour strikeCountWindow was zeroed on a fixed interval, so lightning_strike_count_1h reported strikes since the last reset rather than over the preceding hour. RollingCounter is a fixed memory sliding window: one counter per bucket, nothing stored per event, so a storm cannot grow it. The ring holds one bucket more than the window needs so none is recycled while part of it is still inside, and the oldest bucket contributes only the fraction still in range. Both are needed to hold the span at exactly the window length rather than letting it drift by a bucket either way. Expiry is exact to one bucket rather than to the event, which is below the 5 minute floor on mesh telemetry sends. The distance expires with the last strike in the window instead of on the interval reset. Covered by test/test_rolling_counter. * Widen the RollingCounter edge weighting to 64 bit counts * inWindow is a 32 bit product, so a bucket holding more than 2^32 / BucketMs events wraps. At a 5 minute width that is about 14k: a bucket of 50000 reported 11367 instead of 40000 once it reached the window edge. Below the threshold nothing changes, so lightning was unaffected, but the helper is meant to be reused by counters with far higher rates. test_large_burst_at_window_edge covers it. The existing burst test sampled only inside the window, where the bucket is whole and never weighted. * Trim RollingCounter comments to the house limit --------- Signed-off-by: Andrew Yong <me@ndoo.sg> Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com> |
||
|
|
a5fc95f774 |
fix(mesh): coerce coordinate traffic to the position channel on event builds (#11545)
Under USERPREFS_BLOCK_POSITION_ON_EVENT_CHANNEL every coordinate packet a client aimed at the event channel was rejected with the "Location sharing is disabled on this channel" notification - including the phone's own location feed. Both apps hand a GPS-less node its fix as a POSITION_APP packet addressed to the node itself on channel 0; that packet never leaves the device (Router::sendLocal delivers it locally) but resolved to the event channel and was dropped before PositionModule saw it. Result: the toast on every location tick, and nodes without a GPS never learned a position to share on their private channel. Position traffic now converges on the position channel - findPositionChannel(), the first channel with non-zero on-wire precision, which is never the event channel: - From-us-to-us coordinate packets are exempt from the event block. - Local coordinate sends aimed at the event channel (phone share-location, request-position, waypoints, any module/UI originator) are moved onto the position channel in Router::sendLocal and PhoneAPI instead of rejected. The client notification is only sent when no channel carries positions at all. - A position request DM'd to us on the event channel is answered on the position channel at that channel's precision (request_id preserved, same reply throttle); the requester's coordinates are still not stored, forwarded, relayed or published. want_response from the bitfield is merged before the event-channel decode short-circuit so such requests are seen. - PositionModule::sendOurPosition, positionUnchangedSinceLastSend and MeshService::trySendPosition use the shared helper instead of three copies of the same walk. Non-event builds are unaffected: the coercion compiles out and the helper matches the previous walk. Tests: coverage-event-policy (test_event_channel_phone_api, test_event_channel_router, test_position_precision, test_mqtt, test_nexthop_routing) and the same suites with the policy off. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
8fe246e250 |
fix(mesh): relay foreign packets whose channel hash collides with a local channel (#11544)
* fix(mesh): relay foreign packets whose channel hash collides with a local channel
The channel hash is one byte, so a foreign channel's name/PSK can fold to the
same hash as a local channel (~1/256 per local channel held). Since
|
||
|
|
83fd62b756 |
test(native): add 14 suites for routing, persistence, parsing and identity gaps (#11515)
* test(native): add 14 suites for routing, persistence, parsing and identity gaps Coverage audit of the native test tree; adds the highest-value untested logic as 11 new suites and extends 3 existing ones (200 test functions). New: test_stream_framing, test_nodedb_boot_recovery, test_nodedb_legacy_migration, test_nodedb_v25_roundtrip, test_nodedb_identity_hygiene, test_channel_keys, test_reliable_ack_matrix, test_hop_start_policy, test_routing_response_hops, test_phone_api_config_dump, test_observer. Extended: test_rtc, test_mqtt, test_xmodem. Two source changes the audit produced: - StreamAPI::handleRecStream copied stream->read()'s `cInt < 0` EOF check into the buffer-fed path, where there is no EOF sentinel; with signed char any byte >= 0x80 (START1 is 0x94) aborted the parse. Read the byte as uint8_t directly. Latent on develop (no callers), pinned by test_stream_framing. - Extract the post-decode pre-hop predicate from Router::handleReceived into shouldSkipHandleForPostDecodeHop() (NodeDB.h) so test_hop_start_policy drives the exact expression the router calls. No behavior change. test/state-manifest.tsv declares the suites that construct a NodeDB. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): address review - harden observer dispatch, trim comments Review follow-ups on the coverage-audit suites: - Observable::notifyObservers() erased list nodes while holding an iterator into them, so an observer that unobserves itself from onNotify corrupted the dispatch. Today the only self-detacher (PhoneAPI::onNotify -> checkConnectionTimeout -> close -> unobserve) survives solely because it returns -1 and aborts the chain before the increment; that unwritten contract is now gone. Removal during a dispatch nulls the entry and the outermost notify sweeps afterwards, which keeps self-detach, next-detach and destruction-during-notify all safe without an allocation. Hoisting the next iterator instead would have inverted the hazard and broken the existing next-detach case. Two regression tests added. - Correct the documented caller of shouldSkipHandleForPostDecodeHop: the call is in Router::dispatchReceived, not handleReceived. - Cast hop fields to unsigned at the %u call site in test_hop_start_policy. - Trim the new suites' file headers to the one-or-two-line rule in AGENTS.md. - Rename eight test functions whose names were exactly `test_` + 35 chars: that is the shape of a Lob API key, so trufflehog flagged them as secrets and failed the Trunk CI check. Full 68-suite Docker coverage run matches the pre-change baseline. * test(native): revert the observer dispatch change, keep the contract test Backs out the notifyObservers() deferred-removal hardening from the previous commit. It was reviewer-driven scope creep: nothing in the coverage audit needed it, no test required it, and it changes dispatch semantics in a header with ~76 observe() call sites on native verification alone. The hazard it addressed is not reachable today. The only observer that unobserves itself from onNotify is PhoneAPI (onNotify -> checkConnectionTimeout -> close -> unobserve), and it returns -1, which aborts the chain before the iterator is advanced past the erased node. test_self_detach_with_abort_during_notify stays: it passes against the unmodified dispatch and pins that the -1 is load-bearing, so a later cleanup that "simplifies" it away goes red. The unsafe variant (self-detach returning 0) is documented in a comment rather than tested, since asserting it would be asserting UB. * fix(serial): recover the frame behind a stray framing marker A byte that failed the START2 check was discarded rather than re-tested as a possible START1, so 0x94 0x94 0xc3 ... lost the real frame: one corrupted byte on a noisy UART silently dropped the frame behind it. Re-test the byte in place instead. Applied to both copies of the receive state machine. readStream() is the one that matters in the field - it is the serial path every phone client uses - while handleRecStream() still has no callers on develop. Strictly widens what the parser accepts; no frame that parsed before parses differently. test_stream_framing covers it on both receive paths, plus a run of stray markers and a START1-then-unrelated-byte resync. This was originally documented as a known gap in the framing suite. Fixing it instead was NomDeTom's call on review: a passing test asserting the bad behavior is what makes it hard to change later, and it is the same defect shape as the signedness fix three functions away. Also: use Throttle::deadlinePassed() in test_reliable_ack_matrix rather than a bare millis() compare, matching the house deadline rule. * test(native): cover the stray-marker resync on the buffer path too The stray-marker fix went into both copies of the receive state machine, but only test_stray_start1_before_frame_still_delivers drove both. The repeated- marker and unrelated-byte cases drove readStream() alone, so a regression in handleRecStream() would have gone unnoticed by two of the three. Verified load-bearing: reverting only the handleRecStream() half of the fix turns test_repeated_stray_start1_before_frame_still_delivers red on the new assertion. test_start1_then_unrelated_byte_resyncs stays green under that mutation by design - its failing byte is 0x00, where both branches reset to 0 - and covers the other half of the ternary. Also drops the stale header on test_stray_start1_before_frame_still_delivers, which still described the gap as pinned-as-is after the fix landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(native): make the hop-start truth table assert the rows it prints test_truth_table_summary was six TEST_MESSAGE lines and no assertion, so it reported as a case that could not fail - the anti-pattern #11517 names in its unfinished assertion-presence lint, and the one exception to NomDeTom's "no RUN_TEST without an assertion" pass over this PR. The printed row and the checked expectation now come from one struct, so the summary cannot narrate a table the predicates no longer implement. It also covers the consequence columns the per-row tests do not assert together: classifyHopStart, shouldDropPacketForPreHop and shouldSkipHandleForPostDecodeHop for the same packet, with the expectations gated on MESHTASTIC_PREHOP_DROP. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
ef2be877a5 |
Stop breaking TestUtil.cpp on Windows, dangit! (#11529)
* Stop breaking TestUtil.cpp on Windows, dangit! * Update test/TestUtil.cpp Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
|
bca7c0b480 |
Tom fiddles with the test suite - again (#11517)
* test: make every suite run its own binary, and fail the run when it does not PlatformIO links every native test program to the one $BUILD_DIR/$PROGNAME path and attributes Unity output by text alone, never checking that the source file a case came from belongs to the suite it thinks it ran. Both harnesses had been split into a build pass (--without-testing) and a run pass (--without-building), and for a non-embedded platform the run pass never relinks - so all 57 suites executed whichever suite was linked last, each reporting PASSED under its own name. Introduced for CI in |
||
|
|
51eadb77d4 |
fix(NodeDB): reset a persisted event firmware_edition on vanilla builds (#11504)
* fix(NodeDB): reset a persisted event firmware_edition on vanilla builds myNodeInfo lives in devicestate, which survives a firmware reinstall, and the boot-time edition stamp was compiled out entirely on builds without USERPREFS_FIRMWARE_EDITION. A device flashed from an event build back to vanilla therefore kept reporting the event edition forever, and clients kept its branding until a factory reset. Stamp VANILLA in the else branch so the running build is always the source of truth. * Stamp the edition before the boot save decision, and assert the on-disk value Review follow-up: the stamp sat after the devicestate CRC compare, so an edition-only change stayed RAM-only and the persisted event edition survived on disk. Move it next to the other running-build-wins fixups (device_id, min_app_version), which run inside the CRC window, and extend the test to read device.proto back so the persisted value is asserted too. |
||
|
|
f57ee0bd71 |
fix(mesh): restore the implicit ACK for our own overheard PKI DMs (#11502)
* fix(mesh): restore the implicit ACK for our own overheard PKI DMs A DM we originate is PKI-encrypted to the recipient, so when we overhear it being rebroadcast we cannot decrypt it. perhapsHandleReceived() classifies it DECODE_OPAQUE and returns before shouldFilterReceived() runs, which is where the implicit ACK for our own transmission is generated. The client therefore never receives the ROUTING_APP ack it renders as "Delivered to mesh" for a DM, and the message sits in "sending" until it either succeeds outright or times out as max retransmissions. The ACK only needs the packet header (from/id), not the decoded payload, so split it out of shouldFilterReceived() into perhapsGenerateImplicitAckForOwnOverheard() and also call it from the opaque short-circuit for packets that are from us. Behavior on the decodable path is unchanged. Broadcasts on a PSK channel decode normally and always reached the generator, which is why channel messages were unaffected and only DMs showed the symptom. * test: rename implicit-ack tests to avoid a trufflehog false positive The camelCase identifiers tripped trunk's trufflehog/Lob secret detector. * test: shorten one test name past trunk's Lob secret-detector pattern trufflehog's Lob rule matches test_ followed by exactly 35 word characters, which both new test names happened to hit. Unrelated to the fix. |
||
|
|
905482ccce |
fix(serial): don't sleep forever with pending PhoneAPI output on UART consoles (#11500)
* fix(serial): don't sleep forever with pending PhoneAPI output on UART consoles Since #11164 bounded the stream drain, a config dump can end a dispatch with output still queued. On UART-console ESP32 boards runOnce() then returns INT32_MAX with no RX pending, and neither rxInt() nor onNowHasData() fires for the remaining output, so the download wedges mid nodeinfo stream until the client happens to send a byte. Add StreamAPI::hasPendingOutput() (transport-retained frame or queued PhoneAPI data) and have SerialConsole::runOnce() short-poll (<=25ms) while it holds instead of sleeping INT32_MAX. The #11164 write budget is unchanged; idle sleep behavior with a drained queue is unchanged. The retained-frame probe also covers the ESP32-S2 USB-CDC branch, which takes the same INT32_MAX path. * test(serial): restore scratch NodeDB via tearDown, trim comments to house style A failed TEST_ASSERT longjmps out of a Unity test without running destructors, so RAII cannot restore the swapped nodeDB pointer; install the scratch NodeDB explicitly and restore/delete it in tearDown(), which runs after every test outcome. Also shorten the new comments to the two-line house limit. |
||
|
|
a661fd8cd4 |
fixes #11466 (#11487)
* fixes #11466 * Keep locally-addressed routing feedback out of the phone echo filter allocForSending stamps ACK/NAK packets with from == our nodenum and sendLocal defaults to RX_SRC_RADIO, so the loopback gate never applies. Filtering on isFromUs alone dropped implicit rebroadcast ACKs, duty-cycle and NO_INTERFACE NAKs, and PhoneAPI rate-limit errors on their way to the client. Add coverage through the real RoutingModule, which the mocked one used by the rest of the suite cannot exercise, and correct the test seam comment. * Clean up the temporary RoutingModule in tearDown() A failed Unity assertion longjmps out of the test, so the in-test delete never ran and the module stayed registered in MeshModule::modules for every later test. Track it at file scope, as realNeighborInfoModule already is. |
||
|
|
34680833b8 |
fix(test): make the native-windows test suite build and run (#11482)
* fix(test): make the native-windows test suite build and run pio test -e native-windows failed every suite at the build stage. Five independent causes, all Windows-only: - TestUtil.cpp called lstat(), which MinGW-w64 does not provide. The state-checkpoint walk added in #11322 is fenced with ARCH_PORTDUINO, which native-windows also satisfies, so all 53 suites failed to compile. Route it through a stat() shim on _WIN32. - test_default, test_http_content_handler, test_meshpacket_serializer and test_serial define no setUp/tearDown and relied on the weak defaults PlatformIO emits in unity_config.c. GCC lowers a weak definition on PE-COFF to a weak external, leaving the symbol undefined, so it does not satisfy unity.c's reference and the link fails. Define them explicitly, as the other 49 suites already do. - test_mqtt included <arpa/inet.h>, absent on MinGW, for htonl(). Use winsock2.h there. - test_gps_update_scheduling uses TEST_ASSERT_DOUBLE_WITHIN. Unity omits double support unless UNITY_INCLUDE_DOUBLE is defined, so the assertion compiled to an unconditional failure. Define it for the env. - test_getfiles_rejects_overlong_path is excluded on _WIN32. Overrunning the 228-byte file_name needs at least 229 bytes below the portduino root, and that root is already ~34 bytes, so every qualifying path passes the 260-byte MAX_PATH: the nested mkdir() fails, the file is never created, and getFiles() has nothing to drop. No component layout satisfies both limits. Each of the seven suites that failed on Windows was verified individually after the change. test_fscommon_getfiles still fails in a full run, for a cause outside this change: rmDir() does not remove directories on Windows, so empty dirs left by an earlier run survive setUp() and make getFiles() report a depth truncation. That is a pre-existing FSCommon bug, reported separately. No Linux or macOS behaviour changes: every guard is _WIN32-only except UNITY_INCLUDE_DOUBLE, which is scoped to env:native-windows. * fix(test): define UNITY_INCLUDE_DOUBLE for every native env The flag was scoped to env:native-windows, but the gap is not Windows-specific. Verified on Debian with gcc against the Linux env's own Unity 2.6.1 and PlatformIO's generated native unity_config: UNITY_INCLUDE_DOUBLE : NOT defined UNITY_EXCLUDE_DOUBLE : defined test_double_within:FAIL: Unity Double Precision Disabled UNITY_INCLUDE_DOUBLE appears nowhere in the repo, the ini files, the workflow, or PlatformIO's unity runner, which adds only UNITY_INCLUDE_CONFIG_H. So TEST_ASSERT_DOUBLE_* is an always-failing stub on Linux and macOS too, not only on Windows. Moved to portduino_base.build_flags_common, which every native env resolves: native, native-tft, native-fb, native-tft-debug, coverage, coverage-event-policy, native-macos, native-windows and native-wasm. This does change Linux and macOS: TEST_ASSERT_DOUBLE_* becomes a real comparison instead of a stub. test_gps_update_scheduling is the only suite using those macros and its arithmetic is integer-based and bit-identical across platforms, so it should pass wherever it runs. Note it currently reports PASSED on CI in 0.03s while emitting no Unity output at all, so those assertions appear never to execute there; that is tracked separately and is not addressed here. |
||
|
|
a00675e00c | unset can have what it likes (#11496) | ||
|
|
a400143090 |
fix: improve acknowledged unicast retry reliability (#11320)
* Improve acknowledged unicast retry reliability * Fix merged next-hop routing tests --------- Co-authored-by: Ben Meadors <benmmeadors@gmail.com> |
||
|
|
b565a07a83 |
Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 (#11381)
* Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 BSEC2 cost ~37-39 KB flash and ~4-5 KB static RAM on ~190 of ~240 build targets, linked whether or not a BME680 was attached, and was a no-source proprietary archive inside GPLv3 release binaries. The firmware consumed exactly one BSEC-exclusive output: the IAQ value. - New BME680IaqEstimator: clean-room log-domain baseline tracker (humidity-compensated gas resistance vs a rise-fast/decay-slow ceiling, 0-500 scale matching the existing UI bands), pure math, unit-tested on native (test_bme680_iaq, 15 tests incl. a deep-sleep reboot simulation). Warm-up/burn-in progress persists to /prefs/bme680.dat via SafeFile so one-sample-per-wake SENSOR nodes converge across reboots; stale /prefs/bsec.dat is removed once. - BME680Sensor: single-path rewrite on Adafruit_BME680 with async once-per-minute sampling (~20x lower heater duty than BSEC LP mode), a hard 2-minute publish-freshness bound (a dead sensor stops reporting instead of freezing its last reading on the wire), and suppression of bogus gas_resistance=0 points from heater-unstable cycles. - platformio.ini: environmental_extra_common/_extra/_no_bsec collapsed into one section; Bosch BSEC2 + BME68x deps deleted; per-variant BSEC link-path hacks and the TEMPORARY promicro lib_ignore removed. nrf52_promicro_diy_tcxo regains BME680 support at 36 KB clear of the warm-store cap; rak4631 lands at 75 KB clear. - EnvironmentTelemetry: iaq rendering gates on has_iaq (a genuine IAQ of 0 now displays); stale BSEC comments rewritten. - rak4631 size budgets tightened (113000->108000 RAM, 786000->746000 flash) to lock in the reclaimed headroom. - bin/bme680_iaq_replay.cpp: host-side replay harness for tuning the estimator against captured BSEC traces (mean abs error + band agreement), no reflashing needed. Measured (develop -> this branch): rak4631 -38.8 KB flash / -4.9 KB RAM; heltec-v3 -36.4 KB / -4.0 KB; tlora-v2-1-1_6 +1.3 KB (its IAQ approximation had been dead code since #9663 due to an inverted isfinite check and now actually runs). Note: gas_resistance stays kOhm on the wire for fleet compatibility; the proto comment claiming MOhm gets a separate meshtastic/protobufs docs PR. * Address CodeRabbit review feedback - Use Throttle::isWithinTimespanMs for all elapsed-time predicates in BME680Sensor per coding guidelines (deadline math for the async reading completion stays raw, as it targets an absolute timestamp) - Make the state file name members static constexpr - Replay tool: cast uint16_t before %u (default argument promotion), report malformed input lines instead of silently skipping, and fail non-zero on stream read errors * Address CodeRabbit nitpicks - Replace the local clampf helper with std::clamp (meshUtils.h's clamp drags in Arduino.h, which would break the estimator's standalone host build that the replay harness depends on) - Trim the replay tool's file header to a two-line summary; the full build, capture, and tuning workflow moves to docs/bme680_iaq_replay.md |
||
|
|
230da77642 |
fix(time): convert the millis() rollover sites #11291's CI guard cannot see (#11483)
* MeshPacketQueue: fix millis() rollover in the late-packet drop test replaceLowerPriorityPacket() read `backPacket->tx_after < now`, with `now` taken from millis() on the line above. tx_after is an absolute deadline, so that comparison inverts while the deadline sits on the far side of the 32-bit wrap: a queued late packet reads as not-yet-due for the rest of the wrap window, or every late packet reads as droppable at once. The same statement ordered two deadlines against each other with `backPacket->tx_after > p->tx_after`, which has the same problem. #11291 swept every site where millis() sits next to the comparison operator, and its CI guard matches that shape. Stashing the clock in a local first is the same bug written so the guard cannot see it. Both tests now subtract before comparing: the due test through Throttle::deadlinePassedAt(), and the ordering through the elapsed-since-now form already used in AdminModule's oldest-slot scan. The snapshot comes from Time::getMillis() so the deadlines and the test read one clock, per the convention deadlinePassedAt() documents. The `dt` the log line reports is now derived from the same elapsed value rather than recomputed. Behaviour is otherwise unchanged, save the boundary: deadlinePassedAt() is inclusive, so a deadline landing exactly on `now` reads as due rather than one millisecond early. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * RadioLibInterface: don't widen a uint32_t deadline delta into a 64-bit long TRANSMIT_DELAY_COMPLETED tested whether the front packet was still waiting with long delay_remaining = txp->tx_after ? txp->tx_after - millis() : 0; if (delay_remaining > 0) ... The subtraction is uint32_t. Where long is 32-bit - every embedded target - an already-due deadline lands negative and the packet transmits, which is why this has never been visible on device. Where long is 64-bit (portduino, and the native test build) the same value zero-extends to ~4.29e9, reads as positive, and the packet is rescheduled 49.7 days out. It stays parked until some later notifyLater() with overwrite happens to reset the timer. That is not an edge case. notifyLater() schedules through setIntervalFromNow(), so the thread wakes at or after the deadline; being a millisecond past due is the ordinary path through this branch. Ask Throttle instead. deadlinePassedAt() is the unsigned half-range test, so there is no signed conversion to get wrong at any width, and the remaining delay handed to notifyLater() is computed from the same snapshot. On 32-bit the behaviour is identical, including at the boundary: a deadline equal to now transmitted before and still does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ExpressLRSFiveWay: convert the two remaining raw window checks to Throttle runOnce() dismissed the alert frame with `now > alertingSinceMs + 2000` and chose its poll rate with `now < keyDownStart + 20000`, both against a millis() snapshot in a local. Same rollover inversion as any other naive compare, and invisible to the millis-deadline-check guard because millis() is not adjacent to the operator. update() in the same file was already on Throttle. hasElapsed()/isWithinTimespanMs() with the stored event give the full ~49.7 day range and need no snapshot. Sentinels are unchanged in meaning: `alerting` is the armed flag for alertingSinceMs and is tested first, and keyDownStart == 0 reads as "recent" for the first 20s of uptime exactly as `now < 0 + 20000` did - a poll rate either way. The arm sites move to Time::getMillis() so the writes land on the clock Throttle reads, which also puts them within reach of Time::setTestMillis(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * GPSUpdateScheduling: record whether a search is running, don't infer it elapsedSearchMs() answered "am I searching?" by ordering two raw millis() stamps: searchStartedMs > searchEndedMs. Whichever stamp lands on the far side of the 32-bit wrap reads as the larger one, so the answer inverts once per wrap cycle, in both directions: - a search that started before the wrap and ended after it keeps reading as "searching". elapsedSearchMs() then grows without bound and searchedTooLong() aborts a search that is not running. - a search that started after the wrap, following one that ended before it, reads as "idle". elapsedSearchMs() returns 0, so an unproductive search is never aborted and the receiver stays powered until it locks. Both self-heal at the next informSearching(), which bounds the damage to one GPS cycle - but the ordering test cannot be made wrap-correct, because the two stamps carry no information about which wrap they belong to. It does not need to be. Whether a search is in progress is a fact the three inform*() calls already have in hand; the ordering was only ever standing in for it. Add the flag and set it there. elapsedSearchMs() keeps its unsigned subtraction, which was always the correct part. The file's clock reads move to Time::getMillis() so the suite can drive them across the wrap. Behaviour-preserving in production - Time::getMillis() is millis() unless a test injects a clock. test_gps_update_scheduling/ gains seven cases: the idle/searching/ended states, elapsed exactness across the wrap, both inversion directions above, and reset(). The two wrap cases fail on the old predicate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * MessageStore: date boot-relative messages in uptime seconds A message received before the wall clock is trustworthy is stamped boot-relative and healed by upgradeBootRelativeTimestamps() once the RTC arrives. Both the stamp and the "same boot?" test were millis() / 1000, which wraps every 49.7 days: a stamp taken before the wrap reads as newer than `bootNow` afterwards, so `m.timestamp <= bootNow` declines to heal it and the message shows "???" until it ages out. MessageRenderer's own copy of the test falls the same way and prints invalidTime. Neither produces a wrong time - the guard is what fails safe - but Time::getUptimeSecs() landed in #11291 for exactly this, and does not wrap for 136 years. Both sites take it, which makes the comparison exact rather than merely fail-safe. While here, the autosave tick had its own hand-rolled deadline helper - `reachedMs(now, target)` as `(int32_t)(now - target) >= 0`. Wrap-correct, but a competing idiom for what Throttle::isWithinTimespanMs() already answers, and the signed cast is the form #11291 replaced everywhere else. Deleted; the stamps read Time::getMillis() so the whole path is on one clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * WebServer: drop the hand-rolled millis() wrap branch getAdaptiveInterval() special-cased the wrap by hand: if (currentTime >= lastActivityTime) timeSinceActivity = currentTime - lastActivityTime; else timeSinceActivity = (UINT32_MAX - lastActivityTime) + currentTime + 1; Those two expressions are the same number - unsigned subtraction already computes the difference modulo 2^32 - so this is not a bug, just eight lines reimplementing what Throttle does. It also reads like a site that has thought about the wrap and settled it, which makes it a bad example to copy. Two isWithinTimespanMs() calls against the stored activity stamp, matching ethApiServer's shape for the same adaptive-interval decision. The stamps move to Time::getMillis() so the writes and the reads share a clock. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * MeshPacketQueue: only order elapsed times once both deadlines have passed The late-packet eviction I rewrote compared how long ago each deadline passed: backElapsed < (uint32_t)(now - p->tx_after) That is only an ordering when both deadlines are in the past. An incoming packet whose tx_after is still in the future subtracts to a near-2^32 elapsed, which reads as the most overdue packet in the queue rather than the least - so a full queue would drop the overdue packet it was about to transmit in favour of one that is not ready yet. The comparison it replaced, `backPacket->tx_after > p->tx_after`, got this right away from the wrap; I lost it in the conversion. Classify before ordering: p->tx_after must be unset, or passed, before its elapsed time means anything. Two expired deadlines still order by which is further overdue, which is what the branch is for. Caught by CodeRabbit on #11483. test/test_meshpacket_queue/ pins the branch: the future-dated arrival that started this, both directions of the both-expired ordering, the undelayed arrival, and all of it again with the deadlines and `now` on opposite sides of the wrap. maxLen is 1 so the suite reaches the branch without dragging in CompareMeshPacketFunc and a NodeDB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * ExpressLRSFiveWay: treat "no key pressed yet" as no activity keyDownStart is 0 until the first press of a boot, and the fast-poll window read that as a press at time zero: 100ms polling for the first 20s of uptime with no activity at all, re-triggering once per millis() wrap. The arithmetic this replaced (`now < keyDownStart + 20000`) did the same, so it is not a regression - but the sentinel is exactly what the conventions say to test before the elapsed comparison, and "has there been recent key activity" has an honest answer here. 250ms is the documented floor for not missing presses, so an idle node simply starts there and moves to 100ms on the first press. Also trims the wrap-cases comment in test_gps_update_scheduling to the two-line house limit. Both from CodeRabbit review on #11483. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
3b608b8fc5 |
fix(mesh): keep ROUTING_APP responses when toPhoneQueue is full (#11480)
* fix(mesh): keep ROUTING_APP responses when toPhoneQueue is full #2918 narrowed the overflow policy to evict the oldest entry only for TEXT_MESSAGE_APP and RANGE_TEST_APP, dropping every other portnum. A dropped ROUTING_APP response leaves the phone with no delivery confirmation for a message it sent. Add ROUTING_APP to the eviction list and pin the policy in test/test_tophone_queue. Fixes #11439 * fix(mesh): gate the queue-overflow portnum check on the decoded variant decoded.portnum aliases encrypted.size in the payload union, so an encrypted packet could be read as a privileged portnum by its ciphertext length. Restore config.device.rebroadcast_mode in the test teardown. * test: rename a test to avoid a trufflehog false positive test_text_still_admitted_when_queue_full is "test_" followed by exactly 35 characters, which matches the Lob API key shape and fails trunk check. |
||
|
|
f5314148c2 |
Serialise AirTime behind a lock, and stop handing out its buckets (#11362)
* Copy airtime reports into a caller buffer instead of exposing the array
airtimeReport() returned a pointer into the rotating bucket arrays, so the
caller held a handle to state that logAirtime() and every accessor mutate
underneath it. Copy into a caller-supplied buffer instead, and report failure
for a null buffer, a count past the log depth, or an unknown report type.
ContentHandler owns its buffer and hoists getPeriodsToLog() out of the three
calls that repeated it.
* Cover the AirTime report API and log-dispatch contract
Half of AirTime's surface had no tests: which store each report type feeds,
what airtimeReport() does when misused, how the first sync seeds itself, and
whether calling several entry points in one interval compounds the rotation.
Eighteen tests, asserted through the public API rather than the public bucket
arrays - those arrays are meant to become private, and a test that reads them
would have to be rewritten rather than pinning a contract.
Two of them state a convention that was never written down: the report arrays
are shift-ordered with slot 0 newest, and slot 0 covers only the time since the
last rotation. channelUtilization and utilizationTX use the opposite convention
- a modular ring indexed by uptime phase - and reading one as if it were the
other is a defect that has already happened once.
* Characterise AirTime window decay, TX gates, and sleep behaviour
Thirty-three tests in three kinds. Invariants must hold forever; boundaries pin
off-by-ones a refactor would move; five characterisations encode today's wrong
numbers, each tagged with the phase that will flip it.
Readings are asserted against an event-log oracle - airtime physically on air
inside (now - window, now], computed from a list of completed packets - rather
than against hand-worked constants, so a test states "this matches the
definition" instead of "this looked right when I wrote it".
The characterisations, all measured rather than assumed:
- the window covers (N-1)p + phase but divides by Np, so a steady 10% load
reads 8.33% right after a bucket boundary -> phase 5
- the same load sweeps across bucket phase instead of holding -> phase 5
- the hour window carries the same defect, 10x smaller -> phase 5
- a packet longer than its bucket is credited whole to the bucket
it completed in, so a saturated LONG_SLOW channel reads >100% -> phase 4b
- getSilentMinutes() reads a modular ring as if the index were an
age, so identical airtime gives different answers by phase -> phase 6
Two tests needed correcting during the write, both my expectations rather than
the code: a six-bucket ring sheds whole buckets, so a 30s gap drops three of
five survivors and not "half"; and the oracle sees 59 completions in a 60s
window, not 60, because the one on the lower edge is outside it.
Not written: the planned RX_LOG/RX_ALL_LOG disjointness test. That is a
property of the two radio drivers, which choose one or the other per packet -
it is not observable from AirTime, which records what it is told. The
AirTime-side half is already covered by the routing tests.
* Drop write-only and undefined AirTime members
None of this was reachable:
air_period_tx / air_period_rx file-scope mirrors of airtimes.periodTX/RX,
accumulated, rotated and memset in lockstep
with them but never read out or serialised.
Orphaned when #2552 re-pointed the writes at
bare globals instead of deleting them.
lastUtilPeriod, lastUtilPeriodTX written on every sync, read nowhere
airtimes.lastPeriodIndex written on every rotation, read nowhere
currentPeriodIndex() computes (secs / 3600) % 8 - a modular-ring
index for the one array that is shift-ordered
rather than a ring. Its only two uses were the
dead field above and a log line. It is the
fossil of the same confusion that makes
getSilentMinutes() wrong.
UtilizationPercentTX() declared, never defined
free logAirtime()/airtimeReport() declared, never defined; the latter still
carried the array-returning signature the
previous commit removed, so it actively misled
Also fixes the rotation log line, which read currentPeriodIndex() from inside
the loop although the index is advanced before it - on a multi-hour wake it
printed the same final value once per rotation. It now reports which of the
crossed hours is being rotated.
airtimeRotatePeriod() is kept: it has no caller in the tree either, but unlike
the above it is a defined public method, so out-of-tree callers are plausible.
Measured, not estimated: sizeof(AirTime) 464 -> 456 B, plus 64 B of globals, so
-72 B of static RAM. Padding accounts for the difference from the 66 B the plan
predicted by counting declared bytes.
The whole point of writing the tests first: the suite is green here with zero
test changes.
* Document what the AirTime figures measure and how they are stored
Comments only, but four of the things they replace were false.
The header's example analytics claimed RX_ALL_LOG was "all received lora
packets" and offered "RX_ALL_LOG - RX_LOG = other lora radios". Both radio
drivers pick exactly one of the two per packet, so they are disjoint: RX_ALL_LOG
is airtime we could not parse, the subtraction can go negative, and the total is
TX + RX + RX_ALL. Replaced with the actual contract - four inputs, eight
outputs, the window each spans, and the fact that the three thresholds are
hard-coded members rather than the settings they look like.
Names the two storage conventions on their declarations, because mixing them up
is what makes getSilentMinutes() wrong: channelUtilization and utilizationTX are
modular rings indexed by uptime phase, where the oldest bucket is (current + 1)
% N; airtimes.period* is shift-ordered with slot 0 newest, where the index IS an
age and slot 0 is a partial hour.
Defines the measurement as wall time rather than awake time, and says why: a
sleeping node still hears traffic, and per-node redefinition would make two
broadcast readings incomparable. Records that the 60s figure is published to the
mesh at >= 1h cadence, so what other nodes see is a snapshot - at LONG_FAST and
1% occupancy it reads exactly 0 in about 44% of reports - and that the contention
window it feeds moves in 20-percentage-point steps, so small errors never reach
the backoff.
Finally, states that rotation happens on access rather than on the scheduler
tick, names the test that enforces it, and leaves a TODO pointing at the plan
phases that fix the characterised accuracy defects.
* Serialise AirTime behind a lock proven by a private token
Two mechanisms solving different halves. A lock-free inner core (Windows) holds
all state and all logic; it has no lock and no way to reach one, so nesting is
impossible by construction. A private Held token takes the lock in its own
constructor and is the only thing that can be passed where a core method demands
one, so the lock cannot be forgotten either.
The rule is now uniform with no exceptions to remember: every public method
takes the lock once and delegates. In particular isTxAllowed*() lock like
everything else - before the split they could not, because they called the
public accessors and the lock is not recursive. That asymmetry was the foot-gun
the previous design documented in prose and hoped nobody would trip.
getPeriodsToLog()/getSecondsPerPeriod() still take no lock; they return
compile-time constants and touch no state.
channelUtilization[] and utilizationTX[] were public, so the lock was bypassable
at compile time. They move into the private core. Four test sites reached in;
all four now use logAirtime() plus the virtual clock, and no new test seam was
needed. Nothing in src/ was affected.
The re-entry assert is guarded on PIO_UNIT_TESTING, so it exists in test builds
only. The design sketched #ifdef DEBUG, but nothing in this tree defines DEBUG
or NDEBUG, so either spelling ships the assert to every board - and
nrf52_promicro_diy_tcxo has ~128 bytes of headroom under its 0xEA000 warm-store
cap, which the assert's strings and abort path overrun. It would have worked on
hardware, since the check runs in Held's owner initialiser and so precedes the
blocking take; the objection is that abort()ing a live mesh node is a poor trade
for a bug never seen in the field. Native tests are where it earns its keep
anyway: Portduino compiles Lock::lock() to an empty body, so a nested take there
succeeds silently and nothing else would notice.
Also comments out ScopedBusyAirTime in test_traffic_management. It is inert
twice over: the module holds no reference to airTime at all since hop exhaustion
was shelved, and the fixture never worked anyway - writing the buckets on a
fresh AirTime is undone by the first accessor call, which takes the firstTime
branch and memsets them. It reported 0%, not the 100% it claimed. Left in place,
commented, with both reasons recorded.
Cost on the tightest board in the tree, nrf52_promicro_diy_tcxo: the six phases
together add 96 bytes of flash, leaving it 32 bytes clear of the warm-store
guard. RAM is 72 bytes lower from the dead-state removal. Suite green at 47/47,
with test_airtime unedited apart from the added nesting test.
* Count rotations with the loop variable, not a separate tally
LOG_DEBUG compiles to nothing under DEBUG_MUTE, so the counter's only read
disappeared with it and the tally became write-only. It does not warn today -
this build has -Wunused-but-set-variable on, and it fires for other locals, but
not for one that is only initialised and never read - so it was latent rather
than broken: a stricter flag or -Werror would have failed muted builds only.
Using the loop variable removes the class of problem, since the loop condition
reads it, and drops the elapsedAirtimePeriods-- mutation as a side benefit.
Same iteration count, same output.
Found by compiling nrf52_promicro_diy_tcxo with -D DEBUG_MUTE, which is worth
recording for its own sake: muting logs takes that image from 802 784 to
673 416 bytes, 98.5% to 82.6% of flash. Logging is 16% of the largest nrf52
image, and its 32 bytes of warm-store headroom are a logging-verbosity question
rather than a code-size one.
* Tighten the comments added by this branch
Comment-only: with comments stripped, all five files are byte-identical to the
previous commit.
Removed the references to the planning notes. Those documents are working
material and will go stale; the code should not depend on them. The five
CHARACTERISATION tags now describe the defect they pin and stop there, and the
accuracy TODO names the four defects and points at the tests instead of a plan
file.
Also removed, as noise rather than information:
- comparisons against pre-#11291 behaviour, which nobody reading this needs
- a comment describing the lock restructure as future work, written before it
landed
- speculation ("plausible", "worth pinning so a future...")
- an aside arguing with an arithmetic slip made while writing the test
Kept the mechanical facts that are slow to re-derive: the two storage orderings
and which array uses which, RX_LOG/RX_ALL_LOG disjointness, the locking rule and
the addSpanned() constraint that protects it, why the re-entry assert is
test-only, and the concrete numbers - (N-1)p + phase, 14 164 ms, the 20 pp
contention-window steps.
Net 16 comment lines out of src/, 33 out of test/.
* Gate the AirTime re-entry check on the host, not on testing
PIO_UNIT_TESTING is injected by PlatformIO purely on BUILD_TYPE, with no
platform check, so it is defined on an on-target `pio test` run too. The
check arms before the lock is taken - a nested take blocks forever, so a
later check would never run - which under preemption false-positives on
legitimate contention and races on its own write.
Derive AIRTIME_REENTRY_CHECK once from PIO_UNIT_TESTING && !HAS_FREE_RTOS
and use it at all three sites. Had the three conditions ever diverged, an
on-target test build would fail to compile on a member the header no
longer declares.
* Log AirTime outside the lock it serialises
DEBUG_PORT.log() blocks on a UART write, and `lock` is a plain binary
semaphore with no priority inheritance, so holding it across a log call
lets the main thread stall the radio thread in getTxDelayMsec().
Move logAirtime()'s LOG_DEBUG into the shell, after the Held scope
closes; the shell already has both arguments, so nothing has to be
passed back out of the core. isTxAllowed{ChannelUtil,AirUtil} read into
a local under the lock and warn after it. The log bodies are braced
because LOG_DEBUG compiles away under DEBUG_MUTE and a bare `if (x) ;`
trips -Wempty-body.
Fold the two doubled index calls into `+=` while touching the lines.
* Give each airtime report its own buffer
handleReport() reused one array across the three airtimeReport() calls
and ignored the bool. A failed report would have left the previous
type's data in place and emitted it under the next type's key. Build
each through a lambda whose buffer is zeroed per call, so a failure
emits zeros.
Unreachable today - the count is always PERIODS_TO_LOG and the type is
always valid - but the old shape only read as correct by accident.
* Drop a stray semicolon from the inert-guard comment
* Address external review: name the race, tighten the claims and the tests
The header sold the lock as mechanism without naming a second thread, which
invites the reasonable objection that this is a cooperative OSThread codebase.
There is a real race and it is nRF52-only: NRF52Bluetooth registers its ToRadio
write callback with defer == false, so a phone's packet runs handleToRadio ->
sendToMesh -> Router::send on the Bluefruit BLE task, reading
utilizationTXPercent() and getSilentMinutes() while loopTask may be inside
logAirtime(). ESP32 hands BLE work to the main task and does not have it.
Three claims in the header were wrong or overstated:
- "nesting is impossible by construction" - Windows is a nested class with an
enclosing class's access rights, and `extern AirTime *airTime` is in the
same header, so airTime->anyPublicMethod() from inside it is well-formed
and would hang. Nothing does it; the assert is the backstop. Say that
instead, because the comment below instructs contributors to add helpers
to Windows on the strength of the guarantee.
- "every public method takes the lock exactly once" - two constant accessors
take none and isTxAllowedAirUtil() takes it zero or one times. State the
exceptions where the invariant is stated, not only at the definitions.
- "both radio drivers pick exactly one per packet" - five drop paths log
neither. At most one. Recorded against plan4 rather than fixed here: it
changes a telemetry value.
getPeriodsToLog()/getSecondsPerPeriod() become static constexpr, which removes
them from the locking claim structurally and lets ContentHandler size its
buffer and its count from one constant.
Tests:
- C14's saturated AirTime is installed by a helper and restored in tearDown.
Unity's TEST_ABORT() is longjmp and does not run destructors of automatic
objects, so the scoped guard it replaces would leave airTime dangling into
an abandoned frame on any assertion failure - and the same commit that
added it removed the tearDown reset that did cover that.
- test_getSilentMinutes_counts_minutes_until_enough_ages_out asserted only
`mins <= 60`, which neither return path can violate. The answer is 59.
- test_backwards_uptime_degrades_safely stepped 600s -> 60s, which leaves
elapsedAirtimePeriods at 0, so it never reached the hourly-report branch
its own comment describes. Step by the wrap instead and assert the exact
figures.
- test_airtime leaked EU_868 out of the duty-cycle case into every later one,
and the reentry test's isTxAllowedAirUtil() coverage depended on it.
Restore the region in tearDown and set it explicitly where it is wanted.
- Rename that test to what it can actually check: no single method takes the
lock twice. The calls are sequential, so it cannot catch two methods
nesting.
* trunk: suppress trufflehog/Lob false positives in test_airtime
* Address CodeRabbit review: the rotate trace, the cap warn, the backoff
Four findings from the CodeRabbit pass. Two were introduced by this branch,
one is a real inconsistency it inherited, one is a naming slip.
The rotate trace was the one that mattered. "Log AirTime outside the lock it
serialises" moved the per-packet lines and the two TX-gate warnings out to the
shell, but missed LOG_DEBUG("Rotate airtimes, crossed hour %u") because it does
not sit in the shell at all: it is inside Windows::syncNow(), the lock-free
core, which by construction only ever runs under Held. Nothing at that line
looks like a lock, which is why it survived.
The exposure is smaller than the review suggests - runOnce() syncs at 1 Hz, so
in steady state this is one line an hour, and the PERIODS_TO_LOG - 1 burst
needs an hour of light sleep with no intervening sync - but a UART write under
a plain binary semaphore with no priority inheritance is exactly what the
comment above logAirtime() says this code does not do. syncNow() now
accumulates crossings in rotationsPendingLog and runOnce() drains it inside the
Held scope, then logs after release. Any caller can cross an hour; only that
thread reports it, so a crossing raised elsewhere is traced at most one tick
late. The `if (rotations > 0)` guard keeps the drained value read under
DEBUG_MUTE, where LOG_DEBUG expands to nothing - the write-only tally that
"Count rotations with the loop variable" removed.
addFromContact()'s favorite fallback stamped silently when the protected cap
refused it. The stamp is new on this branch; the two sibling refusals (ignore,
verify) both emit PROTECTED_CAP_WARN_FMT, so the operator lost the only signal
that the cap was hit on the one path that has a fallback.
lfs_assert() mixed clocks: Throttle read Time::getMillis(), the remainder was
computed from a second, bare millis(). The review's stated failure mode - a
native test overriding the clock - cannot happen, since the hook is behind
PIO_UNIT_TESTING and this file is nRF52-only. The real defect is the second
read: a tick landing on the 20-minute boundary between the check and the
subtraction underflows the remainder into delay(~50 days), on a device that has
just found its flash corrupt. One read, clamped, and preFSBegin() stores from
the same clock.
The eviction test is renamed to
test_eviction_prefersCurrentBootStampOverPost2038Epoch. The finding is right
that it was snake_case, but the suggested testEvictionPrefers... does not match
this file either, which is test_<area>_<camelCase> throughout.
Not taken, both pre-existing and out of scope for a rollover branch:
- t5s3_epaper's touchResumeAtMs/suppressFromMs read an active suppression as
inactive if the wake lands in the 1 ms where millis() is 0. Consequence is
one skipped 150 ms touch-settle window per 49.7-day wrap.
- NRF52Bluetooth::onPairingPasskey() busy-waits 30 s in a BLE callback. Worth
saying plainly that this branch makes it more visible: the old
`millis() < start_time + 30000` overflowed at the wrap and cut the wait
short, so the correct Throttle form is what lets it run the full 30 s.
Reworking it into an OSThread is its own change.
Native suite GREEN, 48/48, 672 cases.
|
||
|
|
6745995442 |
docs: move the firmware design docs to the documentation site (#11488)
The five documents under docs/ were written in this repo while their features
were developed. Four of them describe shipped, upstream behaviour and belong on
meshtastic.org, where users and client authors will look for them:
traffic_management_module.md -> configuration/module/traffic-management
+ development/reference/traffic-management-internals
node_info_stores.md -> development/reference/node-info-stores
mesh_beacon_module.md -> configuration/module/mesh-beacon
+ development/reference/mesh-beacon-internals
+ development/device/mesh-beacon-client-interface
lora_region_preset_compatibility_client_spec.md
-> development/device/region-preset-compatibility
Each is split by audience: settings pages carry the config surface in user
terms, reference pages carry firmware mechanism, and the device pages carry the
protocol a client app speaks. The region-preset spec always said it should
graduate out of this repo once its protobuf landed upstream, which it has
(FromRadio.region_presets, field 19).
nexthop-routing-reliability.md is not documentation - it is a working document
with a mitigation plan, a "files to modify" list and commit sequencing. Its
mitigations shipped in #10745, so the plan is history and the analysis is
superseded; it is dropped rather than published.
Comments that cited the deleted files now point at the published pages, and the
NextHop test header cites #10745 instead of the deleted plan.
|
||
|
|
fdb644e0b7 |
Fix millis() rollover in deadline, interval, and timestamp handling (#11291)
* Add native test coverage for the UptimeClock monotonic seam
src/UptimeClock.{h,cpp} shipped without a dedicated test suite. Port the six
tests from the monotonic-time branch (test/test_time), retargeted to the
renamed header.
The wrap test crosses 0xFFFFFFFF via advanceTestMillis() rather than a second
setTestMillis(): setTestMillis() sets clockSourceChanged, which makes
getMillis64() rebase its accumulator and swallow the wrap.
* NextHopRouter: fix 49.7-day millis() rollover in retransmission timing
Resolves the "FIXME, handle 51 day rolloever here!!!" in
NextHopRouter::doRetransmissions() by switching the retransmission-due
comparison from plain unsigned <= to a signed-difference cast.
The previous p.nextTxMsec <= now comparison silently breaks across the
~49.7 day millis() wraparound: pending retransmissions either stall
for the remainder of the wrap window, or all fire simultaneously at
the rollover boundary. Long-running router/infrastructure nodes do hit
this in practice.
The replacement (int32_t)(p.nextTxMsec - now) <= 0 is the standard
Arduino/embedded idiom for rollover-safe deadline checks and behaves
identically to the original for any non-wrap timing.
* Address Copilot review: use unsigned half-range for rollover-safe retransmit check
Review feedback from @Copilot on PR #10227: casting a uint32_t
subtraction to int32_t is implementation-defined in C++ when the
unsigned value exceeds INT32_MAX (even though it works on typical
two's-complement targets).
Switch to the fully well-defined unsigned half-range form:
nextTxMsec is in the past-or-equal iff (now - nextTxMsec) has not
wrapped past 2^31 ms. Future offsets < 2^31 ms wrap into the top
half and read as 'not yet'.
Same semantics as the signed-cast version on every two's-complement
platform we care about, but portable to any conforming C++ impl.
* Use monotonic time for airtime windows
* Document monotonic airtime windows
* Fix test_packet_signing sentinel that #10227's rollover fix inverts
test_C3_invalid_repeated_packet_cannot_ack_or_change_retry_state parked a
pending packet at nextTxMsec = UINT32_MAX to mean "never retransmit", then
asserted that a rejected repeated packet leaves the retry state untouched.
NextHopRouter::doRetransmissions() now tests whether a retransmit is due with
an unsigned half-range compare, (uint32_t)(now - nextTxMsec) < 0x80000000u,
so that retransmission timing survives the ~49.7 day millis() wrap. Under it
now - 0xFFFFFFFF == now + 1, a small positive delta, so UINT32_MAX reads as
~1ms in the past: the retransmit fires and rewrites nextTxMsec, and the test
failed with "Expected 4294967295 Was 6247".
Use a representable future time instead. Production is unaffected either way -
nextTxMsec is only ever written as millis() + d, and UINT32_MAX came from the
test harness alone - so the sentinel is what needs to go, not the comparison.
Special-casing UINT32_MAX in the retransmit path would keep a value that reads
as "expired" under any wrap-correct compare.
The value is held in a local because millis() advances across
runPipelineIngress(), so recomputing it at the assertion would compare against
a different number.
Reported upstream on meshtastic/firmware#10227, whose branch predates this test.
* Make Throttle time-injectable and add hasElapsed()
Throttle backs ~94 call sites, which makes it the highest-leverage place in
the tree to put the clock seam: reading Time::getMillis() instead of millis()
in its three call sites turns all of them into time-injectable code at once,
without touching any of them. The 32-bit millis() wrap is not otherwise
reachable from a native test.
The read is behaviour-preserving - Time::getMillis() returns millis() unless a
test injects a clock - and the full native suite passes with it live.
Also add hasElapsed(), the complement of isWithinTimespanMs(), because 51 of
the 94 call sites are spelled !isWithinTimespanMs and read poorly. Its
boundary is inclusive (>=) since isWithinTimespanMs uses <; both are
documented. It deliberately does not treat lastExecutionMs == 0 as "never
run": call sites pair that test with the interval check themselves, and
absorbing a sentinel into the one helper every module depends on is exactly
the value-overloading hazard being removed elsewhere.
Migrating the existing !isWithinTimespanMs sites is cosmetic and deliberately
left out of this commit.
test/test_throttle/ covers window semantics, both boundaries, the complement
identity, execute()'s first-run and throttled paths, and - the point of the
exercise - a window opened before the wrap closing correctly after it,
including at the 24h interval that is the longest in the tree.
* Stop disarmed deadline sentinels reaching the comparison
Two deadline variables encoded "inactive" as a magic value that only reads as
"never" because the comparison against it is a naive millis() compare. Under
any rollover-correct comparison both invert to "expired ~49 days ago", so they
have to be untangled before those comparisons can be fixed.
Power::reboot() set rebootAtMsec = -1 on platforms with no reboot
implementation, intending "never fire". Every reader already treats 0 as the
disarm value - powerCommandsCheck() tests `if (rebootAtMsec && ...)`, and
AdminModule writes 0 to cancel - so -1 was both wrong and unnecessary. Use 0.
Left as UINT32_MAX it would reboot-loop the moment the comparison is corrected.
ExternalNotificationModule's nag window compared against nagCycleCutoff, which
holds UINT32_MAX once stopped and 1 at boot. isNagging is the real armed flag,
so test it first and short-circuit: a disarmed cutoff can no longer reach the
arithmetic, while an idle module still takes the same sleep path that the
boot-time value of 1 was relying on.
Note this fixes the sentinel only. The comparison itself is still a naive
`nagCycleCutoff < millis()` and remains on the list to convert.
* Fix millis() rollover in every deadline and interval comparison
Roughly 20 sites compared against millis() directly - `millis() > deadline`,
`deadline < millis()`, `last + interval < millis()`. All of them break for
about 24 days after the 32-bit millis() wrap: depending on which side of the
wrap each value sits, the action either stalls for weeks or fires immediately
and repeatedly. The longest affected interval is the 12 hour NTP renewal, a
~50x margin against the wrap, so none of these needed the range - only the
correct comparison.
Add Throttle::deadlinePassed(deadlineMs) for sites that store an absolute
deadline they cannot re-express as "interval since an event". It uses the same
unsigned half-range test as NextHopRouter::doRetransmissions() rather than
introducing a competing signed-cast idiom, and unlike the signed cast it is
defined for every input. Sites that do store an event use the existing
isWithinTimespanMs / hasElapsed. Nothing gained new state.
Because both helpers read Time::getMillis(), every converted site is now
reachable from a native test that drives the clock across the wrap; the
comparison itself is covered directly in test/test_throttle/.
Sentinel handling is the reason this could not be a mechanical rewrite. The
disarm convention is not uniform: 0 means "inactive" for rebootAtMsec,
shutdownAtMsec, alertBannerUntil, fixHoldEnds, suppressUntilMs and
touchResumeBlockUntilMs; 0 means "due now" for ntp_renew, which is forced to 0
at link-up; UINT32_MAX means "inactive" for nagCycleCutoff; and
alertBannerUntil == 0 in isOverlayBannerShowing() means "show indefinitely".
Every inactive marker is arithmetically far in the past, so a correct
comparison fires on it - each site tests its sentinel before the arithmetic,
and keeps the meaning it had.
Two sites carried a second bug found on the way:
BME680Sensor tested (stateUpdateCounter * STATE_SAVE_PERIOD) < millis(). With
a 6 hour period and a uint16_t counter that product overflows uint32_t after
about 198 saves, independently of the millis() wrap. It now measures the
interval since the last save.
EInkDynamicDisplay had `if (previousRunMs > millis()) return;` as a millis()
overflow guard, which skipped rate limiting entirely for the whole post-wrap
period - the bug it meant to prevent. Every check below it already goes
through Throttle, so the guard is removed rather than fixed.
MotionSensor's calibration countdown is converted to a signed delta rather
than deadlinePassed, because it needs the remaining magnitude and not a
boolean; that matches the already-correct check in the same file.
* Remove getMillis64() and use Throttle for the NodeInfo reply window
getMillis64() had exactly one caller and no callers in tests. It also carried
obligations that made it the wrong shape for this firmware: a wrap accumulator
in mutable statics, which is not ISR-safe, and which must be polled at least
once every ~49.7 days or it silently misses a wrap and returns a time ~49 days
short.
Its one caller only wanted to know whether a 12 hour suppression window had
elapsed - which Throttle answers correctly across the wrap without any
accumulator. NodeInfoModule now stores Time::getMillis() in lastNodeInfoSeen
and tests the window with Throttle::isWithinTimespanMs, so the map holds
milliseconds rather than seconds derived from a 64-bit read.
USERPREFS_NODEINFO_REPLY_SUPPRESS_SECS is user-overridable and now feeds a
multiply by 1000, so a static_assert rejects any value too large to express in
milliseconds instead of letting it wrap.
clockSourceChanged goes too. It existed solely to rebase getMillis64()'s
accumulator when a test swapped clock sources, and it made the wrap untestable
through the injection API: setTestMillis() set the flag, so a wrap crossed by
two setTestMillis() calls was swallowed. With the accumulator gone the flag has
nothing to rebase, and the injection API is a plain settable clock.
The three getMillis64 tests are dropped as they no longer describe anything.
One test replaces them, pinning that advanceTestMillis() wraps past
0xFFFFFFFF rather than saturating, since the Throttle wrap tests rely on it.
Also fix eviction in pruneLastNodeInfoCache(): it picked the entry with the
smallest stored stamp, which is the wrong victim once some stamps sit on the
far side of the wrap. It now evicts the largest elapsed time.
* Add CI guard and docs rule against naive millis() comparisons
Fixing the existing sites does not stop the next one being added. The
millis-deadline-check job rejects millis() placed directly next to a comparison
operator, in either order, anywhere in src/. It lives in test_native.yml
alongside suite-count-check, which sets the precedent for a repo-hygiene guard
that CI enforces and bin/run-tests.sh does not.
The correct idioms all subtract before comparing, so none of them match the
pattern. Line comments are stripped first, so documentation is free to name the
broken form - as the guard's own comment and the coding conventions both do.
Writing the check before finishing the sweep turned out to be worth it: it
found roughly 14 sites that a by-hand audit of deadline variables had missed,
including two extra nagCycleCutoff compares, both boot-screen timeouts, and a
6 hour sensor save interval that was also overflowing a uint32_t multiply.
.github/millis-deadline-allowlist.txt covers the cases that are genuinely not
deadline tests. Both current entries are uptime thresholds - "has the device
been up N ms" - with no stored deadline and no event to measure from: a 30s
button holdoff against phantom shutdown from floating pins, and a 10s window
for the OEM boot logo. Each re-crosses its threshold once per wrap, which is
harmless for boot-holdoff logic and not worth new state to avoid. Entries are
keyed on file plus exact source text, without line numbers, so an edit above an
entry does not silently invalidate it.
Locally the guard reports 19 matches before the sweep and 2 after, both
allowlisted.
The Throttle bullet in the coding conventions is rewritten from "prefer
Throttle for rate limiting" to "never compare against millis() directly", lists
all four helpers with when to use which, names the CI guard, and documents the
sentinel hazard with the rebootAtMsec = -1 case that would have become a reboot
loop. Mirrored into AGENTS.md; CLAUDE.md gets a pointer row.
* Trim rollover comments to what the code needs
The comments added with the millis() rollover fixes carried too much of the
investigation that produced them: how many sites were found, which document
recorded them, what the old code used to do. That belongs in the commit history,
not in the source, and some of it was already stale - Power::reboot() still
described the check it disarms as "a naive millis() > deadline" when that
comparison had been fixed in the same series.
What stays is the non-obvious part at each site: which sentinel value the
variable overloads and what it means there, since that differs between call
sites and is what a correct comparison gets wrong. 0 means "not scheduled" for
rebootAtMsec, "renew now" for ntp_renew, and "show indefinitely" in
isOverlayBannerShowing().
Exposition is kept where it earns its place: the Throttle helpers, the uptime
clock's note on why there is no 64-bit variant, and the tests. The Throttle
docs lose only the site count and the "longest interval in the firmware"
statistic, both of which would age badly; the range trade-off between the two
forms is what a caller actually needs.
Comments only - no code changed, verified by diff.
* possible fixes
* Address review feedback on the rollover fixes
- BME680Sensor: checkpoint lastStateSaveMs after a successful write instead of
at the interval test. The first save (IAQ accuracy >= 2) left it at 0, timing
the next save from boot, and stamping before the write deferred the retry a
full period when the write failed. Reads Time::getMillis(), the same clock
Throttle compares against.
- Throttle: add deadlinePassedAt(now, deadline) for loops that snapshot the
clock once and test many deadlines; deadlinePassed() now delegates to it.
NextHopRouter::doRetransmissions() uses it, replacing the inline half-range
compare adopted from #10227 (nightjoker7) - same arithmetic, credited at the
call site - and takes its snapshot from Time::getMillis() so setNextTx()
deadlines and the due test cannot diverge under an injected test clock.
- test_native.yml: set -euo pipefail in the millis-deadline guard, matching the
sibling suite-count job. Without -e a partially failed scan could report "no
violations" from truncated output.
- test_packet_signing: build the not-due deadline from Time::getMillis() rather
than millis(), so the test and the router read one clock.
- test_throttle: cover deadlinePassedAt(), and correct a wrapped-value comment
(0xFFFFFF00 + 400 is 0x00000090, not 0x00000094).
Two review comments were declined: the AirTime mutex (every airTime-> caller
runs in the single cooperative loop, WebServerThread included) and the
MotionSensor 0-sentinel countdown (the calibration frame is only installed
while a window is open).
clod helped out here
* Correct the described failure window of a naive millis() compare
The comments and agent docs said a bare `millis() > deadline` "breaks for ~24
days after the wrap". That figure belongs to the fix, not the bug: it is the
half-range limit of deadlinePassed(), which reads deadlines more than 2^31 ms
ahead as already passed, and the range over which a UINT32_MAX sentinel reads
as passed.
The naive compare's actual failure is an inversion lasting only while the
deadline sits on the far side of the wrap, so it is bounded by the interval:
the action fires immediately and loses its wait, or blocks for about the wait
it should have performed - days for the nRF52 flash-corruption backoff,
one skipped cycle for a seconds-long retransmit timer.
Comments and docs only; the ~24.8 day statements that correctly describe
deadlinePassed()'s own range are left as they were.
clod helped out here
* Restore a monotonic uptime clock and consolidate the wrap counters
Time::getMillisMonotonic() is the getMillis64() shape - a 32-bit wrap
counter carried across reads - promoted to the shared timebase, with
Time::getUptimeSecs() as the derived whole-seconds view. This deliberately
reverses the earlier removal of getMillis64(), and the distinction matters:
removal was right for a lazily-read accumulator with one rare caller, where
a 49.7-day gap between reads silently swallowed a wrap. Here every read is
the poll and AirTime::runOnce() guarantees one per second; the missed-wrap
contract is pinned by a test rather than left as a footnote.
Three private wrap counters collapse into it:
- AirTime::syncNow() takes its seconds from Time::getUptimeSecs() and drops
its lastSyncMsec checkpoint; window rotation is unchanged.
- DeviceTelemetryModule loses refreshUptime()/uptimeWrapCount/uptimeLastMs;
uptime_seconds comes from Time::getUptimeSecs(), which also removes the
0.296s-per-wrap truncation of (0xFFFFFFFF / 1000) * wraps. Its two
interval checks move to Throttle::hasElapsed().
- HostMetricsModule's copies of those members were never read (its uptime
comes from /proc/uptime) - deleted.
Not ISR-safe (unguarded mutable carry): ISRs keep using getMillis(), which
stays a pure read. Audited: no interrupt-context file reads getTime(),
getValidTime(), or the new accessors.
test/native-suite-count 44 -> 45: the bump for test_uptime_clock was lost
in a branch history rewrite, leaving every later value off by one -
run-tests.sh reports AMBER and CI's suite-count-check fails on the current
push until this correction.
* Anchor the wall clock in monotonic milliseconds
getTime() computed elapsed-since-time-set as a 32-bit millis() delta, so a
node that took time once and stayed up past 49.7 days reported a wall clock
one full cycle in the past - and last_heard, rx_time, message and position
stamps all inherited it. The anchor is now the 64-bit monotonic count
(timeStartMsec -> timeStartMs64) and the elapsed term is computed in 64-bit,
so the wall clock is exact at any uptime.
All six anchor writers follow: the five hardware-RTC read branches and
perhapsSetRTC(), which keeps a truncated 32-bit copy of the same instant for
its Throttle-checked rate-limit stamps. The test seams anchor the same way.
Two native regression tests drive getTime() across the wrap through the
Time seam - one anchored before the wrap and read after it, one anchored
after a counted wrap - with the test epoch derived from BUILD_EPOCH so the
plausibility window cannot rot as the build date advances.
* Stamp the rx_time placeholder in monotonic uptime seconds
computeRxTimeStamp() stamped Time::getMillis() when the clock was untrusted,
and reconcilePendingRxTimes() back-calculated with a 32-bit millis() delta -
correct within one wrap, but a placeholder older than 49.7 days aliased to a
small elapsed value and reconciled to a plausible-but-wrong recent epoch:
the exact failure has_rx_time exists to prevent, reachable by an ordinary
unattended router whose phone connects two months in.
The placeholder is now Time::getUptimeSecs(). Both stamps come off the
monotonic counter, so the elapsed term is exact at any age and the aliasing
window is gone outright rather than widened. If elapsed somehow exceeds the
epoch itself, the packet stays un-dated (absent, never wrong) instead of
clamping to a pre-1970 value. Defence in depth: a placeholder that leaks
needs ~50 years of uptime to cross MIN_PLAUSIBLE_EPOCH, where milliseconds
took 18.3 days.
The stream-API reconciliation tests keep their scenarios with the placeholder
unit switched, and ScopedTimeFixture resets the monotonic carry so uptime
seconds are deterministic per case.
* Date nodes heard before the clock arrives, without polluting last_heard
A node first heard while the wall clock was untrusted got no last_heard at
all, and nothing backfilled it once time arrived - the phone showed "Last
heard: unknown" for a node it had just announced. The arrival instant now
waits in a RAM-only sidecar (NodeNum -> uptime seconds, 32 slots,
reuse-oldest - the RouteHealth shape) and is converted to a real epoch on
the clock-becoming-trusted transition, beside the existing rx_time
reconciliation. last_heard itself never holds anything but a real epoch or
0: it persists to flash and the warm tier, where an uptime-relative value
would be meaningless after reboot.
The sidecar's write sites are updateFrom()'s no-trusted-clock path (the
rx_time placeholder already carries the arrival instant, so this is a store,
not a second clock read) and addFromContact's anti-eviction stamps, which
previously wrote a bare getTime() - boot-relative seconds on a clockless
node, the exact value lastHeardIsWallClock() exists to catch. Eviction
ranking honours the stamps: heard-this-boot outranks every stored epoch,
ordered among themselves, so a stamped contact is not the first victim.
PhoneAPI re-reads last_heard at nodeinfo send time: a record prefetched
before the clock became trusted can carry 0 while the store has since been
backfilled, and re-reading at the pop makes handshake ordering (time-set vs
node-list download) irrelevant. Backfill never moves last_heard backwards
and skips the pathological elapsed-exceeds-epoch case. A node evicted to
the warm tier before time arrives is still absorbed with last_heard 0 -
same as before, bounded to the untrusted window.
* Update the agent docs for the monotonic timebase
The conventions bullet asserted there is deliberately no 64-bit millis; the
monotonic uptime clock restored for timestamps changes that contract. State
the split explicitly: Throttle for deadlines and intervals (no carry state),
Time::getMillisMonotonic()/getUptimeSecs() for timestamps, polled by
construction and not ISR-safe.
* Publish the monotonic wrap carry from a single writer
getMillisMonotonic() was a read-modify-write on two unguarded statics, and it
is reached off the main loop: the nRF52 Bluefruit task via
onFromRadioAuthorize() -> PhoneAPI::getFromRadio -> getValidTime(), and the
portduino civetweb workers via the same path. Two readers interleaving inside
the wrap window could each increment the carry, putting every uptime and
wall-clock reading 2^32 ms ahead for the rest of the boot - a permanent ~49.7
day jump in rx_time, last_heard and ClientNotification.time.
Readers no longer write. serviceMonotonic() publishes a snapshot behind a
seqlock and is the only writer; a reader adds its own unsigned elapsed time to
that snapshot, which is exact across the wrap, so it never inspects the
boundary and cannot miscount it. The main loop publishes every iteration, so
the once-per-49.7-days obligation now has the whole window of margin instead of
resting on an instruction-wide race.
AirTime was the guaranteed poller and is now a pure reader, so the two airtime
wrap tests step the clock the way loop() does. The test clock itself is atomic
so a suite can drive it from one thread while others read.
* Re-arm the GPS ephemeris hold when none is in force
The rollover sweep guarded the hold re-arm with `fixHoldEnds != 0 &&`, which
reads like the sentinel rule but inverts this site. The comparison it replaced,
`(fixHoldEnds + GPS_THREAD_INTERVAL) < millis()`, was always true when nothing
was armed - that was the point, since 0 means "not holding" and so is a reason
to arm. With the guard, a publish that cleared the hold without sleeping (the
`shouldPublish && !tooLong && !holdExpired` path, which does not call down())
left hasValidLocation set and prev_fixQual non-zero, so no disjunct held:
nothing re-armed, nothing published, and the receiver stayed powered at the
200ms poll until searchedTooLong() fired.
State the question positively instead. fixHoldInForce() is the only place the
sentinel is interpreted, and both of runOnce()'s decisions derive from it - the
asymmetry is now visible rather than implied, since arming does not require a
prior hold but expiring does. Its `!= 0` test is not redundant with the
arithmetic: deadlinePassed() is an unsigned half-range test, so past 2^31 ms of
uptime the sentinel reads as a deadline ~24.9 days in the future.
Kept beside its caller rather than in a header; the native test build compiles
GPS.cpp, so the suite declares the prototypes.
Also converts the getACK() wait to isWithinTimespanMs(start, interval): it has
both the start instant and the interval in hand, which gives the full 49.7-day
range instead of 24.8 days ahead, and takes its anchor from Time::getMillis()
so the wait is injectable.
* Date the NodeInfo reply window in uptime seconds
The 12h reply-suppression stamp regressed from wrap-immune 64-bit seconds to
raw 32-bit milliseconds, and pruneLastNodeInfoCache() evicts only by node count
and DB membership - never by age. A stable mesh under the node cap therefore
keeps every stamp indefinitely, and once uptime passes 49.7 days an old one
aliases back into the window: `now - stamp` computes as ~0 and a legitimate
NodeInfo request goes unanswered for up to 12h. It self-heals and repeats once
per wrap cycle.
Store Time::getUptimeSecs() instead, which does not wrap for 136 years, and
drop the millisecond conversion the previous shape needed. Entries past the
window are now evicted too: they can only ever decide "don't suppress".
N8-N11 cover the window from both sides, and N10 pins the regression - it needs
a full 2^32 ms of uptime to elapse, not merely a crossing of the boundary,
because that is when a millisecond stamp reads as "answered this instant".
tearDown() now restores the injected clock and C14's region and TX bucket. A
failing assertion aborts the test body, so restoring at the end of it leaked
that state into every later case.
* Update the agent docs for the single-writer clock and sentinel direction
Two rules the preceding three commits changed.
The monotonic clock is no longer maintained by whoever happens to read it:
serviceMonotonic() is the only writer, readers are pure, and calling it from
anywhere but the main loop reintroduces the double-count.
The sentinel guidance gained the half it was missing. It named UINT32_MAX as a
sentinel while prescribing an idiom that only covers 0, and it assumed the
sentinel always means "suppress" - at the GPS fix-hold site it meant "fire",
which is how that regression passed review looking like the rule.
* Name the fix-hold expiry predicate and arm it from the injected clock
holdJustExpired() gives the second reading of the fixHoldEnds sentinel a
name beside the first, so both are pinned by test/test_gps_fix_hold/ and
neither can be respelled at the call site. The old inline form could not
be tested: written as a literal, its guard folds at compile time and the
assertion asserts nothing.
The arm site used bare millis() while the evaluation reads the Throttle
clock; same value in production, but it kept that write out of reach of
Time::setTestMillis(). Remap a deadline that lands on 0, which would
otherwise read as no hold at all.
* Share the extend formula between the clock's reader and writer
getMillisMonotonic() and serviceMonotonic() carried byte-identical wrap
arithmetic. A one-sided edit to either would drift the published carry
from what readers report, so keep one copy.
* Trim the NodeInfo dedup comment to the house limit
* todo note for potential future imrpovments
* fix some simple deadlines
* Trim the hold-expiry test comment to the house limit
* Fix non-blocking uptime publication and pre-clock recency edges (#29)
* fix(time): avoid blocking monotonic readers
* test(time): make paused-publisher check deterministic
* fix(time): address review portability gaps
* Init the eviction sentinel to the newest possible recency
EvictionRecency{} is {0, false}, which evictionRecencyOlder() ranks as older than
every candidate: without the oldestIndex/oldestBoringIndex guards nothing would
ever be selected and a full node DB would stop evicting entirely.
Init to the genuine maximum instead, so the sentinel is correct on its own. The
index guards stay: two independent reasons the scan is right beats one.
* Keep the deadline-guard check name branch protection matches
The guard was widened to cover Time::getMillis() and unqualified getMillis(),
and renamed to suit. Upstream branch protection matches required checks by name,
so a rename means the old name never reports and merges block on a check that
will never arrive.
Widen the guard, keep the name; the descriptive text carries the broader scope.
* Correct native-suite-count to 47 after the develop merge
Upstream #11293 added test_nmea_wpl and took develop's count to 43; this branch
had independently reached 46. Merging develop resolved the counter textually,
keeping 46, while the directory set became the union of both sides at 47.
The suite-count CI gate fails on the mismatch, and it gates the native test jobs,
so the tests themselves were being skipped.
* test(uptime): make the wrap fall where the comment says it does
The concurrent-reader case started at 0xFFFFF000, leaving 0x1000 to the wrap, so
the 0x800 advance annotated "cross the wrap" fell short and the wrap actually
happened during the following 60s advance.
Start at 0xFFFFF800 instead, so the first advance lands exactly on the wrap while
the readers are running and the second is the ordinary time after it - the shape
both comments already described. Total elapsed is unchanged, so the closing
assertion still holds.
* Respond to human comments
* Did I ever tell you about the time I went to Shelbyville? I wore an onion on my belt, which was the style at the time.
* Convert the I2S nag deadline develop dragged in
The HAS_I2S_SPEAKER_NRF52 RTTTL block arrived from develop with a raw
nagCycleCutoff >= millis(), which the deadline guard rejects. Use the same
Throttle::deadlinePassed() form as the two sibling paths in this function.
* Arm the LittleFS format guard with a flag, not a zero timestamp
preFSBegin() runs in the first millisecond of boot, so millis() can legitimately
return 0 there. Both readers of last_format_ms treated 0 as "nothing formatted
this boot", which would skip the repeat-corruption escalation and let a dead
flash reformat-loop instead of reporting FLASH_CORRUPTION_UNRECOVERABLE.
* Note the single-thread contract on AirTime
* Note the AirTime locking TODO, and tighten the thread note
The two constant getters are not constrained, and getSilentMinutes() reads the
buckets without rotating them, so "the accessors mutate" was not accurate.
* trunk: ignore trufflehog false positives on millis-wrap test constants
test_throttle and test_uptime_clock pin dense clusters of hex boundary
constants (0xFFFFFF00u and neighbors) to exercise 32-bit millis()
rollover. trufflehog's Lob detector stitches nearby hex literals into
one candidate string, and the result happens to match a Lob API key
shape - not a secret, just test fixtures.
Same pattern already used for the gitleaks/nodedb-fixture false
positive in this file.
---------
Co-authored-by: nightjoker7 <mattdeering7@gmail.com>
Co-authored-by: Clive Blackledge <clive@ansible.org>
Co-authored-by: Benjamin Faershtein <119711889+RCGV1@users.noreply.github.com>
Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
|
||
|
|
54d6ce833e |
gps: avoid pow() in GPS_HARDSLEEP threshold heuristic (#11179)
* gps: avoid pow() in GPS_HARDSLEEP threshold heuristic GPS::down() used pow(seconds, 1.22) to pick between GPS_SOFTSLEEP and GPS_HARDSLEEP - a curve fit the surrounding comment already describes as "not particularly accurate". On flash-constrained builds where this was the only pow() call site (e.g. wio-e5), it single-handedly pulled in the full double-precision libm pow/rem_pio2 chain for a heuristic threshold decision. Replaces it with gpsHardsleepThresholdMs(), a piecewise-linear lookup over the same curve, sampled at 16 points and verified to track the original formula within ~0.5% for inputs >=10s and ~1.6% for 5-10s (worse only in relative terms below 5s, where the absolute difference is at most a couple of seconds - negligible against update intervals measured in tens of seconds to hours). Signed-off-by: Andrew Yong <me@ndoo.sg> Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> * gps: trim comments to repo convention (1-2 lines) Addresses a CodeRabbit nitpick: the explanatory comments in GPSUpdateScheduling.cpp and test_gps_update_scheduling/test_main.cpp had grown into multi-line blocks with provenance detail that belongs in the commit message, not inline. Trims each to 1-2 lines, keeping only the essential rationale/bounds. Signed-off-by: Andrew Yong <me@ndoo.sg> Assisted-by: Claude Sonnet 5 <noreply@anthropic.com> * Refactor main function to setup and loop for tests Signed-off-by: Thomas Göttgens <tgoettgens@gmail.com> * gps: extend the hardsleep threshold table below 5s and tighten its tests The 0s-to-5s chord read 42% high at 1s, 22% at 2s and 12% at 3s, against the ~1.6% the comment claimed. Adding 1s, 2s and 3s sample points brings the worst error below 10s to 1.60% at 7s. Above 10s it is 0.55% at 728s, unchanged. Tests: sample off-breakpoint values only, including both worst-error inputs (7s and 728s). Replace the 3000ms absolute floor, which made the 1s assertion unfalsifiable given a true value of 2750ms, with 2% and 0.75% bounds. Add breakpoint-exactness and clamp-boundary coverage. --------- Signed-off-by: Andrew Yong <me@ndoo.sg> Signed-off-by: Thomas Göttgens <tgoettgens@gmail.com> Co-authored-by: Austin <vidplace7@gmail.com> Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com> |
||
|
|
2f6906974e | gps: replace GeoCoord::latLongToMeter's spherical trig with equirectangular approximation (#11184) | ||
|
|
af56a11f00 |
Replace native-suite-count file with dynamic test discovery (#11413)
* Derive the native suite count on the fly instead of registering it in a file test/native-suite-count was a manually-maintained register of the test_* directory count, reconciled against the actual directories by bin/run-tests.sh (as an AMBER verdict) and by a dedicated suite-count-check CI job. The reconciliation only ever guarded the file itself: the check that matters - suites that actually ran vs. the test_* directories on disk - already derives its expected count from a directory walk, so the file added a bookkeeping step to every suite addition/removal without adding signal. Remove the file and everything that existed to keep it honest: - bin/run-tests.sh: drop the canonical-count file read, the count-mismatch AMBER verdict, and the [canonical: x/y] suffix; the verdict lines already carry ran/expected from the directory walk. The shuffle seed suffix stays. - test_native.yml: delete the suite-count-check job and its needs: edges. - Docs (copilot-instructions.md, AGENTS.md, test/README.md) and the test-script comments now describe the count as derived from test/test_* at run time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Add suite-shrinkage-check: fail a PR that silently loses a test_* suite With test/native-suite-count gone, nothing in CI noticed the suite set shrinking: platformio test discovers and runs whatever test_* directories exist, and bin/run-tests.sh derives its expected count from the same walk, so a suite directory lost in a bad rebase or an overzealous cleanup just means fewer suites run - every remaining check stays green. Restore that tripwire git-aware instead of file-based: on pull_request runs, compare the test_* directory list at the PR's merge base against the PR result. A vanished suite fails the job unless its name appears in the PR title, PR body, or a commit message in the PR's range - a deliberate removal satisfies that by stating what it removes; an accidental loss cannot. Other events skip: they have no natural base, and PRs are where accidents arrive. No job depends on this one (a skipped job would skip its dependents). Incidentally: test/ currently holds 47 test_* directories while the deleted count file said 46 - the manual register had already drifted, which is exactly the bookkeeping failure mode this replaces. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd * Re-pad the verdict table after shortening the AMBER row Shrinking the AMBER cell left the table's column padding inconsistent, which trunk (prettier + markdownlint MD060) rejects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APCEfNjd1X7ErDHEzT6Dqd --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
d765bd99ca |
Fix the all-zero MAC address for own node (#11409)
* Fix the all-zero MAC address for own node * Increment native suite count from 46 to 47 * Fix condition for copying MAC address in PhoneAPI |