521 Commits

Include renames

Author	SHA1	Message	Date
Ben Meadors	b565a07a83	Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 (#11381) ... * Remove proprietary Bosch BSEC blob; open in-tree IAQ estimator for BME680 BSEC2 cost ~37-39 KB flash and ~4-5 KB static RAM on ~190 of ~240 build targets, linked whether or not a BME680 was attached, and was a no-source proprietary archive inside GPLv3 release binaries. The firmware consumed exactly one BSEC-exclusive output: the IAQ value. - New BME680IaqEstimator: clean-room log-domain baseline tracker (humidity-compensated gas resistance vs a rise-fast/decay-slow ceiling, 0-500 scale matching the existing UI bands), pure math, unit-tested on native (test_bme680_iaq, 15 tests incl. a deep-sleep reboot simulation). Warm-up/burn-in progress persists to /prefs/bme680.dat via SafeFile so one-sample-per-wake SENSOR nodes converge across reboots; stale /prefs/bsec.dat is removed once. - BME680Sensor: single-path rewrite on Adafruit_BME680 with async once-per-minute sampling (~20x lower heater duty than BSEC LP mode), a hard 2-minute publish-freshness bound (a dead sensor stops reporting instead of freezing its last reading on the wire), and suppression of bogus gas_resistance=0 points from heater-unstable cycles. - platformio.ini: environmental_extra_common/_extra/_no_bsec collapsed into one section; Bosch BSEC2 + BME68x deps deleted; per-variant BSEC link-path hacks and the TEMPORARY promicro lib_ignore removed. nrf52_promicro_diy_tcxo regains BME680 support at 36 KB clear of the warm-store cap; rak4631 lands at 75 KB clear. - EnvironmentTelemetry: iaq rendering gates on has_iaq (a genuine IAQ of 0 now displays); stale BSEC comments rewritten. - rak4631 size budgets tightened (113000->108000 RAM, 786000->746000 flash) to lock in the reclaimed headroom. - bin/bme680_iaq_replay.cpp: host-side replay harness for tuning the estimator against captured BSEC traces (mean abs error + band agreement), no reflashing needed. Measured (develop -> this branch): rak4631 -38.8 KB flash / -4.9 KB RAM; heltec-v3 -36.4 KB / -4.0 KB; tlora-v2-1-1_6 +1.3 KB (its IAQ approximation had been dead code since #9663 due to an inverted isfinite check and now actually runs). Note: gas_resistance stays kOhm on the wire for fleet compatibility; the proto comment claiming MOhm gets a separate meshtastic/protobufs docs PR. * Address CodeRabbit review feedback - Use Throttle::isWithinTimespanMs for all elapsed-time predicates in BME680Sensor per coding guidelines (deadline math for the async reading completion stays raw, as it targets an absolute timestamp) - Make the state file name members static constexpr - Replay tool: cast uint16_t before %u (default argument promotion), report malformed input lines instead of silently skipping, and fail non-zero on stream read errors * Address CodeRabbit nitpicks - Replace the local clampf helper with std::clamp (meshUtils.h's clamp drags in Arduino.h, which would break the estimator's standalone host build that the replay harness depends on) - Trim the replay tool's file header to a two-line summary; the full build, capture, and tuning workflow moves to docs/bme680_iaq_replay.md	2026-08-13 13:21:16 -04:00
Tom	6745995442	docs: move the firmware design docs to the documentation site (#11488) ... The five documents under docs/ were written in this repo while their features were developed. Four of them describe shipped, upstream behaviour and belong on meshtastic.org, where users and client authors will look for them: traffic_management_module.md -> configuration/module/traffic-management + development/reference/traffic-management-internals node_info_stores.md -> development/reference/node-info-stores mesh_beacon_module.md -> configuration/module/mesh-beacon + development/reference/mesh-beacon-internals + development/device/mesh-beacon-client-interface lora_region_preset_compatibility_client_spec.md -> development/device/region-preset-compatibility Each is split by audience: settings pages carry the config surface in user terms, reference pages carry firmware mechanism, and the device pages carry the protocol a client app speaks. The region-preset spec always said it should graduate out of this repo once its protobuf landed upstream, which it has (FromRadio.region_presets, field 19). nexthop-routing-reliability.md is not documentation - it is a working document with a mitigation plan, a "files to modify" list and commit sequencing. Its mitigations shipped in #10745, so the plan is history and the analysis is superseded; it is dropped rather than published. Comments that cited the deleted files now point at the published pages, and the NextHop test header cites #10745 instead of the deleted plan.	2026-08-13 16:25:03 +02:00
Tom	de6b23190a	Test suite rebuild (#11322) ... * docs(nodedb): make the native node cap unambiguous The native node cap was stated in four places that disagreed, and the disagreement already caused a wrong diagnosis: a saturated 200-node database looked arithmetically impossible because the cap had been read as 248, computed from a header that does not apply on this platform. The real value is 198. On portduino MAX_NUM_NODES is not a compile-time constant at all - the variant defines it as `portduino_config.MaxNodes`, resolved at runtime, default 200 and settable per host with `General: MaxNodes`. variant.h is reached before mesh-pb-constants.h, so that header's ARCH_PORTDUINO branch never fires and its plausible-looking 250 is dead code. - #error-guard the dead branch rather than leave a wrong number where people grep. The guard found a real defect: seven translation units reach mesh-pb-constants.h without configuration.h (SerialConsole.cpp, StreamAPI.cpp, PacketAPI.cpp, ServerAPI.cpp, PiWebServer.cpp, ServiceEnvelope.cpp, MeshtasticOTA.cpp, and test/TestUtil.cpp), so each was compiling with a different MAX_NUM_NODES - and therefore a different PACKETHISTORY_MAX - than the rest of the build. Each now includes configuration.h first. It cannot be included from mesh-pb-constants.h itself: that reaches SerialConsole.h through DebugConfiguration.h and closes a cycle. - Name the bare 250 in getMaxNodesAllocatedSize() NODEDB_MIGRATION_LOAD_CEILING. It is a decode allowance for files written by larger-cap firmware, not a cap, and it read like one. - Fix docs/node_info_stores.md, which named the wrong source and a "10-250" range that is wrong for native, and the copilot-instructions tunables line that said "portduino 250". * test(harness): give each suite its own scratch HOME and report leftovers Native suites shared one directory. Every suite that constructs a NodeDB loads and saves ~/.portduino/default/prefs/ - nodes.proto, config.proto, channels.proto, module.proto, device.proto, warm.dat, transmit_history.dat - and nothing cleared it, so state leaked suite -> suite within a run and run -> every run after it. A test run could also rewrite a real meshtasticd node database on the same machine. Per-run isolation does not fix this: the leak is generated inside a single run, so the boundary has to be per suite. bin/pio-test-isolate.sh runs each suite in its own scratch $HOME, registered as test_testing_command for env:native and env:coverage so a bare `pio test` and CI get the same boundary, not just bin/run-tests.sh. It runs the binary unchanged and exits with its exit code, so PlatformIO's pass/fail is untouched. Overriding HOME here rather than around `pio` also sidesteps the blocker that a bare HOME= breaks pio's own ~/.platformio/penv/bin/pio lookup. Leftovers are reported as a second axis, PASS/FAIL x CLEAN/DIRTY, because an unintended write has no matching assertion by definition - nobody writes TEST_ASSERT for a save they do not know is happening. The harness asserts it from outside, so it applies to every suite without the author opting in. - Only the *set of changed paths* is asserted, never contents. Hashes answer the boolean "did this change?" and nothing more; content baselines over protobuf bytes would churn on every NodeInfoLite field added, which is how snapshot suites become noise. - Deliberate writes are declared in test/state-manifest.tsv - one central file, suite / flags / mandatory reason. run-tests.sh prints the opt-out count on every run. - Granularity follows the state flag, so the two ship together: per-test by default (TestUtil redefines RUN_TEST to checkpoint after each test, naming the exact test that dirtied things), suite boundary for state=per-suite, where carrying state across test cases is the declared behaviour. - A declared write that does NOT happen is reported as MISSING, not folded into DIRTY. It catches silently broken persistence; a warning for now, since some are conditional. - Graded AMBER, not RED. With isolation in place DIRTY means "undeclared", not "dangerous", and a check that lands red on day one gets switched off. Guard the guard, both halves: state_assert_empty() refuses to run a suite against a sandbox that is not empty (otherwise the after-diff measures against the wrong baseline and reports CLEAN while meaning nothing), and bin/test-state-check.sh drives the real wrapper with fixtures asserting CLEAN / CLEAN / DIRTY / MISSING plus both directions of the empty assertion. A checker that silently matches everything would otherwise pass forever. --write-manifest proposes entries for a human to paste and justify; it never applies them, and neither does CI. * test(harness): stop reporting Unity's exit code as a signal A native suite ends in exit(UNITY_END()), and UNITY_END() returns the failure count. PlatformIO's native runner reads that non-zero exit code as a POSIX signal number, so four failures print "Program received signal SIGILL", five print "SIGTRAP", and the suite is classified [ERRORED] rather than [FAILED]. There is no crash. The signal name tracks the failure count and nothing else - it moved SIGILL -> SIGTRAP when a diagnostic probe added a fifth failure - and it cost hours of hunting a memory bug that did not exist, on an env (native) that carries no sanitizer at all. It also explains the phantom extra test case in the totals: the runner adds a synthetic entry for the signal it thinks it saw. run-tests.sh now says so inline whenever a signal line appears, and the three agent-facing docs say it too. * test(admin): isolate NodeDB and globals per test setUp() did `if (!nodeDB) nodeDB = new NodeDB();` and never deleted it, so 83 of the 85 tests shared one never-reset database and never restored config, owner, devicestate or channelFile. The fixture that does restore them was opt-in and armed by exactly two tests. The setUp comment claiming the rest "set their own config/region state and are unaffected" was not true - the admin handlers under test write all four globals. Route every test through the fixture instead: setUp saves the globals and installs a fresh NodeDB, tearDown restores and deletes it. The two tests that armed it themselves no longer need to. All 85 pass, so nothing was silently relying on the shared state. It costs about 7% of the suite's runtime (a NodeDB construction is a loadFromDisk plus, with a region set, key generation) - worth paying to write the phase 3 tests against a clean fixture rather than 83 tests' residue. Also cap the per-test attribution in the run summary at five entries; the full list stays in the suite's sandbox. * test(fs): cover the bounded file-manifest walk getFiles() runs on every phone sync via STATE_SEND_FILEMANIFEST, and nothing asserted any of its bounding behaviour. It does execute unasserted from test_stream_api's handshakes, but the cap, the depth limit, the wasLimited paths, overlong-path rejection and capacity release were all unguarded. Eight tests, all describing what the code does today: today's code is already correct here, since #10778 landed the by-reference collectFiles(), the 64-entry cap, the strlcpy bounds and the swap-idiom release. They pass on arrival, which is the point - this is the baseline a later change has to leave alone. Two things they do not cover, and cannot: - Moving reserve() outside the __cpp_exceptions guard. Exceptions are on natively, so the #else branch is not compiled. The suite's job there is to prove that change alters nothing observable. - The file.name() null guard. No in-tree backend returns null; the guard is defensive. The manifest-release test pins the swap idiom rather than calling PhoneAPI's releaseFilesManifest(), which is file-local. It asserts capacity() == 0, not just size() == 0 - a size-only check passes on clear(), which is the bug #7924 shipped. Suite count 43 -> 44, recounted against the directories rather than copied. * test(admin): assert node-DB metadata saves skip the radio reload set_favorite_node, set_ignored_node and toggle_muted_node each persist a NodeInfoLite bit and nothing else. MeshService::reloadConfig() gates its region re-derivation and configChanged notification on saveWhat & (SEGMENT_CONFIG | SEGMENT_CHANNELS), so a SEGMENT_NODEDATABASE-only save already skips the live radio reconfigure. Pure characterization - all three pass on develop. Worth pinning because that reconfigure is the path implicated in the WisMesh Tag favourite-node crash, and develop asserts nothing about it: widening the saveWhat mask or reordering the check would currently go unnoticed. Ported from the config-save series along with ConfigChangedCounter (an Observer<void *> counting configChanged notifications, the only externally visible signal that the reload branch was taken) and TEST_NODE_NUM. They join the existing suite, so no suite-count change. * refactor(menu): extract the mute toggle into a named function The node menu's mute action was inline in a banner-callback lambda, and that lambda only ever runs via screen->showOverlayBanner() - which is why nothing in MenuHandler.cpp was reachable from a test. Lift the `selected == Mute` branch into menuHandler::toggleNodeMuted(uint32_t) and call it from the lambda. Behaviour-neutral by construction: same statements, same order, same bare saveToDisk(). The null check moves into the function, so the call site no longer needs its own lookup. Verified by the native build and suite; the byte-identical-image check on a headroom-constrained nRF52 board was not run locally - CI's firmware-size comment covers it. Three tests come with it, all describing today's behaviour: - the bit flips both ways and no configChanged fires (develop never calls reloadConfig on this path); - an unknown node is a no-op rather than a write; - and the segment mask. Flipping one NodeInfoLite bit currently rewrites all five segments via bare saveToDisk(). That is asserted deliberately, with the comment naming it as characterization of a known defect: a pending fix narrows it to SEGMENT_NODEDATABASE, and when it lands this assertion is expected to change, which makes the improvement visible in the diff instead of silent. saveToDisk() is not virtual, so the mask is observed through its effect - remove the five prefs files, toggle, and see which reappear. * docs(test): make every suite count a pointer to the canonical one test/native-suite-count is the registered total and is machine-checked against test/test_* on every full run and by the suite-count-check CI job. Every other statement of the count is a copy that drifts: copilot-instructions said 12, AGENTS.md said 19, and the real number is 44. Replace both literals with a pointer to the file, say explicitly that no document should state the count as a literal, and reframe the two suite listings as descriptions rather than inventories - they carry per-suite information the count does not, so they stay, but nothing should infer completeness from their length. Register the new FS suite in both. * test(harness): randomise suite order, reproducibly Landed last, deliberately. Randomising an order-dependent suite set does not find bugs so much as convert a silent pass into intermittent red, and the first instinct is to revert the randomisation rather than fix the coupling. Phases 1-2 removed the coupling; this keeps it removed. Both runners previously hid order dependence behind a fixed order that happened to differ between them, and neither order was chosen: CI's area rules put admin first, PlatformIO's local discovery is reverse alphabetical and put it last. CI was green by accident. - bin/run-tests.sh --shuffle / --seed <n>. The seed defaults to HEAD's short SHA: one order per commit, so a red is replayable and attributable to the diff instead of flaky, while the project keeps exploring orders. Printed at the start and carried into the RESULT line, so a verdict is replayable from that line alone; the full order is printed on failure, because for an order-dependent failure the order is the diagnostic. - The shuffle is a Fisher-Yates over a MINSTD generator rather than awk's rand(), whose sequence differs between gawk and mawk. A seed that does not reproduce the same order on another machine is not a seed. - Shuffling needs one `pio test -f <suite>` invocation per suite - PlatformIO orders by its own os.walk() over test/ and filters only select - which measures at about 4.7s per suite of extra startup. - CI shuffles its area order, seeded from GITHUB_SHA and printed with the command to replay it locally. Intra-area order stays PlatformIO's; controlling it there would mean per-suite invocations, which is a cost worth deciding separately. Also records the 16 measured entries in test/state-manifest.tsv, each with its reason, taken from a full run's --write-manifest output rather than guessed. * test(default): cover the region-throttle interval overload getConfiguredOrDefaultMsScaled(configured, default, nodes, TrafficType) is the overload every telemetry and position module actually calls, and nothing referenced TrafficType anywhere under test/. All four of its behaviours were unguarded: the no-region guard, the throttle <= 1 short-circuit, the multiply, and the 64-bit overflow clamp. The throttles are real, not hypothetical - EU_866 carries PROFILE_LITE, which sets both positionThrottle and telemetryThrottle to 10, so a change here moves broadcast spacing in that region by an order of magnitude. Each test pins numOnlineNodes at the congestion threshold and uses ROUTER, which never congestion-scales, so the coefficient is 1 and the throttle is the only variable. The overflow case needs a base above INT32_MAX/10, hence three days rather than one. * ci(test): keep pull-request suite order fixed, seed the rest Shuffling the area order on every run - including pull_request - would turn a contributor's PR red for an ordering they did not choose, which is how a randomisation gets reverted instead of the coupling being fixed. That is the exact dynamic the ordering work was sequenced last to avoid, and the previous commit walked straight into it. - pull_request keeps the fixed declared area order. - push and schedule shuffle, seeded from the commit SHA: deterministic per commit, printed, attributable, and never blocking someone else's PR. - A suite_order_seed input on workflow_call and workflow_dispatch overrides both, so a specific failing order can be replayed anywhere, including on a PR. The run log prints which mode it took, the resulting order, and the local command to replay it. * ci(test): satisfy CKV_GHA_7 and yamllint on the seed input The seed is reachable through workflow_call, which callers can pass programmatically. The workflow_dispatch copy tripped checkov's "workflow_dispatch inputs MUST be empty" rule, and suppressing it was not worth it: replaying a specific order is a local operation, and the run log already prints the exact bin/run-tests.sh command to do it. * style(menu): apply the node-ID format convention RadioInterface.cpp documents the rule: 0x%08x in logs, !%08x in user-facing display. MenuHandler held every remaining exception - seven logs printing bare %08X, and two display labels doing the same. Repo-wide there are now no bare %08X node IDs left in log calls. * ci(test): pass workflow inputs through env, not shell interpolation suite_order_seed and github.event_name were spliced into the run: script as ${{ }} text, so a value carrying shell metacharacters would execute as code on the runner rather than being read as data. semgrep (run-shell-injection) and zizmor (template-injection) both flag it. Both now arrive as environment variables and are read as "$VAR". * refactor(test): share the seeded shuffle between the harness and CI bin/run-tests.sh and test_native.yml each carried a byte-identical copy of the MINSTD Fisher-Yates awk. The workflow prints "replay locally: ./bin/run-tests.sh --shuffle --seed $seed" after a shuffled CI run, and that instruction is only true while the two agree - drift would be announced by a replay quietly reproducing a different order than the one that failed. Extract shuffle_suites() to bin/lib/shuffle.sh and source it from both. Permutations verified identical across seeds before and after the move. * fix(test): correct the shared-state MISSING check and summary join Three defects in the new harness: state_classify() matched declarations two different ways - state_path_declared() for "undeclared", a hand-rolled regex for "missing". Interpolating an entry into an ERE also let a metacharacter in a manifest name match a file that is not the declared one. Both directions now go through the one helper. `paste -sd'; '` does not join with "; ": with -s, paste cycles through a multi-character delimiter one character per join, so paths rendered as "a;b c;d e". Replaced with an awk join. test-state-check.sh ran on after a failed cd instead of stopping (SC2164). ./bin/test-state-check.sh: 6/6 fixtures pass, MISSING included. * fix(portduino): bound General.MaxNodes MaxNodes was validated only for <= 0. Any positive value, including a typo'd or pasted-in one, propagates to MAX_NUM_NODES and scales both the node DB and the nodes.proto decode ceiling - failing at boot with no obvious cause. The ceiling is a sanity bound, not a capability limit; raise it if a host genuinely needs more. * docs(nodedb): reconcile the capacity tables The property matrix omitted the ESP32-S3 100-node flash tier that the platform table above it lists, and neither mentioned that the WASM build overrides MaxNodes to 80 in wasm_config_apply(). * fix(nodedb): make mesh-pb-constants.h self-sufficient on portduino The ARCH_PORTDUINO #error assumed it was unreachable in a normal build. It is not: the vendored device-ui sources include this header without configuration.h, which broke both native-tft docker builds. Include configuration.h here instead, ahead of every compile-time default - variant.h overrides MAX_RX_TOPHONE as well as MAX_NUM_NODES, so placing it lower in the file just moves the divergence to a redefinition. The #error stays as a backstop for the case where that include genuinely stops providing the cap. Verified with the native env's own flags: a TU including only this header now compiles, normal-order use of both macros compiles, and NodeDB.cpp compiles. * fix(portduino): raise the MaxNodes ceiling to 16000 Marked artificial: nothing in the node DB fails at 16001. 16000 sits just under the 16384 (128 x 128) population where HopScalingModule saturates its sampling denominator and starts dropping nodes, so a host inside the bound still gets meaningful hop recommendations. * lint(trunk): advise on node IDs logged as bare %08x RadioInterface.cpp documents the convention - 0x%08x in logs, !%08x in display - but nothing enforced it, which is how the MenuHandler cluster drifted. 22 call sites in PacketHistory, NodeInfoModule and PositionModule are still off it. A trunk linter rather than a CI grep job, because trunk checks changed files: new violations get flagged without a 22-site cleanup landing in an unrelated PR. Modelled on the existing too-many-defined definition. Scoped to values it can tell are IDs - an ID-shaped argument (->num, .from, getNodeNum) or message text naming one. A 32-bit hex that is not an ID is out of scope, so the CRC32 logs in ethOTA.cpp are correctly ignored. Emits "note", trunk's only non-blocking level: "warning" and "info" both exit non-zero and would gate CI, which is not what a log-format nit deserves. The pre-existing sites are line-scoped in the allowlist, so a new bad call in those same files is still caught. * lint(trunk): stop exempting the known node-id-format sites The seeded allowlist made the rule green by declaring the backlog acceptable. Empty it instead, so the 22 pre-existing sites are reported and get cleaned up by whoever next edits those files. Costs nothing to do: the rule emits "note", so these are non-blocking either way. The allowlist stays for its real purpose - a value the linter misreads as an ID. * style: log node and packet IDs as 0x%08x Clears the 22 sites the node-id-format linter reports, so the rule starts from zero rather than from a backlog nobody can see - trunk suppresses pre-existing findings by default, so left alone these would not have surfaced on edit the way an empty allowlist implies. Format strings only; no argument or control flow changes. The !%08x user-facing display forms are deliberately untouched - that is the other half of the same convention. * test(harness): build once up front, so suite timings mean something run-tests.sh fused build and run in a single pio invocation, so whichever suite PlatformIO's directory walk reached first absorbed the entire src compile and reported it as its own duration. On a real run that made a 0.03s suite report 13m21s, and hid the build cost from every other number in the summary. Do what .github/workflows/test_native.yml already does: one --without-testing build pass, then run with --without-building. Measured on a full 44-suite run - the build is now a single reported figure and 968 test cases execute in 1.9s, with no suite above 0.084s. Build output goes to its own log rather than $LOG: the outcome regexes match "error:" and "[ERRORED]", so a compiler diagnostic sharing that file would read as a test failure. Both red paths now keep the log they quote from. $LOG and the build log are mktemps the EXIT trap removes, so the three grepped lines were previously all anyone ever saw - and the cause is usually further up than the first [FAILED]. * test(harness): keep the run log on every red path bin/pio-test-isolate.sh already keeps a failing or DIRTY suite's sandbox and log under .pio/test-state/<suite>/. What was missing is the cross-suite view: $LOG is a mktemp the EXIT trap deletes, so run-tests.sh quoted three grepped lines from a file that no longer existed by the time anyone looked. Preserve it as .pio/build/<env>/test-failure.log from both red paths - including "no success summary found", which said "see log" while preserving nothing, and which is exactly the case where the build died before any suite ran and so left no per-suite sandbox either. Cleared at the start of every run, so a green run cannot leave a red one's log lying around looking current. * fix(test): report the real failure count on a shuffled red A shuffled run is one `pio test` invocation per suite, all appending to the same log, so the log carries one PlatformIO "N test cases:" summary per suite. verdict_red() took `tail -1`, which reports whatever the LAST suite did: a failure in suite 3 printed a "0 failed" summary from suite 44 directly under "RED - failures detected:". Sum the summaries instead. A single summary line - every unshuffled run - is passed through verbatim, so the familiar output is byte-identical. The patterns are passed to the awk helper as strings rather than /regex/ literals: awk evaluates a regex literal in argument position as `$0 ~ /re/`, so the callee would receive 0 or 1 and silently sum garbage. * fix(test): do not emit an empty suite name for an empty shuffle `printf '%s\n' "$@"` with no arguments still writes one empty line, and both callers read shuffle_suites through mapfile, so an empty suite list arrived as a single suite named "". Return before the printf when there is nothing to shuffle. * test(harness): state and enforce the Linux host requirement The native harness is a Linux tool: bash 4+ (mapfile), GNU coreutils and GNU find (-printf, md5sum, -executable). Most of that predates this branch - mapfile and both find predicates are already on develop - but none of it was written down, so the requirement was there to be discovered rather than read. Refuse to start on a non-Linux uname instead of degrading. On a BSD userland this would not fail cleanly: it would mis-hash the sandbox and mis-read the suite list, and still print a verdict. A state check that silently measures the wrong thing is worse than one that declines to run. Carrying a per-host fallback was the alternative, and it buys a second code path that nothing in CI exercises. bin/test-native-docker.sh already exists for macOS and non-Linux hosts, and the native-macos PlatformIO env is a build target for meshtasticd, not a test host - the isolation wrapper is registered for env:native and env:coverage only. Documented in the script header, test/README.md, and both agent docs. * fix(test): terminate every suite with exit(UNITY_END()) Two sites across two suites ended on a bare UNITY_END(). That ends the reporting, not the suite: setup() returns, the runtime goes on calling loop(), and the process runs forever. PlatformIO does not notice - it reports a suite from its Unity output, not from process exit - so the suite passes, the run goes green, and the binary stays resident. Thirteen of them had accumulated on one dev box, the oldest 19 hours old. The costs are quiet by construction: - the per-suite sandbox is deleted underneath a live process, so its CLEAN/DIRTY verdict describes what the suite had written when the harness stopped looking, not what it left behind; - .gcda coverage and LeakSanitizer's report both flush from atexit handlers, so a suite that never exits contributes no coverage and gets no leak check; - each survivor pins its own deleted 94 MB binary, which du cannot see. One of the two is the #else of an architecture guard, which is the easiest one to get wrong - it looks like there is nothing to clean up. test_mqtt has a correct exit(UNITY_END()) in its live branch, so a "does this file call exit() anywhere" check passes the file whole. test_serial had two more. develop's serial-config validation rework restructured that suite - the architecture guard is gone and both remaining branches now exit correctly - so this commit no longer has anything to change there; bin/lint-unity-exit.sh, added later on this branch, is what keeps it that way. test/README.md gets a section on it, since the skeleton showing the right shape had not stopped this happening. * test(harness): detect and reap suites that outlive their run A suite that never exits was invisible: PlatformIO reports a suite from its Unity output, so the run stayed green while the binary kept running. Two checks, because they fail differently. Runtime, in bin/pio-test-isolate.sh: the sandbox $HOME is mktemp-unique per suite, so any process still holding it is a survivor of that suite. Matching on the environment rather than a remembered PID identifies one whatever its parentage - a fork, a grandchild, a process already reparented to init - none of which a $! comparison catches. Reaped before the after-fingerprint is taken, so that fingerprint measures a tree nobody is still writing to, and so a run cannot leave processes accumulating on the host. Recorded as a sixth summary column and graded AMBER: the tests did pass, but the CLEAN verdict and the coverage were measured under a false assumption. Author-time, as bin/lint-unity-exit.sh, wired into trunk at "note" like node-id-format: every UNITY_END() must be wrapped in exit(). The rule is per occurrence, and that is the point - a file-level "calls exit() somewhere" check passes test_serial and test_mqtt, which have a correct one in their live branch and a bare one in the #else. Running it over the tree turned up test_mqtt, which the file-level pass had missed. It allows `int rc = UNITY_END(); ...; exit(rc)`, used by test_packet_signing to restore globals between the summary and the exit. That is where the rule gives ground: capturing and never exiting would leak and is not flagged. Flagging a correct idiom would push someone to "fix" working code. bin/test-state-check.sh gains a survivor fixture, asserting the wrapper both reports and reaps - a detector that only reports leaves the host accumulating processes, which is half the harm. 8/8. * fix(lint): make the unity-exit scanner statement-aware The rule judged one physical line at a time, which reports two kinds of correct code as bare: /* a comment that happens to mention UNITY_END() */ <- interior lines were never stripped exit( UNITY_END()); <- exit( and the macro never met On a probe of both, two of three findings were wrong. This is a note-level rule whose whole job is advice, and bin/lint-node-id-format.sh already says why that matters: a false positive costs more than a miss. One that cries wolf gets ignored, and the real finding goes with it. Carry /* ... */ state across lines and accumulate logical statements before testing, with a 12-line cap so one unclosed call cannot swallow the rest of the file - the same structure lint-node-id-format.sh uses, so the two custom linters in bin/ work alike rather than each having its own idea. Verified both directions: the develop-era sources still produce the same four findings, the fixed tree produces none, and a probe covering block-comment interiors, wrapped exit(), line comments, return UNITY_END() and capture-then- exit reports only the genuinely bare calls - including a complete block comment followed by real bare code on the same line, which the state machine has to keep live. Reported by CodeRabbit on #11322. * fix(lint): tokenise instead of pattern-matching, and self-test it Second round of review findings on the same scanner, all confirmed by direct test before changing anything. Six defects, one root cause: layered regexes cannot tokenise C++. False positives (correct code reported): - UNITY_END() inside a string literal read as code False negatives (real leaks missed): - a string containing "/*" opened comment state and swallowed later lines - greedy .* removed everything between two block comments on one line, taking a bare call with it - myexit(UNITY_END()) matched the exit() exemption as a substring - x == UNITY_END() and total += UNITY_END() matched the assignment exemption Replaced with a character-level scan carrying comment state, and token-bounded exemptions: exit must be a whole identifier, and the capture form must be a plain `=`. Raw string literals are still not modelled - there are none under test/, and delimiter tracking for a case that does not occur would be untested code guarding untested code, so it is documented rather than guessed at. Also drops the `return UNITY_END()` exemption. It only terminates from main(), there is no main() under test/, and from a helper it just returns a count. bin/test-lint-unity-exit.sh pins all fifteen cases, every false positive and false negative found in review among them. The rule has been wrong twice in a way that looked fine by inspection; it needed a self-test more than it needed another careful reading. Two further findings in the same review: - bin/run-tests.sh dropped PASSTHRU in shuffled mode, so `--shuffle -vvv` built verbosely and then ran quietly. The shuffled loop now forwards EXTRA_ARGS, which is PASSTHRU minus the -f pair it supplies per suite. - bin/run-tests.sh did not guard `cd "$ROOT_DIR"`. And one that did not reproduce: the survivor fixture's glob does find the pid file (verified with the lookup instrumented - the earlier failure was an artifact of running the script from /tmp, where SCRIPT_DIR cannot resolve). The assertion was still weak, because an empty pid took the "not running" branch and passed vacuously. It now fails if the pid was never recorded, and finds the file by search rather than assuming a directory depth. Reported by CodeRabbit on #11322. * fix(lint): report each UNITY_END occurrence at its own location The self-test only asked "did the linter say anything", so it could not have caught a wrong line, a wrong column, or a missing second finding. Fixtures now assert the exact diagnostics as line:col, and the first run of that assertion found two real problems. The caret pointed at the wrong occurrence. For `exit(UNITY_END()); UNITY_END();` the verdict was right but the column was 17 - the wrapped call - because the scanner stripped terminating forms out of the whole statement and then reported the first occurrence it had seen. Two bare calls on one line reported once. Judged per occurrence now, by looking back through whitespace at what wraps it, so both the count and the caret are right. That also needed a position map from strip_noncode(): removing a comment or collapsing a literal shifts every later column, and counting occurrences in the raw line does not recover it either - TEST_MESSAGE("... UNITY_END() ..."); UNITY_END(); has two occurrences in the raw text and one in the code. Four of the expected columns I wrote by hand were also wrong, off by one. The linter was right in every case; the assertions were not. They are computed from the fixture text now rather than pasted from output, because a baseline accepted from the tool it is testing asserts nothing. 17 fixtures, including the two-on-one-line case from review and its mirror. Reported by CodeRabbit on #11322.	2026-08-06 14:05:07 +00:00
TomandClaude Opus 4.8	e26b6bfc5a	TMM & Warmstore key source naming (#11119) ... * one becomes two * warmstore clarify * Address PR review: key-provenance terminology consistency - Log line now says "not key-proven" (gate is XEdDSA OR manual, not just signer) - Rename markKeySignerProvenForTest -> markKeyXeddsaSignedForTest (sets only the XEdDSA bit) - Docs + test comments: "signer bit" -> "XEdDSA-signed bit" clod helped too * Rename signer-proven -> key-proven for broadened provenance predicate Address PR #11119 review: the copyPublicKey()/copyUser() out-parameter and the cache-path replay gate now report entry->keyProven() (XEdDSA-signed OR manually verified), so the "signerProven" name and "signer-proven" comments were misleading. Rename the public out-param to keyProven, the local cachedKeySignerProven to cachedKeyProven, and update coupled callers, log strings, docs headings, and comments to say "key-proven". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * nitpicks --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-31 10:39:39 +00:00
Tom	06577e6b36	tying up some loose ends (#11292)	2026-07-30 13:33:31 +00:00
3 people	1872e5b306	TMM extend to ephemeral 3rd tier store (#11050) ... * Harden TMM NodeInfo direct-response: staleness, key hygiene, throttle The NodeInfo direct-response path answered queries on behalf of other nodes from an unauthenticated, never-expiring cache while suppressing the genuine request (STOP). That let a long-gone or forged identity be served as a fresh-looking, authoritative reply indefinitely. Three fixes: - Staleness gate: refuse to spoof a reply for a node not actually heard within ~6h (PSRAM cache via lastObservedMs; NodeDB fallback via sinceLastSeen, which tolerates last_heard==0 when the clock is unset). A stale entry now falls through so the real request propagates instead of being answered for a dead node. - Key hygiene: mirror NodeDB::updateUser() PKI checks when caching overheard NodeInfo - reject a packet advertising our own public key, and pin keys (drop a NodeInfo whose key mismatches an already-known key from NodeDB or from our own cache). Prevents cache poisoning / key substitution via the spoofed-reply path. - Response throttle: at most one spoofed reply per target node per 30s. Direct responses bypass the per-sender rate limiter (they STOP the request first) and the reply target is attacker-controlled, so this bounds airtime exhaustion and reflected floods. Recorded in a new NodeInfoPayloadEntry.lastResponseMs (PSRAM; self-expiring timestamp compare, no sweep needed). Tests: native NodeDB-fallback staleness (stale drops, fresh serves) plus PSRAM-guarded staleness, throttle, and key-mismatch cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Add TODO(T1-T9) markers for code-review findings on NodeInfo hardening Comment-only. Tags each site flagged in the review of the direct-response throttle/staleness/key-hygiene change so the follow-ups are visible in-context: T1 throttle black-holes distinct requestors; T2 per-target key doesn't bound aggregate TX; T3 non-PSRAM fallback unthrottled; T4 millis-wrap defeats the staleness gate; T5 redundant second O(n) scan for the throttle stamp; T6 duplicated 32-byte key compares; T7 sinceLastSeen() called twice; T8 two 6h constants can desync; T9 clockMs()==0 collides with the 0 sentinel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Address code-review cleanups T6, T7, T8 in NodeInfo direct-response T8: derive kNodeInfoMaxServeAgeSecs from kNodeInfoMaxServeAgeMs so the PSRAM and NodeDB-fallback staleness windows cannot desync. T6: collapse the three duplicated 32-byte key compares in cacheNodeInfoPacket (owner-impersonation + NodeDB pin + cache TOFU pin) into one pubKeysEqual() helper, so the compare lives in one place. T7: compute sinceLastSeen(node) once on the fallback staleness path so the tested age and the logged age cannot diverge (it calls getTime() each time). Remaining review items still marked in-code: T1/T2/T3 (throttle design), T4/T9 (millis-wrap + 0-sentinel), T5 (redundant throttle-stamp scan). Native test_traffic_management suite: 48/48 passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Throttle the NodeDB fallback direct-response path (T3) The per-target NodeInfo response throttle lived only in the PSRAM NodeInfoPayloadEntry, so non-PSRAM boards - which answer from the NodeDB fallback path - emitted spoofed direct replies with no rate limit at all, leaving the airtime/reflection surface the throttle exists to close. Add a single module-global fallback stamp (nodeInfoFallbackLastResponseMs, 4 bytes, guarded by cacheLock) that throttles the fallback path to one spoofed reply per window across all targets. Coarser than the PSRAM per-target throttle, but the fallback path has no per-node slot to stamp, and a global cap still bounds spoofed transmissions - which is the point. Add a native test exercising the fallback throttle window. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Make NodeInfo staleness/throttle wrap- and sentinel-safe (T4, T9) T9: the millis-based fields that use 0 as a "never" sentinel (lastObservedMs, lastResponseMs, nodeInfoFallbackLastResponseMs) could be stamped with a literal 0 during the one-millisecond clockMs()==0 instant at each ~49.7-day wrap, momentarily colliding with the sentinel and disabling the staleness/throttle gate for that entry. Route all such stamps through a new nowStampMs() that maps 0 -> 1; the 1 ms skew is irrelevant to every window these fields feed. T4: the staleness gate's modular age compare is only unambiguous while true age < 2^32 ms (~49.7 days); an entry that lingered unrefreshed that long could wrap to a small age and read as fresh, defeating the gate. Add a NodeInfo eviction pass to the maintenance sweep that drops entries past the serve window, so an entry is removed long before its age can approach the wrap boundary. This also frees slots holding stale identities. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Stamp PSRAM throttle entry by captured index, not a rescan (T5) The post-send throttle stamp did a second full O(n) scan of the PSRAM NodeInfo array under a fresh lock, even though findNodeInfoEntry() had already located the slot at the top of shouldRespondToNodeInfo(). Capture the slot index during that initial lookup and address the entry directly on the stamp path. Because the cache lock is released between the two accesses, the slot could have been evicted or reused, so re-validate node == p->to under the lock before stamping. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Document accepted per-target throttle tradeoffs (T1, T2) Convert the T1/T2 TODO markers into a NOTE: the per-target PSRAM throttle keying and its lack of an aggregate bound are deliberate design choices, not pending work. A distinct requestor being throttled for one window is harmless on a redundant mesh, and keeping the PSRAM path per-target preserves throughput for legitimate multi-target responders (the fallback path already bounds aggregate via its global stamp). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Track signed-provenance of cached NodeInfo public keys Add keySignerProven to NodeInfoPayloadEntry, set when an observed NODEINFO frame's XEdDSA signature was verified (mp.xeddsa_signed). This distinguishes a trust-on-first-use key from one proven to belong to a signer. The flag is monotonic - once proven it stays proven, and the existing key-pin checks forbid the underlying key from changing - so a later unsigned frame cannot downgrade it. A signature can only be verified against a key we already held, so a first-contact key is always TOFU until a later signed frame upgrades it. Groundwork for using the TMM cache as a last-resort public-key source, where this flag serves as a trust tier. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Draw public keys from the TMM NodeInfo cache as a last resort Add TrafficManagementModule::copyPublicKey() and consult it from NodeDB::copyPublicKey() after the hot (NodeInfoLite) and warm (WarmNodeStore) tiers miss. This extends the pool of peers the node can encrypt to: a key the NodeInfo direct-response cache overheard for a node that has since aged out of both NodeDB tiers can still be used. The getter reports whether the key is signer-proven or trust-on-first-use. NodeDB serves TOFU keys here too - the same first-contact trust NodeDB already applies in updateUser() - so the pool actually expands to new long-tail nodes rather than only re-confirming known keys. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Retain keyed NodeInfo entries as a pubkey pool; trust-tier eviction Now that NodeDB::copyPublicKey() draws keys from the NodeInfo cache, the flat 6 h serve-window eviction would discard useful keys the moment a node stops being servable. Split retention: an entry carrying a 32-byte public key is kept for kNodeInfoKeyRetentionMs (7 d), while a keyless entry still expires at the serve window. Both windows stay well under the ~49.7-day millis wrap, preserving the T4 wrap-safety guarantee. Make LRU victim selection trust-tiered to match: a keyless slot is sacrificed before any keyed one, and a trust-on-first-use key before a signer-proven key; within a tier the oldest loses. Mirrors WarmNodeStore's keyed-first admission so the most valuable keys are the stickiest under memory pressure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Test signed-provenance flag and copyPublicKey key-pool source PSRAM-gated tests (run on the ESP32 build, alongside the existing NodeInfo PSRAM tests): - copyPublicKey serves a TOFU key learned from an unsigned NodeInfo and reports signerProven=false - a later signature-verified NodeInfo upgrades provenance to signer-proven while the pinned key bytes stay unchanged - copyPublicKey reports a miss for an uncached node Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Inherit signer provenance from NodeDB when caching a re-found node When the TrafficManagement NodeInfo cache re-caches a node, mark its key signer-proven if NodeDB already knows the node as a verified signer for that same key - even if this particular (unicast/unsigned) frame carried no signature. Previously the flag only upgraded on a frame we verified ourselves, so a node we had already proven elsewhere looked TOFU here. Add NodeDB::isVerifiedSignerForKey(), which checks both tiers - the hot store's signed bitfield and the warm tier's cached signer bit - and requires the key to match so a rotated/mismatched key never inherits a stale verdict. Add WarmNodeStore::isVerifiedSigner() to expose the warm signer bit (the rebase onto develop added the bit itself; this surfaces it for lookups). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Lock NodeInfoPayloadEntry packing with a size assert keySignerProven cost zero bytes: sourceChannel, the two bools, and decodedBitfield are four 1-byte fields that fill a single 4-byte tail word (struct alignment is 4), so the flag consumed former padding rather than growing the 2000-entry PSRAM array. Add a static_assert pinning the entry to sizeof(meshtastic_User) + 20 so a future 5th trailing byte - which would open a fresh word (~8 KB PSRAM across the array) - fails the build instead of silently costing memory, prompting new flags to be packed into existing bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Pack NodeInfo cache booleans into 1-bit fields Fold hasDecodedBitfield and keySignerProven into adjacent uint8_t:1 bitfields so they share a single byte, reserving 6 spare bits for future flags without growing the 2000-entry PSRAM array. sizeof is unchanged (still one packed tail word, sizeof(meshtastic_User) + 20); access is by name exactly as before, so no call sites change. Reorders decodedBitfield ahead of the flags so the two 1-bit members stay adjacent and the compiler packs them together. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Gate NodeInfo replay on signer-proven provenance (compile-time, default on) Add TMM_NODEINFO_REPLAY_REQUIRE_SIGNED (default 1): the direct-response path now only spoofs a reply for a node whose key is signer-proven, in addition to the staleness gate - so replay is based on flagging AND staleness. Replay is a courtesy feature, and vouching for an unverified (trust-on-first-use) identity to other nodes is the risk this closes, so signed-only is the safer default. Define the macro to 0 at build time to also serve fresh TOFU-only nodes. Both paths are gated: the PSRAM path checks the cached keySignerProven flag, the NodeDB fallback checks nodeInfoLiteHasXeddsaSigned(node). The effective gate (TMM_NODEINFO_REPLAY_SIGNED_GATE) is bypassed when PKI is excluded from the build, since nothing can be signed there and the courtesy feature would otherwise be disabled outright. Tests: existing reply-expecting tests establish signer-proven state (a new markKeySignerProvenForTest hook for the PSRAM cache, the NodeDB signed bit for the fallback path); add fallback and PSRAM tests asserting an unsigned/TOFU node is withheld under the default gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Rehydrate re-admitted node names from the TMM NodeInfo cache The warm tier keeps an evicted node's key but not its name (the 40-byte record has no room), so a re-admitted long-tail node is nameless until its next NodeInfo broadcast. The TrafficManagement NodeInfo cache is much larger (2000 PSRAM entries) and commonly still holds the full User, so use it as a name reservoir the warm tier structurally cannot be. On re-admission (getOrCreateMeshNode), after the warm-tier restore, copy the cached identity from the TMM cache via a new copyUser() getter - but only when its cached key matches the key just restored from warm, so a name never attaches to a different identity than the one we encrypt to (key-matched trust). Guarded by HAS_TRAFFIC_MANAGEMENT and a null check; a no-op without the PSRAM cache or when no key is present. CopyUserToNodeInfoLite sets only user-related bits, so the warm-restored signer bit is preserved. Limitations (by design): PSRAM-only, and the TMM cache is RAM-only, so this helps within an uptime session, not across reboot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Drop non-portable NodeInfoPayloadEntry size assert (fixes pico build) The static_assert pinned sizeof(NodeInfoPayloadEntry) to sizeof(meshtastic_User) + 20, but nanopb packs the generated User struct differently per platform: on pico its size is not a multiple of 4, so alignment padding before the uint32 timestamps makes the overhead 22, not 20, and the assert failed the build (native happened to be +20 and passed, hiding it). The bitfield packing it was guarding still stands; replace the fixed-count assert with a comment, since no portable byte count exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G7QggdwMM1CJkguuCmMfZs * Run the NodeInfo cache paths in native tests (TMM_HAS_NODEINFO_CACHE) The NodeInfo direct-response cache and everything layered on it - key pinning, signer provenance, staleness, throttle - was compiled only for ARCH_ESP32 + BOARD_HAS_PSRAM, so its entire test set was skipped by the native CI suite and ran nowhere but real hardware. Fold the scattered platform guards into one TMM_HAS_NODEINFO_CACHE macro that also enables the cache (plain heap, delete[] path already existed) for ARCH_PORTDUINO unit-test builds. Production portduino and embedded test builds are unchanged. Tests that exercise the NodeDB fallback path drop the cache explicitly via a new dropNodeInfoCacheForTest() hook, so both response paths are covered in one binary. Native suite: 60/60 (was 50 with the 10 cache tests compiled out). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit b2dafbbbd6501b42fc27175e599fb38be292b0e3) * Pin NodeInfo cache keys against the warm tier, not just the hot store cacheNodeInfoPacket() mirrored NodeDB::updateUser()'s key pin but checked only getMeshNode() - the hot store. updateUser's own pin effectively covers the warm tier too (getOrCreateMeshNode rehydrates the warm key before the check), so the mirror had a gap: for a node evicted to the warm tier whose cache slot was also gone, an attacker's NodeInfo with a bogus key passed the pin, and the cache's TOFU pin then locked the genuine node's frames out until the poisoned entry aged away. Split the authoritative lookup out of NodeDB::copyPublicKey() as copyPublicKeyAuthoritative() (hot store, then warm tier - never the opportunistic TMM tier, which would compare the cache against itself) and pin against that. Test: warm-tier-only key rejects a mismatching NodeInfo and accepts the matching one. Native suite: 61/61. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit 28e880875ef1f8bf16bdf301a4e6036537a84b4b) * Convert NodeInfo cache clocks to uint8 tick counters Replace the entry's uint32 millis stamps (lastObservedMs, lastResponseMs) with three uint8 free-running tick clocks - the same modular scheme the UnifiedCache counters already use: obsTick 3 min/tick (12.8 h period) - replay staleness gate respTick 5 s/tick (21.3 min) - per-target response throttle retTick 1 h/tick (10.67 d) - retention TTL + LRU age Validity is an explicit flag bit (hasObserved / hasResponded), not a 0 sentinel, and the maintenance sweep saturates a stamp (clears its flag) once its window passes, so no stamp can age toward its ~256-tick aliasing horizon. That retires the wrap/sentinel special cases (T4, T9) - nowStampMs() survives only for the fallback path's module-global millis stamp. obsTick is separated from retention by design: only a genuinely heard NODEINFO frame stamps it, so later membership-based retention refresh can never make a silent node look servable. Also drops lastObservedRxTime, which was only echoed in a debug log. Entry shrinks 136 -> 128 B; the 2000-entry array 272 kB -> 256 kB, 8 kB below the original develop footprint while keeping the throttle and retention features. Tick granularity costs at most one tick per window (+-5 s on 30 s, +-3 min on 6 h, +-1 h on 7 d). Native suite: 61/61. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit e48ed9cfacfed0131b77ed2f1992e8720720d387) * Keep the NodeInfo cache a superset of NodeDB (seed + keep-alive) The cache learned identities only from overheard NODEINFO frames, so its membership was opportunistic: a NodeDB-tier node whose frame was never heard this boot had no entry (no key pin to protect it, no name to rehydrate), and the retention TTL evicted entries for nodes that still lived in the hot store or warm tier. Add an anti-entropy pass to the maintenance sweep (reconcileNodeInfoMembershipLocked): every hot-store / warm-tier node gets an entry - full identity from the hot store (hasFullUser), key-only from the warm tier - with signer provenance inherited key-matched, and NodeDB's key adopted wholesale on conflict. Member entries (isMember) are re-stamped each sweep, so they never age toward the retention TTL; when a node leaves both tiers the keep-alive stops and its entry ages out from its final member re-stamp. Membership also outranks key trust in LRU victim selection, and the reconcile pass skips seeding rather than churn one member out for another when hot+warm exceeds the cache. Two properties are load-bearing: - seeding/keep-alive never touch obsTick/hasObserved, so a seeded or retained entry is never served as a spoofed reply - only genuinely heard frames make a node servable; - copyUser() now requires hasFullUser, so a key-only warm seed can never stamp HAS_USER onto a nameless node via name-rehydration. Native suite: 64/64. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit 4f1b30d838b2f7b11173373b8f1ba97cf91bef6b) * Write-through NodeDB identity commits into the NodeInfo cache updateUser() is the single chokepoint through which every remote identity/key write enters NodeDB (key-verification keys stay in CryptoEngine's pending buffer until a NodeInfo carries them here), so one hook at its tail lets the NodeInfo cache reflect a commit immediately instead of waiting up to a minute for the reconcile sweep - which stays in place as the anti-entropy backstop. onNodeIdentityCommitted() upserts the full User: NodeDB's key is authoritative (a conflicting cached key is stale residue - replaced, provenance dropped), a keyless commit keeps an already-cached TOFU key, and signer provenance transfers only for the committed key. The observation stamp is never touched: knowledge is not observation, so a hook write can never make a never-heard node servable - covered by the new test alongside immediate copyUser/copyPublicKey visibility. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit e160b899728be8d951d4e7bdf5106c777832b917) * Purge TrafficManagement caches on explicit node removal NodeDB::removeNodeByNum() already forgets the node everywhere NodeDB owns - hot store, satellite stores, warm tier - but the TrafficManagement caches kept the deleted identity: the NodeInfo entry went on feeding the key pool (NodeDB::copyPublicKey) and name rehydration, and the unified slot kept its role/next-hop/dedup state, resurrecting the node on next contact. Add purgeNode(): clears both the unified cache slot and the NodeInfo cache entry, called from removeNodeByNum() alongside the warm-tier removal. Removal means full removal; passive eviction never calls this, and the reconcile sweep will not re-seed a node absent from both NodeDB tiers. Test: an observed identity plus a next-hop hint both vanish after removeNodeByNum(). Left for CI to execute per request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT (cherry picked from commit b5564c173337de99e15ef998a5c698550238fd3c) * Document the tmm-fix-2 / tmm-fix-superset reconciliation decisions Two parallel implementations of the same superset design are being combined on this branch. This decision matrix records every divergence and which side each reconciled feature takes, ahead of the code commits that apply them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT * Reconcile retention: no timed eviction, obsTick LRU, hourly seeding Adopt tmm-fix-superset's retention model (reconciliation decisions #3, #4, #9 in .notes/tmm-super-superset-reconciliation.md): - Entries are never evicted on a timer. The 7-day retention TTL and its third tick clock (retTick) are gone; wrap-safety rests entirely on the sweep's presence-bit saturation, and a quiet entry keeps its value (pubkey pool, name rehydration). Slots are reclaimed only by trust/membership-tiered LRU on insert - age scored by obsTick, with never-observed entries counting oldest - or by an explicit purge. - Membership refresh (entry -> NodeDB contains checks) runs every sweep; the heavy seeding pass runs at boot and then hourly, with the write-through hooks carrying immediacy in between. - Tick constants and helpers move into the header beside the existing UnifiedCache tick idiom, with static_asserts tying the tick windows to the fallback path's seconds/ms forms. Kept from tmm-fix-2 (decision #4): the warm tier is still seeded (key-only records via WarmNodeStore::entryAt) so the invariant remains cache superset-of hot AND warm, and the spareMembers guard still stops seeding from churning one member out for another when hot+warm exceeds the cache. Entry shrinks to 128 B (2000 entries = 256 kB). The TTL-based retention test is superseded by a no-timed-eviction test: a quiet keyed entry survives 9 days of sweeps while the serve gate saturates as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT * Reconcile hooks: key-commit write-through, purgeAll, key-matched signer Adopt tmm-fix-superset's hook coverage (reconciliation decisions #5-#8): - onNodeKeyCommitted(): write-through for the two key-write sites that bypass NodeDB::updateUser - the Router's admin-key learn (TOFU-grade) and KeyVerificationModule's manual-verification commit (proven=true, the strongest provenance the cache can carry). Both were coverage gaps in the tmm-fix-2 write-through design. - updateUser's hook call now transfers signer status key-matched (isVerifiedSignerForKey) instead of the node's bare signed flag, and runs on acceptance rather than only on change. - purgeAll(): factoryReset() and resetNodes() clear both TMM tables - removal-is-full-removal applies to bulk resets too, without relying on the usual post-reset reboot. - purgeNode() reuses the existing finders instead of open-coded scans and logs the (user-initiated) purge. - peekNodeInfoFlagsForTest(): flag introspection so saturation and membership tests can assert sweep effects directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT * Merge the two branches' test suites for the reconciled design Port tmm-fix-superset's unique tests, adapted to run natively under TMM_HAS_NODEINFO_CACHE (reconciliation decision #11): - keyHook_upsertsAndGovernsProvenance: TOFU learn -> manual-verification upgrade -> NodeDB-senior rotation resets provenance. - tickSaturation_sweepClearsObserved: the sweep saturates hasObserved past the serve window, the entry persists (no TTL), and a full 256-tick clock wrap cannot alias a saturated stamp back to fresh. - sweepMembershipMarking: reconciliation seeds a hot identity as member+fullUser+unobserved; the next sweep clears membership once the node leaves NodeDB, while the entry itself persists. tmm-fix-2's tests (warm pin, hot/warm seeding, updateUser write-through, removal purge - now also asserting the entry is gone via the new peek hook - and the no-timed-eviction rewrite) were already on this branch. Suite now counts 70 test functions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT * Adapt cherry-picked and seeding tests to the reconciled semantics Two tests needed updating for interactions the validation run surfaced: - The #11035 test (ignoresUnsignedSignerIdentity) was written for a world without the native NodeInfo cache: it needs the NodeDB fallback path (dropNodeInfoCacheForTest) and the signed replay gate satisfied for its target, like the other fallback tests on this branch. - The hot-seed test's "observed frame makes it servable" step now sends a signature-verified frame: its node is a known signer, and per #11035 an unsigned frame from a signer must not (and does not) drive cache writes - the gate working as designed. Full native suite: 70/70 under ASan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWHDDxY4U2XLYW2WJJ2oLT * trunk fml * Doc tidyup * TrafficManagement: key NodeInfo-cache maintenance to its own guard runOnce() nested the entire NodeInfo maintenance block (tick-stamp saturation, membership refresh, boot/hourly reconcile) inside #if TRAFFIC_MANAGEMENT_CACHE_SIZE > 0, so a variant overriding the unified cache to 0 on a PSRAM board would compile the NodeInfo cache without its maintenance: hasObserved would never saturate, obsTick would alias past its 12.8 h wrap, and the 6 h staleness gate - whose wrap-safety explicitly depends on the sweep - would serve spoofed replies for long-gone nodes. Extract maintainNodeInfoCacheLocked(), guarded by TMM_HAS_NODEINFO_CACHE alone, and give runOnce() the same independent-guard structure purgeAll() already has. Also close the runtime cousin of the same mismatch: the write-through hooks (onNodeIdentityCommitted / onNodeKeyCommitted) now no-op while moduleConfig.has_traffic_management is off. Previously they kept filling the cache from NodeDB commits while runOnce() returned INT32_MAX, accumulating entries that were never swept or reconciled. Purges and reads stay ungated: removal must always work, and reads just miss an empty cache. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TrafficManagement: forward throttled NodeInfo requests instead of consuming them The direct-response throttle returned true, which made handleReceived() STOP the request: within the 30 s window a repeat request was neither answered nor forwarded toward the genuine target, and the call site still logged "respond" and bumped nodeinfo_cache_hits for a reply that was never sent. A requester whose first reply was lost on a noisy link got silence for the whole window. Return false instead: the request flows through normal relay handling, so the genuine node (or another cache-holder) can answer, while our own spoofed TX stays bounded. Repeats of the same packet id are already absorbed by the router's duplicate detection, and the stats counter now only counts replies actually sent. Also log purgeNode() only when a slot was actually cleared - it runs for every NodeDB removal, including nodes the caches never tracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TrafficManagement: consolidate NodeInfo-cache #if sprawl into one region Fourteen scattered #if TMM_HAS_NODEINFO_CACHE guards (one per function, each with its own #else stub of (void) casts) collapse into a single guarded region holding every NodeInfo-cache-only function, terminated by one #else block of no-op stub definitions. Because the stubs now exist on every build, the call sites in runOnce(), purgeNode() and purgeAll() drop their guards too; inner guards remain only for orthogonal features (PKI, warm tier) and for real conditional work (constructor allocation, PSRAM paths in shouldRespondToNodeInfo). No behavior change. Compile-checked on both sides of the macro: the native app build (stubs) and the native unit-test build (region). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Router: resolve sender key only for PKI-decrypt candidates perhapsDecode() resolved the sender's public key unconditionally for every encrypted packet, before even checking whether the packet could be PKI-decrypted (channel 0, unicast to us). Since copyPublicKey() grew its TrafficManagement fallback tier, a full hot+warm miss - the common case for channel traffic from senders outside both NodeDB tiers - additionally walked the 2000-entry NodeInfo cache under its lock, per packet, for a key that was then discarded. Move the resolution inside the PKI-candidate branch; remotePublic and haveRemoteKey had no consumers outside it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TrafficManagement: refresh NodeInfo membership hourly, not per-sweep The 60 s sweep re-derived isMember with a per-entry NodeDB lookup: O(entries x members) - about 700k node-number comparisons per minute on a full PSRAM cache (2000 entries x 250 hot + 100 warm), with strided PSRAM reads, all while holding cacheLock against the packet path. Move the refresh into the hourly reconcile pass, which is already O(members x entries): after the seeding loops (so the upsert pass still sees last pass's bits for spareMembers protection), clear every isMember bit and re-mark from both NodeDB tiers - including keyless warm records, which seed nothing but are still members. Accepted tradeoff, now documented on the field: membership lags a passive NodeDB eviction by up to an hour (the entry just stays LRU-sticky slightly longer). Additions stay immediate via the write-through hooks, explicit removals via purgeNode(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * NodeDB: centralize bare-key commits in commitRemoteKey() The two key-write sites that bypass updateUser() (admin-channel learn in Router::perhapsDecode, manual verification in KeyVerificationModule) each carried their own #if HAS_TRAFFIC_MANAGEMENT write-through boilerplate - the pattern this PR itself had to retrofit twice, and which any future direct key write would silently miss, leaving the TrafficManagement cache divergent until the next hourly reconcile. Add NodeDB::commitRemoteKey(n, key32, KeyCommitTrust): writes the key to the hot store and routes the TrafficManagement write-through in one place, with provenance explicit at the call site (AdminChannelProven maps to TOFU-grade, ManuallyVerified to proven). The bypass sites keep their reason for existing - bare-key commits with provenance that updateUser's User-payload/TOFU-pin path cannot express - but no longer know about TrafficManagement at all; both stale includes are dropped (Router's was already unused on develop). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: tick wrap-safety analysis, updated cadences, CodeRabbit doc fixes node_info_stores.md gains a "Tick clocks and wrap safety" section recording which mechanism keeps each modular tick clock honest: the unified-cache clocks pair the 60 s sweep with read-time window resets, the NodeInfo obs/resp clocks are sweep-only (hence the compile-time invariant that maintainNodeInfoCacheLocked() is guarded by TMM_HAS_NODEINFO_CACHE alone), and the warm tier is immune by design - absolute unix-seconds, no wrap until 2106. Also brought current with this series: membership refresh moved to the hourly reconcile, throttled direct-response requests forward instead of being consumed, the commitRemoteKey() bare-key funnel, and the module-disabled gate on the write-through hooks. CodeRabbit doc review (PR #11050): present the NodeInfo payload cache as the third *identity* tier with the unified cache beside the chain, and describe NodeInfoLite's flattened fields / satellite copy-out accessors instead of the removed nested members. Two stale docs/tmm_node_stores.md references in the header now point at the real file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * TrafficManagement: adapt tests to forwarded throttles and hourly membership Throttle tests (PSRAM and NodeDB-fallback paths): a request inside the 30 s window now CONTINUEs into normal relay handling instead of being consumed - assert no spoofed TX, no NAK suppression, and (PSRAM path) that nodeinfo_cache_hits counts only replies actually sent. Membership test (renamed reconcileMembershipMarking): the per-minute sweep no longer refreshes isMember, so the test now pins down both halves of the new contract - the very next sweep after a passive NodeDB eviction still shows the stale member bit (the documented up-to-an-hour lag), and a reconcile interval's worth of sweeps clears it. Disabled-module test additionally proves the write-through hooks share the has_traffic_management gate: a key commit while disabled must not land in the NodeInfo cache. Not covered here: a build permutation with TRAFFIC_MANAGEMENT_CACHE_SIZE overridden to 0 (the configuration the maintenance-guard fix protects) would need its own PlatformIO env plus guards on every unified-cache test - left as a follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * expand test coverage * nitpicks and docs update * more comment fixes * nitpicks * test: make TMM fixture cleanup abort-safe in tearDown() Several tests reset per-test global state (trafficManagementModule pointing at a stack `module`, owner.public_key, the RTC fake clock) only after their assertions. A TEST_ASSERT_* failure longjmps out and skips that trailing cleanup, leaving the state to dangle into later cases - concretely, the next setUp()'s resetNodes() would call trafficManagementModule->purgeAll() on a destroyed object. Move the resets into tearDown(), which runs unconditionally between tests, and drop the now-redundant per-test cleanups. Addresses a CodeRabbit review comment on PR #11050. clod helped too * docs tidy * Throttle NodeInfo direct responses A direct response is addressed to the requesting packet's from field, which is unauthenticated, and is sent by every neighbour that holds the target in cache. One request therefore makes several nodes transmit at an address the requester chose, and nothing limited how often that could be repeated. Replies are now spaced per requester, which bounds how much any single node can be made to receive, plus a global floor on how much airtime the feature can consume. The check sits where a reply is about to be sent, so requests declined for other reasons do not consume the budget. * test: reconcile TMM direct-response tests with #11104 throttle Makes the suite green for now. #11104's per-requester + global-floor throttle (60 s) is layered over the branch's per-target throttle, so the three existing "served again" checks (psram, fallback, sweep) now use a fresh requester to avoid the per-requester window masking the mechanism each one actually exercises. Adds a dedicated test for the new per-requester + 1 s global-floor behaviour (the reflector-flood gap the per-target throttle leaves open, and the bound that survives on non-PSRAM builds). Deferred: the branch's now-redundant fallback global stamp (nodeInfoFallbackLastResponseMs) is left in place as a follow-up cleanup. Clod wants credit. * TMM: unify direct-response throttles into per-sender + per-target RAM tables Replace the two path-specific NodeInfo-spoof throttles (per-entry respTick on the PSRAM cache path, single global stamp on the fallback path) with three symmetric bounds that behave identically with and without PSRAM: - per requester (60 s): how much any one node can be made to receive; - per target (60 s): how often we vouch for the same identity; - global floor (1 s): total airtime, the backstop once an attacker cycles requester/target past the 8-slot tables. Both axes are fixed 8-slot LRU tables in internal RAM (not the PSRAM NodeInfo cache), compared by wrap-safe uint32 ms subtraction, so there is no tick clock and no sweep to maintain. directResponseAllowed() resolves both slots before stamping either, so a reply one axis throttles never consumes the other's budget, and records the send itself. Retires: nodeInfoFallbackLastResponseMs, kNodeInfoResponseThrottleMs, nowStampMs, the respTick byte + hasResponded bit on every cache entry, currentRespTick/kNodeInfoRespTickMs/kNodeInfoThrottleTicks, and the sweep's respTick clear. Renumbers peekNodeInfoFlagsForTest (drops the responded bit). Tests: the psram/fallback throttle tests now assert the per-target axis at 60 s, isolated via a different requester, proving it holds with and without PSRAM; perRequesterAndGlobalFloor isolates the other two axes; the obsolete respTick wrap-safety test is removed. 83/83 native cases pass. clod helped too * whats up, doc? * trimming the comments * test: bump native-suite-count to 38 after upstream rebase Upstream develop carries 38 test_* suite directories but its native-suite-count file still reads 37 (a suite was added without bumping the count). Rebasing onto develop inherits that stale file, so the runner flags AMBER. Correct the registered total to 38. clod helped too --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Thomas Göttgens <tgoettgens@gmail.com>	2026-07-21 12:19:43 +02:00
Tom	b8b5582943	Regioninfo (#11056) ... * advertise a superset of EU regional presets to client devices. * trunk	2026-07-19 06:22:39 -05:00
Oleksandr KravchukandTom	36bfb86937	Remove UA_868 as obsolete (#10994) ... Decree of the Cabinet of Ministers of Ukraine dated February 25, 2026,* No. 262, on amendments to Section 2 of the Plan for the Allocation and Use of the Radio Frequency Spectrum in Ukraine, has harmonised Ukrainian regulations on the 868 MHz spectrum with the EU's, thereby making UA_868 legally obsolete. * https://zakon.rada.gov.ua/laws/show/262-2026-п Co-authored-by: Tom <116762865+NomDeTom@users.noreply.github.com>	2026-07-16 08:52:50 -05:00
Ben Meadors	c5355641d3	Add MEDIUM_TURBO modem preset (#10988) ... * Protobufs * Wire up MEDIUM_TURBO modem preset MEDIUM_TURBO (500 kHz, SF9, CR 4/5) already existed in the protobuf enum but was never wired into firmware, so selecting it silently fell through to the LONG_FAST default and rendered an "Invalid" display name. Add its bw/sf/cr mapping (modemPresetToParams), display name (MediumTurbo/MedT), PRESETS_STD membership (standard regions only — 500 kHz does not fit EU868's 250 kHz band, so it stays out of PRESETS_EU_868 and is rejected/clamped there), and the MEDIUM SNR-grading bucket. Includes positive coverage in test_radio, EU868-reject + US-accept coverage in test_admin_radio and test_mesh_beacon, the STD preset count 9->10, an extended fuzz range, and the client-spec doc. * Address review feedback on MEDIUM_TURBO tests - test_mesh_beacon: assert has_mesh_beacon before checking the invalid preset was cleared, so the EU868-cleared test can't pass on a dropped message (matches the existing SHORT_TURBO test). - test_fuzz_packets: draw modem presets from _ModemPreset_ARRAYSIZE instead of a hard-coded 17 so the fuzz range tracks future enum additions automatically.	2026-07-11 08:24:35 -05:00
Ben MeadorsandClaude Opus 4.8	510e9796f9	Extract mcp-server to its own repo (meshtastic/meshtastic-mcp) (#10861) ... The Python MCP server + hardware test harness that lived under mcp-server/ now has its own home at https://github.com/meshtastic/meshtastic-mcp (published, versioned independently). Remove the in-tree copy and wire the firmware repo to the standalone server externally. - Delete mcp-server/ (96 files) and the 8 harness-coupled AI workflow files under .claude/commands/ and .github/prompts/ that drove ./mcp-server/ run-tests.sh — those workflows now ship with meshtastic-mcp as skills. - .mcp.json: register the server via `uvx --from git+https://github.com/meshtastic/meshtastic-mcp meshtastic-mcp` instead of a local ./mcp-server/.venv, keeping MESHTASTIC_FIRMWARE_ROOT="." so the MCP tools still work from this checkout with no local build. - Repoint the remaining references (AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, bin/regen-*.sh, docs, Screen.h, userPrefs.jsonc, test/fixtures/nodedb/README.md, .trunk/configs/.bandit) at the standalone repo. The MCP tool surface is unchanged — only the pytest harness moves out; run it from a meshtastic-mcp checkout with MESHTASTIC_FIRMWARE_ROOT pointed here. No build/CI/platformio coupling existed, so nothing in the firmware build changes. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-02 14:33:04 -05:00
Tom	3becaf2d95	emdashes begone (#10847)	2026-07-01 19:01:27 -05:00
5 people	ec5d230305	Feat/mesh beacon (#10618) ... * Tips robot virtual node / relayer to different LoRa modes & channels Note that this commit has details hardcoded for the Wellington (NZ) mesh, and also requires the following patch to the protobufs: ----- diff --git a/meshtastic/mesh.proto b/meshtastic/mesh.proto index 03162d8..ec54c99 100644 --- a/meshtastic/mesh.proto +++ b/meshtastic/mesh.proto @@ -1393,6 +1393,21 @@ message MeshPacket { * Set by the firmware internally, clients are not supposed to set this. */ uint32 tx_after = 20; + + /* + * The modem preset to use fo rthis packet + */ + uint32 modem_preset = 21; + + /* + * The frequency slot to use for this packet + */ + uint32 frequency_slot = 22; + + /* + * Whether the packet has a nonstandard radio config + */ + bool nonstandard_radio_config = 23; } /* ----- * fix: repair mesh tips CI build * feat: add MeshBeacon module (Phase 1 — proto + generated code + initial stub) * feat(beacon): implement broadcaster + listener (phases 2-5) * feat(beacon): wire RadioLibInterface hooks + admin validation (phases 6-7) * fix(beacon): fix LocalModuleConfig flat access (no payload_variant), add localonly proto field * feat(beacon): fix broadcaster inheritance, add preset/region validation + proto cache - MeshBeaconBroadcastModule now inherits ProtobufModule<meshtastic_MeshBeacon> (alongside private MeshBeaconModule + OSThread), giving it allocDataPacket() and setStartDelay() without extra includes. - Payload cache: rebuildCache() encodes the MeshBeacon protobuf once and stores it in payloadCache[]/payloadCacheSize; sendBeacon() only calls rebuildCache() when payloadCacheDirty==true. AdminModule calls invalidateCache() after saving new config so the next broadcast picks up changes. - Region/preset validation in handleSetModuleConfig (mesh_beacon_tag): broadcast_on_preset is validated against the device's current region via RadioInterface::validateConfigLora(); broadcast_offer_region is validated via RadioInterface::validateConfigRegion(). Invalid values are zeroed with a LOG_WARN before saving. * feat(beacon): add unit tests for MeshBeaconModule and AdminModule configuration validation * remove old meshtips * more validation in NodeDB and AdminModule, and userprefs for baked in goodness * copilot is my gravity * mmmmm... beacon * oops * Enhance unit tests for MeshBeaconModule with detailed validation checks and output formatting * new lines. Why not? * finally * legacy mode activate! * Update protobufs (#17) Co-authored-by: NomDeTom <116762865+NomDeTom@users.noreply.github.com> * better logic, fixed a test * updated for packet signing fixed a test added guards for licensed/ham mode * channel numbers * beacon: encrypt on the beacon channel PSK; fix split note When broadcast_on_channel overrides the primary channel's name/PSK, the beacon was encrypted with the PRIMARY PSK: perhapsEncode keys encryption off the primary slot, but the radio-thread channel switch happens only after encryption. sendBeaconPacket() now installs the beacon channel into the primary slot for the synchronous duration of send() (cooperative threading => no interleaving) so encryption/hash use the beacon channel, then restores it. A shared beaconChannelSettings() helper builds the channel for both the encrypt-time swap and the RF-time swap so the key+hash cannot drift. Also: correct the legacy-split comments (both packets go out on the same beacon radio settings, not the normal config) and merge the two consecutive `if (hasText)` blocks in the listener (cppcheck duplicateCondition). Tests: add channelPskOverride_swapsBeaconChannelAndRestores and noChannelOverride_doesNotSwapPrimary; MockRouter snapshots the primary channel at send() time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test/beacon: drain toPhoneQueue in tearDown to fix LSan leak abort The listener delivers received text via MeshService::sendToPhone(), which enqueues the packet into toPhoneQueue and takes ownership. Nothing dequeues it in tests, so the three listener tests carrying message text stranded a MeshPacket each — 1272 bytes / 3 allocations that LeakSanitizer flagged at process exit, aborting the coverage run (surfaced by pio as [ERRORED] / SIGHUP even though all 40 assertions passed). Drain the phone queue in tearDown (getForPhone()/releaseToPool) so the packets return to packetPool. Suite is now GREEN with no sanitizer abort. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * legacy hop override for zero-hoppers * ever more beacons * beacon: comment out broadcast_send_as_node pending further review Functionality preserved in comments with full signing/has_bitfield notes for when it is re-enabled. Proto tag 3 retained on the wire. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test/beacon: fix fromIsCustomNodeWhenSet now that send-as-node is disabled broadcast_send_as_node is commented out; from is always the local node. Update the test assertion and doc comment to match current behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update protobufs (#21) Co-authored-by: NomDeTom <116762865+NomDeTom@users.noreply.github.com> * flags for beacons * beacon: do more with less — slot-index targets + validation Multi-target beacons embedded a full ChannelSettings in every BroadcastTarget, blowing ModuleConfig past the 512-byte BLE FromRadio budget so the firmware would not compile. Targets now reference an existing channel-table slot by channel_index and the broadcaster resolves it via channels.getByIndex() at TX time. Net effect: the same multi-target capability for a fraction of the bytes — FromRadio 609 -> 510 B, MeshBeaconConfig 596 -> 324 B, AdminMessage 615 -> 511 B. - proto: BroadcastTarget.channel (embedded) -> channel_index (uint32 ref); regen all generated headers (size constants propagate to admin/localonly/deviceonly/mesh). - broadcaster: resolve channel_index from the channel table; an out-of-range or blank slot falls back to the default channel for the target preset rather than borrowing the primary's name/PSK. - AdminModule: validate broadcast_targets entries on write (region/preset sanitised like the single-target fields; channel_index range-checked). - userPrefs: TARGET_<n>_CHANNEL_{NAME,NUM,PSK} collapse to a single CHANNEL_INDEX. - docs: two-step (set_channel -> set_module_config) multi-target setup, inline-vs- reference distinction, and single-/multi-target are equal (not "legacy") options. - tests: target validation + channel-index resolution incl. blank-slot fallback (47/47 green on `./bin/run-tests.sh -e native -f test_mesh_beacon`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRAF5csgsMn6p1zEcFL8Qz * throttling after reboot * address copilot review * simplify * fix(beacon): use 0x%08x for node/packet IDs in logs; register test suite The %#08lx log specifiers passed uint32_t (NodeNum/PacketId) to a %lx length modifier — undefined behaviour on 64-bit (native test) targets and non-standard width. Switch to the project-standard 0x%08x. Also bump test/native-suite-count to 25 for the added test_mesh_beacon suite. clod helped too * copilot & clarity clod helped too * refactor(beacon): use auto for the sanitized config copy clod helped too * fix(beacon): guard empty-payload sends; gate has_mesh_beacon on build flag; document ISR_TX pre-switch clod helped too --------- Co-authored-by: Steve Gilberd <steve@erayd.net> Co-authored-by: Darafei Praliaskouski <me@komzpa.net> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-27 08:20:51 -05:00
Ben Meadors	dd1ec9d462	Lora region preset map (#10736) ... * Added lora region and preset maps * Protos * Address PR review feedback - Log (and break/skip) when the region preset map exceeds its array bounds instead of silently dropping regions - Derive test bounds from the generated nanopb array sizes via sizeof() instead of hard-coded magic numbers - Fix want_config sequence comment (missing comma, STATE_SEND_MODULECONFIG) - Specify a language on the spec's fenced code blocks (markdownlint MD040) * Fix want_config stall: handle STATE_SEND_REGION_PRESETS in PhoneAPI::available() available() had a separate per-state switch that wasn't updated for the new state, so it returned false ('unexpected state 5') and getFromRadio() was never called - the config handshake stalled after metadata and the client timed out. Verified via the native simulator integration test.	2026-06-19 19:56:24 -05:00
3 people	22072c5f4b	Pr1.5 tmm nexthop (#10745) ... * TrafficManagement: flat unified cache + persistent next-hop overflow store Reworks the TrafficManagementModule cache layer (policing behaviour unchanged from upstream) and adds a routing-hint overflow store: - Flatten the ring: replace the cuckoo-hashed unified cache and the bucketed PSRAM NodeInfo index with plain flat arrays + linear scan (same idiom as WarmNodeStore). At LoRa packet rates an O(n) scan of the cache is negligible, and it removes a large amount of hashing/displacement complexity. The cache entry is 11 B; timestamps use a uniform +1 presence-offset so a 0 byte always means "empty" across every sub-store. Adds rebaseEpoch() so cached state survives the ~19 h relative-timestamp horizon instead of being flushed. - Next-hop overflow cache: setNextHop/getNextHopHint store a confirmed last-byte relay for a destination, written only from NextHopRouter's ACK-confirmed decision (and mirrored from TraceRoute). NextHopRouter::getNextHop falls back to this cache when the hot NodeDB has no hint, so DMs/relays to long-tail nodes keep routing after the node ages out of NodeInfoLite. - Persistence: preloadNextHopsFromNodeDB warm-starts the cache from persisted NodeInfoLite hints on first maintenance pass; next_hop entries are kept alive across the maintenance sweep (no TTL) and never clobbered by a stale preload. All packet-policing logic (rate limit, position dedup, unknown-packet drop, NodeInfo direct response, hop exhaustion) is the existing upstream behaviour, untouched. HAS_TRAFFIC_MANAGEMENT defaults on so the module is compiled in. (see note). Tests: upstream policing suite now actually runs (adds the MeshTypes.h include that gates HAS_TRAFFIC_MANAGEMENT) plus 4 next-hop tests. Role-aware throttles, politeness, precision clamp, port-interval and mesh-radius gating — and the rate-limit >255 saturation fix — are deferred to the advanced-TMM branch. Note: default dedup movement grid moves to ~91m, which also means 1.5km required to end up with the same signature position - coarser and therefore further than before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * TrafficManagement: fix cppcheck constVariablePointer warning `node` in preloadNextHopsFromNodeDB() is never written through — mark it const to satisfy cppcheck's constVariablePointer check in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add multi-hop NextHop recovery tests and unit tests for routing reliability - Introduced a new test suite for multi-hop NextHop directed-message delivery and relay recovery in `test_nexthop_multihop_recovery.py`. This includes tests for end-to-end delivery and recovery after relay drop. - Implemented unit tests in `test_main.cpp` for NextHop routing reliability mitigations, covering: - M1: Ambiguity-aware last-byte resolution. - M2: NextHopRouter's strict-neighbor gate and hop limit checks. - M3: Route-health freshness and failure decay. - Enhanced mock classes to facilitate controlled testing of node behaviors and routing logic. * grafting fixed * Address Copilot review for PR #10735 (NextHop improvements) - docs/nexthop-routing-reliability.md: update status from "no code changes yet" to reflect that mitigations and tests are implemented RAM pressure and MIGRATION_VERBOSE concerns addressed upstream in PR2.5 (per-platform TRAFFIC_MANAGEMENT_CACHE_SIZE) and PR2 (verbose default=0) respectively; (0,0) sentinel fixed in PR2.5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * CI: fix cppcheck constVariablePointer and test include path - NextHopRouter.cpp: qualify two RouteHealth *h locals as const — only read for stale-route checks, never mutated through the pointer - Router.cpp: qualify meshtastic_NodeInfoLite *node as const in shouldDecrementHopLimit — only read for favorite/role predicate - test_position_module/test_main.cpp: change bare PositionModule.h to modules/PositionModule.h — build_flags sets -Isrc, not -Isrc/modules, so the bare form fails to resolve in the native PlatformIO test env Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * WarmStore: cache device role + protected category in last_heard low bits Steal the low 6 bits of WarmNodeEntry.last_heard to carry an evicted node's device role (4 bits) and a protected category (2 bits) for the hop-trim path, at zero record-size cost (entry stays 40 B; no RAM/flash growth). The high bits remain a real unix-seconds timestamp, quantised to 64 s — ample for warm LRU ordering of long-tail nodes. - absorb() packs role/protectedCat; place()/ring replay store the raw word so metadata round-trips through flash. LRU compares masked time (warmTimeOf). - take() rehydration masks the metadata bits and restores the cached role so a re-admitted node isn't stuck at CLIENT until its next NodeInfo. - NodeDB classifies the category (favorite/ignored/verified -> Flag; tracker/sensor/tak_tracker -> Role) at each eviction site. - WarmNodeStore::lookupMeta() exposes role/category to consumers. - Bump WARM_RING_MAGIC (WRNG->WRN2): old rings read as erased and rebuild; warm data is a non-critical evictee cache, so discard-on-upgrade is safe. Tests: test_warm_store 11/11 (new meta round-trip + quantisation-aware ordering); NodeDB compiles (test_nodedb_blocked 4/4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: migrate v1 rings/files by discarding last_heard, not the data Previously the WRNG->WRN2 magic bump treated old rings as erased, discarding all warm entries — including the PKI public keys that let evicted nodes keep decrypting DMs. Instead, read v1 (WRNG / WRM1) records and keep each node's identity + public key, discarding only last_heard (its low bits would otherwise be misread as the new role/protected metadata). Records re-rank and re-learn their role on next contact. - Ring backend (nRF52840): ringReadHeader accepts both magics and reports v1 via an out-param; replay zeroes last_heard for v1 records. If the active head page is v1, force a rotation so new v2 records never land in a v1-headered page (which would discard their freshly-set role on the next load). Legacy pages convert to v2 as the ring rotates. - File backend (warm.dat): bump WARM_STORE_MAGIC WRM1->WRM2; accept WRM1, verify CRC against the stored bytes, then discard last_heard and mark dirty so the next save rewrites as v2. Tests: test_warm_store 12/12 (adds test_ws_v1_migration_discardsLastHeard: key survives, role/protected reset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: guard role bit-width + test eviction carries role/protected - static_assert that the device role enum still fits the 4-bit warm metadata field (WARM_ROLE_MASK); fails the build loudly if a new role is added past 15 rather than silently truncating role on eviction. (Max role today = 12.) - Add test_migration_carriesRoleAndProtectedIntoWarm: a demoted TRACKER lands in the warm tier with its key, role=TRACKER and protected category=Role; a demoted CLIENT carries role=CLIENT/None. Exercises the NodeDB eviction path + warmProtectedCategory classification (the warm-store unit tests only cover absorb() directly). Tests: test_nodedb_blocked 5/5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix copilot comments * fix(test): restore #if HAS_TRAFFIC_MANAGEMENT guard in TMM test The rebase onto PR1.5 lost the top-level HAS_TRAFFIC_MANAGEMENT guard that PR1.5 introduced, leaving the #else/#endif tail orphaned and causing compile errors on non-TMM builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com>	2026-06-19 19:52:58 -05:00
Sacha Weatherstone	48fe68050b	Remove docs, moved to main repo	2021-04-15 17:24:28 +10:00
Kevin Hester	7a86d150a3	Delete CNAME	2021-04-15 14:59:24 +08:00
Kevin Hester	5e85e6b7d2	Create CNAME	2021-04-15 13:21:55 +08:00
Kevin Hester	d22f2501b6	Delete CNAME	2021-04-15 13:19:07 +08:00
Kevin Hester	680eb18015	Create CNAME	2021-04-15 13:16:31 +08:00
Kevin Hester	4e33de780d	Delete CNAME	2021-04-15 13:16:23 +08:00
Kevin Hester	89d6046ccc	Create CNAME	2021-04-15 13:15:02 +08:00
Kevin Hester	d51b8bbe06	Delete CNAME	2021-04-15 13:11:21 +08:00
Kevin Hester	7d3115bb9b	1.2.28	2021-04-15 12:30:46 +08:00
Kevin Hester	ec18efbe7e	Add beginnings of RAK4600 and update to latest nrf52 arduino sources	2021-04-15 11:45:58 +08:00
Kevin Hester	f1376406fa	Add RAK4631 to build	2021-04-12 13:39:45 +08:00
Kevin Hester	55b38a7b02	RAK4631 is almost feature complete	2021-04-12 13:25:55 +08:00
Kevin Hester	7224782d23	rak4631: gps now works	2021-04-11 15:17:56 +08:00
Kevin Hester	f7d199a3be	rak4631: kinda runs	2021-04-11 13:58:48 +08:00
Kevin Hester	328b24537f	rak4631: builds but WIP	2021-04-11 13:33:27 +08:00
Kevin Hester	a43a04986d	document peer_info	2021-04-11 09:24:36 +08:00
Kevin Hester	e71758457b	use production mqtt server	2021-04-10 10:55:56 +08:00
Kevin Hester	5dfd387b21	use real MQTT server DNS name	2021-04-09 16:56:28 +08:00
Kevin Hester	57bf4073c5	Merge remote-tracking branch 'root/master' into mqtt	2021-04-06 10:58:01 +08:00
Kevin Hester	cbfd80f893	fix #779. Add basic GPIO documentation	2021-04-06 10:56:23 +08:00
Jm Casler	4382caad88	Update RangeTestPlugin.md	2021-04-05 19:25:53 -07:00
Kevin Hester	276526005b	run-both should target tbeams	2021-04-05 16:27:44 +08:00
Kevin Hester	63c976d4f1	update docs	2021-04-05 12:11:07 +08:00
Kevin Hester	e84edc676f	mqtt: automatically start or stop as neede & attempt reconnect	2021-04-05 07:53:07 +08:00
Kevin Hester	d19af8b83d	mqtt: send packets after they are encrypted	2021-04-04 09:20:37 +08:00
Kevin Hester	638cec7f25	we now send packets to mqtt server	2021-04-03 22:27:06 +08:00
Kevin Hester	f3f09f0dcf	MQTT WIP	2021-04-03 16:06:40 +08:00
Kevin Hester	dcf64dfacd	MQTT at least talks to server, works in native and esp32	2021-04-03 14:54:10 +08:00
Kevin Hester	2acde3333c	todo updates	2021-04-03 10:40:56 +08:00
Kevin Hester	8d47e4f3e0	1.2.20	2021-04-02 13:44:53 +08:00
Kevin Hester	2c5ea03b74	fix VFS creation bug in native. fix heap corruption in Fsm free	2021-04-01 20:32:12 +08:00
Kevin Hester	9d452ebf29	add WIP notes about running github actions locally...	2021-04-01 14:12:02 +08:00
Kevin Hester	8a20155214	simplify build	2021-04-01 14:05:27 +08:00
Kevin Hester	2594ea0c2c	test fix for CI tool	2021-03-31 16:28:16 +08:00
Kevin Hester	5f45a10db5	fix sign comparsion	2021-03-31 11:14:55 +08:00
Kevin Hester	5f948c09fe	update libs to fix CI build, thanks @meehow!	2021-03-31 11:05:43 +08:00