mirror of
https://github.com/meshtastic/firmware.git
synced 2026-09-20 13:32:56 -04:00
* fix(nrf52): serialise the warm-node ring against LittleFS on the shared flash cache On nRF52840 the warm-node store writes its 3-page record ring straight through flash_nrf5x_write/erase/flush, holding only spiLock. Every LittleFS writer instead holds Adafruit_LittleFS's own mutex, and two of them run on other tasks entirely: Bluefruit's bond saves on the callback task, and - since phone config writes moved into BLE context - a whole saveToDisk on the BLE task. Neither takes spiLock. Both writers share one 4 KB page cache, one SoftDevice flash semaphore and one result word. flash_cache_write repoints that cache when the requested page differs from the cached one, so a second writer arriving mid-write flushes the first writer's page and re-points the buffer; the first writer's remaining memcpy then lands in the wrong page's image. Ring records end up inside LittleFS metadata, or the reverse. The collision also exhausts the flash layer's 20 x 1 ms busy-retry budget against an 85 ms page erase, and flash_cache_flush discards the failure, so 32 LittleFS blocks vanish with no error reaching the filesystem. What the user sees is a torn directory pair on the next mount, a format, and critical error 13. Take the filesystem mutex in the five ring entry points that reach flash, after spiLock and never before - the order every existing path already uses. The ring touches no LittleFS call itself, so the non-recursive mutex is never re-entered. Longest new hold is a page rotation at roughly half a second, against a 2 s supervision timeout and a 90 s watchdog. Non-nRF52840 backends are untouched. * fix(nodedb): make saveProto report a failed readback or rename SafeFile::close() already verifies the .tmp by hash and renames it over the live file, and saveProto captured that result, logged it, and then returned the pb_encode status alone. A torn or half-programmed page therefore counted as a successful save: for the fullAtomic files the old contents silently survived, for nodes.proto (written in place) the file was simply gone, and saveToDisk's recovery path never fired for the one failure it exists for. * fix(nodedb): retry a failed save before formatting, and never format on a low rail saveToDisk answered any failed write with an immediate fsFormat(), which is where most "critical error 12/13" reports and the total config wipe behind them come from. A write that fails once is far more often a busy SoftDevice or a VDD dip mid-save than a corrupt filesystem, so: - retry twice, 150 ms apart, re-checking powerHAL_isPowerLevelSafe() before each attempt and before the format; on a low rail return false and leave the filesystem alone (the next save lands once the rail recovers, and boot already waits for a safe level) - check fsFormat()'s result instead of assuming it worked - after a successful format rewrite every segment, not only the ones this call asked for: the format took config.proto and the node identity with it, so a nodes-only save that ended in a format used to come back up as a new node - with encrypted storage a format also destroys the DEK; skip the resave rather than land the private key and PSKs on flash in plaintext RP2040 feeds its watchdog across the delays, as the neighbouring code does. * fix(nrf52): quiesce flash before every software reset and power-off The Adafruit flash layer keeps one 4 KB page image and one SoftDevice flash semaphore for the whole chip. Every reset path we own - Power::reboot(), enterDfuMode() (admin enter_dfu_mode_request, which arrives on the BLE task since #10967), cpuDeepSleep()'s reset and system-off arms, and the wio-t1000-s secure DFU handler - went straight to NVIC_SystemReset or sd_power_system_off while another task could be half-way through a page program or erase. A reset in that window leaves the page erased or partly programmed; LittleFS finds the torn metadata on the next mount and the corruption handler formats the filesystem. nrf52FlashQuiesce() takes spiLock and the LittleFS mutex, waits out whatever write is in flight, flushes the page cache, and keeps both locks because the caller resets next. The corruption-reboot handler and __assert_func are left alone: they run inside the filesystem call stack or a fault, where taking the mutex would deadlock. nRF54L is a second copy of these paths since #11867 and still defines ARCH_NRF52, so it gets the same function on the same core flash layer. * fix(nrf52): quiesce flash before the library BLE DFU handler jumps to the bootloader On every board except wio-t1000-s the Nordic DFU service is the framework's BLEDfu, whose START_DFU handler runs on the callback task and jumps to the bootloader with no regard for a flash write in progress on the loop task. That is the OTA path the Apple app and nRF Connect use (Android sends enter_dfu_mode_request instead, which the previous commit covers). QuiescingBLEDfu re-installs the control-point write callback after BLEDfu::begin() and wraps the library's: flush under both locks, then drop the LittleFS mutex before handing over, because the library reloads the bond keys through LittleFS on its way to the jump and the mutex is not recursive. spiLock stays held across the handler: every LittleFS writer on the BLE task takes it first, the loop task cannot preempt the callback task, and the handler never blocks after the flush, so nothing can dirty flash before bootloader_util_app_start(). If the handler returns, nothing jumped, and the lock is released. The library callback is a file-static, so it is read back out of the characteristic through a pointer-to-member obtained via a using-declaration; that is well-formed C++ and compiles under the pinned GCC 9.3 with LTO. * test(nodedb): pin the save-failure contract of saveProto and saveToDisk A failed rename must come back as false from saveProto, a one-off unsafe rail reading during a write must be retried and land, and a rail still unsafe at the retry gate must make saveToDisk return false with the filesystem untouched. The rail is scripted through a strong powerHAL_isPowerLevelSafe() over the weak native default; on Windows the default is strong, so only the rename case runs there. The format branch itself is unreachable natively (a FLASH_CORRUPTION critical error exits the portduino process), which is what the survival assertions pin. * fix(nodedb): only format when the filesystem itself is unreadable Making saveProto honest about write failures gave the recovery path a new way in: any persistent write failure now reached fsFormat(), which takes every file with it. A busy or lock-protected nRF52 flash fails every write for as long as it lasts, so two retries are not enough to tell that apart from a corrupt filesystem, and guessing wrong costs the node its config, keys and bonds. Reads settle it. They never touch the SoftDevice write path that a busy flash fails on, so if /prefs still walks and a stored proto still opens and reads, the metadata chain is intact and the write failure was transient - return false and let the caller try again later. Genuine corruption is not silently tolerated: lfs asserts on it, and the nRF52 handler reboots and formats on the way back up. Covered by a test that fails without this: a save whose rename cannot succeed, against an otherwise healthy filesystem, must leave devicestate untouched. * trunk: exempt Unity test entry points from trufflehog trufflehog's Lob detector matches "test_" followed by alphanumerics, which describes every Unity test function name. It fired on a new test in test_nodedb_save_retry and will fire again on the next suite added. Scoped to test/**/test_main.cpp, alongside the existing gitleaks exemption for the synthetic node-DB fixtures. * fix(nodedb): feed the RP2040 watchdog around the format and the resave saveToDisk() only feeds the watchdog at the top of each retry. The last retry, the readable probe, fsFormat() and the five-segment resave then share one 8 s budget (watchdog_enable in main-rp2xx0.cpp) with no loop left to feed it. A timeout during the resave leaves the filesystem empty and the node boots on defaults with a new identity - the exact outcome this PR exists to prevent, reached by a different road. Feed once before the probe and again before the resave. Both feeds sit outside any lock: filesystemStillReadable() takes spiLock itself, and the format has already released it. ARCH_RP2040 covers rp2040 and rp2350 alike, and the blocks compile out everywhere else, so no other platform and no native test changes. Raised by @caveman99 in review. * fix(nodedb): narrow the save-probe comment and name the full-filesystem case The comment on filesystemStillReadable() claimed "real corruption asserts in lfs and formats on reboot". That does hold on nRF52 - nrf52.ini builds with -DLFS_NO_ASSERT and force-includes cpp_overrides/lfs_util.h, whose LFS_NO_ASSERT arm routes LFS_ASSERT to the lfs_assert() in main-nrf52.cpp, which stamps NRF52_MAGIC_LFS_IS_CORRUPT and resets into the format - but NodeDB.cpp compiles for ESP32, RP2040 and portduino too, where nothing of the sort is wired up. It is also not true on nRF52 under POFWARN, where lfs_assert() deliberately skips the stamp. Drop the claim rather than qualify it three ways. The log line now names what a field log actually needs to tell apart: a filesystem that still reads but cannot be written is either busy or full. Raised by @caveman99 in review. * fix(sx128x): quiesce flash before the 2.4GHz region reset reinitChip() saves the region, waits 2 s and resets. On nRF52 that was the last software reset still going straight to NVIC_SystemReset with a page program possibly in flight, so "every software reset" in the earlier commit did not quite hold. The quiesce stays inside the ARCH_NRF52 arm on purpose. The #else arm logs and falls through to lora.setCRC() further down, which re-enters spiBeginTransaction(); a quiesce hoisted above the #if would take spiLock and never give it back, self-deadlocking portduino and stm32wl. Routing this through Power::reboot() is wrong for the same class of reason: setupModules() runs before initLoRa, so its notifyReboot observers and waypointStore.saveToFlash() are live and would add a flash write to an aborted radio init. Raised by @caveman99 in review. * fix(nrf52): only quiesce on the DFU control write that actually resets QuiescingBLEDfu wrapped every control-point write, so a write that was never going to reset still blocked the Bluefruit callback task on spiLock, forced an early page-cache commit and held back the GATT authorize reply. Only START_DFU resets; gate on that. Deliberately no "request->len &&" term. The library's own test is `request->data[0] == START_DFU` with no length check (BLEDfu.cpp:110 in both the nRF52 and nRF54 cores), and Bluefruit hands the callback a copy of a reused event buffer, so a zero-length write carrying a stale 0x01 still resets inside the library. A len term here would let exactly that reset run unquiesced, which is the case this wrapper exists for. Reading data[0] is always in bounds: ble_gatts_evt_write_t declares uint8_t data[1] and the copy covers it. Raised by @caveman99 in review.
271 lines
9.6 KiB
C++
271 lines
9.6 KiB
C++
// NodeDB save-failure contract, NodeDB::saveProto() and NodeDB::saveToDisk() in src/mesh/NodeDB.cpp.
|
|
//
|
|
// saveProto() must report a write that did not land. SafeFile::close() verifies the .tmp by hash and
|
|
// renames it over the live file; saveProto() used to capture that result and then return the
|
|
// pb_encode status alone, so a torn page or a failed rename counted as a successful save and the
|
|
// caller's recovery never ran. The regression guarded: saveProto() returning true while the live
|
|
// file was never replaced.
|
|
//
|
|
// saveToDisk() must not answer a failed write with fsFormat(). A write that fails once is far more
|
|
// often a rail dip or a busy SoftDevice than a corrupt filesystem, and the format takes every file on
|
|
// the device with it; on nRF52 that is the "critical error 12/13" report followed by a new node
|
|
// identity. The contract pinned here: a transient failure is retried and the retry lands, and a rail
|
|
// that is still unsafe at the retry gate makes saveToDisk() return false and leave the filesystem
|
|
// alone. The regression guarded: any path from a single failed write straight into fsFormat().
|
|
//
|
|
// The rail is driven through powerHAL_isPowerLevelSafe(), whose native default is a weak "always
|
|
// safe"; this suite supplies a strong, scripted definition. Windows links the default strongly, so
|
|
// the gate-driven cases are compiled out there and only the rename case runs. The format branch
|
|
// itself cannot be reached in a native test: on portduino a FLASH_CORRUPTION critical error exits the
|
|
// process, which is exactly the outcome the assertions here prove is not taken.
|
|
#include "MeshTypes.h" // Include BEFORE TestUtil.h
|
|
#include "TestUtil.h"
|
|
#include <unity.h>
|
|
|
|
#if defined(ARCH_PORTDUINO)
|
|
#define NSR_TEST_ENTRY extern "C"
|
|
#else
|
|
#define NSR_TEST_ENTRY
|
|
#endif
|
|
|
|
#include "FSCommon.h" // defines FSCom; must precede the feature guard below
|
|
|
|
#if defined(FSCom)
|
|
|
|
#include "mesh/NodeDB.h"
|
|
#include "power/PowerHAL.h"
|
|
#include <cstdio>
|
|
#include <cstdlib>
|
|
#include <cstring>
|
|
#include <string>
|
|
#include <unistd.h>
|
|
#include <vector>
|
|
|
|
namespace
|
|
{
|
|
|
|
// --- Scripted rail -------------------------------------------------------------------------------
|
|
// Each powerHAL_isPowerLevelSafe() call consumes one entry; past the end the rail reads safe.
|
|
// saveToDisk() asks at entry, saveToDiskNoRetry() and saveProto() ask again, and the retry gate asks
|
|
// before each retry, so the second reading is the first one a write can see.
|
|
std::vector<bool> railScript;
|
|
size_t railCalls = 0;
|
|
|
|
void scriptRail(std::initializer_list<bool> readings)
|
|
{
|
|
railScript.assign(readings);
|
|
railCalls = 0;
|
|
}
|
|
|
|
// --- File helpers ----------------------------------------------------------------------------------
|
|
|
|
bool readFileBytes(const char *path, std::vector<uint8_t> &out)
|
|
{
|
|
out.clear();
|
|
File f = FSCom.open(path, FILE_O_READ);
|
|
if (!f)
|
|
return false;
|
|
uint8_t buf[512];
|
|
size_t n;
|
|
while ((n = f.read(buf, sizeof(buf))) > 0)
|
|
out.insert(out.end(), buf, buf + n);
|
|
f.close();
|
|
return true;
|
|
}
|
|
|
|
void writeFileBytes(const char *path, const std::vector<uint8_t> &bytes)
|
|
{
|
|
FSCom.remove(path); // FILE_O_WRITE is append on some backends; start clean
|
|
File f = FSCom.open(path, FILE_O_WRITE);
|
|
TEST_ASSERT_TRUE_MESSAGE(f, path);
|
|
TEST_ASSERT_EQUAL_size_t(bytes.size(), f.write(bytes.data(), bytes.size()));
|
|
f.close();
|
|
}
|
|
|
|
// FNV-1a content fingerprint; answers only "did this file change?". 0 == missing file.
|
|
uint64_t fileFingerprint(const char *path)
|
|
{
|
|
std::vector<uint8_t> bytes;
|
|
if (!readFileBytes(path, bytes))
|
|
return 0;
|
|
uint64_t h = 1469598103934665603ULL;
|
|
for (uint8_t b : bytes) {
|
|
h ^= b;
|
|
h *= 1099511628211ULL;
|
|
}
|
|
return h;
|
|
}
|
|
|
|
// Make the next persisted config differ from the last one, so a landed write is visible.
|
|
void bumpConfig()
|
|
{
|
|
config.device.node_info_broadcast_secs += 1;
|
|
}
|
|
|
|
// PortduinoFS::rmdir() is unlink() underneath and cannot remove a directory, so go to the host.
|
|
bool removeHostDirectory(const char *fsPath)
|
|
{
|
|
std::string host = std::string(getenv("HOME")) + "/.portduino/default" + fsPath;
|
|
return ::rmdir(host.c_str()) == 0;
|
|
}
|
|
|
|
} // namespace
|
|
|
|
#if !defined(_WIN32)
|
|
// Strong definition shadows the weak native default in src/power/PowerHAL.cpp.
|
|
bool powerHAL_isPowerLevelSafe()
|
|
{
|
|
if (railCalls < railScript.size())
|
|
return railScript[railCalls++];
|
|
railCalls++;
|
|
return true;
|
|
}
|
|
#endif
|
|
|
|
// --- saveProto ------------------------------------------------------------------------------------
|
|
|
|
// A directory squatting on the live path makes the final rename fail after the .tmp was written and
|
|
// verified, which is the only failure SafeFile::close() reports that pb_encode cannot see.
|
|
static void test_saveProto_failedRename_returnsFalse(void)
|
|
{
|
|
TEST_MESSAGE("=== saveProto: a failed rename is reported, not swallowed ===");
|
|
std::vector<uint8_t> live;
|
|
TEST_ASSERT_TRUE(readFileBytes(configFileName, live));
|
|
TEST_ASSERT_TRUE(FSCom.remove(configFileName));
|
|
TEST_ASSERT_TRUE(FSCom.mkdir(configFileName));
|
|
|
|
const bool saved = nodeDB->saveProto(configFileName, meshtastic_LocalConfig_size, &meshtastic_LocalConfig_msg, &config);
|
|
|
|
// Restore the sandbox before asserting so a failure does not leak the directory into later tests.
|
|
std::string tmp = std::string(configFileName) + ".tmp";
|
|
FSCom.remove(tmp.c_str());
|
|
TEST_ASSERT_TRUE(removeHostDirectory(configFileName));
|
|
writeFileBytes(configFileName, live);
|
|
|
|
TEST_ASSERT_FALSE(saved);
|
|
}
|
|
|
|
#if !defined(_WIN32)
|
|
// --- saveToDisk retry gate -------------------------------------------------------------------------
|
|
|
|
static void test_saveToDisk_railDipDuringWrite_retriesAndLands(void)
|
|
{
|
|
TEST_MESSAGE("=== saveToDisk: one unsafe reading mid-write is retried, no format ===");
|
|
const uint64_t before = fileFingerprint(configFileName);
|
|
TEST_ASSERT_NOT_EQUAL_UINT64(0, before);
|
|
bumpConfig();
|
|
|
|
scriptRail({true, false}); // entry safe, the write itself sees the dip, the retry gate is safe
|
|
|
|
TEST_ASSERT_TRUE(nodeDB->saveToDisk(SEGMENT_CONFIG));
|
|
|
|
TEST_ASSERT_GREATER_THAN_size_t(2, railCalls); // the retry gate was consulted
|
|
TEST_ASSERT_NOT_EQUAL_UINT64(before, fileFingerprint(configFileName));
|
|
TEST_ASSERT_TRUE(FSCom.exists(deviceStateFileName)); // an fsFormat() would have taken this too
|
|
}
|
|
|
|
static void test_saveToDisk_railStillUnsafeAtRetry_bailsWithoutFormat(void)
|
|
{
|
|
TEST_MESSAGE("=== saveToDisk: rail unsafe at the retry gate returns false, no format ===");
|
|
const uint64_t before = fileFingerprint(configFileName);
|
|
TEST_ASSERT_NOT_EQUAL_UINT64(0, before);
|
|
bumpConfig();
|
|
|
|
scriptRail({true, false, false, false, false}); // never recovers while this save is in progress
|
|
|
|
// On portduino a FLASH_CORRUPTION critical error exits the process, so surviving this call is
|
|
// itself the proof that no format was attempted.
|
|
TEST_ASSERT_FALSE(nodeDB->saveToDisk(SEGMENT_CONFIG));
|
|
|
|
TEST_ASSERT_EQUAL_UINT64(before, fileFingerprint(configFileName));
|
|
TEST_ASSERT_TRUE(FSCom.exists(deviceStateFileName));
|
|
}
|
|
|
|
// The whole point of the retry gate: a write that keeps failing while the filesystem still reads is a
|
|
// busy or lock-protected flash, not corruption, and formatting would take every other file with it.
|
|
static void test_saveToDisk_writeFailsButFsReadable_doesNotFormat(void)
|
|
{
|
|
TEST_MESSAGE("=== saveToDisk: unwritable but readable filesystem must not be formatted ===");
|
|
std::vector<uint8_t> live;
|
|
TEST_ASSERT_TRUE(readFileBytes(configFileName, live));
|
|
const uint64_t deviceStateBefore = fileFingerprint(deviceStateFileName);
|
|
|
|
// A directory on the live path fails every rename, so all retries fail - while /prefs and the
|
|
// other protos stay perfectly readable.
|
|
TEST_ASSERT_TRUE(FSCom.remove(configFileName));
|
|
TEST_ASSERT_TRUE(FSCom.mkdir(configFileName));
|
|
|
|
const bool saved = nodeDB->saveToDisk(SEGMENT_CONFIG);
|
|
|
|
std::string tmp = std::string(configFileName) + ".tmp";
|
|
FSCom.remove(tmp.c_str());
|
|
TEST_ASSERT_TRUE(removeHostDirectory(configFileName));
|
|
writeFileBytes(configFileName, live);
|
|
|
|
TEST_ASSERT_FALSE(saved);
|
|
// The decisive assertion: an fsFormat() would have taken devicestate with it.
|
|
TEST_ASSERT_TRUE(FSCom.exists(deviceStateFileName));
|
|
TEST_ASSERT_EQUAL_UINT64(deviceStateBefore, fileFingerprint(deviceStateFileName));
|
|
}
|
|
|
|
static void test_saveToDisk_railUnsafeAtEntry_returnsFalseImmediately(void)
|
|
{
|
|
TEST_MESSAGE("=== saveToDisk: rail unsafe at entry writes nothing ===");
|
|
const uint64_t before = fileFingerprint(configFileName);
|
|
bumpConfig();
|
|
|
|
scriptRail({false});
|
|
|
|
TEST_ASSERT_FALSE(nodeDB->saveToDisk(SEGMENT_CONFIG));
|
|
|
|
TEST_ASSERT_EQUAL_size_t(1, railCalls);
|
|
TEST_ASSERT_EQUAL_UINT64(before, fileFingerprint(configFileName));
|
|
}
|
|
#endif // !_WIN32
|
|
|
|
void setUp(void)
|
|
{
|
|
scriptRail({});
|
|
}
|
|
|
|
void tearDown(void) {}
|
|
|
|
NSR_TEST_ENTRY void setup()
|
|
{
|
|
initializeTestEnvironment();
|
|
nodeDB = new NodeDB(); // first boot on the pristine per-suite sandbox persists the default set
|
|
|
|
UNITY_BEGIN();
|
|
|
|
printf("\n=== saveProto ===\n");
|
|
RUN_TEST(test_saveProto_failedRename_returnsFalse);
|
|
|
|
#if !defined(_WIN32)
|
|
printf("\n=== saveToDisk retry gate ===\n");
|
|
RUN_TEST(test_saveToDisk_railDipDuringWrite_retriesAndLands);
|
|
RUN_TEST(test_saveToDisk_railStillUnsafeAtRetry_bailsWithoutFormat);
|
|
RUN_TEST(test_saveToDisk_writeFailsButFsReadable_doesNotFormat);
|
|
RUN_TEST(test_saveToDisk_railUnsafeAtEntry_returnsFalseImmediately);
|
|
#endif
|
|
|
|
exit(UNITY_END());
|
|
}
|
|
|
|
NSR_TEST_ENTRY void loop() {}
|
|
|
|
#else // !FSCom
|
|
|
|
void setUp(void) {}
|
|
void tearDown(void) {}
|
|
|
|
NSR_TEST_ENTRY void setup()
|
|
{
|
|
initializeTestEnvironment();
|
|
UNITY_BEGIN();
|
|
exit(UNITY_END());
|
|
}
|
|
|
|
NSR_TEST_ENTRY void loop() {}
|
|
|
|
#endif
|