From 2b751a44ecf90d4e690bdec2ae920858ce5f1fbf Mon Sep 17 00:00:00 2001 From: Jamie Pine Date: Sun, 20 Sep 2026 17:08:41 -0700 Subject: [PATCH] Design how a replica draws tiles it never baked A replica carries the index alone, so titan's 99,137 tiles are invisible from the Mac. The cache is machine-local because the thumbnail request answers with a path the client maps, which is also why the client needs no change: fill a local cache file for the remote volume and the same answer works. Adds the fetch design (where a remote tile lands, why the identities already line up, why the owner's version has to travel with the tile, FetchTile and WebP on the wire) as P6, and the durable sidecar tier as P7 behind L5. Records the deployed rebuild and thumbnail results in the register. Co-Authored-By: Claude Opus 5 (1M context) --- PROJECT_STATUS.md | 4 +- .../2026-09-19-incremental-replication.md | 76 ++++++++++++++++++- 2 files changed, 75 insertions(+), 5 deletions(-) diff --git a/PROJECT_STATUS.md b/PROJECT_STATUS.md index 91776ba49..e5f6aefbc 100644 --- a/PROJECT_STATUS.md +++ b/PROJECT_STATUS.md @@ -1,6 +1,6 @@ # Project Status -> Last updated: 2026-09-19 +> Last updated: 2026-09-20 > Code baseline: `e8febd196` > Purpose: short-term working context across development sessions while > `.tasks/` is frozen. @@ -152,7 +152,7 @@ Recent landed. | Work | Status | Owner | Next proof or action | | ------------------------------------------------- | -------------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Source runtime reliability | R1-R3, R6 and R7 landed and verified live | in progress | Execute `docs/plans/2026-09-15-source-runtime-reliability.md`. Landed: destructive paths removed, flush made a durability barrier, self-nudge loop stopped and hashing progress labeled (`c63359a9e`, `adf729cd9`, `92bb4412e`); R1 ancestor synthesis (`3a425c629`); R2 anchored volume roots (`4b3eb7389`); R3 persisted replica inventory (`18304e041`) plus the fetch-generation fix (`0650e0acb`: header names the generation and blake3 of the bytes delivered, a delivery must parse before it replaces a good artifact, one failed source no longer aborts the device sync, fetch failures back off, an unchanged generation still refreshes owner facts, revocation unloads and quarantines replicas, snapshot decode is bounded); the coverage heal (`fd40a5b87`); watchers made unconditional on a bare daemon (`88f496891`, `1d8d23962`, `0e605105b`); dirty-driven refetches paced to five minutes (`b57cb2857`). R6 executed 2026-09-16, results in the plan: `51e4873d4` opened stores read-only with listings, lookups and Unicode-folded name search; `3d8c44394` routes one backend per source (arena when restored or walked, SQLite otherwise, no empty-result retry, `total_is_exact` on the wire, detached directories list from their stores); `475a6b0b3` delivers a nested source's replica as its own database with the snapshot header contract, verified live — dev-tools ships as an 11.3 MB database, the sibling leak is gone, and the replica lists and searches identically; the 100-store/1M-record cold fan-out measures 1.7 s (`b88bfe4cf`). R7 executed 2026-09-16, results in the plan: all ten stores validated by direct SQL with zero dangling ancestry, the home store needed no repair, cold-restart with titan down serves and searches all nine replicas, byte read, remote op and remote log stream verified, reconvergence transferred zero bytes for titan's nine sources. Titan runs this build (deployed 2026-09-18). Its daemon had aborted at 03:27 that morning with `memory allocation of 53520 bytes failed`: musl's allocator gives each allocation group its own memory mapping, and the same code reached 45,641 of the kernel's 65,530 maps and 10.5 GB RSS within 43 minutes of a restart. With mimalloc as the musl global allocator it holds about 260 maps and a flat 1.7 GB. The runbook's start command also sent stdout to `~/spacedrive/daemon.log`, inside the jamie-nas source, so titan re-hashed its own log every 30 seconds; stdout now goes to `data/logs/stdout.log` in the excluded data dir. The deployed `sd-server` matches, but its relaunch stays with James and needs `DATA_DIR` plus `SD_AUTH`. On 2026-09-18 titan re-copied the Mac's unchanged home store in full four times: a nested source's generation came from `data.db` and WAL size and mtime, which move whenever the store's pool drains, and each round cost a 736 MB transfer and a 57-minute arena rebuild. The generation is now the store's revision, a count of committed row changes kept by triggers (`crates/store/src/revision.rs`); after one switch-over copy titan transferred nothing for 19.5 hours, where the old scheme would have made about 18 more copies. Titan runs this build since 2026-09-19 21:34, with `bin/ffmpeg` and `bin/ffprobe` (n8.1.2, copied from the CCTV remux tools) wired in through `SPACEDRIVE_FFMPEG_PATH` and `SPACEDRIVE_FFPROBE_PATH`: a forced tile on a CCTV clip spawned that ffmpeg and baked a 384px frame, so video tiles work on the NAS. That restart exposed the next defect: a replica restart re-copies what it already holds, because a share publishes only after its arena rebuild, so the first sync pass finds no share and refetches 736 MB it already had. It is P1 in `docs/plans/2026-09-19-incremental-replication.md`. The hour-long rebuild behind that share is gone: `Arena::add_entry` built a file type registry per row, parsing 92 KB of built-in TOML each time, and the registry is now built once for the process. Inserts went from 3.00 ms to 7.2 µs, putting a 1.76M-entry rebuild near 13 s in a debug build, with the SQL under it measured at 3 s. Remaining: R4 compaction and memory honesty (Mac 3.1 GB / titan 3.4 GB RSS as its baseline), R5 typed health and coverage, R8 acceptance matrix and the docs sweep, delta replication and transfer cleanup per `docs/plans/2026-09-19-incremental-replication.md` (its `temp_store` decision is James's), plus R6's other registered follow-ons (request cancellation, the deferred title index). | +| Source runtime reliability | R1-R3, R6 and R7 landed and verified live | in progress | Execute `docs/plans/2026-09-15-source-runtime-reliability.md`. Landed: destructive paths removed, flush made a durability barrier, self-nudge loop stopped and hashing progress labeled (`c63359a9e`, `adf729cd9`, `92bb4412e`); R1 ancestor synthesis (`3a425c629`); R2 anchored volume roots (`4b3eb7389`); R3 persisted replica inventory (`18304e041`) plus the fetch-generation fix (`0650e0acb`: header names the generation and blake3 of the bytes delivered, a delivery must parse before it replaces a good artifact, one failed source no longer aborts the device sync, fetch failures back off, an unchanged generation still refreshes owner facts, revocation unloads and quarantines replicas, snapshot decode is bounded); the coverage heal (`fd40a5b87`); watchers made unconditional on a bare daemon (`88f496891`, `1d8d23962`, `0e605105b`); dirty-driven refetches paced to five minutes (`b57cb2857`). R6 executed 2026-09-16, results in the plan: `51e4873d4` opened stores read-only with listings, lookups and Unicode-folded name search; `3d8c44394` routes one backend per source (arena when restored or walked, SQLite otherwise, no empty-result retry, `total_is_exact` on the wire, detached directories list from their stores); `475a6b0b3` delivers a nested source's replica as its own database with the snapshot header contract, verified live — dev-tools ships as an 11.3 MB database, the sibling leak is gone, and the replica lists and searches identically; the 100-store/1M-record cold fan-out measures 1.7 s (`b88bfe4cf`). R7 executed 2026-09-16, results in the plan: all ten stores validated by direct SQL with zero dangling ancestry, the home store needed no repair, cold-restart with titan down serves and searches all nine replicas, byte read, remote op and remote log stream verified, reconvergence transferred zero bytes for titan's nine sources. Titan runs this build (deployed 2026-09-18). Its daemon had aborted at 03:27 that morning with `memory allocation of 53520 bytes failed`: musl's allocator gives each allocation group its own memory mapping, and the same code reached 45,641 of the kernel's 65,530 maps and 10.5 GB RSS within 43 minutes of a restart. With mimalloc as the musl global allocator it holds about 260 maps and a flat 1.7 GB. The runbook's start command also sent stdout to `~/spacedrive/daemon.log`, inside the jamie-nas source, so titan re-hashed its own log every 30 seconds; stdout now goes to `data/logs/stdout.log` in the excluded data dir. The deployed `sd-server` matches, but its relaunch stays with James and needs `DATA_DIR` plus `SD_AUTH`. On 2026-09-18 titan re-copied the Mac's unchanged home store in full four times: a nested source's generation came from `data.db` and WAL size and mtime, which move whenever the store's pool drains, and each round cost a 736 MB transfer and a 57-minute arena rebuild. The generation is now the store's revision, a count of committed row changes kept by triggers (`crates/store/src/revision.rs`); after one switch-over copy titan transferred nothing for 19.5 hours, where the old scheme would have made about 18 more copies. Titan runs this build since 2026-09-19 21:34, with `bin/ffmpeg` and `bin/ffprobe` (n8.1.2, copied from the CCTV remux tools) wired in through `SPACEDRIVE_FFMPEG_PATH` and `SPACEDRIVE_FFPROBE_PATH`: a forced tile on a CCTV clip spawned that ffmpeg and baked a 384px frame, so video tiles work on the NAS. That restart exposed the next defect: a replica restart re-copies what it already holds, because a share publishes only after its arena rebuild, so the first sync pass finds no share and refetches 736 MB it already had. It is P1 in `docs/plans/2026-09-19-incremental-replication.md`. The hour-long rebuild behind that share is gone: `Arena::add_entry` built a file type registry per row, parsing 92 KB of built-in TOML each time, and the registry is now built once for the process. Inserts went from 3.00 ms to 7.2 µs, putting a 1.76M-entry rebuild near 13 s in a debug build, with the SQL under it measured at 3 s. Remaining: R4 compaction and memory honesty (Mac 3.1 GB / titan 3.4 GB RSS as its baseline), R5 typed health and coverage, R8 acceptance matrix and the docs sweep, delta replication and transfer cleanup per `docs/plans/2026-09-19-incremental-replication.md` (its `temp_store` decision is James's), plus R6's other registered follow-ons (request cancellation, the deferred title index).. Deployed to titan 2026-09-20: the replica restore that took 56m 2s now takes 19.3 s, and the refetch after a restart 8m 10s against 60m 15s, most of it now the download. The restart refetch is still unfixed; the restore publishes its share ten seconds after the first sync pass asks, so speed alone will not close it. Thumbnails then ran over titan's three real sources, leaving 99,137 tiles (jamie-nas 52,503, cctv 45,851, footage 783) in 21 GB of pvcache, after two defects. The `thumbnail` filter ran ahead of the scale, so it buffered thirty frames at source resolution: one 4264x2408 ProRes frame peaked at 1,290 MB and eight parallel decodes got the daemon SIGKILLed mid-job, against 233 MB with the scale first and one decode thread (`7847728c1`). The decode pool also sized itself from cores alone and is now bounded by free memory as well (`3b1bb2ba3`). Behind both, `NodeArena` mapped a `NamedTempFile`, which only `Drop` removes, so every daemon exit left about ten arena files behind; on titan `/tmp` is a tmpfs with no swap and 1,949 of them held 9.4 GB, leaving the daemon under 2 GB free. The backing file is now unnamed, so the kernel reclaims it however the process ends (`19b62033b`), and clearing the orphans took available memory from 1.9 GB to 11.4 GB. A replica still carries the index alone, so none of titan's tiles are visible from the Mac; that is now P6 and P7 of the replication plan. | | Mac client operating titan | live acceptance reopened | unowned | Pairing, remote CLI operations, jobs and logs remain functional. The September 15 evening audit found only eight titan replicas active on the Mac: dev-tools remains registered on titan with 28,079 records, but its snapshot is unavailable and its arena/watcher are absent. Preserve the earlier NAS freezes. Recovery and renewed acceptance are specified in `docs/plans/2026-09-15-source-runtime-reliability.md`; the earlier nine-source proof is historical. | | Peer device facts | live on both machines (2026-09-18) | unowned | Titan runs `30d15bfa4` and publishes its hardware and nine volumes (eight datasets and the pool); on the Mac, `devices.list` carries titan's Ryzen 5 5500, 6 cores and 31 GB, `volumes.list` lists the nine volumes under titan, and all nine replicas fold into them, dev-tools onto `pool`. Next: look at titan's card in the app. The periodic reconnect now also dials peers still in the `Paired` state, so a peer that was down when this device started is reached once it returns; verified live, the Mac reconnected to titan 20 seconds after titan came back, with no restart. Open follow-up: a rename over an existing file inside a source fails to land in the store on `UNIQUE(parent_uuid, title)`; replacing `sd-server` on titan failed one content_identity job and left `sd-server.new` and the old `sd-server` as separate rows in jamie-nas. | | Preserve NAS and pool evidence | pending | James | Save the TrueNAS config, encryption result, pool topology, GUID, and SMART output described by the runbook. | diff --git a/docs/plans/2026-09-19-incremental-replication.md b/docs/plans/2026-09-19-incremental-replication.md index 0fc5d5e84..644d37d53 100644 --- a/docs/plans/2026-09-19-incremental-replication.md +++ b/docs/plans/2026-09-19-incremental-replication.md @@ -2,13 +2,14 @@ > Status: proposal. P0 landed 2026-09-19. > Captured: 2026-09-19 -> Owns: delta replication, registered as a follow-on of R6 in -> `2026-09-15-source-runtime-reliability.md` +> Owns: delta replication and tiles for a replica, registered as a follow-on +> of R6 in `2026-09-15-source-runtime-reliability.md` > Register: `PROJECT_STATUS.md` > Companions: `docs/core/design/mounts.md` (replication by generation), > `2026-09-15-source-runtime-reliability.md` (R3 generations, R6 database > delivery), `docs/core/design/source-durability.md` (what sync needs from -> stores), `crates/store/src/revision.rs` (the store revision) +> stores), `crates/store/src/revision.rs` (the store revision), +> `2026-09-08-locations-demoted.md` (L5, the durable sidecar tier) ## Outcome @@ -18,6 +19,9 @@ fallen further behind than the owner keeps changes for. A one-file change costs a one-file transfer and a one-file update to the replica's arena. An unchanged source costs one listing request. +The same replica draws thumbnails for files it has never read, from tiles its +owner already baked. + ## Why Today every replication is a full copy. `FetchSnapshot` and `FetchDatabase` @@ -256,6 +260,67 @@ store replication as well. That is R6's registered follow-on "database delivery for non-nested sources", and it retires snapshot replication and its pacing. +### Tiles a replica did not bake + +A replica carries the index alone. Titan holds 99,137 tiles for its own +sources; the Mac's copy of titan is eight arena snapshots and one database, +and the Mac's own caches cover only the Mac's volumes. Browsing titan from the +Mac draws no thumbnails, and generating more on titan never changes that. + +The cache is machine-local by construction. `ThumbRequestAction` answers with +a `cache_path` the client maps read-only and copies pixels out of, and a local +path is the whole reason tiles stay where they were baked. That contract does +not have to change: if the daemon fills a local cache file for the remote +volume, the same `cache_path` and `TileIdentity` answer works and the client +is none the wiser. + +**Where a remote tile lands.** Caches are keyed by volume, and a volume id is +derived from the volume rather than from the host that mounted it, so it names +the same volume on every device. `RemoteSourceInfo.volume_uuid` already +carries it. A receiver writes remote tiles to `volumes//thumbs.pvcache`, beside its own, under the existing layout. + +**Identity already lines up.** A tile is keyed by record uuid and a content +version, and a replica holds the owner's uuids because it is a copy of the +owner's database. Both sides name a tile the same way with nothing new to +agree on. + +The version is the one thing to get right. `thumbnail_version` folds in +whether the baking host had FFmpeg, so a receiver that computes its own +version for a file it cannot read disagrees with the owner and reads every +remote tile as stale. The owner's version travels with the tile and the +replica stores it as given. A replica never derives a version for a file it +cannot open. + +**Protocol.** `FetchTile { source_id, uuid, version }`, appended after the +existing variants so their wire indices hold. The owner answers from its own +cache through `PvcacheReader::get`: a header carrying the frame and the +version it actually holds, then pixels. `TileState::Absent` answers absent, +and the owner may bake on demand before answering. + +Raw pixels are the wrong wire format. A 384 px slot is 576 KB of BGRA, against +tens of KB encoded. Send WebP, which `core` already depends on, and decode on +receipt; the receiver writes BGRA into its cache, so the client's mmap path is +untouched. + +**Requesting.** `ThumbRequestInput` takes `SdPath`, which can already name +another device. A path under a replica resolves through `remote_share` to the +owning device, and the request enqueues a fetch rather than a bake. Completion +is the `Thumbnail` event clients already listen for. Draw order is request +order, so a viewport is served before its prefetch margin, and an owner that +is offline answers absent rather than stalling the batch. + +**What it does not give you.** All of this is cache. Tiles vanish with a cache +wipe, and nothing is available while the owner is asleep. L5 in +`2026-09-08-locations-demoted.md` is the durable tier: content-addressed files +scoped to the source, which replicate with the source and serve every +duplicate of a file from one tile. That is what a library holding thumbnails +for a whole NAS actually rests on. + +Fetch comes first because it needs no durable tier and answers the case in +front of us. When sidecars land, the fetch becomes the fallback for what no +sidecar covers. + ## Phases | Phase | Scope | Exit proof | @@ -266,6 +331,8 @@ pacing. | P3 | `FetchChanges` and replica database apply | A one-file change on the Mac reaches titan's replica database within one refresh interval, with bytes proportional to the change. An owner restart transfers nothing. A recreated store and a replica behind compaction each take exactly one full copy | | P4 | Arena apply and bootstrap profiling | A delta updates titan's arena in time proportional to the delta, and the result matches an arena rebuilt from the replica database. Bootstrap time measured and bounded | | P5 | Volume-root sources onto store replication | No replication path reads file metadata as a version | +| P6 | `FetchTile`, and remote tiles into a local cache | Browsing titan from the Mac draws thumbnails for files the Mac has never read. An offline owner draws no tile and blocks nothing. Fetched tiles survive a restart on both sides | +| P7 | Sidecars carry tiles with the source, after L5 | A replica draws tiles for an owner that is asleep, and a cache wipe on either side costs no rebake | ## Acceptance @@ -283,6 +350,9 @@ copies and apply time, the counters R3 asks for. every known replica acknowledges a revision. 3. Whether adapter sources replicate to peers. Today only filesystem sources are listed. +4. Whether a remote volume's tile cache is capped and evicted, or left to + grow. Titan's own tiles are 21 GB across 99,137 slots, and a replica that + fetches everything it ever drew converges on the same order. ## Relationship to sync