mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2026-11-03 13:55:30 -05:00
Jamie Pineandcapy-ai-staging[bot]
08eb622e29
* Library backup and restore with manifest verification libraries.backup copies library.db, sync.db, every owned source store and its sidecars with VACUUM INTO while the daemon runs, hashes each file with blake3 and writes manifest.json to a directory or .tar.zst archive. libraries.backup.verify rehashes a backup without restoring it. libraries.restore verifies every hash, refuses a backup from a newer schema, stages under the data dir, swaps files into place with the open store and sidecar pools released first, parks the displaced state in restore-trash/, and reopens the library. Replacing a library with other member devices needs force, since their device rows and sync watermarks would diverge. CLI: sd library backup --to, sd library backup verify --from, sd library restore --from [--as-new | --replace] [--force]. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * CLI help: restore-trash is kept, not pruned Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: keep stores closed across the swap, clear the arena, serialize library.json Review findings on #3115: the volume index kept the pre-restore arena and drive snapshots after a replace, a watcher event could reopen the old data.db between the release and the rename, and library.json was copied byte for byte while the daemon rewrites it in place. quiesce_stores takes every affected source's open gate and holds it until the renames are done, closes the cached handles, detaches the library's registrations and partitions, and removes the drive snapshots so the reopened library rebuilds from the restored stores. Backup serializes the open library's config instead of copying the file, and restore parses the staged config before touching the data directory. A failure after the first displacement now names the restore-trash directory. The test feeds source b's watcher during the replace and proves a later write reaches the restored file on disk. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: load the staged config as LibraryConfig, hold sidecars across the swap Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
111 lines
3.8 KiB
Plaintext
111 lines
3.8 KiB
Plaintext
---
|
|
title: Database Architecture
|
|
description: SQLite ownership and transaction boundaries in Spacedrive
|
|
sidebarTitle: Database
|
|
---
|
|
|
|
Spacedrive uses embedded SQLite databases so users can run the complete system
|
|
without a database server. Database boundaries follow data authority and
|
|
lifetime.
|
|
|
|
## Database roles
|
|
|
|
| Database | Owns |
|
|
| --- | --- |
|
|
| Core state | Known libraries, daemon configuration, and local identity |
|
|
| Library state | Source registry, devices, policy, organization, and assertions |
|
|
| Source store | One source generation and its irreplaceable assertion layer |
|
|
| Jobs | Durable execution state, progress, output, and failure |
|
|
| Catalog | Rebuildable cross-source content and availability projection |
|
|
|
|
One giant library file does not own every source record.
|
|
|
|
## Source-store schema
|
|
|
|
A source store is self-contained. Its generation includes records, provider
|
|
facets, content evidence, edges, search projection, and ingest state. Its
|
|
assertion layer includes durable overlays and extension knowledge that must
|
|
survive a scan.
|
|
|
|
Generation replacement and assertion retention are separate transaction
|
|
concerns. A reindex can publish a complete new generation without deleting
|
|
human decisions.
|
|
|
|
## SQLite behavior
|
|
|
|
The daemon is the writer. Clients use registered operations instead of opening
|
|
database files directly.
|
|
|
|
WAL mode supports concurrent daemon reads while bounded write transactions
|
|
commit. A job advances its durable watermark only after the corresponding
|
|
transaction succeeds.
|
|
|
|
Database pragmas, connection limits, and checkpoint policy are selected per
|
|
role. A write-heavy source ingest does not need the same pool behavior as a
|
|
small library registry.
|
|
|
|
## Migrations
|
|
|
|
Each database records its schema version. Migrations:
|
|
|
|
1. verify the current version;
|
|
2. run inside a transaction where SQLite permits it;
|
|
3. preserve unknown or extension-owned data;
|
|
4. update the version only after success;
|
|
5. leave a clear recovery error on failure.
|
|
|
|
A migration that changes public types also regenerates client bindings.
|
|
|
|
## Backup
|
|
|
|
Copying a live database file without its WAL can lose committed state. Use
|
|
the library backup, which snapshots every database with `VACUUM INTO` and
|
|
records a blake3 hash per file, or a source freeze for one store.
|
|
|
|
~~~bash
|
|
sd-cli library backup --to <DIR|FILE.tar.zst>
|
|
sd-cli library backup verify --from <DIR|FILE.tar.zst>
|
|
sd-cli library restore --from <DIR|FILE.tar.zst> [--as-new | --replace]
|
|
sd-cli sources freeze <SOURCE_ID>
|
|
~~~
|
|
|
|
The library can be small while still containing permissions, tags, and
|
|
provenance that no origin can reproduce, so a backup always takes the
|
|
library and its source stores together. See [Library](/core/library).
|
|
|
|
## Corruption and recovery
|
|
|
|
Recovery follows ownership:
|
|
|
|
- Rebuild a catalog projection from attached source stores.
|
|
- Reingest generation data only if the origin still answers.
|
|
- Restore assertions and library state from a verified backup.
|
|
- Preserve a damaged store for investigation before attempting repair.
|
|
|
|
Never solve source-store corruption by deleting the store and hoping the origin
|
|
still exists.
|
|
|
|
## Extension data
|
|
|
|
Extensions use declared model storage and host operations. They do not receive
|
|
an unrestricted SQLite connection.
|
|
|
|
Source-derived extension facts follow the source generation. User or software
|
|
assertions survive reindexing. Standalone extension models retain their own
|
|
identity and provenance.
|
|
|
|
## Testing
|
|
|
|
Database tests cover:
|
|
|
|
- migration from every supported schema;
|
|
- transaction rollback and process interruption;
|
|
- batch salvage after one invalid row;
|
|
- WAL-safe backup and restore;
|
|
- generation replacement with assertion survival;
|
|
- concurrent reads during ingest;
|
|
- large-source query plans.
|
|
|
|
See [Data Model](/core/data-model) for ownership and [Indexing](/core/indexing)
|
|
for batch and sweep safety.
|