Extension file kinds K1+K2: manifest kinds, layered registry, stored kind_name (#3135)

* Extension kinds K1: manifest kinds, layered FileTypeRegistry::current, conflicts in extensions.list

A manifest may declare kinds (name, built-in parent, extensions, MIME
types, magic bytes, preview). The registry becomes the built-in table plus
every loaded extension's kinds in load order, swapped by the plugin
manager on load and unload; the five builtin() callers and the context
field read current(). A kind that would redefine a built-in extension is
refused at load. load_all sorts directory names so a contested file
extension resolves the same way on every machine; the dropped claim is
reported by extensions.list and its magic patterns stay available.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Extension kinds K2: store schema v2 kind_name, identity phase writes kinds, reidentification on load and open

content.kind_name (nullable, no backfill) and idx_facet_file_extension
over lower(extension) arrive with schema version 2; a replica still at
version 1 reads through a per-connection temp view that adds the column
as NULL. The content identity phase stores the kind for every file and
the kind name for extension kinds, reading the first 8 KB only when an
extension kind's magic patterns are in play. Rows identified before an
extension existed are named by extension when the extension loads and
when a store opens. File gains content_kind_name and reads content_kind
from the row when present. Photos declares raw; the test extension
declares fake with a second fixture contesting it for the WASM suite.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Carry the stored kind across a verification write, filter store search on it, order unclaimed magic candidates by load

bind_content takes the previous row's kind and kind_name onto the
confirmed row when the write carries none, so a verification pass no
longer drops what the identity phase stored. passes_store_filters answers
a content_types filter from the stored kind before deriving from the
name, matching the listing. magic_candidates for an extension nothing
claims walks extension kinds in load order, so a header matching two
kinds resolves the same way on every start.

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Carry the stored kind only when the write describes the same bytes

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

* Add kind_name to the dedupe test's content identities

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
This commit is contained in:
Jamie Pineandcapy-ai-staging[bot] authored and GitHub committed 2026-10-08 03:40:58 +00:00
1 parent ad6452dfbb
commit 0e5340394e
41 files changed
+1977 -100

No files matched your search

+1 -1
View File
@@ -212,7 +212,7 @@ Recent landed.
| Library backup and restore | in progress 2026-10-06 (Fable, SPAC-23); one PR against `main` | Fable | Release gate "Database backup and restore pass from release candidates" and candidate validation step 7. `libraries.backup` copies `library.db`, `sync.db`, every owned source store and its `sidecars.db` with `VACUUM INTO` on a fresh connection while the daemon runs, hashes each file with blake3 and writes `manifest.json` (device, build sha, applied migrations by name, each source's store id, revision and schema hash), into a directory or `.tar.zst`; `libraries.backup.verify` rehashes without restoring; `libraries.restore` verifies every hash first, refuses a backup whose library applied a migration this build does not know, stages under the data dir, renames into place (replace mode closes the library, releases open store and sidecar pools so the swapped `data.db` is what the next open reads, and parks the displaced state in `restore-trash/`), and reopens. Replacing a library other devices are members of needs `force`, because the device rows and sync watermarks in it would diverge from what those peers hold. CLI: `sd library backup --to`, `sd library backup verify --from`, `sd library restore --from [--as-new|--replace] [--force]`. `core/tests/library_backup_test.rs` backs up while a watcher writes, restores into a fresh data dir and proves counts, tag assertions, space items and store revisions equal, replaces over a mutated library and proves the mutation gone, and detects a one-byte tamper. Documented in `docs/core/library.mdx` (Backup and restore). Next: run the gate on the MacBook library before the beta candidate. |
| Extension loading | in progress 2026-10-07 (Fable, SPAC-28); one PR against `main` | Fable | Product sequence step 5, first half. A daemon built with the `wasm` feature discovers every `<data dir>/extensions/<dir>/manifest.json` at startup (before libraries open, so a resumed job finds its extension), instantiates it with wasmer 6 (4.4 stopped linking on Rust 1.89+, wasmer #5610), and keeps the instance for its lifetime. `plugin_init` registers jobs; `extensions.list` reports them and `extensions.run_job` starts one as a `wasm_job` through the normal job manager. The guest's host calls are bridged to the job context: progress, warnings, metrics, an interrupt flag the pause path sets, and checkpoints that reach the checkpoint table before the guest continues, so both a graceful stop (paused row) and a SIGKILL (running row plus checkpoint) resume from the guest's last state on restart; proven live on a Linux daemon and in `core/tests/wasm_extension_test.rs` (acceptance group, with its own `--features wasm` build: carrying the feature across the integration group added minutes to every test binary link, 51 to 78 minutes on the first CI run). The SDK's `#[job]` export now runs the job function (async ones through a poll loop) and `plugin_init` installs a panic hook so a guest panic reaches the host log. Phase B: the photos extension builds to wasm32 without wasm-bindgen imports, loads beside the test extension and registers four jobs; its first job stops at `JobContext::run` (task execution has no host function), and the SDK has no file-kind or preview registration at all, so step 5's second half is a host-function brief (report SPAC-28 lists the gaps: task execution, vdfs entry and sidecar access, AI inference, model registry, tags, entropy, clock, file-kind and preview registration). The `wasm` feature stays opt-in; whether the shipped daemon turns it on is Jamie's call. |
| Extension host functions, phase 1 | in progress 2026-10-07 (Fable, SPAC-30); one PR against `main`, one commit per gap | Fable | Product sequence step 5, second half, in SPAC-28's gap order. Every guest request that returns data now goes through one import, `spacedrive_op(name, payload)`, answered on the job's async side with permission checked against a strict manifest (`read_records` with an optional glob, `read_sidecars`, `write_sidecars` per kind, `use_models` per category; unknown fields are refused at load). Landed: task execution (`#[task]` attempts bracketed by `task.begin`/`task.end` with the SDK's deadline and an attempt line in the job log), entropy and clock (`spacedrive_random` backs `getrandom` so `Uuid::new_v4` works in the guest; `spacedrive_now_ms` backs `clock::now`), records over source stores (`Record` replaces the SDK's `Entry`; `records.get/query/read` resolve through the library's sources and read bytes from the store root; tag and metadata filters are refused), sidecars as one JSON document per extension kind under `SidecarKind::Extension { extension_id, kind }`, models (`#[model]` emits a definition `register_model` collects; the registry builds a `DataTypeSchema` and opens an sd-store db at `<library>/extensions/<id>/data.db`; `models.put/get/list` keyed by uuid or `content:<uuid>`), `config.get` for `config.json` beside the manifest, and `ai.infer`, which checks the `use_models` grant and answers `not_available` because the core has no inference provider. The photos extension handles that with one warning and a skip, so `analyze_photos` completes over a fixture library of twelve JPEGs in `core/tests/wasm_extension_test.rs` (the test extension's new `catalog` job covers records, sidecars, models and tasks in the same suite). Still unbacked: inference itself (so scenes, places and moments have nothing to run on), tags, custom fields, `dispatch_jobs`, agents, actions, queries, and file-kind or preview registration (gap 9, untouched). |
| Extension file kinds and previews | design proposed 2026-10-07 (Fable, SPAC-31); nothing built | unowned | Plan: `docs/plans/2026-10-07-extension-kinds-and-previews.md`. Product sequence step 5, second half, and SPAC-28 gap 9. A manifest `kinds` section (name, built-in parent, extensions, magic bytes, preview) folds into a layered `FileTypeRegistry::current()` the plugin manager swaps on load and unload; the content identity phase writes `content.kind` and a new `content.kind_name` (store schema v2), so a file keeps its kind name after the extension unloads and the client falls back to the parent renderer; `ContentRenderer.tsx` becomes a lookup in a renderer registry fed by `extensions.list`, with bundle previews served from `/extension/:id/*` under a plain `mount(el, ctx)` contract. Nothing is stored to migrate: `content.kind` is NULL on every row today and kinds are derived at read time. Five phases K1 to K5, about eight days, two PRs. Jamie decides: kinds stay one level deep under a built-in parent; conflicts resolve by load order and are reported, no picker yet. |
| Extension file kinds and previews | K1 and K2 in PR 2026-10-07 (Fable, SPAC-35), one PR against `main`; K3 to K5 unowned | Fable | Plan: `docs/plans/2026-10-07-extension-kinds-and-previews.md`. Product sequence step 5, second half, and SPAC-28 gap 9. Built: `manifest.json` `kinds` (name, built-in parent, extensions, MIME, magic, preview; `core/src/filetype/kinds.rs`), validated at load and refused when a claim would redefine a built-in extension; `FileTypeRegistry::current()` is the built-in table plus every loaded extension's kinds in load order, swapped by the plugin manager on load and unload, and every former `builtin()` caller and the context field read it; two extensions claiming one file extension resolve by sorted load order and `extensions.list` reports `kinds` and `conflicts`; store schema v2 adds `content.kind_name` and `idx_facet_file_extension` (no backfill, a v1 replica reads through a temp view); the content identity phase writes `kind` for every file and `kind_name` for extension kinds, checking extension-kind magic bytes; rows identified earlier are named by extension when an extension loads and when a store opens; `File.content_kind_name` reads from the row and `content_kind` from the stored kind when present. Photos declares `raw`. Tests: registry layering and conflicts (lib), manifest refusals (lib), store migration v1 to v2 and the v1 replica (sd-store), and a WASM acceptance test over `.fake` with a second contesting fixture: columns set, lone magic match on the contested kind, name kept after unload, reidentification on load and after a restart. Next: K3 (client renderer registry over `content_kind_name` and `extensions.list`), K4 (bundle previews), K5 (photos viewer bundle). Jamie decides: kinds one level deep; conflicts by load order, no picker yet. |
| MCP server over the registry | PR open 2026-10-07 (Fable, SPAC-33); one PR against `main` | Fable | Product sequence step 2, second half. `sd mcp` speaks MCP over stdio with rmcp and exposes every registered query, action, validate and preview as a tool named after the op, with the input schema walked from the op's specta input type (`core/src/infra/wire/json_schema.rs`), `library_id` on library ops defaulting to the CLI's selection when the daemon still lists it, else the first library, read-only annotations on queries, daemon errors as tool errors, and a synthetic `jobs.wait` that polls `jobs.info` to a terminal status. The server holds no state and no auth beyond the daemon socket. `apps/cli/tests/mcp_test.rs` (acceptance group) starts a core and RPC server in-process, spawns `sd-cli mcp`, handshakes, lists tools, calls `libraries.list` and `jobs.list`, tracks a source and waits for its walk. Docs: `docs/cli/mcp.mdx` (Claude Code, Cursor, Capy configs) and the CLI skill. Next: try it from Claude Code against the MacBook daemon; decide whether actions should require a confirmed preview. |
| Repository release gates in CI | in progress 2026-10-07 (Fable, SPAC-26); one PR against `main` | Fable | `docs/core/releases.mdx` lists five repository gates; only formatting, clippy, typecheck and tests had jobs. `ci.yml` gains four more on every PR to `main`: Docs (bun 1.3.4, `docs/` frozen lockfile, `types:check` and `next build`), Generated types (`generate_typescript_types` plus `cargo check -p spacedrive` for the Tauri schemas, fails on a diff under `packages/ts-client/src/generated` or `apps/tauri/src-tauri/gen/schemas`, own rust-cache key), Secrets scan (pinned gitleaks over the PR's commits and the tree, allowlist in `.gitleaks.toml` including the AWS docs example key in `cloud_credentials.rs`), and License and notices (LICENSE is Apache-2.0, NOTICE exists and carries the same AGPL-3.0 cut-over date as README). Found and fixed on the way: the docs build failed on an unescaped `<name>` in `cli/library-sync-setup.mdx`, and the committed Tauri schemas predated the updater plugin. The gates in `releases.mdx` now name their job. Next: confirm the four jobs green on the PR and the warm wall time under 10 min. |
+4 -6
View File
@@ -50,8 +50,6 @@ pub struct CoreContext {
pub external_tools: Arc<ExternalTools>,
// Remote job cache for cross-device job visibility
pub remote_job_cache: Arc<RemoteJobCache>,
// File type registry (loaded once at startup, never changes)
pub file_type_registry: Arc<FileTypeRegistry>,
// Job logging configuration
pub job_logging_config: Option<JobLoggingConfig>,
pub job_logs_dir: Option<PathBuf>,
@@ -109,7 +107,6 @@ impl CoreContext {
thumbs,
external_tools,
remote_job_cache: Arc::new(RemoteJobCache::new()),
file_type_registry: Arc::new(FileTypeRegistry::new()),
job_logging_config: None,
job_logs_dir: None,
data_dir,
@@ -122,9 +119,10 @@ impl CoreContext {
&self.volume_index
}
/// Get the file type registry
pub fn file_type_registry(&self) -> &Arc<FileTypeRegistry> {
&self.file_type_registry
/// The file type registry every lookup shares: the built-in types plus
/// the kinds of every loaded extension, swapped by the plugin manager.
pub fn file_type_registry(&self) -> Arc<FileTypeRegistry> {
FileTypeRegistry::current()
}
/// Get the library manager
+19 -2
View File
@@ -79,6 +79,11 @@ pub struct File {
/// Additional computed fields
pub content_kind: ContentKind, // Populated by the indexer, for when a File does not have a ContentIdentity
/// The extension kind id (`<extension id>:<name>`) the content identity
/// phase stored, whether or not that extension is loaded now;
/// `content_kind` is then its parent. `None` for a built-in kind.
#[serde(default)]
pub content_kind_name: Option<String>,
pub is_local: bool, // this is redundant with SdPath
/// Video duration (for grid display optimization)
@@ -237,7 +242,7 @@ impl File {
.and_then(|extension| extension.to_str())
.map(|extension| extension.to_lowercase());
let content_kind =
crate::filetype::FileTypeRegistry::builtin().identify_by_extension(path);
crate::filetype::FileTypeRegistry::current().identify_by_extension(path);
(stem, extension, EntryKind::File, content_kind)
};
let now = Utc::now();
@@ -260,11 +265,20 @@ impl File {
modified_at: now,
accessed_at: None,
content_kind,
content_kind_name: None,
duration_seconds: None,
thumbnail_path: None,
}
}
/// A file as its store row describes it.
///
/// The stored content kind is the answer when the identity phase has
/// written one; a row identified before kinds were stored derives its
/// parent from the name through the current registry, as every row did
/// before. The stored kind name is reported whether or not the extension
/// that declared it is loaded, so the client can fall back to the parent
/// until the extension returns.
pub fn from_store_entry(entry: &sd_store::FsEntry, sd_path: SdPath) -> Self {
let is_local = sd_path.is_local();
@@ -293,7 +307,7 @@ impl File {
.filter(|kind| *kind != ContentKind::Unknown)
.unwrap_or_else(|| {
if entry.kind == sd_store::FileKind::File {
crate::filetype::FileTypeRegistry::builtin()
crate::filetype::FileTypeRegistry::current()
.identify_by_extension(std::path::Path::new(&entry.name))
} else {
ContentKind::Unknown
@@ -316,6 +330,7 @@ impl File {
modified_at: from_ms(entry.mtime_ms).unwrap_or_else(Utc::now),
accessed_at: from_ms(entry.atime_ms),
content_kind,
content_kind_name: entry.content_kind_name.clone(),
extension,
kind,
is_local,
@@ -408,6 +423,7 @@ impl File {
modified_at,
accessed_at,
content_kind: ContentKind::Unknown,
content_kind_name: None,
extension,
kind,
is_local,
@@ -519,6 +535,7 @@ mod tests {
sampled_hash: None,
integrity_hash: None,
content_kind,
content_kind_name: None,
content_error: None,
}
}
+250
View File
@@ -0,0 +1,250 @@
//! # Extension file kinds
//!
//! An extension's `manifest.json` may declare file kinds: a name, a parent
//! from the built-in [`ContentKind`] set, the file extensions and magic bytes
//! that identify it, and how the client previews it. A kind is one level deep
//! under its parent so every existing consumer of a content kind (search
//! filters, kind stats, icons) keeps understanding the parent while a loaded
//! extension refines it. The kind's id everywhere else is
//! `<extension id>:<name>`, the convention jobs already use, so two
//! extensions can both declare `raw` without colliding.
//!
//! These types live beside the registry rather than in the extension module
//! because the registry, `extensions.list` and the content identity phase all
//! read them, and all three exist in a build without the `wasm` feature.
use serde::{Deserialize, Serialize};
use specta::Type;
use crate::domain::ContentKind;
use crate::filetype::magic::MagicBytePattern;
/// Built-in renderers a kind may name through `preview.renderer`. The client
/// registers these first; `ContentRenderer` falls back to the parent's
/// renderer for a name it does not know.
pub const BUILTIN_RENDERERS: [&str; 7] = [
"image", "video", "audio", "mesh", "document", "text", "default",
];
/// Priority an extension kind registers with, above every built-in type, so a
/// kind that refines a built-in extension wins the lookup for it.
pub const EXTENSION_KIND_PRIORITY: u8 = 110;
/// A file kind as an extension manifest declares it.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(deny_unknown_fields)]
pub struct ExtensionKind {
/// Lowercase `[a-z0-9-]`, unique within the manifest.
pub name: String,
#[serde(default)]
pub display_name: Option<String>,
/// A built-in kind other than `unknown`, `model_entry` and `memory`.
pub parent: ContentKind,
/// Lowercase, no leading dot, at least one.
pub extensions: Vec<String>,
#[serde(default)]
pub mime_types: Vec<String>,
#[serde(default)]
pub magic: Vec<MagicPatternSpec>,
/// Absent means the parent's renderer.
#[serde(default)]
pub preview: Option<PreviewSpec>,
}
/// A magic byte pattern in the shape the built-in TOML definitions use.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(deny_unknown_fields)]
pub struct MagicPatternSpec {
/// Hex bytes separated by spaces; `??` matches any byte.
pub pattern: String,
#[serde(default)]
pub offset: usize,
}
/// How the client previews a kind: a built-in renderer by name, or a
/// `ui_manifest.json` `file_viewers` entry by id.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(rename_all = "snake_case")]
pub enum PreviewSpec {
Renderer(String),
Viewer(String),
}
/// A file extension two loaded extensions both claimed. The kind loaded
/// first holds the claim; the other kind's claim on that extension is
/// dropped from the lookup table and its patterns stay available to the
/// content identity phase as a tie-breaker.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
pub struct KindConflict {
/// The file extension, lowercase, no dot.
pub extension: String,
/// The kind id whose claim was dropped.
pub kind: String,
/// The kind id that holds the extension.
pub claimed_by: String,
}
impl ExtensionKind {
/// The kind's id under its extension: `<extension id>:<name>`.
pub fn id(&self, extension_id: &str) -> String {
format!("{extension_id}:{}", self.name)
}
/// What the host relies on beyond the shape serde enforces.
///
/// A parent outside the built-in set would leave the client with no
/// renderer to fall back to, and an extension with a dot or uppercase
/// would never match the lowercase lookup the registry does.
pub fn validate(&self) -> Result<(), String> {
if self.name.is_empty()
|| !self
.name
.chars()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')
{
return Err(format!(
"kind name {:?} must be lowercase [a-z0-9-]",
self.name
));
}
if matches!(
self.parent,
ContentKind::Unknown | ContentKind::ModelEntry | ContentKind::Memory
) {
return Err(format!(
"kind {:?} cannot have {} as its parent",
self.name, self.parent
));
}
if self.extensions.is_empty() {
return Err(format!("kind {:?} claims no file extensions", self.name));
}
for ext in &self.extensions {
if ext.is_empty()
|| !ext
.chars()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_')
{
return Err(format!(
"kind {:?} extension {ext:?} must be lowercase with no leading dot",
self.name
));
}
}
for magic in &self.magic {
MagicBytePattern::from_hex_string(&magic.pattern, magic.offset, 0)
.map_err(|e| format!("kind {:?} magic {:?}: {e}", self.name, magic.pattern))?;
}
if let Some(PreviewSpec::Renderer(renderer)) = &self.preview {
if !BUILTIN_RENDERERS.contains(&renderer.as_str()) {
return Err(format!(
"kind {:?} names no built-in renderer {renderer:?}; one of {}",
self.name,
BUILTIN_RENDERERS.join(", ")
));
}
}
Ok(())
}
/// The magic patterns, parsed. Call after [`Self::validate`].
pub fn magic_patterns(&self) -> Vec<MagicBytePattern> {
self.magic
.iter()
.filter_map(|m| {
MagicBytePattern::from_hex_string(&m.pattern, m.offset, EXTENSION_KIND_PRIORITY)
.ok()
})
.collect()
}
}
/// Validate every kind in one manifest together: each on its own, and the
/// names unique within the manifest.
pub fn validate_kinds(kinds: &[ExtensionKind]) -> Result<(), String> {
let mut seen = std::collections::HashSet::new();
for kind in kinds {
kind.validate()?;
if !seen.insert(kind.name.as_str()) {
return Err(format!("kind {:?} is declared twice", kind.name));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn kind(json: &str) -> Result<ExtensionKind, String> {
let kind: ExtensionKind = serde_json::from_str(json).map_err(|e| e.to_string())?;
kind.validate()?;
Ok(kind)
}
#[test]
fn a_well_formed_kind_parses() {
let raw = kind(
r#"{"name":"raw","display_name":"RAW photo","parent":"image",
"extensions":["cr2","nef"],"mime_types":["image/x-canon-cr2"],
"magic":[{"pattern":"49 49 2A 00","offset":0}],"preview":{"renderer":"image"}}"#,
)
.unwrap();
assert_eq!(raw.id("com.spacedrive.photos"), "com.spacedrive.photos:raw");
assert_eq!(raw.parent, ContentKind::Image);
assert_eq!(raw.preview, Some(PreviewSpec::Renderer("image".into())));
assert_eq!(raw.magic_patterns().len(), 1);
}
#[test]
fn the_rules_are_enforced() {
let bad = [
(
r#"{"name":"Raw","parent":"image","extensions":["cr2"]}"#,
"lowercase",
),
(
r#"{"name":"raw","parent":"unknown","extensions":["cr2"]}"#,
"parent",
),
(
r#"{"name":"raw","parent":"image","extensions":[]}"#,
"no file extensions",
),
(
r#"{"name":"raw","parent":"image","extensions":[".cr2"]}"#,
"leading dot",
),
(
r#"{"name":"raw","parent":"image","extensions":["cr2"],"magic":[{"pattern":"ZZ"}]}"#,
"magic",
),
(
r#"{"name":"raw","parent":"image","extensions":["cr2"],"preview":{"renderer":"hologram"}}"#,
"renderer",
),
(
r#"{"name":"raw","parent":"image","extensions":["cr2"],"icon":"x"}"#,
"unknown field",
),
(
r#"{"name":"raw","parent":"kind-of-image","extensions":["cr2"]}"#,
"unknown variant",
),
];
for (json, reason) in bad {
let err = kind(json).expect_err(json);
assert!(err.contains(reason), "{json}: {err}");
}
}
#[test]
fn names_are_unique_within_a_manifest() {
let kinds: Vec<ExtensionKind> = serde_json::from_str(
r#"[{"name":"raw","parent":"image","extensions":["cr2"]},
{"name":"raw","parent":"image","extensions":["nef"]}]"#,
)
.unwrap();
assert!(validate_kinds(&kinds).unwrap_err().contains("twice"));
}
}
+7
View File
@@ -12,9 +12,11 @@ use thiserror::Error;
use uuid::Uuid;
pub mod builtin;
pub mod kinds;
pub mod magic;
pub mod registry;
pub use kinds::{ExtensionKind, KindConflict, MagicPatternSpec, PreviewSpec};
pub use magic::{MagicByte, MagicBytePattern};
pub use registry::FileTypeRegistry;
@@ -47,6 +49,11 @@ pub struct FileType {
/// Extensible metadata
pub metadata: JsonValue,
/// The kind id (`<extension id>:<name>`) when an extension declared this
/// type; `None` for every built-in type. `category` is then the parent.
#[serde(default)]
pub kind_name: Option<String>,
}
/// Result of file type identification
+415 -22
View File
@@ -1,16 +1,18 @@
//! File type registry - the main API for file type identification
use super::kinds::{ExtensionKind, KindConflict, EXTENSION_KIND_PRIORITY};
use super::{FileType, FileTypeError, IdentificationMethod, IdentificationResult, Result};
use crate::domain::ContentKind;
use crate::filetype::magic::MagicBytePattern;
use serde::Deserialize;
use std::collections::HashMap;
use std::path::Path;
use std::sync::{Arc, OnceLock, RwLock};
use tokio::fs::File;
use tokio::io::{AsyncReadExt, AsyncSeekExt};
/// Maximum bytes to read for magic byte identification
const MAX_MAGIC_BYTES: usize = 8192;
pub const MAX_MAGIC_BYTES: usize = 8192;
/// Maximum bytes to read for content analysis
const MAX_CONTENT_BYTES: usize = 4096;
@@ -47,6 +49,14 @@ struct MagicByteDefinition {
}
/// Registry of all known file types
///
/// Two layers exist for the process: the built-in registry, parsed once from
/// the embedded definitions, and the current registry, which is the built-in
/// one plus every loaded extension's kinds. The plugin manager swaps the
/// current one on load and unload; every lookup goes through
/// [`Self::current`] so a kind filter and a preview never disagree about the
/// same file.
#[derive(Clone)]
pub struct FileTypeRegistry {
/// All registered file types by ID
types: HashMap<String, FileType>,
@@ -56,18 +66,232 @@ pub struct FileTypeRegistry {
/// MIME type to type ID mapping
mime_map: HashMap<String, String>,
/// Extension kinds whose claim on a file extension lost to a kind loaded
/// earlier, by file extension. Their types stay in `types` so the content
/// identity phase can still check their magic patterns.
contested: HashMap<String, Vec<String>>,
/// Every dropped claim, in load order, for `extensions.list`.
conflicts: Vec<KindConflict>,
/// Extension kind ids in load order, so a tie between kinds that
/// nothing but their magic bytes distinguishes resolves the same way
/// the extension table does.
extension_kind_order: Vec<String>,
}
static BUILTIN: OnceLock<Arc<FileTypeRegistry>> = OnceLock::new();
static CURRENT: OnceLock<RwLock<Arc<FileTypeRegistry>>> = OnceLock::new();
impl FileTypeRegistry {
/// Create a new registry with built-in types
/// The built-in registry, built once for the process.
///
/// Building one parses every built-in definition. That is affordable once
/// and not once per entry: an arena rebuilt from a delivered database adds
/// more than a million rows one at a time.
pub fn builtin() -> &'static Self {
static BUILTIN: std::sync::OnceLock<FileTypeRegistry> = std::sync::OnceLock::new();
BUILTIN.get_or_init(Self::new)
BUILTIN.get_or_init(|| Arc::new(Self::new()))
}
/// The registry every lookup should use: the built-in types plus the
/// kinds of every loaded extension. The built-in registry alone until an
/// extension with kinds loads.
pub fn current() -> Arc<Self> {
CURRENT
.get_or_init(|| RwLock::new(BUILTIN.get_or_init(|| Arc::new(Self::new())).clone()))
.read()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.clone()
}
/// Replace the current registry with the built-in types plus these
/// extensions' kinds, in load order. The plugin manager calls this after
/// every load and unload; an empty list restores the built-in registry.
pub fn install_current(extensions: &[(String, Vec<ExtensionKind>)]) -> Arc<Self> {
let registry = Arc::new(Self::with_extension_kinds(extensions));
let slot = CURRENT
.get_or_init(|| RwLock::new(BUILTIN.get_or_init(|| Arc::new(Self::new())).clone()));
*slot
.write()
.unwrap_or_else(|poisoned| poisoned.into_inner()) = registry.clone();
registry
}
/// The built-in registry plus these extensions' kinds, in load order.
pub fn with_extension_kinds(extensions: &[(String, Vec<ExtensionKind>)]) -> Self {
let mut registry = Self::builtin().clone();
for (extension_id, kinds) in extensions {
for kind in kinds {
registry.register_extension_kind(extension_id, kind);
}
}
registry
}
/// Why a kind may not load against the built-in table, or `None`.
///
/// A kind may refine a file extension the built-in table maps to its own
/// parent, or claim one the table does not know. Redefining `.pdf` as an
/// image would change what every consumer of the parent sees, so that is
/// refused rather than layered.
pub fn refusal_for(&self, kind: &ExtensionKind) -> Option<String> {
for ext in &kind.extensions {
let builtin = self
.get_by_extension(ext)
.into_iter()
.filter(|ft| ft.kind_name.is_none())
.max_by_key(|ft| ft.priority)
.map(|ft| ft.category)
.unwrap_or(ContentKind::Unknown);
if builtin != ContentKind::Unknown && builtin != kind.parent {
return Some(format!(
"kind {:?} claims .{ext} with parent {} but the built-in table maps it to {}",
kind.name, kind.parent, builtin
));
}
}
None
}
/// Add one extension kind. A file extension another extension kind
/// already holds stays with it; this kind's claim is recorded as a
/// conflict and the type is kept for its magic patterns.
fn register_extension_kind(&mut self, extension_id: &str, kind: &ExtensionKind) {
let id = kind.id(extension_id);
for ext in &kind.extensions {
let ext = ext.to_lowercase();
let holder = self
.extension_map
.get(&ext)
.into_iter()
.flatten()
.find(|id| self.types.get(*id).is_some_and(|ft| ft.kind_name.is_some()))
.cloned();
match holder {
Some(claimed_by) => {
self.contested
.entry(ext.clone())
.or_default()
.push(id.clone());
self.conflicts.push(KindConflict {
extension: ext,
kind: id.clone(),
claimed_by,
});
}
None => self.extension_map.entry(ext).or_default().push(id.clone()),
}
}
for mime in &kind.mime_types {
self.mime_map
.entry(mime.clone())
.or_insert_with(|| id.clone());
}
self.extension_kind_order.push(id.clone());
self.types.insert(
id.clone(),
FileType {
id: id.clone(),
name: kind
.display_name
.clone()
.unwrap_or_else(|| kind.name.clone()),
extensions: kind.extensions.clone(),
mime_types: kind.mime_types.clone(),
uti: None,
magic_bytes: kind.magic_patterns(),
category: kind.parent,
priority: EXTENSION_KIND_PRIORITY,
metadata: serde_json::Value::Null,
kind_name: Some(id),
},
);
}
/// Every extension kind with the file extensions it holds, contested
/// claims left out, in no particular order.
pub fn extension_kinds(&self) -> Vec<(&FileType, Vec<String>)> {
self.types
.values()
.filter(|ft| ft.kind_name.is_some())
.map(|ft| {
let held = ft
.extensions
.iter()
.map(|e| e.to_lowercase())
.filter(|ext| {
self.extension_map
.get(ext)
.is_some_and(|ids| ids.contains(&ft.id))
})
.collect();
(ft, held)
})
.collect()
}
/// Every dropped claim, in load order.
pub fn conflicts(&self) -> &[KindConflict] {
&self.conflicts
}
/// The extension kinds whose magic patterns decide a file with this
/// extension: the kind holding the extension when it declares patterns,
/// and every contested kind for it. For an extension no type claims,
/// every extension kind with patterns, since the name says nothing.
/// Empty means the extension lookup is the whole answer.
pub fn magic_candidates(&self, extension: Option<&str>) -> Vec<&FileType> {
let with_magic = |ft: &&FileType| ft.kind_name.is_some() && !ft.magic_bytes.is_empty();
let Some(ext) = extension.map(str::to_lowercase) else {
return Vec::new();
};
let claimed = self.get_by_extension(&ext);
if claimed.is_empty() {
return self
.extension_kind_order
.iter()
.filter_map(|id| self.types.get(id))
.filter(with_magic)
.collect();
}
let mut candidates: Vec<&FileType> = claimed.into_iter().filter(with_magic).collect();
candidates.extend(
self.contested
.get(&ext)
.into_iter()
.flatten()
.filter_map(|id| self.types.get(id))
.filter(with_magic),
);
candidates
}
/// The kind a file's header decides, given the candidates
/// [`Self::magic_candidates`] named and the type its extension mapped to.
///
/// A lone match assigns that kind, which is how a contested kind still
/// wins the files whose bytes it recognizes. Several matches keep the
/// extension's holder when it is among them. No match keeps the
/// extension result.
pub fn resolve_by_magic<'a>(
&'a self,
by_extension: Option<&'a FileType>,
candidates: &[&'a FileType],
header: &[u8],
) -> Option<&'a FileType> {
let matched: Vec<&FileType> = candidates
.iter()
.copied()
.filter(|ft| ft.magic_bytes.iter().any(|p| p.matches(header)))
.collect();
match matched.as_slice() {
[] => by_extension,
[only] => Some(only),
several => by_extension
.filter(|holder| several.iter().any(|ft| ft.id == holder.id))
.or(Some(several[0])),
}
}
pub fn new() -> Self {
@@ -75,6 +299,9 @@ impl FileTypeRegistry {
types: HashMap::new(),
extension_map: HashMap::new(),
mime_map: HashMap::new(),
contested: HashMap::new(),
conflicts: Vec::new(),
extension_kind_order: Vec::new(),
};
// Load built-in types
@@ -163,25 +390,19 @@ impl FileTypeRegistry {
///
/// Returns `ContentKind::Unknown` if the extension is not recognized.
pub fn identify_by_extension(&self, path: &Path) -> ContentKind {
let extension = match path.extension().and_then(|s| s.to_str()) {
Some(ext) => ext,
None => return ContentKind::Unknown,
};
self.type_by_extension(path)
.map(|ft| ft.category)
.unwrap_or(ContentKind::Unknown)
}
let candidates = self.get_by_extension(extension);
match candidates.len() {
0 => ContentKind::Unknown,
1 => candidates[0].category,
_ => {
// Multiple matches - pick highest priority
candidates
.iter()
.max_by_key(|ft| ft.priority)
.map(|ft| ft.category)
.unwrap_or(ContentKind::Unknown)
}
}
/// The highest priority type the path's extension maps to, which is an
/// extension kind when one holds the extension and a built-in type
/// otherwise. `None` for no extension or one nothing claims.
pub fn type_by_extension(&self, path: &Path) -> Option<&FileType> {
let extension = path.extension().and_then(|s| s.to_str())?;
self.get_by_extension(extension)
.into_iter()
.max_by_key(|ft| ft.priority)
}
/// Identify a file type from a path
@@ -402,6 +623,7 @@ impl FileTypeRegistry {
category,
priority: def.priority,
metadata: def.metadata,
kind_name: None,
})
}
}
@@ -434,4 +656,175 @@ mod tests {
let ts_types = registry.get_by_extension("ts");
assert_eq!(ts_types.len(), 2); // TypeScript and MPEG-TS
}
fn kind(name: &str, parent: ContentKind, extensions: &[&str], magic: &[&str]) -> ExtensionKind {
ExtensionKind {
name: name.into(),
display_name: None,
parent,
extensions: extensions.iter().map(|e| e.to_string()).collect(),
mime_types: Vec::new(),
magic: magic
.iter()
.map(|pattern| super::super::kinds::MagicPatternSpec {
pattern: pattern.to_string(),
offset: 0,
})
.collect(),
preview: None,
}
}
#[test]
fn an_extension_kind_refines_a_built_in_extension_and_goes_away_on_unload() {
let raw = kind("raw", ContentKind::Image, &["cr2", "dng"], &[]);
let layered = FileTypeRegistry::with_extension_kinds(&[("photos".to_string(), vec![raw])]);
let path = Path::new("shot.dng");
let ft = layered.type_by_extension(path).unwrap();
assert_eq!(ft.id, "photos:raw");
assert_eq!(ft.kind_name.as_deref(), Some("photos:raw"));
assert_eq!(layered.identify_by_extension(path), ContentKind::Image);
assert_eq!(
layered.type_by_extension(Path::new("a.jpg")).unwrap().id,
"image/jpeg",
"other built-in types are untouched"
);
let unloaded = FileTypeRegistry::with_extension_kinds(&[]);
assert_eq!(
unloaded.type_by_extension(path).unwrap().id,
"image/x-adobe-dng"
);
assert!(unloaded.conflicts().is_empty());
}
#[test]
fn a_kind_may_not_redefine_a_built_in_extension() {
let builtin = FileTypeRegistry::builtin();
let bad = kind("pdf-as-image", ContentKind::Image, &["pdf"], &[]);
let reason = builtin.refusal_for(&bad).unwrap();
assert!(reason.contains(".pdf"), "{reason}");
assert!(reason.contains("document"), "{reason}");
let refine = kind("raw", ContentKind::Image, &["cr2"], &[]);
assert_eq!(builtin.refusal_for(&refine), None);
let fresh = kind("thing", ContentKind::Database, &["zzznoone"], &[]);
assert_eq!(builtin.refusal_for(&fresh), None);
}
#[test]
fn two_extensions_claiming_one_extension_resolve_by_load_order() {
let first = kind("fake", ContentKind::Image, &["xyz"], &["46 41 4B 45"]);
let second = kind(
"other",
ContentKind::Image,
&["xyz", "abc"],
&["4F 54 48 52"],
);
let layered = FileTypeRegistry::with_extension_kinds(&[
("one".to_string(), vec![first]),
("two".to_string(), vec![second]),
]);
assert_eq!(
layered.type_by_extension(Path::new("f.xyz")).unwrap().id,
"one:fake"
);
assert_eq!(
layered.type_by_extension(Path::new("f.abc")).unwrap().id,
"two:other",
"an uncontested extension of the loser still loads"
);
assert_eq!(
layered.conflicts(),
&[KindConflict {
extension: "xyz".into(),
kind: "two:other".into(),
claimed_by: "one:fake".into(),
}]
);
let candidates = layered.magic_candidates(Some("xyz"));
let ids: Vec<&str> = candidates.iter().map(|ft| ft.id.as_str()).collect();
assert_eq!(ids, ["one:fake", "two:other"]);
let holder = layered.type_by_extension(Path::new("f.xyz"));
assert_eq!(
layered
.resolve_by_magic(holder, &candidates, b"OTHR....")
.unwrap()
.id,
"two:other",
"a lone match on a contested kind wins the file"
);
assert_eq!(
layered
.resolve_by_magic(holder, &candidates, b"FAKE....")
.unwrap()
.id,
"one:fake"
);
assert_eq!(
layered
.resolve_by_magic(holder, &candidates, b"????....")
.unwrap()
.id,
"one:fake",
"no match keeps the extension result"
);
assert!(
layered.magic_candidates(Some("jpg")).is_empty(),
"a built-in extension nobody refines reads no bytes"
);
let unknown: Vec<&str> = layered
.magic_candidates(Some("zzzunknown"))
.iter()
.map(|ft| ft.id.as_str())
.collect();
assert_eq!(
unknown,
["one:fake", "two:other"],
"an unknown extension checks every kind with magic, in load order"
);
let both = FileTypeRegistry::with_extension_kinds(&[
(
"one".to_string(),
vec![kind("a", ContentKind::Text, &["qqq"], &["4F 54"])],
),
(
"two".to_string(),
vec![kind("b", ContentKind::Text, &["rrr"], &["4F 54 48"])],
),
]);
let candidates = both.magic_candidates(Some("zzzunknown"));
assert_eq!(
both.resolve_by_magic(None, &candidates, b"OTHR")
.unwrap()
.id,
"one:a",
"several matches on an unclaimed extension go to the kind loaded first"
);
assert!(layered.magic_candidates(None).is_empty());
}
#[test]
fn install_current_swaps_the_process_registry() {
let before = FileTypeRegistry::current();
assert!(before.type_by_extension(Path::new("a.zzzkind")).is_none());
let installed = FileTypeRegistry::install_current(&[(
"ext".to_string(),
vec![kind("zz", ContentKind::Text, &["zzzkind"], &[])],
)]);
assert_eq!(
FileTypeRegistry::current()
.type_by_extension(Path::new("a.zzzkind"))
.unwrap()
.id,
"ext:zz"
);
assert!(Arc::ptr_eq(&installed, &FileTypeRegistry::current()));
FileTypeRegistry::install_current(&[]);
assert!(FileTypeRegistry::current()
.type_by_extension(Path::new("a.zzzkind"))
.is_none());
}
}
+24
View File
@@ -37,6 +37,30 @@ reports `supported: false` and `extensions.run_job` is refused.
reject unknown fields, so a typo in a grant fails at load instead of
silently granting nothing.
## File kinds
A manifest may declare `kinds`: a name, a built-in `ContentKind` parent, the
file extensions and optional MIME types and magic bytes that identify it,
and a `preview` naming a built-in renderer or a `ui_manifest.json` viewer
(`core/src/filetype/kinds.rs`). The kind's id is `<extension id>:<name>`.
On load the manager rebuilds `FileTypeRegistry::current()` from the built-in
table plus every loaded extension's kinds in load order, and on unload it
rebuilds without them. A kind may refine a file extension the built-in table
maps to its own parent, or claim one the table does not know; a claim that
would redefine a built-in extension is refused at load. Two extensions
claiming one file extension resolve by load order (`load_all` sorts the
directory names), the later claim is dropped from the lookup and reported by
`extensions.list` as a conflict, and its magic patterns still decide a file
whose bytes match only them.
The content identity phase writes the kind and, for an extension kind, its
name onto the content row (`content.kind`, `content.kind_name`, store schema
version 2). Rows identified before an extension existed are named by
extension when the extension loads and when a store opens
(`core/src/ops/indexing/kinds.rs`). A row keeps its name after the extension
unloads; `File.content_kind` is then the parent and `File.content_kind_name`
the name, so a client can fall back until the extension returns.
## Running a job
1. `extensions.run_job` looks the job up in the registry and dispatches a
+61 -5
View File
@@ -17,6 +17,8 @@ use super::host_functions::{self, host_spacedrive_log, JobBridge, PluginEnv};
use super::job_registry::ExtensionJobRegistry;
use super::model_registry::ExtensionModelRegistry;
use super::types::{ExtensionManifest, LoadedPlugin};
use crate::filetype::{ExtensionKind, FileTypeRegistry};
use crate::ops::indexing::VolumeIndex;
#[derive(Error, Debug)]
pub enum PluginError {
@@ -135,16 +137,58 @@ pub struct PluginManager {
plugin_dir: PathBuf,
job_registry: Arc<ExtensionJobRegistry>,
model_registry: Arc<ExtensionModelRegistry>,
/// The kinds of every loaded extension, in load order, which is the
/// order the file type registry resolves a contested file extension in.
kinds: Vec<(String, Vec<ExtensionKind>)>,
/// Where open stores are found, so a load with kinds can name the rows
/// those stores already hold. `None` only in tests of the manager alone.
volume_index: Option<Arc<VolumeIndex>>,
}
impl PluginManager {
/// Create new plugin manager
pub fn new(plugin_dir: PathBuf) -> Self {
pub fn new(plugin_dir: PathBuf, volume_index: Option<Arc<VolumeIndex>>) -> Self {
Self {
plugins: Arc::new(RwLock::new(HashMap::new())),
plugin_dir,
job_registry: Arc::new(ExtensionJobRegistry::new()),
model_registry: Arc::new(ExtensionModelRegistry::new()),
kinds: Vec::new(),
volume_index,
}
}
/// Put this extension's kinds into the registry every lookup uses, and
/// name the rows open stores already hold after them.
async fn add_kinds(&mut self, plugin_id: &str, kinds: &[ExtensionKind]) {
if kinds.is_empty() {
return;
}
self.kinds.push((plugin_id.to_string(), kinds.to_vec()));
let registry = FileTypeRegistry::install_current(&self.kinds);
for conflict in registry.conflicts() {
if conflict.kind.starts_with(&format!("{plugin_id}:")) {
tracing::warn!(
extension = %plugin_id,
file_extension = %conflict.extension,
kind = %conflict.kind,
claimed_by = %conflict.claimed_by,
"file extension already claimed by an earlier extension; claim dropped"
);
}
}
if let Some(volume_index) = &self.volume_index {
crate::ops::indexing::kinds::name_kinds_in_open_stores(volume_index).await;
}
}
/// Take this extension's kinds back out of the registry. Rows already
/// named keep their name; readers fall back to the parent kind.
fn remove_kinds(&mut self, plugin_id: &str) {
let before = self.kinds.len();
self.kinds.retain(|(id, _)| id != plugin_id);
if self.kinds.len() != before {
FileTypeRegistry::install_current(&self.kinds);
}
}
@@ -177,12 +221,19 @@ impl PluginManager {
}
};
let mut loaded = Vec::new();
// Sorted, so the extension that wins a contested file extension is
// the same on every machine rather than whichever the filesystem
// listed first.
let mut dir_names = Vec::new();
while let Ok(Some(entry)) = dirs.next_entry().await {
if !entry.path().join("manifest.json").is_file() {
continue;
if entry.path().join("manifest.json").is_file() {
dir_names.push(entry.file_name().to_string_lossy().to_string());
}
let dir_name = entry.file_name().to_string_lossy().to_string();
}
dir_names.sort();
let mut loaded = Vec::new();
for dir_name in dir_names {
match self.load_plugin(&dir_name).await {
Ok(id) => loaded.push(id),
Err(e) => tracing::error!(extension = %dir_name, "Extension did not load: {e}"),
@@ -395,7 +446,9 @@ impl PluginManager {
"Extension loaded"
);
let kinds = plugin.manifest.kinds.clone();
self.plugins.write().await.insert(plugin_id.clone(), plugin);
self.add_kinds(&plugin_id, &kinds).await;
Ok(plugin_id)
}
@@ -411,6 +464,7 @@ impl PluginManager {
.ok_or_else(|| PluginError::NotFound(plugin_id.to_string()))?;
self.job_registry.unregister_extension_jobs(plugin_id);
self.model_registry.unregister_extension(plugin_id);
self.remove_kinds(plugin_id);
// A trapped guest cannot be trusted to run cleanup, and a guest still
// running a job keeps its instance alive until that job returns; the
@@ -465,10 +519,12 @@ impl PluginManager {
},
);
}
let kinds = previous.manifest.kinds.clone();
self.plugins
.write()
.await
.insert(plugin_id.to_string(), previous);
self.add_kinds(plugin_id, &kinds).await;
return Err(e);
}
Ok(())
+62
View File
@@ -6,6 +6,8 @@ use std::path::PathBuf;
use std::sync::atomic::AtomicBool;
use std::sync::{Arc, Mutex};
use crate::filetype::{kinds::validate_kinds, ExtensionKind, FileTypeRegistry};
/// Extension manifest (`manifest.json`).
///
/// Unknown fields are refused rather than ignored, so a grant the host does
@@ -44,6 +46,11 @@ pub struct ExtensionManifest {
/// Configuration schema (JSON Schema)
#[serde(default)]
pub config_schema: Option<serde_json::Value>,
/// File kinds this extension declares, folded into the file type
/// registry while it is loaded. See `crate::filetype::kinds`.
#[serde(default)]
pub kinds: Vec<ExtensionKind>,
}
/// Permission declaration in manifest.
@@ -104,6 +111,12 @@ impl ExtensionManifest {
));
}
}
validate_kinds(&self.kinds)?;
for kind in &self.kinds {
if let Some(reason) = FileTypeRegistry::builtin().refusal_for(kind) {
return Err(reason);
}
}
Ok(())
}
}
@@ -223,6 +236,15 @@ mod tests {
assert!(perms.can_read_sidecar("exif"));
assert!(!perms.can_write_sidecar("exif"));
assert_eq!(manifest.models.len(), 2);
assert_eq!(
manifest.validate(),
Ok(()),
"photos' kinds load against the built-in table"
);
assert_eq!(
manifest.kinds[0].id(&manifest.id),
"com.spacedrive.photos:raw"
);
}
#[test]
@@ -260,6 +282,46 @@ mod tests {
assert!(manifest.validate().is_err());
}
#[test]
fn the_loader_refuses_a_kind_that_redefines_a_built_in_extension() {
let mut manifest: ExtensionManifest = serde_json::from_str(
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
"kinds":[{"name":"raw","parent":"image","extensions":["cr2","dng"],
"preview":{"renderer":"image"}}]}"#,
)
.unwrap();
assert_eq!(
manifest.validate(),
Ok(()),
"refining .dng under image is allowed"
);
manifest.kinds[0].extensions = vec!["pdf".into()];
let err = manifest.validate().unwrap_err();
assert!(err.contains(".pdf") && err.contains("document"), "{err}");
manifest.kinds[0].extensions = vec!["cr2".into()];
manifest.kinds[0].parent = crate::domain::ContentKind::Unknown;
assert!(manifest.validate().unwrap_err().contains("parent"));
let twice = serde_json::from_str::<ExtensionManifest>(
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
"kinds":[{"name":"raw","parent":"image","extensions":["cr2"]},
{"name":"raw","parent":"image","extensions":["nef"]}]}"#,
)
.unwrap();
assert!(twice.validate().unwrap_err().contains("twice"));
let unknown_field = serde_json::from_str::<ExtensionManifest>(
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
"kinds":[{"name":"raw","parent":"image","extensions":["cr2"],"icon":"x"}]}"#,
);
assert!(unknown_field
.unwrap_err()
.to_string()
.contains("unknown field"));
}
#[test]
fn a_glob_without_a_list_names_one_extension() {
let perms = ManifestPermissions {
+4 -1
View File
@@ -186,7 +186,10 @@ impl Core {
warn!("Could not create extensions directory: {e}");
}
let mut pm = crate::infra::extension::PluginManager::new(plugin_dir);
let mut pm = crate::infra::extension::PluginManager::new(
plugin_dir,
Some(context.volume_index().clone()),
);
let loaded = pm.load_all().await;
info!("Loaded {} extension(s): {:?}", loaded.len(), loaded);
+45 -1
View File
@@ -1,4 +1,5 @@
//! List loaded extensions and the jobs they registered
//! List loaded extensions, the jobs they registered, the file kinds they
//! declare, and the file extensions two of them both claimed.
use std::sync::Arc;
@@ -7,6 +8,8 @@ use specta::Type;
use crate::{
context::CoreContext,
domain::ContentKind,
filetype::{KindConflict, PreviewSpec},
infra::{
api::SessionContext,
query::{CoreQuery, QueryResult},
@@ -25,12 +28,29 @@ pub struct ExtensionJobInfo {
pub resumable: bool,
}
/// A file kind an extension declares, as the client resolves previews and
/// icons against it. The kinds of every loaded extension are the client's
/// only source of extension kinds, which is what lets a stored kind name
/// fall back to its parent once the extension is gone.
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
pub struct ExtensionKindInfo {
/// `<extension id>:<name>`, the value `File.content_kind_name` carries
pub id: String,
pub name: String,
pub display_name: String,
pub parent: ContentKind,
/// Every extension the manifest claims, contested ones included
pub extensions: Vec<String>,
pub preview: Option<PreviewSpec>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
pub struct ExtensionInfo {
pub id: String,
pub name: String,
pub version: String,
pub jobs: Vec<ExtensionJobInfo>,
pub kinds: Vec<ExtensionKindInfo>,
}
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
@@ -38,6 +58,9 @@ pub struct ListExtensionsOutput {
/// Whether this build can load extensions at all
pub supported: bool,
pub extensions: Vec<ExtensionInfo>,
/// File extensions two loaded extensions both claimed, resolved by load
/// order: the kind loaded first holds each one.
pub conflicts: Vec<KindConflict>,
}
pub struct ListExtensionsQuery;
@@ -61,6 +84,7 @@ impl CoreQuery for ListExtensionsQuery {
return Ok(ListExtensionsOutput {
supported: true,
extensions: Vec::new(),
conflicts: Vec::new(),
});
};
let pm = plugin_manager.read().await;
@@ -80,17 +104,36 @@ impl CoreQuery for ListExtensionsQuery {
})
.collect();
jobs.sort_by(|a, b| a.name.cmp(&b.name));
let kinds = manifest
.kinds
.iter()
.map(|kind| ExtensionKindInfo {
id: kind.id(&id),
name: kind.name.clone(),
display_name: kind
.display_name
.clone()
.unwrap_or_else(|| kind.name.clone()),
parent: kind.parent,
extensions: kind.extensions.clone(),
preview: kind.preview.clone(),
})
.collect();
extensions.push(ExtensionInfo {
id,
name: manifest.name.clone(),
version: manifest.version.clone(),
jobs,
kinds,
});
}
extensions.sort_by(|a, b| a.id.cmp(&b.id));
Ok(ListExtensionsOutput {
supported: true,
extensions,
conflicts: crate::filetype::FileTypeRegistry::current()
.conflicts()
.to_vec(),
})
}
#[cfg(not(feature = "wasm"))]
@@ -99,6 +142,7 @@ impl CoreQuery for ListExtensionsQuery {
Ok(ListExtensionsOutput {
supported: false,
extensions: Vec::new(),
conflicts: Vec::new(),
})
}
}
+1
View File
@@ -318,6 +318,7 @@ pub(super) async fn integrity(file: &Keyed) -> (Result<String, String>, Option<L
integrity_hash: Some(integrity.clone()),
size: file.entry.size,
kind: None,
kind_name: None,
},
};
(Ok(integrity), Some(learned))
+1 -1
View File
@@ -283,7 +283,7 @@ pub fn folder_for_kind(entry: &FsEntry) -> String {
.map(ContentKind::from_id)
.filter(|kind| *kind != ContentKind::Unknown)
.unwrap_or_else(|| {
crate::filetype::FileTypeRegistry::builtin()
crate::filetype::FileTypeRegistry::current()
.identify_by_extension(Path::new(&entry.name))
});
match kind {
+3 -7
View File
@@ -291,7 +291,7 @@ impl Arena {
uuid: Uuid,
metadata: EntryMetadata,
) -> std::io::Result<(Option<ContentKind>, Uuid)> {
self.add_entry_with_registry(path, Some(uuid), metadata, FileTypeRegistry::builtin())
self.add_entry_with_registry(path, Some(uuid), metadata, &FileTypeRegistry::current())
}
fn add_entry_with_registry(
@@ -451,14 +451,10 @@ impl Arena {
entries: Vec<(PathBuf, Option<Uuid>, EntryMetadata)>,
) -> std::io::Result<Vec<(Option<ContentKind>, Uuid)>> {
let mut results = Vec::with_capacity(entries.len());
let registry = FileTypeRegistry::current();
for (path, uuid, metadata) in entries {
results.push(self.add_entry_with_registry(
path,
uuid,
metadata,
FileTypeRegistry::builtin(),
)?);
results.push(self.add_entry_with_registry(path, uuid, metadata, &registry)?);
}
Ok(results)
+113 -20
View File
@@ -13,16 +13,28 @@
//! decision that needs it, which is the same rule the ladder in
//! `sd_store::content` exists to keep: never delete one copy of two on the
//! strength of a guess.
//!
//! The pass also decides the content's kind, since it is the one place that
//! opens every file. The file extension answers for a built-in kind. A file
//! an extension kind claims with magic patterns, or one no type claims, has
//! its first bytes checked against the extension kinds' patterns; the
//! built-in patterns stay unused as before. Both the kind and, for an
//! extension kind, its name land on the content row.
use crate::{
domain::content_identity::ContentHashGenerator,
domain::{content_identity::ContentHashGenerator, ContentKind},
filetype::{registry::MAX_MAGIC_BYTES, FileTypeRegistry},
infra::job::{generic_progress::GenericProgress, prelude::*, types::JobPriority},
ops::indexing::SourceStore,
};
use futures::StreamExt;
use sd_store::ContentIdentity;
use serde::{Deserialize, Serialize};
use std::{path::PathBuf, sync::Arc};
use std::{
path::{Path, PathBuf},
sync::Arc,
};
use tokio::io::AsyncReadExt;
use uuid::Uuid;
/// Files claimed from the store per pass. Large enough that the queue round
@@ -166,6 +178,7 @@ impl JobHandler for ContentIdentityJob {
let mut identified = 0u64;
let mut unreadable = 0u64;
let registry = FileTypeRegistry::current();
loop {
ctx.check_interrupt().await?;
@@ -180,7 +193,7 @@ impl JobHandler for ContentIdentityJob {
for chunk in batch.chunks(PROGRESS_CHUNK) {
ctx.check_interrupt().await?;
let (identities, failures) = hash_batch(chunk.to_vec()).await;
let (identities, failures) = hash_batch(chunk.to_vec(), &registry).await;
// Failures leave the pending set with their reason recorded,
// so the loop always advances and the store can say
@@ -240,19 +253,24 @@ pub(crate) fn source_label(root: &std::path::Path) -> String {
/// on every pass.
async fn hash_batch(
batch: Vec<(Uuid, PathBuf, u64)>,
registry: &FileTypeRegistry,
) -> (Vec<(Uuid, ContentIdentity)>, Vec<(Uuid, String)>) {
let results: Vec<_> = futures::stream::iter(batch)
.map(|(uuid, path, size)| async move {
match ContentHashGenerator::generate_content_hash(&path).await {
Ok(hash) => Ok((
uuid,
ContentIdentity {
sampled_hash: Some(hash),
integrity_hash: None,
size: Some(size as i64),
kind: None,
},
)),
Ok(hash) => {
let (kind, kind_name) = identify_kind(&path, registry).await;
Ok((
uuid,
ContentIdentity {
sampled_hash: Some(hash),
integrity_hash: None,
size: Some(size as i64),
kind,
kind_name,
},
))
}
Err(error) => {
tracing::debug!(path = %path.display(), %error, "could not hash");
Err((uuid, error.to_string()))
@@ -274,6 +292,36 @@ async fn hash_batch(
(identities, failures)
}
/// The kind a file's name and, when an extension kind's patterns are in
/// play, its first bytes decide: the `ContentKind` discriminant to store,
/// and the extension kind's id when one claimed the file.
///
/// Bytes are read only when the registry has patterns to check for this
/// extension; a plain `.jpg` costs no second open. A header that cannot be
/// read keeps the extension result, since the hash just succeeded and the
/// file was there a moment ago.
async fn identify_kind(path: &Path, registry: &FileTypeRegistry) -> (Option<i64>, Option<String>) {
let by_extension = registry.type_by_extension(path);
let extension = path.extension().and_then(|e| e.to_str());
let candidates = registry.magic_candidates(extension);
let file_type = if candidates.is_empty() {
by_extension
} else {
let mut header = vec![0u8; MAX_MAGIC_BYTES];
let read = match tokio::fs::File::open(path).await {
Ok(mut file) => file.read(&mut header).await.unwrap_or(0),
Err(_) => 0,
};
header.truncate(read);
registry.resolve_by_magic(by_extension, &candidates, &header)
};
let Some(file_type) = file_type else {
return (None, None);
};
let kind = (file_type.category != ContentKind::Unknown).then_some(file_type.category as i64);
(kind, file_type.kind_name.clone())
}
/// Queue the hashing of every source on this machine, behind whatever else is
/// running.
///
@@ -323,10 +371,13 @@ mod tests {
let real = dir.path().join("real.bin");
std::fs::write(&real, vec![7u8; 4096]).unwrap();
let (identities, failures) = hash_batch(vec![
(Uuid::now_v7(), dir.path().join("gone.bin"), 10),
(Uuid::now_v7(), real, 4096),
])
let (identities, failures) = hash_batch(
vec![
(Uuid::now_v7(), dir.path().join("gone.bin"), 10),
(Uuid::now_v7(), real, 4096),
],
FileTypeRegistry::builtin(),
)
.await;
assert_eq!(identities.len(), 1);
@@ -342,10 +393,10 @@ mod tests {
std::fs::write(&one, vec![3u8; 8192]).unwrap();
std::fs::write(&two, vec![3u8; 8192]).unwrap();
let (identities, _) = hash_batch(vec![
(Uuid::now_v7(), one, 8192),
(Uuid::now_v7(), two, 8192),
])
let (identities, _) = hash_batch(
vec![(Uuid::now_v7(), one, 8192), (Uuid::now_v7(), two, 8192)],
FileTypeRegistry::builtin(),
)
.await;
assert_eq!(identities.len(), 2);
@@ -354,4 +405,46 @@ mod tests {
"two copies of the same bytes are one content row"
);
}
#[tokio::test]
async fn a_built_in_kind_is_stored_and_an_extension_kind_reads_the_header() {
use crate::filetype::{ExtensionKind, MagicPatternSpec};
let dir = tempfile::tempdir().unwrap();
let photo = dir.path().join("a.jpg");
std::fs::write(&photo, vec![0xFF; 4096]).unwrap();
let nothing = dir.path().join("b.zzzkind");
std::fs::write(&nothing, b"ZZKIND rest of file".repeat(300)).unwrap();
let other = dir.path().join("c.zzzkind");
std::fs::write(&other, vec![0u8; 4096]).unwrap();
let builtin = FileTypeRegistry::builtin();
assert_eq!(identify_kind(&photo, builtin).await, (Some(1), None));
assert_eq!(identify_kind(&nothing, builtin).await, (None, None));
let layered = FileTypeRegistry::with_extension_kinds(&[(
"ext".into(),
vec![ExtensionKind {
name: "zz".into(),
display_name: None,
parent: ContentKind::Text,
extensions: vec!["zzzkind".into()],
mime_types: Vec::new(),
magic: vec![MagicPatternSpec {
pattern: "5A 5A 4B 49 4E 44".into(),
offset: 0,
}],
preview: None,
}],
)]);
assert_eq!(
identify_kind(&nothing, &layered).await,
(Some(ContentKind::Text as i64), Some("ext:zz".into()))
);
assert_eq!(
identify_kind(&other, &layered).await,
(Some(ContentKind::Text as i64), Some("ext:zz".into())),
"no magic match keeps the extension result"
);
assert_eq!(identify_kind(&photo, &layered).await, (Some(1), None));
}
}
+56
View File
@@ -0,0 +1,56 @@
//! Naming stored content rows after the kinds loaded extensions declare.
//!
//! The content identity phase writes a row's kind when it reads the bytes.
//! Rows identified before an extension existed carry a built-in kind, or
//! nothing at all, and the bytes are not read again for them: one update
//! per declared kind names every row whose file carries one of the kind's
//! extensions. The pass runs when a store opens, so every store sees the
//! kinds loaded at startup, and when an extension with kinds loads while
//! stores are open. It is idempotent, so running it twice costs one indexed
//! probe per kind and changes nothing.
use std::sync::Arc;
use crate::filetype::FileTypeRegistry;
use crate::ops::indexing::VolumeIndex;
/// Name the rows of one store after every extension kind the registry
/// holds. Returns how many rows gained a name.
pub async fn name_kinds_in_store(
db: &sd_store::SourceDb,
registry: &FileTypeRegistry,
) -> Result<u64, sd_store::Error> {
let mut named = 0;
for (kind, extensions) in registry.extension_kinds() {
let Some(kind_name) = kind.kind_name.as_deref() else {
continue;
};
named += sd_store::name_content_kind_by_extension(
db.pool(),
kind_name,
kind.category as i32 as i64,
&extensions,
)
.await?;
}
Ok(named)
}
/// Run the pass over every store this machine has open.
pub async fn name_kinds_in_open_stores(volume_index: &Arc<VolumeIndex>) {
let registry = FileTypeRegistry::current();
if registry.extension_kinds().is_empty() {
return;
}
for store in volume_index.open_stores() {
match name_kinds_in_store(store.db(), &registry).await {
Ok(0) => {}
Ok(named) => {
tracing::info!(source = %store.id(), named, "named content rows after extension kinds")
}
Err(error) => {
tracing::warn!(source = %store.id(), %error, "could not name content rows after extension kinds")
}
}
}
}
+1
View File
@@ -17,6 +17,7 @@ pub mod descriptor;
pub mod handlers;
pub mod input;
pub mod job;
pub mod kinds;
pub mod lens;
pub mod metadata;
pub mod metrics;
+20
View File
@@ -138,6 +138,20 @@ impl SourceStore {
.with_context(|| format!("load ledger for {id}"))?;
tracing::debug!(source = %id, records = ledger.len(), "source store attached");
// Rows identified before the loaded extensions existed take their
// kinds now, so a store opened after startup is never behind them.
match super::kinds::name_kinds_in_store(&db, &crate::filetype::FileTypeRegistry::current())
.await
{
Ok(0) => {}
Ok(named) => {
tracing::info!(source = %id, named, "named content rows after extension kinds")
}
Err(error) => {
tracing::warn!(source = %id, %error, "could not name content rows after extension kinds")
}
}
let (tx, rx) = mpsc::channel(QUEUE_DEPTH);
tokio::spawn(write_loop(db.clone(), ledger, rx));
@@ -1364,6 +1378,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: None,
size: Some(5),
kind: None,
kind_name: None,
},
)])
.await;
@@ -1528,6 +1543,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: None,
size: Some(*size as i64),
kind: None,
kind_name: None,
},
)
})
@@ -1580,6 +1596,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: None,
size: Some(*size as i64),
kind: None,
kind_name: None,
},
)
})
@@ -1624,6 +1641,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: Some("integrity-hash".to_string()),
size: Some(*size as i64),
kind: None,
kind_name: None,
},
)
})
@@ -1726,6 +1744,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: None,
size: Some(size),
kind: None,
kind_name: None,
};
let identified = pending
.iter()
@@ -1776,6 +1795,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
integrity_hash: None,
size: Some(4),
kind: None,
kind_name: None,
},
)
};
+1
View File
@@ -196,6 +196,7 @@ async fn verify_batch(
integrity_hash: Some(hash),
size: Some(size as i64),
kind: None,
kind_name: None,
},
size,
)),
+5
View File
@@ -1352,6 +1352,11 @@ impl VolumeIndex {
Some(store)
}
/// Every writable store this machine has opened so far.
pub fn open_stores(&self) -> Vec<Arc<SourceStore>> {
self.stores.read().values().cloned().collect()
}
/// A read-only handle to a source's store, cached per source.
///
/// `None` means the source has no store on disk or it refused to open,
+3 -2
View File
@@ -128,7 +128,7 @@ impl LibraryQuery for MediaSearchQuery {
};
let registry = context.file_type_registry();
let extensions = media_extensions(registry, &input.filters);
let extensions = media_extensions(&registry, &input.filters);
if extensions.is_empty() {
return Ok(finished);
}
@@ -159,7 +159,7 @@ impl LibraryQuery for MediaSearchQuery {
needle: input.query.to_lowercase(),
filters: &input.filters,
tags: tags.as_ref(),
registry,
registry: &registry,
extensions: &extensions,
device_slug: crate::device::get_current_device_slug(),
};
@@ -534,6 +534,7 @@ mod tests {
sampled_hash: None,
integrity_hash: None,
content_kind: None,
content_kind_name: None,
content_error: None,
}
}
+2 -2
View File
@@ -81,7 +81,7 @@ impl FileSearchQuery {
&self.input.pagination,
&context,
cache,
registry,
&registry,
)
.await?
}
@@ -93,7 +93,7 @@ impl FileSearchQuery {
&self.input.pagination,
&context,
cache,
registry,
&registry,
)
.await?
}
+10 -2
View File
@@ -6,7 +6,7 @@
//! other backend. Matching, filters and scoring mirror the arena path so the
//! two backends return the same hits for the same capture.
use crate::domain::{File, SdPath};
use crate::domain::{ContentKind, File, SdPath};
use crate::filetype::FileTypeRegistry;
use crate::infra::query::QueryError;
use crate::ops::search::input::{DateField, SearchFilters};
@@ -146,7 +146,15 @@ pub(super) fn passes_store_filters(
}
if let Some(ref content_types) = filters.content_types {
let identified_kind = file_type_registry.identify_by_extension(absolute);
// The stored kind is what the listing shows, so the filter agrees
// with it; a row identified before kinds were stored derives one
// from the name as the listing does.
let identified_kind = entry
.content_kind
.and_then(|kind| i32::try_from(kind).ok())
.map(ContentKind::from_id)
.filter(|kind| *kind != ContentKind::Unknown)
.unwrap_or_else(|| file_type_registry.identify_by_extension(absolute));
if !content_types.contains(&identified_kind) {
return false;
}
@@ -230,6 +230,7 @@ fn record_to_file(
modified_at: captured_at,
accessed_at: None,
content_kind,
content_kind_name: None,
is_local: true,
duration_seconds: None,
thumbnail_path: str_field("thumb_path").map(|p| absolutize(PathBuf::from(p))),
+2
View File
@@ -94,6 +94,7 @@ async fn dedupe_keeps_a_file_whose_own_bytes_were_never_read_in_full(
integrity_hash: Some(integrity_of_a.clone()),
size: Some(4096),
kind: None,
kind_name: None,
},
)
.await?;
@@ -106,6 +107,7 @@ async fn dedupe_keeps_a_file_whose_own_bytes_were_never_read_in_full(
integrity_hash: None,
size: Some(4096),
kind: None,
kind_name: None,
},
)
.await?;
@@ -0,0 +1,17 @@
{
"id": "zz-second-kind",
"name": "Second kind",
"version": "0.1.0",
"description": "Test fixture: a second extension claiming the .fake extension, loaded after the test extension so its claim is contested. Runs the test extension's module.",
"wasm_file": "test_extension.wasm",
"kinds": [
{
"name": "other",
"display_name": "Other fake file",
"parent": "text",
"extensions": ["fake", "fake2"],
"magic": [{ "pattern": "4F 54 48 52", "offset": 0 }],
"preview": { "renderer": "text" }
}
]
}
+280 -1
View File
@@ -15,12 +15,18 @@ use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
use sd_core::{
domain::{ContentKind, File, SdPath},
filetype::{FileTypeRegistry, KindConflict, PreviewSpec},
infra::{
action::LibraryAction,
api::SessionContext,
job::{database::checkpoint, types::JobStatus},
query::CoreQuery,
},
ops::{
extensions::{RunExtensionJobAction, RunExtensionJobInput},
extensions::{
ListExtensionsInput, ListExtensionsQuery, RunExtensionJobAction, RunExtensionJobInput,
},
sources::track::{TrackSourceAction, TrackSourceInput},
},
Core,
@@ -55,6 +61,87 @@ fn install_test_extension(data_dir: &Path) {
install_extension(data_dir, "test-extension", "test_extension.wasm");
}
/// A second extension claiming the test extension's `.fake` kind, under a
/// directory name that sorts after it. It runs the test extension's module;
/// only its manifest differs.
fn install_second_kind_extension(data_dir: &Path) {
let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/extensions/zz-second-kind/manifest.json");
let wasm = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.parent()
.unwrap()
.join("extensions/test-extension/test_extension.wasm");
let target = data_dir.join("extensions/zz-second-kind");
std::fs::create_dir_all(&target).unwrap();
std::fs::copy(manifest, target.join("manifest.json")).unwrap();
std::fs::copy(wasm, target.join("test_extension.wasm")).unwrap();
}
/// Track `dir` in `library` and wait until every file in it has a content
/// identity. Returns the store and its file entries by name.
async fn track_and_identify(
core: &Core,
library: &Arc<sd_core::library::Library>,
dir: PathBuf,
expected: u64,
) -> (
Arc<sd_core::ops::indexing::SourceStore>,
Vec<sd_store::FsEntry>,
) {
let tracked = TrackSourceAction::from_input(TrackSourceInput {
path: dir,
name: None,
overrides: Default::default(),
})
.unwrap()
.execute(library.clone(), core.context.clone())
.await
.unwrap();
let store = core
.context
.volume_index()
.store_for(&tracked.root)
.await
.expect("the tracked source has a store");
let deadline = tokio::time::Instant::now() + Duration::from_secs(60);
loop {
let contents = store.counts().await.map_or(0, |counts| counts.contents);
if contents == expected {
break;
}
assert!(
tokio::time::Instant::now() < deadline,
"{contents} of {expected} files were hashed"
);
tokio::time::sleep(Duration::from_millis(50)).await;
}
(store.clone(), store_files(&store).await)
}
async fn store_files(store: &sd_core::ops::indexing::SourceStore) -> Vec<sd_store::FsEntry> {
store.flush().await.unwrap();
let mut files = sd_store::read::files_beneath(
store.db().pool(),
"",
sd_store::read::Start::First,
None,
false,
100,
)
.await
.unwrap();
files.sort_by(|a, b| a.name.cmp(&b.name));
files
}
fn kind_of(files: &[sd_store::FsEntry], name: &str) -> (Option<i64>, Option<String>) {
let entry = files
.iter()
.find(|f| f.name == name)
.unwrap_or_else(|| panic!("{name} is in the store"));
(entry.content_kind, entry.content_kind_name.clone())
}
/// Start an extension job and wait for it to end, whichever way.
async fn run_to_end(
core: &Core,
@@ -671,3 +758,195 @@ fn walk(dir: &Path) -> Vec<PathBuf> {
}
out
}
/// The test extension declares a `fake` kind over `.fake` with magic
/// bytes, and a second fixture loaded after it contests the extension. The
/// content identity phase stores the kind and its name for every file,
/// built-in kinds included; the row keeps its name after the extension
/// unloads while the registry forgets it; rows identified before an
/// extension existed are named when it loads and when a store opens after
/// a restart.
#[tokio::test(flavor = "multi_thread")]
async fn extension_kinds_are_stored_and_survive_unload() {
guest_log();
let temp_dir = TempDir::new().unwrap();
let data_dir = temp_dir.path().join("core");
install_test_extension(&data_dir);
install_second_kind_extension(&data_dir);
let core = Core::new(data_dir.clone()).await.unwrap();
let pm = core
.plugin_manager
.as_ref()
.expect("plugin manager")
.clone();
// Both loaded, in directory order; the list reports the kinds and the
// one contested claim.
let session =
SessionContext::device_session(Uuid::now_v7(), sd_core::device::get_current_device_slug());
let list = ListExtensionsQuery::from_input(ListExtensionsInput {})
.unwrap()
.execute(core.context.clone(), session.clone())
.await
.unwrap();
let ids: Vec<&str> = list.extensions.iter().map(|e| e.id.as_str()).collect();
assert_eq!(ids, ["test-extension", "zz-second-kind"]);
let fake = &list.extensions[0].kinds[0];
assert_eq!(fake.id, "test-extension:fake");
assert_eq!(fake.display_name, "Fake file");
assert_eq!(fake.parent, ContentKind::Text);
assert_eq!(fake.preview, Some(PreviewSpec::Renderer("text".into())));
assert_eq!(
list.conflicts,
vec![KindConflict {
extension: "fake".into(),
kind: "zz-second-kind:other".into(),
claimed_by: "test-extension:fake".into(),
}]
);
assert_eq!(
FileTypeRegistry::current()
.type_by_extension(Path::new("x.fake2"))
.unwrap()
.id,
"zz-second-kind:other",
"the loser keeps its uncontested extension"
);
// Index a folder: the identity phase writes kind and kind_name.
let library = core
.libraries
.create_library("Kinds", None, core.context.clone())
.await
.unwrap();
let first = temp_dir.path().join("first");
std::fs::create_dir_all(&first).unwrap();
std::fs::write(first.join("a.fake"), b"FAKE".repeat(300)).unwrap();
std::fs::write(first.join("b.fake"), b"OTHR".repeat(300)).unwrap();
std::fs::write(first.join("c.fake"), vec![0u8; 1200]).unwrap();
let mut jpeg = vec![0xFF, 0xD8, 0xFF, 0xE0];
jpeg.extend(std::iter::repeat_n(7u8, 1200));
std::fs::write(first.join("d.jpg"), jpeg).unwrap();
std::fs::write(first.join("e.txt"), b"plain text".repeat(100)).unwrap();
let (first_store, files) = track_and_identify(&core, &library, first.clone(), 5).await;
let text = ContentKind::Text as i64;
assert_eq!(
kind_of(&files, "a.fake"),
(Some(text), Some("test-extension:fake".into()))
);
assert_eq!(
kind_of(&files, "b.fake"),
(Some(text), Some("zz-second-kind:other".into())),
"a lone magic match on the contested kind wins the file"
);
assert_eq!(
kind_of(&files, "c.fake"),
(Some(text), Some("test-extension:fake".into())),
"no magic match keeps the extension holder"
);
assert_eq!(
kind_of(&files, "d.jpg"),
(Some(ContentKind::Image as i64), None),
"a built-in kind is stored too"
);
assert_eq!(kind_of(&files, "e.txt"), (Some(text), None));
let a = files.iter().find(|f| f.name == "a.fake").unwrap();
let file = File::from_store_entry(a, SdPath::local(first.join("a.fake")));
assert_eq!(file.content_kind, ContentKind::Text);
assert_eq!(
file.content_kind_name.as_deref(),
Some("test-extension:fake")
);
// Unload the holder: the registry moves on, the rows do not.
pm.write()
.await
.unload_plugin("test-extension")
.await
.unwrap();
assert_eq!(
FileTypeRegistry::current()
.type_by_extension(Path::new("x.fake"))
.unwrap()
.id,
"zz-second-kind:other",
"the contested claim holds the extension once the winner is gone"
);
let files = store_files(&first_store).await;
assert_eq!(
kind_of(&files, "a.fake"),
(Some(text), Some("test-extension:fake".into()))
);
let a = files.iter().find(|f| f.name == "a.fake").unwrap();
let file = File::from_store_entry(a, SdPath::local(first.join("a.fake")));
assert_eq!(file.content_kind, ContentKind::Text, "the stored parent");
assert_eq!(
file.content_kind_name.as_deref(),
Some("test-extension:fake")
);
pm.write()
.await
.unload_plugin("zz-second-kind")
.await
.unwrap();
assert!(
FileTypeRegistry::current()
.type_by_extension(Path::new("x.fake"))
.is_none(),
"no extension loaded, the built-in registry is back"
);
assert!(FileTypeRegistry::current().conflicts().is_empty());
// Rows identified with no extension loaded have no kind; the load that
// follows names them by extension without reading the bytes.
let second = temp_dir.path().join("second");
std::fs::create_dir_all(&second).unwrap();
std::fs::write(second.join("f.FAKE"), vec![1u8; 1200]).unwrap();
std::fs::write(second.join("g.txt"), b"plain".repeat(300)).unwrap();
let (second_store, files) = track_and_identify(&core, &library, second.clone(), 2).await;
assert_eq!(kind_of(&files, "f.FAKE"), (None, None));
assert_eq!(kind_of(&files, "g.txt"), (Some(text), None));
pm.write()
.await
.load_plugin("test-extension")
.await
.unwrap();
let files = store_files(&second_store).await;
assert_eq!(
kind_of(&files, "f.FAKE"),
(Some(text), Some("test-extension:fake".into())),
"a load with kinds names the rows open stores hold, whatever the case of the extension"
);
assert_eq!(kind_of(&files, "g.txt"), (Some(text), None));
// A store opened after a restart is named by the kinds loaded at startup.
pm.write()
.await
.unload_plugin("test-extension")
.await
.unwrap();
let third = temp_dir.path().join("third");
std::fs::create_dir_all(&third).unwrap();
std::fs::write(third.join("h.fake"), vec![2u8; 1200]).unwrap();
let (_, files) = track_and_identify(&core, &library, third.clone(), 1).await;
assert_eq!(kind_of(&files, "h.fake"), (None, None));
core.shutdown().await.unwrap();
let core = Core::new(data_dir).await.unwrap();
let store = core
.context
.volume_index()
.store_for(&third)
.await
.expect("the third source reopens");
let files = store_files(&store).await;
assert_eq!(
kind_of(&files, "h.fake"),
(Some(text), Some("test-extension:fake".into()))
);
core.shutdown().await.unwrap();
}
+56 -14
View File
@@ -1235,6 +1235,18 @@ impl NeighborRow {
}
}
/// The row a record pointed at before a content write, for what the write
/// has to carry across.
#[derive(sqlx::FromRow)]
struct PreviousContent {
id: i64,
uuid: Uuid,
sampled_hash: Option<String>,
integrity_hash: Option<String>,
kind: Option<i64>,
kind_name: Option<String>,
}
/// The content write itself, against whatever connection the caller holds: a
/// pooled one for a single file, a transaction for a batch.
///
@@ -1264,8 +1276,8 @@ async fn bind_content(
.as_deref()
.map(crate::content::uuid_for);
let previous: Option<(i64, Uuid, Option<String>, Option<String>)> = sqlx::query_as(
"SELECT c.id, c.uuid, c.sampled_hash, c.integrity_hash
let previous: Option<PreviousContent> = sqlx::query_as(
"SELECT c.id, c.uuid, c.sampled_hash, c.integrity_hash, c.kind, c.kind_name
FROM record r JOIN content c ON c.id = r.content_id WHERE r.uuid = ?",
)
.bind(uuid)
@@ -1275,21 +1287,42 @@ async fn bind_content(
// A sampled-only write says nothing against a full read of the same
// bytes, so a record already confirmed under this sampled hash stays
// where it is rather than walking back down to a guess.
if let Some((old_id, _, old_sampled, Some(_))) = &previous {
if identity.integrity_hash.is_none() && *old_sampled == identity.sampled_hash {
return Ok(*old_id);
if let Some(old) = &previous {
if old.integrity_hash.is_some()
&& identity.integrity_hash.is_none()
&& old.sampled_hash == identity.sampled_hash
{
return Ok(old.id);
}
}
// The kind travels with the record when the write describes the same
// bytes: a verification pass carries no kind of its own, and the
// confirmed row it moves the record to must not lose the one the
// identity phase read. A write with a new hash is a changed file, whose
// kind the old bytes say nothing about.
let same_bytes = previous.as_ref().filter(|old| {
(old.sampled_hash.is_some() && old.sampled_hash == identity.sampled_hash)
|| (old.integrity_hash.is_some() && old.integrity_hash == identity.integrity_hash)
});
let kind = identity
.kind
.or_else(|| same_bytes.and_then(|old| old.kind));
let kind_name = identity
.kind_name
.clone()
.or_else(|| same_bytes.and_then(|old| old.kind_name.clone()));
let content_id: i64 = match identity.integrity_hash.as_deref() {
Some(integrity) => sqlx::query_scalar(
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind)
VALUES (?, ?, ?, ?, ?, ?)
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind, kind_name)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (integrity_hash) DO UPDATE SET
candidate_uuid = COALESCE(content.candidate_uuid, excluded.candidate_uuid),
sampled_hash = COALESCE(content.sampled_hash, excluded.sampled_hash),
size = COALESCE(excluded.size, content.size),
kind = COALESCE(excluded.kind, content.kind)
kind = COALESCE(excluded.kind, content.kind),
kind_name = COALESCE(excluded.kind_name, content.kind_name)
RETURNING id",
)
.bind(content_uuid)
@@ -1297,22 +1330,25 @@ async fn bind_content(
.bind(&identity.sampled_hash)
.bind(integrity)
.bind(identity.size)
.bind(identity.kind)
.bind(kind)
.bind(&kind_name)
.fetch_one(&mut *conn)
.await?,
None => sqlx::query_scalar(
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind)
VALUES (?, ?, ?, NULL, ?, ?)
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind, kind_name)
VALUES (?, ?, ?, NULL, ?, ?, ?)
ON CONFLICT (sampled_hash) WHERE integrity_hash IS NULL DO UPDATE SET
size = COALESCE(excluded.size, content.size),
kind = COALESCE(excluded.kind, content.kind)
kind = COALESCE(excluded.kind, content.kind),
kind_name = COALESCE(excluded.kind_name, content.kind_name)
RETURNING id",
)
.bind(content_uuid)
.bind(candidate_uuid)
.bind(&identity.sampled_hash)
.bind(identity.size)
.bind(identity.kind)
.bind(kind)
.bind(&kind_name)
.fetch_one(&mut *conn)
.await?,
};
@@ -1323,7 +1359,13 @@ async fn bind_content(
.execute(&mut *conn)
.await?;
let Some((old_id, old_uuid, _, old_integrity)) = previous else {
let Some(PreviousContent {
id: old_id,
uuid: old_uuid,
integrity_hash: old_integrity,
..
}) = previous
else {
return Ok(content_id);
};
if old_id == content_id {
+31
View File
@@ -922,6 +922,37 @@ pub async fn mark_content_unreadable(
Ok(())
}
/// Name the kind of every content row whose files carry one of these
/// extensions and that no extension kind has named yet.
///
/// This is the reidentification an extension's load runs over rows that
/// were identified before it existed: one statement per declared kind, by
/// extension, since the bytes are not read again. A row another extension
/// already named keeps that name; magic-only claims wait for the next
/// identity pass over those files. Returns the rows named.
pub async fn name_content_kind_by_extension(
pool: &sqlx::SqlitePool,
kind_name: &str,
kind: i64,
extensions: &[String],
) -> Result<u64> {
if extensions.is_empty() {
return Ok(0);
}
let sql = format!(
"UPDATE content SET kind_name = ?, kind = ? WHERE kind_name IS NULL AND id IN (
SELECT r.content_id FROM record r
JOIN facet_file f ON f.record_uuid = r.uuid
WHERE r.content_id IS NOT NULL AND lower(f.extension) IN ({}))",
vec!["?"; extensions.len()].join(", ")
);
let mut query = sqlx::query(&sql).bind(kind_name).bind(kind);
for ext in extensions {
query = query.bind(ext.to_lowercase());
}
Ok(query.execute(pool).await?.rows_affected())
}
/// One copy of some bytes: a record, and where it is.
#[derive(Debug, Clone)]
pub struct ContentCopy {
+2 -2
View File
@@ -62,8 +62,8 @@ pub use error::{Error, Result};
pub use file::{
content_of, copies_of_content, count_files_needing_content, count_files_needing_verification,
duplicate_copies, files_needing_content, files_needing_verification, filesystem_schema,
mark_content_unreadable, ContentCopy, FileKind, FileWrite, Ledger, Observation, PendingContent,
PendingVerification, Resolution, SubtreeRename, Watermark,
mark_content_unreadable, name_content_kind_by_extension, ContentCopy, FileKind, FileWrite,
Ledger, Observation, PendingContent, PendingVerification, Resolution, SubtreeRename, Watermark,
};
pub use migrate::SCHEMA_VERSION;
pub use read::{FsEntry, TitleMatches};
+47 -1
View File
@@ -28,7 +28,11 @@ use crate::error::{Error, Result};
/// 1. `content` rebuilt so a row's integrity hash belongs to every record on
/// it: candidate rows keyed by sampled hash, confirmed rows by integrity
/// hash, `candidate_uuid` beside `uuid`.
pub const SCHEMA_VERSION: i64 = 1;
/// 2. `content.kind_name` for the extension kind that identified the bytes,
/// nullable and never backfilled, and an index over the lowercase file
/// extension so a reidentification pass by extension does not walk the
/// facet table.
pub const SCHEMA_VERSION: i64 = 2;
/// The version a store carries. `0` for a store written before versions
/// existed.
@@ -76,6 +80,7 @@ pub(crate) async fn run(pool: &SqlitePool) -> Result<()> {
async fn apply(pool: &SqlitePool, next: i64) -> Result<()> {
match next {
1 => content_rows_own_their_hash(pool).await,
2 => content_rows_name_their_kind(pool).await,
_ => Err(Error::Other(format!(
"no migration defined for schema version {next}"
))),
@@ -103,6 +108,47 @@ const CONTENT_V1_INDEXES: [&str; 4] = [
"CREATE INDEX idx_content_candidate_uuid ON content(candidate_uuid)",
];
/// The index a reidentification pass by extension needs. Lowercase because
/// the walk writes the extension as the name carries it and every lookup
/// compares lowercase. Created here for a store that predates it and by the
/// open path for a fresh one; `IF NOT EXISTS` makes both idempotent.
pub const FACET_FILE_EXTENSION_INDEX: &str =
"CREATE INDEX IF NOT EXISTS idx_facet_file_extension ON facet_file(lower(extension))";
/// Migration 2: `content.kind_name`, nullable, with no backfill. Before this
/// no row carried a kind at all; the kind was derived from the file name on
/// every read. Rows identified from here on get `kind` and `kind_name` at
/// the content identity phase, and an extension that loads with kinds runs
/// one reidentification update by extension over the rows it can name.
async fn content_rows_name_their_kind(pool: &SqlitePool) -> Result<()> {
let mut conn = pool.acquire().await?;
let mut tx = conn.begin().await?;
let present: Option<i64> =
sqlx::query_scalar("SELECT 1 FROM pragma_table_info('content') WHERE name = 'kind_name'")
.fetch_optional(&mut *tx)
.await?;
if present.is_none() {
sqlx::query("ALTER TABLE content ADD COLUMN kind_name TEXT")
.execute(&mut *tx)
.await?;
}
let has_facet_file: Option<i64> = sqlx::query_scalar(
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'facet_file'",
)
.fetch_optional(&mut *tx)
.await?;
if has_facet_file.is_some() {
sqlx::query(FACET_FILE_EXTENSION_INDEX)
.execute(&mut *tx)
.await?;
}
sqlx::query("PRAGMA user_version = 2")
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(())
}
/// Migration 1: rebuild `content` so that no record carries an integrity
/// hash its own bytes were never read for.
///
+7 -1
View File
@@ -53,6 +53,9 @@ pub struct FsEntry {
pub sampled_hash: Option<String>,
pub integrity_hash: Option<String>,
pub content_kind: Option<i64>,
/// The extension kind id the identity phase stored, loaded extension or
/// not; `content_kind` is then its parent.
pub content_kind_name: Option<String>,
pub content_error: Option<String>,
}
@@ -80,7 +83,8 @@ macro_rules! entry_columns {
COALESCE(f.is_hidden, 0) AS is_hidden, f.extension AS extension, \
f.link_target AS link_target, f.inode AS inode, f.mode AS mode, f.uid AS uid, f.gid AS gid, \
f.content_error AS content_error, c.uuid AS content_uuid, \
c.sampled_hash AS sampled_hash, c.integrity_hash AS integrity_hash, c.kind AS content_kind"
c.sampled_hash AS sampled_hash, c.integrity_hash AS integrity_hash, c.kind AS content_kind, \
c.kind_name AS content_kind_name"
};
}
@@ -130,6 +134,7 @@ struct EntryRow {
sampled_hash: Option<String>,
integrity_hash: Option<String>,
content_kind: Option<i64>,
content_kind_name: Option<String>,
}
fn entry_from_row(row: EntryRow) -> Option<FsEntry> {
@@ -155,6 +160,7 @@ fn entry_from_row(row: EntryRow) -> Option<FsEntry> {
sampled_hash: row.sampled_hash,
integrity_hash: row.integrity_hash,
content_kind: row.content_kind,
content_kind_name: row.content_kind_name,
content_error: row.content_error,
})
}
+8 -1
View File
@@ -95,7 +95,8 @@ CREATE TABLE IF NOT EXISTS content (
sampled_hash TEXT,
integrity_hash TEXT UNIQUE,
size INTEGER,
kind INTEGER
kind INTEGER,
kind_name TEXT
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_content_candidate ON content(sampled_hash)
WHERE integrity_hash IS NULL;
@@ -238,7 +239,13 @@ pub struct ContentIdentity {
/// Full tier — hash over every byte.
pub integrity_hash: Option<String>,
pub size: Option<i64>,
/// The built-in kind, or an extension kind's parent, as a `ContentKind`
/// discriminant.
pub kind: Option<i64>,
/// The extension kind id (`<extension id>:<name>`) when an extension's
/// kind identified the bytes. Kept after the extension unloads, so a
/// reader can fall back to `kind` until it returns.
pub kind_name: Option<String>,
}
/// Facet table name for a model. Namespaced so a model can never collide with a
+28
View File
@@ -37,6 +37,13 @@ async fn open_pool(db_path: &Path, create: bool) -> Result<SqlitePool> {
/// A pool that can only read. No journal-mode pragma runs: the store's own
/// files are already WAL, and a delivered artifact may legitimately carry a
/// rollback journal that a read-only connection could not convert anyway.
///
/// A replica keeps its owner's schema version, and an owner on an older
/// build has no `content.kind_name`. The entry readers name that column, so
/// each connection to a store below version 2 gets a temporary view named
/// `content` that adds it as `NULL`; an unqualified name resolves to the
/// temp schema first, so every reader sees the current shape and the file
/// is untouched. The owner's upgrade replaces the replica wholesale.
async fn open_pool_read_only(db_path: &Path) -> Result<SqlitePool> {
let options = SqliteConnectOptions::from_str(&format!("sqlite:{}", db_path.display()))
.map_err(|e| Error::Other(format!("invalid database path: {e}")))?
@@ -46,6 +53,22 @@ async fn open_pool_read_only(db_path: &Path) -> Result<SqlitePool> {
Ok(SqlitePoolOptions::new()
.max_connections(4)
.after_connect(|conn, _meta| {
Box::pin(async move {
let version: i64 = sqlx::query_scalar("PRAGMA user_version")
.fetch_one(&mut *conn)
.await?;
if version < 2 {
sqlx::query(
"CREATE TEMP VIEW IF NOT EXISTS content AS \
SELECT *, NULL AS kind_name FROM main.content",
)
.execute(&mut *conn)
.await?;
}
Ok(())
})
})
.connect_with(options)
.await?)
}
@@ -103,6 +126,11 @@ impl SourceManager {
for sql in &generate_ddl(schema) {
sqlx::query(sql).execute(pool).await?;
}
if schema.models.contains_key("file") {
sqlx::query(crate::migrate::FACET_FILE_EXTENSION_INDEX)
.execute(pool)
.await?;
}
Ok(())
}
+161
View File
@@ -471,3 +471,164 @@ async fn migration_1_timing_on_a_million_records() {
assert_eq!(count, rows);
println!("migration 1 over {rows} content rows and records: {elapsed:?}");
}
/// Turn a fresh store into one written at schema version 1: the content
/// table without `kind_name`, no extension index, `user_version` 1.
async fn downgrade_to_v1(path: &Path) -> SqlitePool {
let pool = raw(path).await;
for sql in [
"DROP TABLE content",
migrate::CONTENT_V1,
"CREATE UNIQUE INDEX idx_content_candidate ON content(sampled_hash) WHERE integrity_hash IS NULL",
"CREATE INDEX idx_content_sampled ON content(sampled_hash)",
"CREATE INDEX idx_content_uuid ON content(uuid)",
"CREATE INDEX idx_content_candidate_uuid ON content(candidate_uuid)",
"DROP INDEX IF EXISTS idx_facet_file_extension",
"PRAGMA user_version = 1",
] {
sqlx::query(sql).execute(&pool).await.expect(sql);
}
pool
}
async fn insert_v1_content(pool: &SqlitePool, id: i64, sampled: &str, kind: Option<i64>) {
sqlx::query(
"INSERT INTO content (id, uuid, candidate_uuid, sampled_hash, size, kind) VALUES (?, ?, ?, ?, 10, ?)",
)
.bind(id)
.bind(uuid_for(sampled))
.bind(uuid_for(sampled))
.bind(sampled)
.bind(kind)
.execute(pool)
.await
.expect("content row");
}
async fn insert_file_facet(pool: &SqlitePool, record: Uuid, extension: &str) {
sqlx::query("INSERT INTO facet_file (record_uuid, size, extension) VALUES (?, 10, ?)")
.bind(record)
.bind(extension)
.execute(pool)
.await
.expect("facet row");
}
async fn kind_columns(pool: &SqlitePool) -> Vec<(i64, Option<i64>, Option<String>)> {
sqlx::query_as("SELECT id, kind, kind_name FROM content ORDER BY id")
.fetch_all(pool)
.await
.expect("kind columns")
}
/// Migration 2 on a version 1 store: the column arrives empty, the
/// extension index exists, every row and its kind survive, and the
/// reidentification statement names rows by extension without touching a
/// row already named.
#[tokio::test]
async fn migration_2_adds_kind_name_without_a_backfill() {
let fixture = Fixture::new().await;
let pool = downgrade_to_v1(&fixture.db_path()).await;
insert_v1_content(&pool, 1, "s1", Some(1)).await;
insert_v1_content(&pool, 2, "s2", None).await;
insert_v1_content(&pool, 3, "s3", Some(4)).await;
let cr2 = insert_record(&pool, "shot.CR2", Some(1)).await;
let nef = insert_record(&pool, "shot.nef", Some(2)).await;
let pdf = insert_record(&pool, "paper.pdf", Some(3)).await;
insert_file_facet(&pool, cr2, "CR2").await;
insert_file_facet(&pool, nef, "nef").await;
insert_file_facet(&pool, pdf, "pdf").await;
pool.close().await;
let db = fixture
.manager
.open("source-1")
.await
.expect("open migrates");
assert_eq!(db.schema_version(), 2);
assert_eq!(
migrate::version(db.pool()).await.expect("version"),
2,
"the file carries the new version"
);
assert_eq!(
kind_columns(db.pool()).await,
vec![(1, Some(1), None), (2, None, None), (3, Some(4), None)],
"rows keep their kind and gain an empty kind_name"
);
let index: Option<String> = sqlx::query_scalar(
"SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_facet_file_extension'",
)
.fetch_optional(db.pool())
.await
.expect("index lookup");
assert_eq!(index.as_deref(), Some("idx_facet_file_extension"));
let named = sd_store::name_content_kind_by_extension(
db.pool(),
"photos:raw",
1,
&["cr2".to_string(), "nef".to_string()],
)
.await
.expect("reidentify");
assert_eq!(named, 2, "the uppercase CR2 and the nef row are both named");
let again =
sd_store::name_content_kind_by_extension(db.pool(), "other:raw", 1, &["cr2".to_string()])
.await
.expect("reidentify again");
assert_eq!(again, 0, "a named row keeps its name");
assert_eq!(
kind_columns(db.pool()).await,
vec![
(1, Some(1), Some("photos:raw".into())),
(2, Some(1), Some("photos:raw".into())),
(3, Some(4), None),
]
);
let entry = db.entry_by_uuid(nef).await.expect("read").expect("row");
assert_eq!(entry.content_kind_name.as_deref(), Some("photos:raw"));
assert_eq!(entry.content_kind, Some(1));
// Reopening is a no-op.
drop(db);
let reopened = fixture.manager.open("source-1").await.expect("reopen");
assert_eq!(reopened.schema_version(), 2);
assert_eq!(kind_columns(reopened.pool()).await.len(), 3);
}
/// A replica still at version 1 is read through the current entry shape:
/// its rows report no kind name rather than failing on a column the owner
/// has not written yet.
#[tokio::test]
async fn a_version_1_replica_reads_with_an_empty_kind_name() {
let fixture = Fixture::new().await;
let pool = downgrade_to_v1(&fixture.db_path()).await;
insert_v1_content(&pool, 1, "s1", Some(1)).await;
let a = insert_record(&pool, "a.cr2", Some(1)).await;
insert_file_facet(&pool, a, "cr2").await;
let replica_path = fixture.dir.path().join("replica.db");
sqlx::query("VACUUM INTO ?")
.bind(replica_path.to_string_lossy().to_string())
.execute(&pool)
.await
.expect("vacuum into");
pool.close().await;
let replica = SourceManager::open_file_read_only(&replica_path)
.await
.expect("replica opens");
assert_eq!(replica.schema_version(), 1);
let entry = replica.entry_by_uuid(a).await.expect("read").expect("row");
assert_eq!(entry.content_kind, Some(1));
assert_eq!(entry.content_kind_name, None);
let (entries, _) = sd_store::read::all_entries_page(replica.pool(), 0, 10)
.await
.expect("page");
assert_eq!(entries.len(), 1);
assert_eq!(
migrate::version(replica.pool()).await.expect("version"),
1,
"nothing migrated the replica"
);
}
+73
View File
@@ -857,3 +857,76 @@ async fn duplicate_discovery_groups_by_sampled_hash_and_splits_by_integrity() {
"the pair is reported under the confirmed uuid"
);
}
/// The kind the identity phase read from the bytes travels with the record
/// to the confirmed row a verification pass moves it to, since that pass
/// carries no kind of its own.
#[tokio::test]
async fn a_full_read_keeps_the_kind_the_sampled_pass_stored() {
let fixture = Fixture::new().await;
let db = fixture.open().await;
db.begin_sync().await.expect("epoch");
let note = db
.upsert("note", "note-1", &json!({ "title": "A" }))
.await
.expect("note");
db.set_content_identity(
note,
&ContentIdentity {
sampled_hash: Some("sampled-1".to_string()),
kind: Some(1),
kind_name: Some("photos:raw".to_string()),
..Default::default()
},
)
.await
.expect("candidate");
let confirmed = db
.set_content_identity(note, &confirmed("sampled-1", "integrity-1"))
.await
.expect("confirmed");
let (kind, kind_name): (Option<i64>, Option<String>) =
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
.bind(confirmed)
.fetch_one(db.pool())
.await
.expect("confirmed row");
assert_eq!(kind, Some(1));
assert_eq!(kind_name.as_deref(), Some("photos:raw"));
// A later identity pass that names the kind itself still wins.
db.set_content_identity(
note,
&ContentIdentity {
sampled_hash: Some("sampled-1".to_string()),
integrity_hash: Some("integrity-1".to_string()),
kind: Some(7),
kind_name: Some("other:thing".to_string()),
..Default::default()
},
)
.await
.expect("renamed");
let (kind, kind_name): (Option<i64>, Option<String>) =
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
.bind(confirmed)
.fetch_one(db.pool())
.await
.expect("confirmed row");
assert_eq!((kind, kind_name.as_deref()), (Some(7), Some("other:thing")));
// Changed bytes with no kind of their own inherit nothing.
let changed = db
.set_content_identity(note, &sampled("sampled-2"))
.await
.expect("rehashed");
let (kind, kind_name): (Option<i64>, Option<String>) =
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
.bind(changed)
.fetch_one(db.pool())
.await
.expect("new row");
assert_eq!((kind, kind_name), (None, None));
}
+4
View File
@@ -237,6 +237,7 @@ async fn content_tag_reaches_every_copy() {
integrity_hash: None,
size: Some(42),
kind: None,
kind_name: None,
};
db.set_content_identity(original, &identity)
.await
@@ -302,6 +303,7 @@ async fn late_binding_fills_the_content_key() {
integrity_hash: None,
size: Some(42),
kind: None,
kind_name: None,
};
db.set_content_identity(original, &identity)
.await
@@ -346,6 +348,7 @@ async fn a_content_tag_applied_before_verification_reaches_every_copy_afterwards
integrity_hash: None,
size: Some(42),
kind: None,
kind_name: None,
};
db.set_content_identity(original, &guess)
.await
@@ -477,6 +480,7 @@ async fn identical_claims_collapse_across_stores() {
integrity_hash: None,
size: Some(7),
kind: None,
kind_name: None,
};
for (db, device, external) in [(&a, device_a, "copy-a"), (&b, device_b, "copy-b")] {
@@ -1,6 +1,6 @@
# Extension File Kinds and Previews
> Status: proposed 2026-10-07; nothing built
> Status: K1 and K2 built 2026-10-07 (SPAC-35); K3 to K5 open
> Captured: 2026-10-07 (SPAC-31), from the tree at `b938526`
> Owns: how an extension's manifest declares file kinds, how identification
> assigns them, how the client renders a preview for them, and what happens
@@ -268,15 +268,29 @@ without the `wasm` feature behaves as today.
| Phase | Work | Exit proof | Size |
| --- | --- | --- | --- |
| K1 Manifest and registry | `kinds` parsed and validated; layered `FileTypeRegistry::current()`; five `builtin()` call sites and the context field moved; `load_all` sorted; conflicts and contested claims recorded; `extensions.list` reports kinds and conflicts; ts-client regenerated | Unit tests: a manifest with a bad parent or a redefining claim is rejected with the reason; two fixtures claiming `.xyz` load with one conflict in the list output; `identify_by_extension` on `foo.cr2` returns the `raw` type with parent `image` while the fixture is loaded and `image` after `unload` | 2 days |
| K2 Persisted kinds | Store schema v2 `kind_name` and `idx_facet_file_extension`; content identity phase writes `kind` and `kind_name` and checks extension-kind magic; `File.content_kind_name`; reidentification pass on load | Acceptance test (wasm group): index a folder with `.cr2` files under a fixture extension, read the store, both columns set; unload the extension, re-read through `files.list`, `content_kind_name` still `raw`, `content_kind` is `image`; a `.cr2` whose bytes carry no TIFF header and whose extension is also claimed by a second fixture with matching magic gets the second kind | 2 days |
| K1 Manifest and registry (done 2026-10-07, SPAC-35) | `kinds` parsed and validated; layered `FileTypeRegistry::current()`; five `builtin()` call sites and the context field moved; `load_all` sorted; conflicts and contested claims recorded; `extensions.list` reports kinds and conflicts; ts-client regenerated | Unit tests: a manifest with a bad parent or a redefining claim is rejected with the reason; two fixtures claiming `.xyz` load with one conflict in the list output; `identify_by_extension` on `foo.cr2` returns the `raw` type with parent `image` while the fixture is loaded and `image` after `unload` | 2 days |
| K2 Persisted kinds (done 2026-10-07, SPAC-35) | Store schema v2 `kind_name` and `idx_facet_file_extension`; content identity phase writes `kind` and `kind_name` and checks extension-kind magic; `File.content_kind_name`; reidentification pass on load | Acceptance test (wasm group): index a folder with `.cr2` files under a fixture extension, read the store, both columns set; unload the extension, re-read through `files.list`, `content_kind_name` still `raw`, `content_kind` is `image`; a `.cr2` whose bytes carry no TIFF header and whose extension is also claimed by a second fixture with matching magic gets the second kind | 2 days |
| K3 Client registry | `renderers.ts`, `useExtensionKinds`, `ContentRenderer` becomes a lookup; `getContentKindName`; inspector and file kinds page show the name | `tsc` clean; Playwright or desktop recording of Quick Preview on a `raw` fixture rendering through the image renderer, then falling back to the same renderer with the extension removed from the data dir and the daemon restarted | 1.5 days |
| K4 Bundle previews | `ui_manifest.json` `file_viewers` parsed for `id` and `bundle`; `/extension/:id/*path` route; `BundleRenderer` with the `mount` contract; a tiny fixture bundle in `extensions/test-extension/ui/` | Acceptance: the fixture declares a kind with `preview.viewer`; Quick Preview mounts the bundle and the bundle's DOM shows the file name; deleting the bundle file makes the preview fall back to the parent renderer with one logged warning | 2 days |
| K5 Photos | Photos' manifest declares `raw` (and `heic` where the built-in table lacks a magic pattern) with `preview.renderer: image`; `ui_manifest.json` `file_viewers` gains `id` and `bundle`, with `raw` pointing at it once a viewer exists | Photos loads with no conflicts on a daemon with the built-in table; `sd-cli op extensions.list` shows its kinds | 0.5 day |
| K5 Photos (manifest half done with K1) | Photos' manifest declares `raw` with `preview.renderer: image` (`heic` already has a built-in magic pattern, so it stays built in); `ui_manifest.json` `file_viewers` gains `id` and `bundle`, with `raw` pointing at it once a viewer exists | Photos loads with no conflicts on a daemon with the built-in table; `sd-cli op extensions.list` shows its kinds | 0.5 day |
K1 and K2 are core; K3 and K4 are client plus one server route; K5 is
manifest edits. K3 can start after K1. Eight days, two PRs (K1+K2, K3 to K5).
As built (K1 and K2, SPAC-35), three details differ from the text above:
- `content.kind` is written for built-in kinds too, closing the first open
question: every row the identity phase touches from now on carries its
kind, and only rows identified before that derive one at read time.
- The reidentification pass runs when a store opens as well as when an
extension with kinds loads, because stores open lazily after the
extensions load at startup. `idx_facet_file_extension` is over
`lower(extension)` since the walk stores the extension as the name
carries it.
- A replica below schema version 2 is read through a per-connection
temporary view that adds `kind_name` as `NULL`, so an older owner's
database still answers the entry readers (`crates/store/src/source.rs`).
## Decisions for Jamie
1. Parent is always built in, and a kind is one level deep. The alternative,
+58
View File
@@ -68,5 +68,63 @@
},
"description": "ResNet50 trained on Places365 dataset"
}
],
"kinds": [
{
"name": "raw",
"display_name": "RAW photo",
"parent": "image",
"extensions": [
"cr2",
"cr3",
"nef",
"arw",
"dng",
"raf",
"orf",
"rw2",
"pef",
"dcr",
"srw"
],
"mime_types": [
"image/x-canon-cr2",
"image/x-canon-cr3",
"image/x-nikon-nef",
"image/x-sony-arw",
"image/x-adobe-dng",
"image/x-fuji-raf",
"image/x-olympus-orf",
"image/x-panasonic-rw2",
"image/x-pentax-pef",
"image/x-kodak-dcr",
"image/x-samsung-srw"
],
"magic": [
{
"pattern": "49 49 2A 00",
"offset": 0
},
{
"pattern": "4D 4D 00 2A",
"offset": 0
},
{
"pattern": "46 55 4A 49 46 49 4C 4D",
"offset": 0
},
{
"pattern": "49 49 52 4F",
"offset": 0
},
{
"pattern": "49 49 55 00",
"offset": 0
}
],
"preview": {
"renderer": "image"
}
}
]
}
+20 -1
View File
@@ -11,5 +11,24 @@
"write_sidecars": [
"digest"
]
}
},
"kinds": [
{
"name": "fake",
"display_name": "Fake file",
"parent": "text",
"extensions": [
"fake"
],
"magic": [
{
"pattern": "46 41 4B 45",
"offset": 0
}
],
"preview": {
"renderer": "text"
}
}
]
}
+56 -3
View File
@@ -1257,7 +1257,7 @@ strategy: string | null; estimated_files: number | null; estimated_bytes: number
export type ExtensionCase = "lower" | "keep";
export type ExtensionInfo = { id: string; name: string; version: string; jobs: ExtensionJobInfo[] };
export type ExtensionInfo = { id: string; name: string; version: string; jobs: ExtensionJobInfo[]; kinds: ExtensionKindInfo[] };
export type ExtensionJobInfo = {
/**
@@ -1269,6 +1269,22 @@ name: string;
*/
full_name: string; resumable: boolean };
/**
* A file kind an extension declares, as the client resolves previews and
* icons against it. The kinds of every loaded extension are the client's
* only source of extension kinds, which is what lets a stored kind name
* fall back to its parent once the extension is gone.
*/
export type ExtensionKindInfo = {
/**
* `<extension id>:<name>`, the value `File.content_kind_name` carries
*/
id: string; name: string; display_name: string; parent: ContentKind;
/**
* Every extension the manifest claims, contested ones included
*/
extensions: string[]; preview: PreviewSpec | null };
/**
* A dependency Spacedrive knows how to discover and use.
*/
@@ -1338,7 +1354,13 @@ created_at: string; modified_at: string; accessed_at: string | null;
/**
* Additional computed fields
*/
content_kind: ContentKind; is_local: boolean;
content_kind: ContentKind;
/**
* The extension kind id (`<extension id>:<name>`) the content identity
* phase stored, whether or not that extension is loaded now;
* `content_kind` is then its parent. `None` for a built-in kind.
*/
content_kind_name?: string | null; is_local: boolean;
/**
* Video duration (for grid display optimization)
*/
@@ -2387,6 +2409,26 @@ export type Keep =
*/
{ kind: "these"; paths: SdPath[] };
/**
* A file extension two loaded extensions both claimed. The kind loaded
* first holds the claim; the other kind's claim on that extension is
* dropped from the lookup table and its patterns stay available to the
* content identity phase as a tie-breaker.
*/
export type KindConflict = {
/**
* The file extension, lowercase, no dot.
*/
extension: string;
/**
* The kind id whose claim was dropped.
*/
kind: string;
/**
* The kind id that holds the extension.
*/
claimed_by: string };
/**
* Latency metrics snapshot
*/
@@ -2915,7 +2957,12 @@ export type ListExtensionsOutput = {
/**
* Whether this build can load extensions at all
*/
supported: boolean; extensions: ExtensionInfo[] };
supported: boolean; extensions: ExtensionInfo[];
/**
* File extensions two loaded extensions both claimed, resolved by load
* order: the kind loaded first holds each one.
*/
conflicts: KindConflict[] };
export type ListLibrariesInput = {
/**
@@ -3683,6 +3730,12 @@ export type PortRange = { from: number; to: number };
*/
export type PreferencesOutput = { theme: string; language: string };
/**
* How the client previews a kind: a built-in renderer by name, or a
* `ui_manifest.json` `file_viewers` entry by id.
*/
export type PreviewSpec = { renderer: string } | { viewer: string };
export type Process = (ProcessStatus) & { id: string };
export type ProcessLeasesInput = Record<string, never>;