mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2026-10-30 04:49:32 -04:00
Extension file kinds K1+K2: manifest kinds, layered registry, stored kind_name (#3135)
* Extension kinds K1: manifest kinds, layered FileTypeRegistry::current, conflicts in extensions.list A manifest may declare kinds (name, built-in parent, extensions, MIME types, magic bytes, preview). The registry becomes the built-in table plus every loaded extension's kinds in load order, swapped by the plugin manager on load and unload; the five builtin() callers and the context field read current(). A kind that would redefine a built-in extension is refused at load. load_all sorts directory names so a contested file extension resolves the same way on every machine; the dropped claim is reported by extensions.list and its magic patterns stay available. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Extension kinds K2: store schema v2 kind_name, identity phase writes kinds, reidentification on load and open content.kind_name (nullable, no backfill) and idx_facet_file_extension over lower(extension) arrive with schema version 2; a replica still at version 1 reads through a per-connection temp view that adds the column as NULL. The content identity phase stores the kind for every file and the kind name for extension kinds, reading the first 8 KB only when an extension kind's magic patterns are in play. Rows identified before an extension existed are named by extension when the extension loads and when a store opens. File gains content_kind_name and reads content_kind from the row when present. Photos declares raw; the test extension declares fake with a second fixture contesting it for the WASM suite. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Carry the stored kind across a verification write, filter store search on it, order unclaimed magic candidates by load bind_content takes the previous row's kind and kind_name onto the confirmed row when the write carries none, so a verification pass no longer drops what the identity phase stored. passes_store_filters answers a content_types filter from the stored kind before deriving from the name, matching the listing. magic_candidates for an extension nothing claims walks extension kinds in load order, so a header matching two kinds resolves the same way on every start. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Carry the stored kind only when the write describes the same bytes Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add kind_name to the dedupe test's content identities Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
ad6452dfbb
commit
0e5340394e
41 files changed
+1977
-100
No files matched your search
+1
-1
@@ -212,7 +212,7 @@ Recent landed.
|
||||
| Library backup and restore | in progress 2026-10-06 (Fable, SPAC-23); one PR against `main` | Fable | Release gate "Database backup and restore pass from release candidates" and candidate validation step 7. `libraries.backup` copies `library.db`, `sync.db`, every owned source store and its `sidecars.db` with `VACUUM INTO` on a fresh connection while the daemon runs, hashes each file with blake3 and writes `manifest.json` (device, build sha, applied migrations by name, each source's store id, revision and schema hash), into a directory or `.tar.zst`; `libraries.backup.verify` rehashes without restoring; `libraries.restore` verifies every hash first, refuses a backup whose library applied a migration this build does not know, stages under the data dir, renames into place (replace mode closes the library, releases open store and sidecar pools so the swapped `data.db` is what the next open reads, and parks the displaced state in `restore-trash/`), and reopens. Replacing a library other devices are members of needs `force`, because the device rows and sync watermarks in it would diverge from what those peers hold. CLI: `sd library backup --to`, `sd library backup verify --from`, `sd library restore --from [--as-new|--replace] [--force]`. `core/tests/library_backup_test.rs` backs up while a watcher writes, restores into a fresh data dir and proves counts, tag assertions, space items and store revisions equal, replaces over a mutated library and proves the mutation gone, and detects a one-byte tamper. Documented in `docs/core/library.mdx` (Backup and restore). Next: run the gate on the MacBook library before the beta candidate. |
|
||||
| Extension loading | in progress 2026-10-07 (Fable, SPAC-28); one PR against `main` | Fable | Product sequence step 5, first half. A daemon built with the `wasm` feature discovers every `<data dir>/extensions/<dir>/manifest.json` at startup (before libraries open, so a resumed job finds its extension), instantiates it with wasmer 6 (4.4 stopped linking on Rust 1.89+, wasmer #5610), and keeps the instance for its lifetime. `plugin_init` registers jobs; `extensions.list` reports them and `extensions.run_job` starts one as a `wasm_job` through the normal job manager. The guest's host calls are bridged to the job context: progress, warnings, metrics, an interrupt flag the pause path sets, and checkpoints that reach the checkpoint table before the guest continues, so both a graceful stop (paused row) and a SIGKILL (running row plus checkpoint) resume from the guest's last state on restart; proven live on a Linux daemon and in `core/tests/wasm_extension_test.rs` (acceptance group, with its own `--features wasm` build: carrying the feature across the integration group added minutes to every test binary link, 51 to 78 minutes on the first CI run). The SDK's `#[job]` export now runs the job function (async ones through a poll loop) and `plugin_init` installs a panic hook so a guest panic reaches the host log. Phase B: the photos extension builds to wasm32 without wasm-bindgen imports, loads beside the test extension and registers four jobs; its first job stops at `JobContext::run` (task execution has no host function), and the SDK has no file-kind or preview registration at all, so step 5's second half is a host-function brief (report SPAC-28 lists the gaps: task execution, vdfs entry and sidecar access, AI inference, model registry, tags, entropy, clock, file-kind and preview registration). The `wasm` feature stays opt-in; whether the shipped daemon turns it on is Jamie's call. |
|
||||
| Extension host functions, phase 1 | in progress 2026-10-07 (Fable, SPAC-30); one PR against `main`, one commit per gap | Fable | Product sequence step 5, second half, in SPAC-28's gap order. Every guest request that returns data now goes through one import, `spacedrive_op(name, payload)`, answered on the job's async side with permission checked against a strict manifest (`read_records` with an optional glob, `read_sidecars`, `write_sidecars` per kind, `use_models` per category; unknown fields are refused at load). Landed: task execution (`#[task]` attempts bracketed by `task.begin`/`task.end` with the SDK's deadline and an attempt line in the job log), entropy and clock (`spacedrive_random` backs `getrandom` so `Uuid::new_v4` works in the guest; `spacedrive_now_ms` backs `clock::now`), records over source stores (`Record` replaces the SDK's `Entry`; `records.get/query/read` resolve through the library's sources and read bytes from the store root; tag and metadata filters are refused), sidecars as one JSON document per extension kind under `SidecarKind::Extension { extension_id, kind }`, models (`#[model]` emits a definition `register_model` collects; the registry builds a `DataTypeSchema` and opens an sd-store db at `<library>/extensions/<id>/data.db`; `models.put/get/list` keyed by uuid or `content:<uuid>`), `config.get` for `config.json` beside the manifest, and `ai.infer`, which checks the `use_models` grant and answers `not_available` because the core has no inference provider. The photos extension handles that with one warning and a skip, so `analyze_photos` completes over a fixture library of twelve JPEGs in `core/tests/wasm_extension_test.rs` (the test extension's new `catalog` job covers records, sidecars, models and tasks in the same suite). Still unbacked: inference itself (so scenes, places and moments have nothing to run on), tags, custom fields, `dispatch_jobs`, agents, actions, queries, and file-kind or preview registration (gap 9, untouched). |
|
||||
| Extension file kinds and previews | design proposed 2026-10-07 (Fable, SPAC-31); nothing built | unowned | Plan: `docs/plans/2026-10-07-extension-kinds-and-previews.md`. Product sequence step 5, second half, and SPAC-28 gap 9. A manifest `kinds` section (name, built-in parent, extensions, magic bytes, preview) folds into a layered `FileTypeRegistry::current()` the plugin manager swaps on load and unload; the content identity phase writes `content.kind` and a new `content.kind_name` (store schema v2), so a file keeps its kind name after the extension unloads and the client falls back to the parent renderer; `ContentRenderer.tsx` becomes a lookup in a renderer registry fed by `extensions.list`, with bundle previews served from `/extension/:id/*` under a plain `mount(el, ctx)` contract. Nothing is stored to migrate: `content.kind` is NULL on every row today and kinds are derived at read time. Five phases K1 to K5, about eight days, two PRs. Jamie decides: kinds stay one level deep under a built-in parent; conflicts resolve by load order and are reported, no picker yet. |
|
||||
| Extension file kinds and previews | K1 and K2 in PR 2026-10-07 (Fable, SPAC-35), one PR against `main`; K3 to K5 unowned | Fable | Plan: `docs/plans/2026-10-07-extension-kinds-and-previews.md`. Product sequence step 5, second half, and SPAC-28 gap 9. Built: `manifest.json` `kinds` (name, built-in parent, extensions, MIME, magic, preview; `core/src/filetype/kinds.rs`), validated at load and refused when a claim would redefine a built-in extension; `FileTypeRegistry::current()` is the built-in table plus every loaded extension's kinds in load order, swapped by the plugin manager on load and unload, and every former `builtin()` caller and the context field read it; two extensions claiming one file extension resolve by sorted load order and `extensions.list` reports `kinds` and `conflicts`; store schema v2 adds `content.kind_name` and `idx_facet_file_extension` (no backfill, a v1 replica reads through a temp view); the content identity phase writes `kind` for every file and `kind_name` for extension kinds, checking extension-kind magic bytes; rows identified earlier are named by extension when an extension loads and when a store opens; `File.content_kind_name` reads from the row and `content_kind` from the stored kind when present. Photos declares `raw`. Tests: registry layering and conflicts (lib), manifest refusals (lib), store migration v1 to v2 and the v1 replica (sd-store), and a WASM acceptance test over `.fake` with a second contesting fixture: columns set, lone magic match on the contested kind, name kept after unload, reidentification on load and after a restart. Next: K3 (client renderer registry over `content_kind_name` and `extensions.list`), K4 (bundle previews), K5 (photos viewer bundle). Jamie decides: kinds one level deep; conflicts by load order, no picker yet. |
|
||||
| MCP server over the registry | PR open 2026-10-07 (Fable, SPAC-33); one PR against `main` | Fable | Product sequence step 2, second half. `sd mcp` speaks MCP over stdio with rmcp and exposes every registered query, action, validate and preview as a tool named after the op, with the input schema walked from the op's specta input type (`core/src/infra/wire/json_schema.rs`), `library_id` on library ops defaulting to the CLI's selection when the daemon still lists it, else the first library, read-only annotations on queries, daemon errors as tool errors, and a synthetic `jobs.wait` that polls `jobs.info` to a terminal status. The server holds no state and no auth beyond the daemon socket. `apps/cli/tests/mcp_test.rs` (acceptance group) starts a core and RPC server in-process, spawns `sd-cli mcp`, handshakes, lists tools, calls `libraries.list` and `jobs.list`, tracks a source and waits for its walk. Docs: `docs/cli/mcp.mdx` (Claude Code, Cursor, Capy configs) and the CLI skill. Next: try it from Claude Code against the MacBook daemon; decide whether actions should require a confirmed preview. |
|
||||
| Repository release gates in CI | in progress 2026-10-07 (Fable, SPAC-26); one PR against `main` | Fable | `docs/core/releases.mdx` lists five repository gates; only formatting, clippy, typecheck and tests had jobs. `ci.yml` gains four more on every PR to `main`: Docs (bun 1.3.4, `docs/` frozen lockfile, `types:check` and `next build`), Generated types (`generate_typescript_types` plus `cargo check -p spacedrive` for the Tauri schemas, fails on a diff under `packages/ts-client/src/generated` or `apps/tauri/src-tauri/gen/schemas`, own rust-cache key), Secrets scan (pinned gitleaks over the PR's commits and the tree, allowlist in `.gitleaks.toml` including the AWS docs example key in `cloud_credentials.rs`), and License and notices (LICENSE is Apache-2.0, NOTICE exists and carries the same AGPL-3.0 cut-over date as README). Found and fixed on the way: the docs build failed on an unescaped `<name>` in `cli/library-sync-setup.mdx`, and the committed Tauri schemas predated the updater plugin. The gates in `releases.mdx` now name their job. Next: confirm the four jobs green on the PR and the warm wall time under 10 min. |
|
||||
|
||||
|
||||
+4
-6
@@ -50,8 +50,6 @@ pub struct CoreContext {
|
||||
pub external_tools: Arc<ExternalTools>,
|
||||
// Remote job cache for cross-device job visibility
|
||||
pub remote_job_cache: Arc<RemoteJobCache>,
|
||||
// File type registry (loaded once at startup, never changes)
|
||||
pub file_type_registry: Arc<FileTypeRegistry>,
|
||||
// Job logging configuration
|
||||
pub job_logging_config: Option<JobLoggingConfig>,
|
||||
pub job_logs_dir: Option<PathBuf>,
|
||||
@@ -109,7 +107,6 @@ impl CoreContext {
|
||||
thumbs,
|
||||
external_tools,
|
||||
remote_job_cache: Arc::new(RemoteJobCache::new()),
|
||||
file_type_registry: Arc::new(FileTypeRegistry::new()),
|
||||
job_logging_config: None,
|
||||
job_logs_dir: None,
|
||||
data_dir,
|
||||
@@ -122,9 +119,10 @@ impl CoreContext {
|
||||
&self.volume_index
|
||||
}
|
||||
|
||||
/// Get the file type registry
|
||||
pub fn file_type_registry(&self) -> &Arc<FileTypeRegistry> {
|
||||
&self.file_type_registry
|
||||
/// The file type registry every lookup shares: the built-in types plus
|
||||
/// the kinds of every loaded extension, swapped by the plugin manager.
|
||||
pub fn file_type_registry(&self) -> Arc<FileTypeRegistry> {
|
||||
FileTypeRegistry::current()
|
||||
}
|
||||
|
||||
/// Get the library manager
|
||||
|
||||
+19
-2
@@ -79,6 +79,11 @@ pub struct File {
|
||||
|
||||
/// Additional computed fields
|
||||
pub content_kind: ContentKind, // Populated by the indexer, for when a File does not have a ContentIdentity
|
||||
/// The extension kind id (`<extension id>:<name>`) the content identity
|
||||
/// phase stored, whether or not that extension is loaded now;
|
||||
/// `content_kind` is then its parent. `None` for a built-in kind.
|
||||
#[serde(default)]
|
||||
pub content_kind_name: Option<String>,
|
||||
pub is_local: bool, // this is redundant with SdPath
|
||||
|
||||
/// Video duration (for grid display optimization)
|
||||
@@ -237,7 +242,7 @@ impl File {
|
||||
.and_then(|extension| extension.to_str())
|
||||
.map(|extension| extension.to_lowercase());
|
||||
let content_kind =
|
||||
crate::filetype::FileTypeRegistry::builtin().identify_by_extension(path);
|
||||
crate::filetype::FileTypeRegistry::current().identify_by_extension(path);
|
||||
(stem, extension, EntryKind::File, content_kind)
|
||||
};
|
||||
let now = Utc::now();
|
||||
@@ -260,11 +265,20 @@ impl File {
|
||||
modified_at: now,
|
||||
accessed_at: None,
|
||||
content_kind,
|
||||
content_kind_name: None,
|
||||
duration_seconds: None,
|
||||
thumbnail_path: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// A file as its store row describes it.
|
||||
///
|
||||
/// The stored content kind is the answer when the identity phase has
|
||||
/// written one; a row identified before kinds were stored derives its
|
||||
/// parent from the name through the current registry, as every row did
|
||||
/// before. The stored kind name is reported whether or not the extension
|
||||
/// that declared it is loaded, so the client can fall back to the parent
|
||||
/// until the extension returns.
|
||||
pub fn from_store_entry(entry: &sd_store::FsEntry, sd_path: SdPath) -> Self {
|
||||
let is_local = sd_path.is_local();
|
||||
|
||||
@@ -293,7 +307,7 @@ impl File {
|
||||
.filter(|kind| *kind != ContentKind::Unknown)
|
||||
.unwrap_or_else(|| {
|
||||
if entry.kind == sd_store::FileKind::File {
|
||||
crate::filetype::FileTypeRegistry::builtin()
|
||||
crate::filetype::FileTypeRegistry::current()
|
||||
.identify_by_extension(std::path::Path::new(&entry.name))
|
||||
} else {
|
||||
ContentKind::Unknown
|
||||
@@ -316,6 +330,7 @@ impl File {
|
||||
modified_at: from_ms(entry.mtime_ms).unwrap_or_else(Utc::now),
|
||||
accessed_at: from_ms(entry.atime_ms),
|
||||
content_kind,
|
||||
content_kind_name: entry.content_kind_name.clone(),
|
||||
extension,
|
||||
kind,
|
||||
is_local,
|
||||
@@ -408,6 +423,7 @@ impl File {
|
||||
modified_at,
|
||||
accessed_at,
|
||||
content_kind: ContentKind::Unknown,
|
||||
content_kind_name: None,
|
||||
extension,
|
||||
kind,
|
||||
is_local,
|
||||
@@ -519,6 +535,7 @@ mod tests {
|
||||
sampled_hash: None,
|
||||
integrity_hash: None,
|
||||
content_kind,
|
||||
content_kind_name: None,
|
||||
content_error: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,250 @@
|
||||
//! # Extension file kinds
|
||||
//!
|
||||
//! An extension's `manifest.json` may declare file kinds: a name, a parent
|
||||
//! from the built-in [`ContentKind`] set, the file extensions and magic bytes
|
||||
//! that identify it, and how the client previews it. A kind is one level deep
|
||||
//! under its parent so every existing consumer of a content kind (search
|
||||
//! filters, kind stats, icons) keeps understanding the parent while a loaded
|
||||
//! extension refines it. The kind's id everywhere else is
|
||||
//! `<extension id>:<name>`, the convention jobs already use, so two
|
||||
//! extensions can both declare `raw` without colliding.
|
||||
//!
|
||||
//! These types live beside the registry rather than in the extension module
|
||||
//! because the registry, `extensions.list` and the content identity phase all
|
||||
//! read them, and all three exist in a build without the `wasm` feature.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use specta::Type;
|
||||
|
||||
use crate::domain::ContentKind;
|
||||
use crate::filetype::magic::MagicBytePattern;
|
||||
|
||||
/// Built-in renderers a kind may name through `preview.renderer`. The client
|
||||
/// registers these first; `ContentRenderer` falls back to the parent's
|
||||
/// renderer for a name it does not know.
|
||||
pub const BUILTIN_RENDERERS: [&str; 7] = [
|
||||
"image", "video", "audio", "mesh", "document", "text", "default",
|
||||
];
|
||||
|
||||
/// Priority an extension kind registers with, above every built-in type, so a
|
||||
/// kind that refines a built-in extension wins the lookup for it.
|
||||
pub const EXTENSION_KIND_PRIORITY: u8 = 110;
|
||||
|
||||
/// A file kind as an extension manifest declares it.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ExtensionKind {
|
||||
/// Lowercase `[a-z0-9-]`, unique within the manifest.
|
||||
pub name: String,
|
||||
#[serde(default)]
|
||||
pub display_name: Option<String>,
|
||||
/// A built-in kind other than `unknown`, `model_entry` and `memory`.
|
||||
pub parent: ContentKind,
|
||||
/// Lowercase, no leading dot, at least one.
|
||||
pub extensions: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub mime_types: Vec<String>,
|
||||
#[serde(default)]
|
||||
pub magic: Vec<MagicPatternSpec>,
|
||||
/// Absent means the parent's renderer.
|
||||
#[serde(default)]
|
||||
pub preview: Option<PreviewSpec>,
|
||||
}
|
||||
|
||||
/// A magic byte pattern in the shape the built-in TOML definitions use.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct MagicPatternSpec {
|
||||
/// Hex bytes separated by spaces; `??` matches any byte.
|
||||
pub pattern: String,
|
||||
#[serde(default)]
|
||||
pub offset: usize,
|
||||
}
|
||||
|
||||
/// How the client previews a kind: a built-in renderer by name, or a
|
||||
/// `ui_manifest.json` `file_viewers` entry by id.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum PreviewSpec {
|
||||
Renderer(String),
|
||||
Viewer(String),
|
||||
}
|
||||
|
||||
/// A file extension two loaded extensions both claimed. The kind loaded
|
||||
/// first holds the claim; the other kind's claim on that extension is
|
||||
/// dropped from the lookup table and its patterns stay available to the
|
||||
/// content identity phase as a tie-breaker.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Type)]
|
||||
pub struct KindConflict {
|
||||
/// The file extension, lowercase, no dot.
|
||||
pub extension: String,
|
||||
/// The kind id whose claim was dropped.
|
||||
pub kind: String,
|
||||
/// The kind id that holds the extension.
|
||||
pub claimed_by: String,
|
||||
}
|
||||
|
||||
impl ExtensionKind {
|
||||
/// The kind's id under its extension: `<extension id>:<name>`.
|
||||
pub fn id(&self, extension_id: &str) -> String {
|
||||
format!("{extension_id}:{}", self.name)
|
||||
}
|
||||
|
||||
/// What the host relies on beyond the shape serde enforces.
|
||||
///
|
||||
/// A parent outside the built-in set would leave the client with no
|
||||
/// renderer to fall back to, and an extension with a dot or uppercase
|
||||
/// would never match the lowercase lookup the registry does.
|
||||
pub fn validate(&self) -> Result<(), String> {
|
||||
if self.name.is_empty()
|
||||
|| !self
|
||||
.name
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')
|
||||
{
|
||||
return Err(format!(
|
||||
"kind name {:?} must be lowercase [a-z0-9-]",
|
||||
self.name
|
||||
));
|
||||
}
|
||||
if matches!(
|
||||
self.parent,
|
||||
ContentKind::Unknown | ContentKind::ModelEntry | ContentKind::Memory
|
||||
) {
|
||||
return Err(format!(
|
||||
"kind {:?} cannot have {} as its parent",
|
||||
self.name, self.parent
|
||||
));
|
||||
}
|
||||
if self.extensions.is_empty() {
|
||||
return Err(format!("kind {:?} claims no file extensions", self.name));
|
||||
}
|
||||
for ext in &self.extensions {
|
||||
if ext.is_empty()
|
||||
|| !ext
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_')
|
||||
{
|
||||
return Err(format!(
|
||||
"kind {:?} extension {ext:?} must be lowercase with no leading dot",
|
||||
self.name
|
||||
));
|
||||
}
|
||||
}
|
||||
for magic in &self.magic {
|
||||
MagicBytePattern::from_hex_string(&magic.pattern, magic.offset, 0)
|
||||
.map_err(|e| format!("kind {:?} magic {:?}: {e}", self.name, magic.pattern))?;
|
||||
}
|
||||
if let Some(PreviewSpec::Renderer(renderer)) = &self.preview {
|
||||
if !BUILTIN_RENDERERS.contains(&renderer.as_str()) {
|
||||
return Err(format!(
|
||||
"kind {:?} names no built-in renderer {renderer:?}; one of {}",
|
||||
self.name,
|
||||
BUILTIN_RENDERERS.join(", ")
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The magic patterns, parsed. Call after [`Self::validate`].
|
||||
pub fn magic_patterns(&self) -> Vec<MagicBytePattern> {
|
||||
self.magic
|
||||
.iter()
|
||||
.filter_map(|m| {
|
||||
MagicBytePattern::from_hex_string(&m.pattern, m.offset, EXTENSION_KIND_PRIORITY)
|
||||
.ok()
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// Validate every kind in one manifest together: each on its own, and the
|
||||
/// names unique within the manifest.
|
||||
pub fn validate_kinds(kinds: &[ExtensionKind]) -> Result<(), String> {
|
||||
let mut seen = std::collections::HashSet::new();
|
||||
for kind in kinds {
|
||||
kind.validate()?;
|
||||
if !seen.insert(kind.name.as_str()) {
|
||||
return Err(format!("kind {:?} is declared twice", kind.name));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn kind(json: &str) -> Result<ExtensionKind, String> {
|
||||
let kind: ExtensionKind = serde_json::from_str(json).map_err(|e| e.to_string())?;
|
||||
kind.validate()?;
|
||||
Ok(kind)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_well_formed_kind_parses() {
|
||||
let raw = kind(
|
||||
r#"{"name":"raw","display_name":"RAW photo","parent":"image",
|
||||
"extensions":["cr2","nef"],"mime_types":["image/x-canon-cr2"],
|
||||
"magic":[{"pattern":"49 49 2A 00","offset":0}],"preview":{"renderer":"image"}}"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(raw.id("com.spacedrive.photos"), "com.spacedrive.photos:raw");
|
||||
assert_eq!(raw.parent, ContentKind::Image);
|
||||
assert_eq!(raw.preview, Some(PreviewSpec::Renderer("image".into())));
|
||||
assert_eq!(raw.magic_patterns().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_rules_are_enforced() {
|
||||
let bad = [
|
||||
(
|
||||
r#"{"name":"Raw","parent":"image","extensions":["cr2"]}"#,
|
||||
"lowercase",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"unknown","extensions":["cr2"]}"#,
|
||||
"parent",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"image","extensions":[]}"#,
|
||||
"no file extensions",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"image","extensions":[".cr2"]}"#,
|
||||
"leading dot",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"image","extensions":["cr2"],"magic":[{"pattern":"ZZ"}]}"#,
|
||||
"magic",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"image","extensions":["cr2"],"preview":{"renderer":"hologram"}}"#,
|
||||
"renderer",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"image","extensions":["cr2"],"icon":"x"}"#,
|
||||
"unknown field",
|
||||
),
|
||||
(
|
||||
r#"{"name":"raw","parent":"kind-of-image","extensions":["cr2"]}"#,
|
||||
"unknown variant",
|
||||
),
|
||||
];
|
||||
for (json, reason) in bad {
|
||||
let err = kind(json).expect_err(json);
|
||||
assert!(err.contains(reason), "{json}: {err}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn names_are_unique_within_a_manifest() {
|
||||
let kinds: Vec<ExtensionKind> = serde_json::from_str(
|
||||
r#"[{"name":"raw","parent":"image","extensions":["cr2"]},
|
||||
{"name":"raw","parent":"image","extensions":["nef"]}]"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(validate_kinds(&kinds).unwrap_err().contains("twice"));
|
||||
}
|
||||
}
|
||||
@@ -12,9 +12,11 @@ use thiserror::Error;
|
||||
use uuid::Uuid;
|
||||
|
||||
pub mod builtin;
|
||||
pub mod kinds;
|
||||
pub mod magic;
|
||||
pub mod registry;
|
||||
|
||||
pub use kinds::{ExtensionKind, KindConflict, MagicPatternSpec, PreviewSpec};
|
||||
pub use magic::{MagicByte, MagicBytePattern};
|
||||
pub use registry::FileTypeRegistry;
|
||||
|
||||
@@ -47,6 +49,11 @@ pub struct FileType {
|
||||
|
||||
/// Extensible metadata
|
||||
pub metadata: JsonValue,
|
||||
|
||||
/// The kind id (`<extension id>:<name>`) when an extension declared this
|
||||
/// type; `None` for every built-in type. `category` is then the parent.
|
||||
#[serde(default)]
|
||||
pub kind_name: Option<String>,
|
||||
}
|
||||
|
||||
/// Result of file type identification
|
||||
|
||||
+415
-22
@@ -1,16 +1,18 @@
|
||||
//! File type registry - the main API for file type identification
|
||||
|
||||
use super::kinds::{ExtensionKind, KindConflict, EXTENSION_KIND_PRIORITY};
|
||||
use super::{FileType, FileTypeError, IdentificationMethod, IdentificationResult, Result};
|
||||
use crate::domain::ContentKind;
|
||||
use crate::filetype::magic::MagicBytePattern;
|
||||
use serde::Deserialize;
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
use std::sync::{Arc, OnceLock, RwLock};
|
||||
use tokio::fs::File;
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt};
|
||||
|
||||
/// Maximum bytes to read for magic byte identification
|
||||
const MAX_MAGIC_BYTES: usize = 8192;
|
||||
pub const MAX_MAGIC_BYTES: usize = 8192;
|
||||
|
||||
/// Maximum bytes to read for content analysis
|
||||
const MAX_CONTENT_BYTES: usize = 4096;
|
||||
@@ -47,6 +49,14 @@ struct MagicByteDefinition {
|
||||
}
|
||||
|
||||
/// Registry of all known file types
|
||||
///
|
||||
/// Two layers exist for the process: the built-in registry, parsed once from
|
||||
/// the embedded definitions, and the current registry, which is the built-in
|
||||
/// one plus every loaded extension's kinds. The plugin manager swaps the
|
||||
/// current one on load and unload; every lookup goes through
|
||||
/// [`Self::current`] so a kind filter and a preview never disagree about the
|
||||
/// same file.
|
||||
#[derive(Clone)]
|
||||
pub struct FileTypeRegistry {
|
||||
/// All registered file types by ID
|
||||
types: HashMap<String, FileType>,
|
||||
@@ -56,18 +66,232 @@ pub struct FileTypeRegistry {
|
||||
|
||||
/// MIME type to type ID mapping
|
||||
mime_map: HashMap<String, String>,
|
||||
|
||||
/// Extension kinds whose claim on a file extension lost to a kind loaded
|
||||
/// earlier, by file extension. Their types stay in `types` so the content
|
||||
/// identity phase can still check their magic patterns.
|
||||
contested: HashMap<String, Vec<String>>,
|
||||
|
||||
/// Every dropped claim, in load order, for `extensions.list`.
|
||||
conflicts: Vec<KindConflict>,
|
||||
|
||||
/// Extension kind ids in load order, so a tie between kinds that
|
||||
/// nothing but their magic bytes distinguishes resolves the same way
|
||||
/// the extension table does.
|
||||
extension_kind_order: Vec<String>,
|
||||
}
|
||||
|
||||
static BUILTIN: OnceLock<Arc<FileTypeRegistry>> = OnceLock::new();
|
||||
static CURRENT: OnceLock<RwLock<Arc<FileTypeRegistry>>> = OnceLock::new();
|
||||
|
||||
impl FileTypeRegistry {
|
||||
/// Create a new registry with built-in types
|
||||
/// The built-in registry, built once for the process.
|
||||
///
|
||||
/// Building one parses every built-in definition. That is affordable once
|
||||
/// and not once per entry: an arena rebuilt from a delivered database adds
|
||||
/// more than a million rows one at a time.
|
||||
pub fn builtin() -> &'static Self {
|
||||
static BUILTIN: std::sync::OnceLock<FileTypeRegistry> = std::sync::OnceLock::new();
|
||||
BUILTIN.get_or_init(Self::new)
|
||||
BUILTIN.get_or_init(|| Arc::new(Self::new()))
|
||||
}
|
||||
|
||||
/// The registry every lookup should use: the built-in types plus the
|
||||
/// kinds of every loaded extension. The built-in registry alone until an
|
||||
/// extension with kinds loads.
|
||||
pub fn current() -> Arc<Self> {
|
||||
CURRENT
|
||||
.get_or_init(|| RwLock::new(BUILTIN.get_or_init(|| Arc::new(Self::new())).clone()))
|
||||
.read()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner())
|
||||
.clone()
|
||||
}
|
||||
|
||||
/// Replace the current registry with the built-in types plus these
|
||||
/// extensions' kinds, in load order. The plugin manager calls this after
|
||||
/// every load and unload; an empty list restores the built-in registry.
|
||||
pub fn install_current(extensions: &[(String, Vec<ExtensionKind>)]) -> Arc<Self> {
|
||||
let registry = Arc::new(Self::with_extension_kinds(extensions));
|
||||
let slot = CURRENT
|
||||
.get_or_init(|| RwLock::new(BUILTIN.get_or_init(|| Arc::new(Self::new())).clone()));
|
||||
*slot
|
||||
.write()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner()) = registry.clone();
|
||||
registry
|
||||
}
|
||||
|
||||
/// The built-in registry plus these extensions' kinds, in load order.
|
||||
pub fn with_extension_kinds(extensions: &[(String, Vec<ExtensionKind>)]) -> Self {
|
||||
let mut registry = Self::builtin().clone();
|
||||
for (extension_id, kinds) in extensions {
|
||||
for kind in kinds {
|
||||
registry.register_extension_kind(extension_id, kind);
|
||||
}
|
||||
}
|
||||
registry
|
||||
}
|
||||
|
||||
/// Why a kind may not load against the built-in table, or `None`.
|
||||
///
|
||||
/// A kind may refine a file extension the built-in table maps to its own
|
||||
/// parent, or claim one the table does not know. Redefining `.pdf` as an
|
||||
/// image would change what every consumer of the parent sees, so that is
|
||||
/// refused rather than layered.
|
||||
pub fn refusal_for(&self, kind: &ExtensionKind) -> Option<String> {
|
||||
for ext in &kind.extensions {
|
||||
let builtin = self
|
||||
.get_by_extension(ext)
|
||||
.into_iter()
|
||||
.filter(|ft| ft.kind_name.is_none())
|
||||
.max_by_key(|ft| ft.priority)
|
||||
.map(|ft| ft.category)
|
||||
.unwrap_or(ContentKind::Unknown);
|
||||
if builtin != ContentKind::Unknown && builtin != kind.parent {
|
||||
return Some(format!(
|
||||
"kind {:?} claims .{ext} with parent {} but the built-in table maps it to {}",
|
||||
kind.name, kind.parent, builtin
|
||||
));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Add one extension kind. A file extension another extension kind
|
||||
/// already holds stays with it; this kind's claim is recorded as a
|
||||
/// conflict and the type is kept for its magic patterns.
|
||||
fn register_extension_kind(&mut self, extension_id: &str, kind: &ExtensionKind) {
|
||||
let id = kind.id(extension_id);
|
||||
for ext in &kind.extensions {
|
||||
let ext = ext.to_lowercase();
|
||||
let holder = self
|
||||
.extension_map
|
||||
.get(&ext)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.find(|id| self.types.get(*id).is_some_and(|ft| ft.kind_name.is_some()))
|
||||
.cloned();
|
||||
match holder {
|
||||
Some(claimed_by) => {
|
||||
self.contested
|
||||
.entry(ext.clone())
|
||||
.or_default()
|
||||
.push(id.clone());
|
||||
self.conflicts.push(KindConflict {
|
||||
extension: ext,
|
||||
kind: id.clone(),
|
||||
claimed_by,
|
||||
});
|
||||
}
|
||||
None => self.extension_map.entry(ext).or_default().push(id.clone()),
|
||||
}
|
||||
}
|
||||
for mime in &kind.mime_types {
|
||||
self.mime_map
|
||||
.entry(mime.clone())
|
||||
.or_insert_with(|| id.clone());
|
||||
}
|
||||
self.extension_kind_order.push(id.clone());
|
||||
self.types.insert(
|
||||
id.clone(),
|
||||
FileType {
|
||||
id: id.clone(),
|
||||
name: kind
|
||||
.display_name
|
||||
.clone()
|
||||
.unwrap_or_else(|| kind.name.clone()),
|
||||
extensions: kind.extensions.clone(),
|
||||
mime_types: kind.mime_types.clone(),
|
||||
uti: None,
|
||||
magic_bytes: kind.magic_patterns(),
|
||||
category: kind.parent,
|
||||
priority: EXTENSION_KIND_PRIORITY,
|
||||
metadata: serde_json::Value::Null,
|
||||
kind_name: Some(id),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// Every extension kind with the file extensions it holds, contested
|
||||
/// claims left out, in no particular order.
|
||||
pub fn extension_kinds(&self) -> Vec<(&FileType, Vec<String>)> {
|
||||
self.types
|
||||
.values()
|
||||
.filter(|ft| ft.kind_name.is_some())
|
||||
.map(|ft| {
|
||||
let held = ft
|
||||
.extensions
|
||||
.iter()
|
||||
.map(|e| e.to_lowercase())
|
||||
.filter(|ext| {
|
||||
self.extension_map
|
||||
.get(ext)
|
||||
.is_some_and(|ids| ids.contains(&ft.id))
|
||||
})
|
||||
.collect();
|
||||
(ft, held)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Every dropped claim, in load order.
|
||||
pub fn conflicts(&self) -> &[KindConflict] {
|
||||
&self.conflicts
|
||||
}
|
||||
|
||||
/// The extension kinds whose magic patterns decide a file with this
|
||||
/// extension: the kind holding the extension when it declares patterns,
|
||||
/// and every contested kind for it. For an extension no type claims,
|
||||
/// every extension kind with patterns, since the name says nothing.
|
||||
/// Empty means the extension lookup is the whole answer.
|
||||
pub fn magic_candidates(&self, extension: Option<&str>) -> Vec<&FileType> {
|
||||
let with_magic = |ft: &&FileType| ft.kind_name.is_some() && !ft.magic_bytes.is_empty();
|
||||
let Some(ext) = extension.map(str::to_lowercase) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let claimed = self.get_by_extension(&ext);
|
||||
if claimed.is_empty() {
|
||||
return self
|
||||
.extension_kind_order
|
||||
.iter()
|
||||
.filter_map(|id| self.types.get(id))
|
||||
.filter(with_magic)
|
||||
.collect();
|
||||
}
|
||||
let mut candidates: Vec<&FileType> = claimed.into_iter().filter(with_magic).collect();
|
||||
candidates.extend(
|
||||
self.contested
|
||||
.get(&ext)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(|id| self.types.get(id))
|
||||
.filter(with_magic),
|
||||
);
|
||||
candidates
|
||||
}
|
||||
|
||||
/// The kind a file's header decides, given the candidates
|
||||
/// [`Self::magic_candidates`] named and the type its extension mapped to.
|
||||
///
|
||||
/// A lone match assigns that kind, which is how a contested kind still
|
||||
/// wins the files whose bytes it recognizes. Several matches keep the
|
||||
/// extension's holder when it is among them. No match keeps the
|
||||
/// extension result.
|
||||
pub fn resolve_by_magic<'a>(
|
||||
&'a self,
|
||||
by_extension: Option<&'a FileType>,
|
||||
candidates: &[&'a FileType],
|
||||
header: &[u8],
|
||||
) -> Option<&'a FileType> {
|
||||
let matched: Vec<&FileType> = candidates
|
||||
.iter()
|
||||
.copied()
|
||||
.filter(|ft| ft.magic_bytes.iter().any(|p| p.matches(header)))
|
||||
.collect();
|
||||
match matched.as_slice() {
|
||||
[] => by_extension,
|
||||
[only] => Some(only),
|
||||
several => by_extension
|
||||
.filter(|holder| several.iter().any(|ft| ft.id == holder.id))
|
||||
.or(Some(several[0])),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn new() -> Self {
|
||||
@@ -75,6 +299,9 @@ impl FileTypeRegistry {
|
||||
types: HashMap::new(),
|
||||
extension_map: HashMap::new(),
|
||||
mime_map: HashMap::new(),
|
||||
contested: HashMap::new(),
|
||||
conflicts: Vec::new(),
|
||||
extension_kind_order: Vec::new(),
|
||||
};
|
||||
|
||||
// Load built-in types
|
||||
@@ -163,25 +390,19 @@ impl FileTypeRegistry {
|
||||
///
|
||||
/// Returns `ContentKind::Unknown` if the extension is not recognized.
|
||||
pub fn identify_by_extension(&self, path: &Path) -> ContentKind {
|
||||
let extension = match path.extension().and_then(|s| s.to_str()) {
|
||||
Some(ext) => ext,
|
||||
None => return ContentKind::Unknown,
|
||||
};
|
||||
self.type_by_extension(path)
|
||||
.map(|ft| ft.category)
|
||||
.unwrap_or(ContentKind::Unknown)
|
||||
}
|
||||
|
||||
let candidates = self.get_by_extension(extension);
|
||||
|
||||
match candidates.len() {
|
||||
0 => ContentKind::Unknown,
|
||||
1 => candidates[0].category,
|
||||
_ => {
|
||||
// Multiple matches - pick highest priority
|
||||
candidates
|
||||
.iter()
|
||||
.max_by_key(|ft| ft.priority)
|
||||
.map(|ft| ft.category)
|
||||
.unwrap_or(ContentKind::Unknown)
|
||||
}
|
||||
}
|
||||
/// The highest priority type the path's extension maps to, which is an
|
||||
/// extension kind when one holds the extension and a built-in type
|
||||
/// otherwise. `None` for no extension or one nothing claims.
|
||||
pub fn type_by_extension(&self, path: &Path) -> Option<&FileType> {
|
||||
let extension = path.extension().and_then(|s| s.to_str())?;
|
||||
self.get_by_extension(extension)
|
||||
.into_iter()
|
||||
.max_by_key(|ft| ft.priority)
|
||||
}
|
||||
|
||||
/// Identify a file type from a path
|
||||
@@ -402,6 +623,7 @@ impl FileTypeRegistry {
|
||||
category,
|
||||
priority: def.priority,
|
||||
metadata: def.metadata,
|
||||
kind_name: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -434,4 +656,175 @@ mod tests {
|
||||
let ts_types = registry.get_by_extension("ts");
|
||||
assert_eq!(ts_types.len(), 2); // TypeScript and MPEG-TS
|
||||
}
|
||||
|
||||
fn kind(name: &str, parent: ContentKind, extensions: &[&str], magic: &[&str]) -> ExtensionKind {
|
||||
ExtensionKind {
|
||||
name: name.into(),
|
||||
display_name: None,
|
||||
parent,
|
||||
extensions: extensions.iter().map(|e| e.to_string()).collect(),
|
||||
mime_types: Vec::new(),
|
||||
magic: magic
|
||||
.iter()
|
||||
.map(|pattern| super::super::kinds::MagicPatternSpec {
|
||||
pattern: pattern.to_string(),
|
||||
offset: 0,
|
||||
})
|
||||
.collect(),
|
||||
preview: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_extension_kind_refines_a_built_in_extension_and_goes_away_on_unload() {
|
||||
let raw = kind("raw", ContentKind::Image, &["cr2", "dng"], &[]);
|
||||
let layered = FileTypeRegistry::with_extension_kinds(&[("photos".to_string(), vec![raw])]);
|
||||
let path = Path::new("shot.dng");
|
||||
let ft = layered.type_by_extension(path).unwrap();
|
||||
assert_eq!(ft.id, "photos:raw");
|
||||
assert_eq!(ft.kind_name.as_deref(), Some("photos:raw"));
|
||||
assert_eq!(layered.identify_by_extension(path), ContentKind::Image);
|
||||
assert_eq!(
|
||||
layered.type_by_extension(Path::new("a.jpg")).unwrap().id,
|
||||
"image/jpeg",
|
||||
"other built-in types are untouched"
|
||||
);
|
||||
|
||||
let unloaded = FileTypeRegistry::with_extension_kinds(&[]);
|
||||
assert_eq!(
|
||||
unloaded.type_by_extension(path).unwrap().id,
|
||||
"image/x-adobe-dng"
|
||||
);
|
||||
assert!(unloaded.conflicts().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_kind_may_not_redefine_a_built_in_extension() {
|
||||
let builtin = FileTypeRegistry::builtin();
|
||||
let bad = kind("pdf-as-image", ContentKind::Image, &["pdf"], &[]);
|
||||
let reason = builtin.refusal_for(&bad).unwrap();
|
||||
assert!(reason.contains(".pdf"), "{reason}");
|
||||
assert!(reason.contains("document"), "{reason}");
|
||||
|
||||
let refine = kind("raw", ContentKind::Image, &["cr2"], &[]);
|
||||
assert_eq!(builtin.refusal_for(&refine), None);
|
||||
let fresh = kind("thing", ContentKind::Database, &["zzznoone"], &[]);
|
||||
assert_eq!(builtin.refusal_for(&fresh), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_extensions_claiming_one_extension_resolve_by_load_order() {
|
||||
let first = kind("fake", ContentKind::Image, &["xyz"], &["46 41 4B 45"]);
|
||||
let second = kind(
|
||||
"other",
|
||||
ContentKind::Image,
|
||||
&["xyz", "abc"],
|
||||
&["4F 54 48 52"],
|
||||
);
|
||||
let layered = FileTypeRegistry::with_extension_kinds(&[
|
||||
("one".to_string(), vec![first]),
|
||||
("two".to_string(), vec![second]),
|
||||
]);
|
||||
|
||||
assert_eq!(
|
||||
layered.type_by_extension(Path::new("f.xyz")).unwrap().id,
|
||||
"one:fake"
|
||||
);
|
||||
assert_eq!(
|
||||
layered.type_by_extension(Path::new("f.abc")).unwrap().id,
|
||||
"two:other",
|
||||
"an uncontested extension of the loser still loads"
|
||||
);
|
||||
assert_eq!(
|
||||
layered.conflicts(),
|
||||
&[KindConflict {
|
||||
extension: "xyz".into(),
|
||||
kind: "two:other".into(),
|
||||
claimed_by: "one:fake".into(),
|
||||
}]
|
||||
);
|
||||
|
||||
let candidates = layered.magic_candidates(Some("xyz"));
|
||||
let ids: Vec<&str> = candidates.iter().map(|ft| ft.id.as_str()).collect();
|
||||
assert_eq!(ids, ["one:fake", "two:other"]);
|
||||
let holder = layered.type_by_extension(Path::new("f.xyz"));
|
||||
assert_eq!(
|
||||
layered
|
||||
.resolve_by_magic(holder, &candidates, b"OTHR....")
|
||||
.unwrap()
|
||||
.id,
|
||||
"two:other",
|
||||
"a lone match on a contested kind wins the file"
|
||||
);
|
||||
assert_eq!(
|
||||
layered
|
||||
.resolve_by_magic(holder, &candidates, b"FAKE....")
|
||||
.unwrap()
|
||||
.id,
|
||||
"one:fake"
|
||||
);
|
||||
assert_eq!(
|
||||
layered
|
||||
.resolve_by_magic(holder, &candidates, b"????....")
|
||||
.unwrap()
|
||||
.id,
|
||||
"one:fake",
|
||||
"no match keeps the extension result"
|
||||
);
|
||||
assert!(
|
||||
layered.magic_candidates(Some("jpg")).is_empty(),
|
||||
"a built-in extension nobody refines reads no bytes"
|
||||
);
|
||||
let unknown: Vec<&str> = layered
|
||||
.magic_candidates(Some("zzzunknown"))
|
||||
.iter()
|
||||
.map(|ft| ft.id.as_str())
|
||||
.collect();
|
||||
assert_eq!(
|
||||
unknown,
|
||||
["one:fake", "two:other"],
|
||||
"an unknown extension checks every kind with magic, in load order"
|
||||
);
|
||||
let both = FileTypeRegistry::with_extension_kinds(&[
|
||||
(
|
||||
"one".to_string(),
|
||||
vec![kind("a", ContentKind::Text, &["qqq"], &["4F 54"])],
|
||||
),
|
||||
(
|
||||
"two".to_string(),
|
||||
vec![kind("b", ContentKind::Text, &["rrr"], &["4F 54 48"])],
|
||||
),
|
||||
]);
|
||||
let candidates = both.magic_candidates(Some("zzzunknown"));
|
||||
assert_eq!(
|
||||
both.resolve_by_magic(None, &candidates, b"OTHR")
|
||||
.unwrap()
|
||||
.id,
|
||||
"one:a",
|
||||
"several matches on an unclaimed extension go to the kind loaded first"
|
||||
);
|
||||
assert!(layered.magic_candidates(None).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn install_current_swaps_the_process_registry() {
|
||||
let before = FileTypeRegistry::current();
|
||||
assert!(before.type_by_extension(Path::new("a.zzzkind")).is_none());
|
||||
let installed = FileTypeRegistry::install_current(&[(
|
||||
"ext".to_string(),
|
||||
vec![kind("zz", ContentKind::Text, &["zzzkind"], &[])],
|
||||
)]);
|
||||
assert_eq!(
|
||||
FileTypeRegistry::current()
|
||||
.type_by_extension(Path::new("a.zzzkind"))
|
||||
.unwrap()
|
||||
.id,
|
||||
"ext:zz"
|
||||
);
|
||||
assert!(Arc::ptr_eq(&installed, &FileTypeRegistry::current()));
|
||||
FileTypeRegistry::install_current(&[]);
|
||||
assert!(FileTypeRegistry::current()
|
||||
.type_by_extension(Path::new("a.zzzkind"))
|
||||
.is_none());
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,30 @@ reports `supported: false` and `extensions.run_job` is refused.
|
||||
reject unknown fields, so a typo in a grant fails at load instead of
|
||||
silently granting nothing.
|
||||
|
||||
## File kinds
|
||||
|
||||
A manifest may declare `kinds`: a name, a built-in `ContentKind` parent, the
|
||||
file extensions and optional MIME types and magic bytes that identify it,
|
||||
and a `preview` naming a built-in renderer or a `ui_manifest.json` viewer
|
||||
(`core/src/filetype/kinds.rs`). The kind's id is `<extension id>:<name>`.
|
||||
On load the manager rebuilds `FileTypeRegistry::current()` from the built-in
|
||||
table plus every loaded extension's kinds in load order, and on unload it
|
||||
rebuilds without them. A kind may refine a file extension the built-in table
|
||||
maps to its own parent, or claim one the table does not know; a claim that
|
||||
would redefine a built-in extension is refused at load. Two extensions
|
||||
claiming one file extension resolve by load order (`load_all` sorts the
|
||||
directory names), the later claim is dropped from the lookup and reported by
|
||||
`extensions.list` as a conflict, and its magic patterns still decide a file
|
||||
whose bytes match only them.
|
||||
|
||||
The content identity phase writes the kind and, for an extension kind, its
|
||||
name onto the content row (`content.kind`, `content.kind_name`, store schema
|
||||
version 2). Rows identified before an extension existed are named by
|
||||
extension when the extension loads and when a store opens
|
||||
(`core/src/ops/indexing/kinds.rs`). A row keeps its name after the extension
|
||||
unloads; `File.content_kind` is then the parent and `File.content_kind_name`
|
||||
the name, so a client can fall back until the extension returns.
|
||||
|
||||
## Running a job
|
||||
|
||||
1. `extensions.run_job` looks the job up in the registry and dispatches a
|
||||
|
||||
@@ -17,6 +17,8 @@ use super::host_functions::{self, host_spacedrive_log, JobBridge, PluginEnv};
|
||||
use super::job_registry::ExtensionJobRegistry;
|
||||
use super::model_registry::ExtensionModelRegistry;
|
||||
use super::types::{ExtensionManifest, LoadedPlugin};
|
||||
use crate::filetype::{ExtensionKind, FileTypeRegistry};
|
||||
use crate::ops::indexing::VolumeIndex;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum PluginError {
|
||||
@@ -135,16 +137,58 @@ pub struct PluginManager {
|
||||
plugin_dir: PathBuf,
|
||||
job_registry: Arc<ExtensionJobRegistry>,
|
||||
model_registry: Arc<ExtensionModelRegistry>,
|
||||
/// The kinds of every loaded extension, in load order, which is the
|
||||
/// order the file type registry resolves a contested file extension in.
|
||||
kinds: Vec<(String, Vec<ExtensionKind>)>,
|
||||
/// Where open stores are found, so a load with kinds can name the rows
|
||||
/// those stores already hold. `None` only in tests of the manager alone.
|
||||
volume_index: Option<Arc<VolumeIndex>>,
|
||||
}
|
||||
|
||||
impl PluginManager {
|
||||
/// Create new plugin manager
|
||||
pub fn new(plugin_dir: PathBuf) -> Self {
|
||||
pub fn new(plugin_dir: PathBuf, volume_index: Option<Arc<VolumeIndex>>) -> Self {
|
||||
Self {
|
||||
plugins: Arc::new(RwLock::new(HashMap::new())),
|
||||
plugin_dir,
|
||||
job_registry: Arc::new(ExtensionJobRegistry::new()),
|
||||
model_registry: Arc::new(ExtensionModelRegistry::new()),
|
||||
kinds: Vec::new(),
|
||||
volume_index,
|
||||
}
|
||||
}
|
||||
|
||||
/// Put this extension's kinds into the registry every lookup uses, and
|
||||
/// name the rows open stores already hold after them.
|
||||
async fn add_kinds(&mut self, plugin_id: &str, kinds: &[ExtensionKind]) {
|
||||
if kinds.is_empty() {
|
||||
return;
|
||||
}
|
||||
self.kinds.push((plugin_id.to_string(), kinds.to_vec()));
|
||||
let registry = FileTypeRegistry::install_current(&self.kinds);
|
||||
for conflict in registry.conflicts() {
|
||||
if conflict.kind.starts_with(&format!("{plugin_id}:")) {
|
||||
tracing::warn!(
|
||||
extension = %plugin_id,
|
||||
file_extension = %conflict.extension,
|
||||
kind = %conflict.kind,
|
||||
claimed_by = %conflict.claimed_by,
|
||||
"file extension already claimed by an earlier extension; claim dropped"
|
||||
);
|
||||
}
|
||||
}
|
||||
if let Some(volume_index) = &self.volume_index {
|
||||
crate::ops::indexing::kinds::name_kinds_in_open_stores(volume_index).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Take this extension's kinds back out of the registry. Rows already
|
||||
/// named keep their name; readers fall back to the parent kind.
|
||||
fn remove_kinds(&mut self, plugin_id: &str) {
|
||||
let before = self.kinds.len();
|
||||
self.kinds.retain(|(id, _)| id != plugin_id);
|
||||
if self.kinds.len() != before {
|
||||
FileTypeRegistry::install_current(&self.kinds);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -177,12 +221,19 @@ impl PluginManager {
|
||||
}
|
||||
};
|
||||
|
||||
let mut loaded = Vec::new();
|
||||
// Sorted, so the extension that wins a contested file extension is
|
||||
// the same on every machine rather than whichever the filesystem
|
||||
// listed first.
|
||||
let mut dir_names = Vec::new();
|
||||
while let Ok(Some(entry)) = dirs.next_entry().await {
|
||||
if !entry.path().join("manifest.json").is_file() {
|
||||
continue;
|
||||
if entry.path().join("manifest.json").is_file() {
|
||||
dir_names.push(entry.file_name().to_string_lossy().to_string());
|
||||
}
|
||||
let dir_name = entry.file_name().to_string_lossy().to_string();
|
||||
}
|
||||
dir_names.sort();
|
||||
|
||||
let mut loaded = Vec::new();
|
||||
for dir_name in dir_names {
|
||||
match self.load_plugin(&dir_name).await {
|
||||
Ok(id) => loaded.push(id),
|
||||
Err(e) => tracing::error!(extension = %dir_name, "Extension did not load: {e}"),
|
||||
@@ -395,7 +446,9 @@ impl PluginManager {
|
||||
"Extension loaded"
|
||||
);
|
||||
|
||||
let kinds = plugin.manifest.kinds.clone();
|
||||
self.plugins.write().await.insert(plugin_id.clone(), plugin);
|
||||
self.add_kinds(&plugin_id, &kinds).await;
|
||||
Ok(plugin_id)
|
||||
}
|
||||
|
||||
@@ -411,6 +464,7 @@ impl PluginManager {
|
||||
.ok_or_else(|| PluginError::NotFound(plugin_id.to_string()))?;
|
||||
self.job_registry.unregister_extension_jobs(plugin_id);
|
||||
self.model_registry.unregister_extension(plugin_id);
|
||||
self.remove_kinds(plugin_id);
|
||||
|
||||
// A trapped guest cannot be trusted to run cleanup, and a guest still
|
||||
// running a job keeps its instance alive until that job returns; the
|
||||
@@ -465,10 +519,12 @@ impl PluginManager {
|
||||
},
|
||||
);
|
||||
}
|
||||
let kinds = previous.manifest.kinds.clone();
|
||||
self.plugins
|
||||
.write()
|
||||
.await
|
||||
.insert(plugin_id.to_string(), previous);
|
||||
self.add_kinds(plugin_id, &kinds).await;
|
||||
return Err(e);
|
||||
}
|
||||
Ok(())
|
||||
|
||||
@@ -6,6 +6,8 @@ use std::path::PathBuf;
|
||||
use std::sync::atomic::AtomicBool;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use crate::filetype::{kinds::validate_kinds, ExtensionKind, FileTypeRegistry};
|
||||
|
||||
/// Extension manifest (`manifest.json`).
|
||||
///
|
||||
/// Unknown fields are refused rather than ignored, so a grant the host does
|
||||
@@ -44,6 +46,11 @@ pub struct ExtensionManifest {
|
||||
/// Configuration schema (JSON Schema)
|
||||
#[serde(default)]
|
||||
pub config_schema: Option<serde_json::Value>,
|
||||
|
||||
/// File kinds this extension declares, folded into the file type
|
||||
/// registry while it is loaded. See `crate::filetype::kinds`.
|
||||
#[serde(default)]
|
||||
pub kinds: Vec<ExtensionKind>,
|
||||
}
|
||||
|
||||
/// Permission declaration in manifest.
|
||||
@@ -104,6 +111,12 @@ impl ExtensionManifest {
|
||||
));
|
||||
}
|
||||
}
|
||||
validate_kinds(&self.kinds)?;
|
||||
for kind in &self.kinds {
|
||||
if let Some(reason) = FileTypeRegistry::builtin().refusal_for(kind) {
|
||||
return Err(reason);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -223,6 +236,15 @@ mod tests {
|
||||
assert!(perms.can_read_sidecar("exif"));
|
||||
assert!(!perms.can_write_sidecar("exif"));
|
||||
assert_eq!(manifest.models.len(), 2);
|
||||
assert_eq!(
|
||||
manifest.validate(),
|
||||
Ok(()),
|
||||
"photos' kinds load against the built-in table"
|
||||
);
|
||||
assert_eq!(
|
||||
manifest.kinds[0].id(&manifest.id),
|
||||
"com.spacedrive.photos:raw"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -260,6 +282,46 @@ mod tests {
|
||||
assert!(manifest.validate().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_loader_refuses_a_kind_that_redefines_a_built_in_extension() {
|
||||
let mut manifest: ExtensionManifest = serde_json::from_str(
|
||||
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
|
||||
"kinds":[{"name":"raw","parent":"image","extensions":["cr2","dng"],
|
||||
"preview":{"renderer":"image"}}]}"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
manifest.validate(),
|
||||
Ok(()),
|
||||
"refining .dng under image is allowed"
|
||||
);
|
||||
|
||||
manifest.kinds[0].extensions = vec!["pdf".into()];
|
||||
let err = manifest.validate().unwrap_err();
|
||||
assert!(err.contains(".pdf") && err.contains("document"), "{err}");
|
||||
|
||||
manifest.kinds[0].extensions = vec!["cr2".into()];
|
||||
manifest.kinds[0].parent = crate::domain::ContentKind::Unknown;
|
||||
assert!(manifest.validate().unwrap_err().contains("parent"));
|
||||
|
||||
let twice = serde_json::from_str::<ExtensionManifest>(
|
||||
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
|
||||
"kinds":[{"name":"raw","parent":"image","extensions":["cr2"]},
|
||||
{"name":"raw","parent":"image","extensions":["nef"]}]}"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(twice.validate().unwrap_err().contains("twice"));
|
||||
|
||||
let unknown_field = serde_json::from_str::<ExtensionManifest>(
|
||||
r#"{"id":"ok","name":"x","version":"1","wasm_file":"x.wasm",
|
||||
"kinds":[{"name":"raw","parent":"image","extensions":["cr2"],"icon":"x"}]}"#,
|
||||
);
|
||||
assert!(unknown_field
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("unknown field"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_glob_without_a_list_names_one_extension() {
|
||||
let perms = ManifestPermissions {
|
||||
|
||||
+4
-1
@@ -186,7 +186,10 @@ impl Core {
|
||||
warn!("Could not create extensions directory: {e}");
|
||||
}
|
||||
|
||||
let mut pm = crate::infra::extension::PluginManager::new(plugin_dir);
|
||||
let mut pm = crate::infra::extension::PluginManager::new(
|
||||
plugin_dir,
|
||||
Some(context.volume_index().clone()),
|
||||
);
|
||||
let loaded = pm.load_all().await;
|
||||
info!("Loaded {} extension(s): {:?}", loaded.len(), loaded);
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
//! List loaded extensions and the jobs they registered
|
||||
//! List loaded extensions, the jobs they registered, the file kinds they
|
||||
//! declare, and the file extensions two of them both claimed.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -7,6 +8,8 @@ use specta::Type;
|
||||
|
||||
use crate::{
|
||||
context::CoreContext,
|
||||
domain::ContentKind,
|
||||
filetype::{KindConflict, PreviewSpec},
|
||||
infra::{
|
||||
api::SessionContext,
|
||||
query::{CoreQuery, QueryResult},
|
||||
@@ -25,12 +28,29 @@ pub struct ExtensionJobInfo {
|
||||
pub resumable: bool,
|
||||
}
|
||||
|
||||
/// A file kind an extension declares, as the client resolves previews and
|
||||
/// icons against it. The kinds of every loaded extension are the client's
|
||||
/// only source of extension kinds, which is what lets a stored kind name
|
||||
/// fall back to its parent once the extension is gone.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
|
||||
pub struct ExtensionKindInfo {
|
||||
/// `<extension id>:<name>`, the value `File.content_kind_name` carries
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub display_name: String,
|
||||
pub parent: ContentKind,
|
||||
/// Every extension the manifest claims, contested ones included
|
||||
pub extensions: Vec<String>,
|
||||
pub preview: Option<PreviewSpec>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
|
||||
pub struct ExtensionInfo {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub version: String,
|
||||
pub jobs: Vec<ExtensionJobInfo>,
|
||||
pub kinds: Vec<ExtensionKindInfo>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Type)]
|
||||
@@ -38,6 +58,9 @@ pub struct ListExtensionsOutput {
|
||||
/// Whether this build can load extensions at all
|
||||
pub supported: bool,
|
||||
pub extensions: Vec<ExtensionInfo>,
|
||||
/// File extensions two loaded extensions both claimed, resolved by load
|
||||
/// order: the kind loaded first holds each one.
|
||||
pub conflicts: Vec<KindConflict>,
|
||||
}
|
||||
|
||||
pub struct ListExtensionsQuery;
|
||||
@@ -61,6 +84,7 @@ impl CoreQuery for ListExtensionsQuery {
|
||||
return Ok(ListExtensionsOutput {
|
||||
supported: true,
|
||||
extensions: Vec::new(),
|
||||
conflicts: Vec::new(),
|
||||
});
|
||||
};
|
||||
let pm = plugin_manager.read().await;
|
||||
@@ -80,17 +104,36 @@ impl CoreQuery for ListExtensionsQuery {
|
||||
})
|
||||
.collect();
|
||||
jobs.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
let kinds = manifest
|
||||
.kinds
|
||||
.iter()
|
||||
.map(|kind| ExtensionKindInfo {
|
||||
id: kind.id(&id),
|
||||
name: kind.name.clone(),
|
||||
display_name: kind
|
||||
.display_name
|
||||
.clone()
|
||||
.unwrap_or_else(|| kind.name.clone()),
|
||||
parent: kind.parent,
|
||||
extensions: kind.extensions.clone(),
|
||||
preview: kind.preview.clone(),
|
||||
})
|
||||
.collect();
|
||||
extensions.push(ExtensionInfo {
|
||||
id,
|
||||
name: manifest.name.clone(),
|
||||
version: manifest.version.clone(),
|
||||
jobs,
|
||||
kinds,
|
||||
});
|
||||
}
|
||||
extensions.sort_by(|a, b| a.id.cmp(&b.id));
|
||||
Ok(ListExtensionsOutput {
|
||||
supported: true,
|
||||
extensions,
|
||||
conflicts: crate::filetype::FileTypeRegistry::current()
|
||||
.conflicts()
|
||||
.to_vec(),
|
||||
})
|
||||
}
|
||||
#[cfg(not(feature = "wasm"))]
|
||||
@@ -99,6 +142,7 @@ impl CoreQuery for ListExtensionsQuery {
|
||||
Ok(ListExtensionsOutput {
|
||||
supported: false,
|
||||
extensions: Vec::new(),
|
||||
conflicts: Vec::new(),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -318,6 +318,7 @@ pub(super) async fn integrity(file: &Keyed) -> (Result<String, String>, Option<L
|
||||
integrity_hash: Some(integrity.clone()),
|
||||
size: file.entry.size,
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
};
|
||||
(Ok(integrity), Some(learned))
|
||||
|
||||
@@ -283,7 +283,7 @@ pub fn folder_for_kind(entry: &FsEntry) -> String {
|
||||
.map(ContentKind::from_id)
|
||||
.filter(|kind| *kind != ContentKind::Unknown)
|
||||
.unwrap_or_else(|| {
|
||||
crate::filetype::FileTypeRegistry::builtin()
|
||||
crate::filetype::FileTypeRegistry::current()
|
||||
.identify_by_extension(Path::new(&entry.name))
|
||||
});
|
||||
match kind {
|
||||
|
||||
@@ -291,7 +291,7 @@ impl Arena {
|
||||
uuid: Uuid,
|
||||
metadata: EntryMetadata,
|
||||
) -> std::io::Result<(Option<ContentKind>, Uuid)> {
|
||||
self.add_entry_with_registry(path, Some(uuid), metadata, FileTypeRegistry::builtin())
|
||||
self.add_entry_with_registry(path, Some(uuid), metadata, &FileTypeRegistry::current())
|
||||
}
|
||||
|
||||
fn add_entry_with_registry(
|
||||
@@ -451,14 +451,10 @@ impl Arena {
|
||||
entries: Vec<(PathBuf, Option<Uuid>, EntryMetadata)>,
|
||||
) -> std::io::Result<Vec<(Option<ContentKind>, Uuid)>> {
|
||||
let mut results = Vec::with_capacity(entries.len());
|
||||
let registry = FileTypeRegistry::current();
|
||||
|
||||
for (path, uuid, metadata) in entries {
|
||||
results.push(self.add_entry_with_registry(
|
||||
path,
|
||||
uuid,
|
||||
metadata,
|
||||
FileTypeRegistry::builtin(),
|
||||
)?);
|
||||
results.push(self.add_entry_with_registry(path, uuid, metadata, ®istry)?);
|
||||
}
|
||||
|
||||
Ok(results)
|
||||
|
||||
@@ -13,16 +13,28 @@
|
||||
//! decision that needs it, which is the same rule the ladder in
|
||||
//! `sd_store::content` exists to keep: never delete one copy of two on the
|
||||
//! strength of a guess.
|
||||
//!
|
||||
//! The pass also decides the content's kind, since it is the one place that
|
||||
//! opens every file. The file extension answers for a built-in kind. A file
|
||||
//! an extension kind claims with magic patterns, or one no type claims, has
|
||||
//! its first bytes checked against the extension kinds' patterns; the
|
||||
//! built-in patterns stay unused as before. Both the kind and, for an
|
||||
//! extension kind, its name land on the content row.
|
||||
|
||||
use crate::{
|
||||
domain::content_identity::ContentHashGenerator,
|
||||
domain::{content_identity::ContentHashGenerator, ContentKind},
|
||||
filetype::{registry::MAX_MAGIC_BYTES, FileTypeRegistry},
|
||||
infra::job::{generic_progress::GenericProgress, prelude::*, types::JobPriority},
|
||||
ops::indexing::SourceStore,
|
||||
};
|
||||
use futures::StreamExt;
|
||||
use sd_store::ContentIdentity;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{path::PathBuf, sync::Arc};
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::Arc,
|
||||
};
|
||||
use tokio::io::AsyncReadExt;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Files claimed from the store per pass. Large enough that the queue round
|
||||
@@ -166,6 +178,7 @@ impl JobHandler for ContentIdentityJob {
|
||||
|
||||
let mut identified = 0u64;
|
||||
let mut unreadable = 0u64;
|
||||
let registry = FileTypeRegistry::current();
|
||||
|
||||
loop {
|
||||
ctx.check_interrupt().await?;
|
||||
@@ -180,7 +193,7 @@ impl JobHandler for ContentIdentityJob {
|
||||
|
||||
for chunk in batch.chunks(PROGRESS_CHUNK) {
|
||||
ctx.check_interrupt().await?;
|
||||
let (identities, failures) = hash_batch(chunk.to_vec()).await;
|
||||
let (identities, failures) = hash_batch(chunk.to_vec(), ®istry).await;
|
||||
|
||||
// Failures leave the pending set with their reason recorded,
|
||||
// so the loop always advances and the store can say
|
||||
@@ -240,19 +253,24 @@ pub(crate) fn source_label(root: &std::path::Path) -> String {
|
||||
/// on every pass.
|
||||
async fn hash_batch(
|
||||
batch: Vec<(Uuid, PathBuf, u64)>,
|
||||
registry: &FileTypeRegistry,
|
||||
) -> (Vec<(Uuid, ContentIdentity)>, Vec<(Uuid, String)>) {
|
||||
let results: Vec<_> = futures::stream::iter(batch)
|
||||
.map(|(uuid, path, size)| async move {
|
||||
match ContentHashGenerator::generate_content_hash(&path).await {
|
||||
Ok(hash) => Ok((
|
||||
uuid,
|
||||
ContentIdentity {
|
||||
sampled_hash: Some(hash),
|
||||
integrity_hash: None,
|
||||
size: Some(size as i64),
|
||||
kind: None,
|
||||
},
|
||||
)),
|
||||
Ok(hash) => {
|
||||
let (kind, kind_name) = identify_kind(&path, registry).await;
|
||||
Ok((
|
||||
uuid,
|
||||
ContentIdentity {
|
||||
sampled_hash: Some(hash),
|
||||
integrity_hash: None,
|
||||
size: Some(size as i64),
|
||||
kind,
|
||||
kind_name,
|
||||
},
|
||||
))
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::debug!(path = %path.display(), %error, "could not hash");
|
||||
Err((uuid, error.to_string()))
|
||||
@@ -274,6 +292,36 @@ async fn hash_batch(
|
||||
(identities, failures)
|
||||
}
|
||||
|
||||
/// The kind a file's name and, when an extension kind's patterns are in
|
||||
/// play, its first bytes decide: the `ContentKind` discriminant to store,
|
||||
/// and the extension kind's id when one claimed the file.
|
||||
///
|
||||
/// Bytes are read only when the registry has patterns to check for this
|
||||
/// extension; a plain `.jpg` costs no second open. A header that cannot be
|
||||
/// read keeps the extension result, since the hash just succeeded and the
|
||||
/// file was there a moment ago.
|
||||
async fn identify_kind(path: &Path, registry: &FileTypeRegistry) -> (Option<i64>, Option<String>) {
|
||||
let by_extension = registry.type_by_extension(path);
|
||||
let extension = path.extension().and_then(|e| e.to_str());
|
||||
let candidates = registry.magic_candidates(extension);
|
||||
let file_type = if candidates.is_empty() {
|
||||
by_extension
|
||||
} else {
|
||||
let mut header = vec![0u8; MAX_MAGIC_BYTES];
|
||||
let read = match tokio::fs::File::open(path).await {
|
||||
Ok(mut file) => file.read(&mut header).await.unwrap_or(0),
|
||||
Err(_) => 0,
|
||||
};
|
||||
header.truncate(read);
|
||||
registry.resolve_by_magic(by_extension, &candidates, &header)
|
||||
};
|
||||
let Some(file_type) = file_type else {
|
||||
return (None, None);
|
||||
};
|
||||
let kind = (file_type.category != ContentKind::Unknown).then_some(file_type.category as i64);
|
||||
(kind, file_type.kind_name.clone())
|
||||
}
|
||||
|
||||
/// Queue the hashing of every source on this machine, behind whatever else is
|
||||
/// running.
|
||||
///
|
||||
@@ -323,10 +371,13 @@ mod tests {
|
||||
let real = dir.path().join("real.bin");
|
||||
std::fs::write(&real, vec![7u8; 4096]).unwrap();
|
||||
|
||||
let (identities, failures) = hash_batch(vec![
|
||||
(Uuid::now_v7(), dir.path().join("gone.bin"), 10),
|
||||
(Uuid::now_v7(), real, 4096),
|
||||
])
|
||||
let (identities, failures) = hash_batch(
|
||||
vec![
|
||||
(Uuid::now_v7(), dir.path().join("gone.bin"), 10),
|
||||
(Uuid::now_v7(), real, 4096),
|
||||
],
|
||||
FileTypeRegistry::builtin(),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(identities.len(), 1);
|
||||
@@ -342,10 +393,10 @@ mod tests {
|
||||
std::fs::write(&one, vec![3u8; 8192]).unwrap();
|
||||
std::fs::write(&two, vec![3u8; 8192]).unwrap();
|
||||
|
||||
let (identities, _) = hash_batch(vec![
|
||||
(Uuid::now_v7(), one, 8192),
|
||||
(Uuid::now_v7(), two, 8192),
|
||||
])
|
||||
let (identities, _) = hash_batch(
|
||||
vec![(Uuid::now_v7(), one, 8192), (Uuid::now_v7(), two, 8192)],
|
||||
FileTypeRegistry::builtin(),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(identities.len(), 2);
|
||||
@@ -354,4 +405,46 @@ mod tests {
|
||||
"two copies of the same bytes are one content row"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn a_built_in_kind_is_stored_and_an_extension_kind_reads_the_header() {
|
||||
use crate::filetype::{ExtensionKind, MagicPatternSpec};
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let photo = dir.path().join("a.jpg");
|
||||
std::fs::write(&photo, vec![0xFF; 4096]).unwrap();
|
||||
let nothing = dir.path().join("b.zzzkind");
|
||||
std::fs::write(¬hing, b"ZZKIND rest of file".repeat(300)).unwrap();
|
||||
let other = dir.path().join("c.zzzkind");
|
||||
std::fs::write(&other, vec![0u8; 4096]).unwrap();
|
||||
|
||||
let builtin = FileTypeRegistry::builtin();
|
||||
assert_eq!(identify_kind(&photo, builtin).await, (Some(1), None));
|
||||
assert_eq!(identify_kind(¬hing, builtin).await, (None, None));
|
||||
|
||||
let layered = FileTypeRegistry::with_extension_kinds(&[(
|
||||
"ext".into(),
|
||||
vec![ExtensionKind {
|
||||
name: "zz".into(),
|
||||
display_name: None,
|
||||
parent: ContentKind::Text,
|
||||
extensions: vec!["zzzkind".into()],
|
||||
mime_types: Vec::new(),
|
||||
magic: vec![MagicPatternSpec {
|
||||
pattern: "5A 5A 4B 49 4E 44".into(),
|
||||
offset: 0,
|
||||
}],
|
||||
preview: None,
|
||||
}],
|
||||
)]);
|
||||
assert_eq!(
|
||||
identify_kind(¬hing, &layered).await,
|
||||
(Some(ContentKind::Text as i64), Some("ext:zz".into()))
|
||||
);
|
||||
assert_eq!(
|
||||
identify_kind(&other, &layered).await,
|
||||
(Some(ContentKind::Text as i64), Some("ext:zz".into())),
|
||||
"no magic match keeps the extension result"
|
||||
);
|
||||
assert_eq!(identify_kind(&photo, &layered).await, (Some(1), None));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
//! Naming stored content rows after the kinds loaded extensions declare.
|
||||
//!
|
||||
//! The content identity phase writes a row's kind when it reads the bytes.
|
||||
//! Rows identified before an extension existed carry a built-in kind, or
|
||||
//! nothing at all, and the bytes are not read again for them: one update
|
||||
//! per declared kind names every row whose file carries one of the kind's
|
||||
//! extensions. The pass runs when a store opens, so every store sees the
|
||||
//! kinds loaded at startup, and when an extension with kinds loads while
|
||||
//! stores are open. It is idempotent, so running it twice costs one indexed
|
||||
//! probe per kind and changes nothing.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::filetype::FileTypeRegistry;
|
||||
use crate::ops::indexing::VolumeIndex;
|
||||
|
||||
/// Name the rows of one store after every extension kind the registry
|
||||
/// holds. Returns how many rows gained a name.
|
||||
pub async fn name_kinds_in_store(
|
||||
db: &sd_store::SourceDb,
|
||||
registry: &FileTypeRegistry,
|
||||
) -> Result<u64, sd_store::Error> {
|
||||
let mut named = 0;
|
||||
for (kind, extensions) in registry.extension_kinds() {
|
||||
let Some(kind_name) = kind.kind_name.as_deref() else {
|
||||
continue;
|
||||
};
|
||||
named += sd_store::name_content_kind_by_extension(
|
||||
db.pool(),
|
||||
kind_name,
|
||||
kind.category as i32 as i64,
|
||||
&extensions,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
Ok(named)
|
||||
}
|
||||
|
||||
/// Run the pass over every store this machine has open.
|
||||
pub async fn name_kinds_in_open_stores(volume_index: &Arc<VolumeIndex>) {
|
||||
let registry = FileTypeRegistry::current();
|
||||
if registry.extension_kinds().is_empty() {
|
||||
return;
|
||||
}
|
||||
for store in volume_index.open_stores() {
|
||||
match name_kinds_in_store(store.db(), ®istry).await {
|
||||
Ok(0) => {}
|
||||
Ok(named) => {
|
||||
tracing::info!(source = %store.id(), named, "named content rows after extension kinds")
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::warn!(source = %store.id(), %error, "could not name content rows after extension kinds")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -17,6 +17,7 @@ pub mod descriptor;
|
||||
pub mod handlers;
|
||||
pub mod input;
|
||||
pub mod job;
|
||||
pub mod kinds;
|
||||
pub mod lens;
|
||||
pub mod metadata;
|
||||
pub mod metrics;
|
||||
|
||||
@@ -138,6 +138,20 @@ impl SourceStore {
|
||||
.with_context(|| format!("load ledger for {id}"))?;
|
||||
tracing::debug!(source = %id, records = ledger.len(), "source store attached");
|
||||
|
||||
// Rows identified before the loaded extensions existed take their
|
||||
// kinds now, so a store opened after startup is never behind them.
|
||||
match super::kinds::name_kinds_in_store(&db, &crate::filetype::FileTypeRegistry::current())
|
||||
.await
|
||||
{
|
||||
Ok(0) => {}
|
||||
Ok(named) => {
|
||||
tracing::info!(source = %id, named, "named content rows after extension kinds")
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::warn!(source = %id, %error, "could not name content rows after extension kinds")
|
||||
}
|
||||
}
|
||||
|
||||
let (tx, rx) = mpsc::channel(QUEUE_DEPTH);
|
||||
tokio::spawn(write_loop(db.clone(), ledger, rx));
|
||||
|
||||
@@ -1364,6 +1378,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: None,
|
||||
size: Some(5),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)])
|
||||
.await;
|
||||
@@ -1528,6 +1543,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: None,
|
||||
size: Some(*size as i64),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
})
|
||||
@@ -1580,6 +1596,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: None,
|
||||
size: Some(*size as i64),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
})
|
||||
@@ -1624,6 +1641,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: Some("integrity-hash".to_string()),
|
||||
size: Some(*size as i64),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
})
|
||||
@@ -1726,6 +1744,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: None,
|
||||
size: Some(size),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
};
|
||||
let identified = pending
|
||||
.iter()
|
||||
@@ -1776,6 +1795,7 @@ COALESCE(own.path, parent.path || '/' || r.title, r.title)
|
||||
integrity_hash: None,
|
||||
size: Some(4),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
};
|
||||
|
||||
@@ -196,6 +196,7 @@ async fn verify_batch(
|
||||
integrity_hash: Some(hash),
|
||||
size: Some(size as i64),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
size,
|
||||
)),
|
||||
|
||||
@@ -1352,6 +1352,11 @@ impl VolumeIndex {
|
||||
Some(store)
|
||||
}
|
||||
|
||||
/// Every writable store this machine has opened so far.
|
||||
pub fn open_stores(&self) -> Vec<Arc<SourceStore>> {
|
||||
self.stores.read().values().cloned().collect()
|
||||
}
|
||||
|
||||
/// A read-only handle to a source's store, cached per source.
|
||||
///
|
||||
/// `None` means the source has no store on disk or it refused to open,
|
||||
|
||||
@@ -128,7 +128,7 @@ impl LibraryQuery for MediaSearchQuery {
|
||||
};
|
||||
|
||||
let registry = context.file_type_registry();
|
||||
let extensions = media_extensions(registry, &input.filters);
|
||||
let extensions = media_extensions(®istry, &input.filters);
|
||||
if extensions.is_empty() {
|
||||
return Ok(finished);
|
||||
}
|
||||
@@ -159,7 +159,7 @@ impl LibraryQuery for MediaSearchQuery {
|
||||
needle: input.query.to_lowercase(),
|
||||
filters: &input.filters,
|
||||
tags: tags.as_ref(),
|
||||
registry,
|
||||
registry: ®istry,
|
||||
extensions: &extensions,
|
||||
device_slug: crate::device::get_current_device_slug(),
|
||||
};
|
||||
@@ -534,6 +534,7 @@ mod tests {
|
||||
sampled_hash: None,
|
||||
integrity_hash: None,
|
||||
content_kind: None,
|
||||
content_kind_name: None,
|
||||
content_error: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,7 +81,7 @@ impl FileSearchQuery {
|
||||
&self.input.pagination,
|
||||
&context,
|
||||
cache,
|
||||
registry,
|
||||
®istry,
|
||||
)
|
||||
.await?
|
||||
}
|
||||
@@ -93,7 +93,7 @@ impl FileSearchQuery {
|
||||
&self.input.pagination,
|
||||
&context,
|
||||
cache,
|
||||
registry,
|
||||
®istry,
|
||||
)
|
||||
.await?
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
//! other backend. Matching, filters and scoring mirror the arena path so the
|
||||
//! two backends return the same hits for the same capture.
|
||||
|
||||
use crate::domain::{File, SdPath};
|
||||
use crate::domain::{ContentKind, File, SdPath};
|
||||
use crate::filetype::FileTypeRegistry;
|
||||
use crate::infra::query::QueryError;
|
||||
use crate::ops::search::input::{DateField, SearchFilters};
|
||||
@@ -146,7 +146,15 @@ pub(super) fn passes_store_filters(
|
||||
}
|
||||
|
||||
if let Some(ref content_types) = filters.content_types {
|
||||
let identified_kind = file_type_registry.identify_by_extension(absolute);
|
||||
// The stored kind is what the listing shows, so the filter agrees
|
||||
// with it; a row identified before kinds were stored derives one
|
||||
// from the name as the listing does.
|
||||
let identified_kind = entry
|
||||
.content_kind
|
||||
.and_then(|kind| i32::try_from(kind).ok())
|
||||
.map(ContentKind::from_id)
|
||||
.filter(|kind| *kind != ContentKind::Unknown)
|
||||
.unwrap_or_else(|| file_type_registry.identify_by_extension(absolute));
|
||||
if !content_types.contains(&identified_kind) {
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -230,6 +230,7 @@ fn record_to_file(
|
||||
modified_at: captured_at,
|
||||
accessed_at: None,
|
||||
content_kind,
|
||||
content_kind_name: None,
|
||||
is_local: true,
|
||||
duration_seconds: None,
|
||||
thumbnail_path: str_field("thumb_path").map(|p| absolutize(PathBuf::from(p))),
|
||||
|
||||
@@ -94,6 +94,7 @@ async fn dedupe_keeps_a_file_whose_own_bytes_were_never_read_in_full(
|
||||
integrity_hash: Some(integrity_of_a.clone()),
|
||||
size: Some(4096),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
@@ -106,6 +107,7 @@ async fn dedupe_keeps_a_file_whose_own_bytes_were_never_read_in_full(
|
||||
integrity_hash: None,
|
||||
size: Some(4096),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"id": "zz-second-kind",
|
||||
"name": "Second kind",
|
||||
"version": "0.1.0",
|
||||
"description": "Test fixture: a second extension claiming the .fake extension, loaded after the test extension so its claim is contested. Runs the test extension's module.",
|
||||
"wasm_file": "test_extension.wasm",
|
||||
"kinds": [
|
||||
{
|
||||
"name": "other",
|
||||
"display_name": "Other fake file",
|
||||
"parent": "text",
|
||||
"extensions": ["fake", "fake2"],
|
||||
"magic": [{ "pattern": "4F 54 48 52", "offset": 0 }],
|
||||
"preview": { "renderer": "text" }
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -15,12 +15,18 @@ use std::sync::{Arc, Mutex, OnceLock};
|
||||
use std::time::Duration;
|
||||
|
||||
use sd_core::{
|
||||
domain::{ContentKind, File, SdPath},
|
||||
filetype::{FileTypeRegistry, KindConflict, PreviewSpec},
|
||||
infra::{
|
||||
action::LibraryAction,
|
||||
api::SessionContext,
|
||||
job::{database::checkpoint, types::JobStatus},
|
||||
query::CoreQuery,
|
||||
},
|
||||
ops::{
|
||||
extensions::{RunExtensionJobAction, RunExtensionJobInput},
|
||||
extensions::{
|
||||
ListExtensionsInput, ListExtensionsQuery, RunExtensionJobAction, RunExtensionJobInput,
|
||||
},
|
||||
sources::track::{TrackSourceAction, TrackSourceInput},
|
||||
},
|
||||
Core,
|
||||
@@ -55,6 +61,87 @@ fn install_test_extension(data_dir: &Path) {
|
||||
install_extension(data_dir, "test-extension", "test_extension.wasm");
|
||||
}
|
||||
|
||||
/// A second extension claiming the test extension's `.fake` kind, under a
|
||||
/// directory name that sorts after it. It runs the test extension's module;
|
||||
/// only its manifest differs.
|
||||
fn install_second_kind_extension(data_dir: &Path) {
|
||||
let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("tests/fixtures/extensions/zz-second-kind/manifest.json");
|
||||
let wasm = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
|
||||
.parent()
|
||||
.unwrap()
|
||||
.join("extensions/test-extension/test_extension.wasm");
|
||||
let target = data_dir.join("extensions/zz-second-kind");
|
||||
std::fs::create_dir_all(&target).unwrap();
|
||||
std::fs::copy(manifest, target.join("manifest.json")).unwrap();
|
||||
std::fs::copy(wasm, target.join("test_extension.wasm")).unwrap();
|
||||
}
|
||||
|
||||
/// Track `dir` in `library` and wait until every file in it has a content
|
||||
/// identity. Returns the store and its file entries by name.
|
||||
async fn track_and_identify(
|
||||
core: &Core,
|
||||
library: &Arc<sd_core::library::Library>,
|
||||
dir: PathBuf,
|
||||
expected: u64,
|
||||
) -> (
|
||||
Arc<sd_core::ops::indexing::SourceStore>,
|
||||
Vec<sd_store::FsEntry>,
|
||||
) {
|
||||
let tracked = TrackSourceAction::from_input(TrackSourceInput {
|
||||
path: dir,
|
||||
name: None,
|
||||
overrides: Default::default(),
|
||||
})
|
||||
.unwrap()
|
||||
.execute(library.clone(), core.context.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
let store = core
|
||||
.context
|
||||
.volume_index()
|
||||
.store_for(&tracked.root)
|
||||
.await
|
||||
.expect("the tracked source has a store");
|
||||
let deadline = tokio::time::Instant::now() + Duration::from_secs(60);
|
||||
loop {
|
||||
let contents = store.counts().await.map_or(0, |counts| counts.contents);
|
||||
if contents == expected {
|
||||
break;
|
||||
}
|
||||
assert!(
|
||||
tokio::time::Instant::now() < deadline,
|
||||
"{contents} of {expected} files were hashed"
|
||||
);
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
}
|
||||
(store.clone(), store_files(&store).await)
|
||||
}
|
||||
|
||||
async fn store_files(store: &sd_core::ops::indexing::SourceStore) -> Vec<sd_store::FsEntry> {
|
||||
store.flush().await.unwrap();
|
||||
let mut files = sd_store::read::files_beneath(
|
||||
store.db().pool(),
|
||||
"",
|
||||
sd_store::read::Start::First,
|
||||
None,
|
||||
false,
|
||||
100,
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
files.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
files
|
||||
}
|
||||
|
||||
fn kind_of(files: &[sd_store::FsEntry], name: &str) -> (Option<i64>, Option<String>) {
|
||||
let entry = files
|
||||
.iter()
|
||||
.find(|f| f.name == name)
|
||||
.unwrap_or_else(|| panic!("{name} is in the store"));
|
||||
(entry.content_kind, entry.content_kind_name.clone())
|
||||
}
|
||||
|
||||
/// Start an extension job and wait for it to end, whichever way.
|
||||
async fn run_to_end(
|
||||
core: &Core,
|
||||
@@ -671,3 +758,195 @@ fn walk(dir: &Path) -> Vec<PathBuf> {
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The test extension declares a `fake` kind over `.fake` with magic
|
||||
/// bytes, and a second fixture loaded after it contests the extension. The
|
||||
/// content identity phase stores the kind and its name for every file,
|
||||
/// built-in kinds included; the row keeps its name after the extension
|
||||
/// unloads while the registry forgets it; rows identified before an
|
||||
/// extension existed are named when it loads and when a store opens after
|
||||
/// a restart.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn extension_kinds_are_stored_and_survive_unload() {
|
||||
guest_log();
|
||||
let temp_dir = TempDir::new().unwrap();
|
||||
let data_dir = temp_dir.path().join("core");
|
||||
install_test_extension(&data_dir);
|
||||
install_second_kind_extension(&data_dir);
|
||||
let core = Core::new(data_dir.clone()).await.unwrap();
|
||||
let pm = core
|
||||
.plugin_manager
|
||||
.as_ref()
|
||||
.expect("plugin manager")
|
||||
.clone();
|
||||
|
||||
// Both loaded, in directory order; the list reports the kinds and the
|
||||
// one contested claim.
|
||||
let session =
|
||||
SessionContext::device_session(Uuid::now_v7(), sd_core::device::get_current_device_slug());
|
||||
let list = ListExtensionsQuery::from_input(ListExtensionsInput {})
|
||||
.unwrap()
|
||||
.execute(core.context.clone(), session.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
let ids: Vec<&str> = list.extensions.iter().map(|e| e.id.as_str()).collect();
|
||||
assert_eq!(ids, ["test-extension", "zz-second-kind"]);
|
||||
let fake = &list.extensions[0].kinds[0];
|
||||
assert_eq!(fake.id, "test-extension:fake");
|
||||
assert_eq!(fake.display_name, "Fake file");
|
||||
assert_eq!(fake.parent, ContentKind::Text);
|
||||
assert_eq!(fake.preview, Some(PreviewSpec::Renderer("text".into())));
|
||||
assert_eq!(
|
||||
list.conflicts,
|
||||
vec![KindConflict {
|
||||
extension: "fake".into(),
|
||||
kind: "zz-second-kind:other".into(),
|
||||
claimed_by: "test-extension:fake".into(),
|
||||
}]
|
||||
);
|
||||
assert_eq!(
|
||||
FileTypeRegistry::current()
|
||||
.type_by_extension(Path::new("x.fake2"))
|
||||
.unwrap()
|
||||
.id,
|
||||
"zz-second-kind:other",
|
||||
"the loser keeps its uncontested extension"
|
||||
);
|
||||
|
||||
// Index a folder: the identity phase writes kind and kind_name.
|
||||
let library = core
|
||||
.libraries
|
||||
.create_library("Kinds", None, core.context.clone())
|
||||
.await
|
||||
.unwrap();
|
||||
let first = temp_dir.path().join("first");
|
||||
std::fs::create_dir_all(&first).unwrap();
|
||||
std::fs::write(first.join("a.fake"), b"FAKE".repeat(300)).unwrap();
|
||||
std::fs::write(first.join("b.fake"), b"OTHR".repeat(300)).unwrap();
|
||||
std::fs::write(first.join("c.fake"), vec![0u8; 1200]).unwrap();
|
||||
let mut jpeg = vec![0xFF, 0xD8, 0xFF, 0xE0];
|
||||
jpeg.extend(std::iter::repeat_n(7u8, 1200));
|
||||
std::fs::write(first.join("d.jpg"), jpeg).unwrap();
|
||||
std::fs::write(first.join("e.txt"), b"plain text".repeat(100)).unwrap();
|
||||
let (first_store, files) = track_and_identify(&core, &library, first.clone(), 5).await;
|
||||
|
||||
let text = ContentKind::Text as i64;
|
||||
assert_eq!(
|
||||
kind_of(&files, "a.fake"),
|
||||
(Some(text), Some("test-extension:fake".into()))
|
||||
);
|
||||
assert_eq!(
|
||||
kind_of(&files, "b.fake"),
|
||||
(Some(text), Some("zz-second-kind:other".into())),
|
||||
"a lone magic match on the contested kind wins the file"
|
||||
);
|
||||
assert_eq!(
|
||||
kind_of(&files, "c.fake"),
|
||||
(Some(text), Some("test-extension:fake".into())),
|
||||
"no magic match keeps the extension holder"
|
||||
);
|
||||
assert_eq!(
|
||||
kind_of(&files, "d.jpg"),
|
||||
(Some(ContentKind::Image as i64), None),
|
||||
"a built-in kind is stored too"
|
||||
);
|
||||
assert_eq!(kind_of(&files, "e.txt"), (Some(text), None));
|
||||
|
||||
let a = files.iter().find(|f| f.name == "a.fake").unwrap();
|
||||
let file = File::from_store_entry(a, SdPath::local(first.join("a.fake")));
|
||||
assert_eq!(file.content_kind, ContentKind::Text);
|
||||
assert_eq!(
|
||||
file.content_kind_name.as_deref(),
|
||||
Some("test-extension:fake")
|
||||
);
|
||||
|
||||
// Unload the holder: the registry moves on, the rows do not.
|
||||
pm.write()
|
||||
.await
|
||||
.unload_plugin("test-extension")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
FileTypeRegistry::current()
|
||||
.type_by_extension(Path::new("x.fake"))
|
||||
.unwrap()
|
||||
.id,
|
||||
"zz-second-kind:other",
|
||||
"the contested claim holds the extension once the winner is gone"
|
||||
);
|
||||
let files = store_files(&first_store).await;
|
||||
assert_eq!(
|
||||
kind_of(&files, "a.fake"),
|
||||
(Some(text), Some("test-extension:fake".into()))
|
||||
);
|
||||
let a = files.iter().find(|f| f.name == "a.fake").unwrap();
|
||||
let file = File::from_store_entry(a, SdPath::local(first.join("a.fake")));
|
||||
assert_eq!(file.content_kind, ContentKind::Text, "the stored parent");
|
||||
assert_eq!(
|
||||
file.content_kind_name.as_deref(),
|
||||
Some("test-extension:fake")
|
||||
);
|
||||
|
||||
pm.write()
|
||||
.await
|
||||
.unload_plugin("zz-second-kind")
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
FileTypeRegistry::current()
|
||||
.type_by_extension(Path::new("x.fake"))
|
||||
.is_none(),
|
||||
"no extension loaded, the built-in registry is back"
|
||||
);
|
||||
assert!(FileTypeRegistry::current().conflicts().is_empty());
|
||||
|
||||
// Rows identified with no extension loaded have no kind; the load that
|
||||
// follows names them by extension without reading the bytes.
|
||||
let second = temp_dir.path().join("second");
|
||||
std::fs::create_dir_all(&second).unwrap();
|
||||
std::fs::write(second.join("f.FAKE"), vec![1u8; 1200]).unwrap();
|
||||
std::fs::write(second.join("g.txt"), b"plain".repeat(300)).unwrap();
|
||||
let (second_store, files) = track_and_identify(&core, &library, second.clone(), 2).await;
|
||||
assert_eq!(kind_of(&files, "f.FAKE"), (None, None));
|
||||
assert_eq!(kind_of(&files, "g.txt"), (Some(text), None));
|
||||
|
||||
pm.write()
|
||||
.await
|
||||
.load_plugin("test-extension")
|
||||
.await
|
||||
.unwrap();
|
||||
let files = store_files(&second_store).await;
|
||||
assert_eq!(
|
||||
kind_of(&files, "f.FAKE"),
|
||||
(Some(text), Some("test-extension:fake".into())),
|
||||
"a load with kinds names the rows open stores hold, whatever the case of the extension"
|
||||
);
|
||||
assert_eq!(kind_of(&files, "g.txt"), (Some(text), None));
|
||||
|
||||
// A store opened after a restart is named by the kinds loaded at startup.
|
||||
pm.write()
|
||||
.await
|
||||
.unload_plugin("test-extension")
|
||||
.await
|
||||
.unwrap();
|
||||
let third = temp_dir.path().join("third");
|
||||
std::fs::create_dir_all(&third).unwrap();
|
||||
std::fs::write(third.join("h.fake"), vec![2u8; 1200]).unwrap();
|
||||
let (_, files) = track_and_identify(&core, &library, third.clone(), 1).await;
|
||||
assert_eq!(kind_of(&files, "h.fake"), (None, None));
|
||||
core.shutdown().await.unwrap();
|
||||
|
||||
let core = Core::new(data_dir).await.unwrap();
|
||||
let store = core
|
||||
.context
|
||||
.volume_index()
|
||||
.store_for(&third)
|
||||
.await
|
||||
.expect("the third source reopens");
|
||||
let files = store_files(&store).await;
|
||||
assert_eq!(
|
||||
kind_of(&files, "h.fake"),
|
||||
(Some(text), Some("test-extension:fake".into()))
|
||||
);
|
||||
core.shutdown().await.unwrap();
|
||||
}
|
||||
+56
-14
@@ -1235,6 +1235,18 @@ impl NeighborRow {
|
||||
}
|
||||
}
|
||||
|
||||
/// The row a record pointed at before a content write, for what the write
|
||||
/// has to carry across.
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct PreviousContent {
|
||||
id: i64,
|
||||
uuid: Uuid,
|
||||
sampled_hash: Option<String>,
|
||||
integrity_hash: Option<String>,
|
||||
kind: Option<i64>,
|
||||
kind_name: Option<String>,
|
||||
}
|
||||
|
||||
/// The content write itself, against whatever connection the caller holds: a
|
||||
/// pooled one for a single file, a transaction for a batch.
|
||||
///
|
||||
@@ -1264,8 +1276,8 @@ async fn bind_content(
|
||||
.as_deref()
|
||||
.map(crate::content::uuid_for);
|
||||
|
||||
let previous: Option<(i64, Uuid, Option<String>, Option<String>)> = sqlx::query_as(
|
||||
"SELECT c.id, c.uuid, c.sampled_hash, c.integrity_hash
|
||||
let previous: Option<PreviousContent> = sqlx::query_as(
|
||||
"SELECT c.id, c.uuid, c.sampled_hash, c.integrity_hash, c.kind, c.kind_name
|
||||
FROM record r JOIN content c ON c.id = r.content_id WHERE r.uuid = ?",
|
||||
)
|
||||
.bind(uuid)
|
||||
@@ -1275,21 +1287,42 @@ async fn bind_content(
|
||||
// A sampled-only write says nothing against a full read of the same
|
||||
// bytes, so a record already confirmed under this sampled hash stays
|
||||
// where it is rather than walking back down to a guess.
|
||||
if let Some((old_id, _, old_sampled, Some(_))) = &previous {
|
||||
if identity.integrity_hash.is_none() && *old_sampled == identity.sampled_hash {
|
||||
return Ok(*old_id);
|
||||
if let Some(old) = &previous {
|
||||
if old.integrity_hash.is_some()
|
||||
&& identity.integrity_hash.is_none()
|
||||
&& old.sampled_hash == identity.sampled_hash
|
||||
{
|
||||
return Ok(old.id);
|
||||
}
|
||||
}
|
||||
|
||||
// The kind travels with the record when the write describes the same
|
||||
// bytes: a verification pass carries no kind of its own, and the
|
||||
// confirmed row it moves the record to must not lose the one the
|
||||
// identity phase read. A write with a new hash is a changed file, whose
|
||||
// kind the old bytes say nothing about.
|
||||
let same_bytes = previous.as_ref().filter(|old| {
|
||||
(old.sampled_hash.is_some() && old.sampled_hash == identity.sampled_hash)
|
||||
|| (old.integrity_hash.is_some() && old.integrity_hash == identity.integrity_hash)
|
||||
});
|
||||
let kind = identity
|
||||
.kind
|
||||
.or_else(|| same_bytes.and_then(|old| old.kind));
|
||||
let kind_name = identity
|
||||
.kind_name
|
||||
.clone()
|
||||
.or_else(|| same_bytes.and_then(|old| old.kind_name.clone()));
|
||||
|
||||
let content_id: i64 = match identity.integrity_hash.as_deref() {
|
||||
Some(integrity) => sqlx::query_scalar(
|
||||
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind, kind_name)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (integrity_hash) DO UPDATE SET
|
||||
candidate_uuid = COALESCE(content.candidate_uuid, excluded.candidate_uuid),
|
||||
sampled_hash = COALESCE(content.sampled_hash, excluded.sampled_hash),
|
||||
size = COALESCE(excluded.size, content.size),
|
||||
kind = COALESCE(excluded.kind, content.kind)
|
||||
kind = COALESCE(excluded.kind, content.kind),
|
||||
kind_name = COALESCE(excluded.kind_name, content.kind_name)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(content_uuid)
|
||||
@@ -1297,22 +1330,25 @@ async fn bind_content(
|
||||
.bind(&identity.sampled_hash)
|
||||
.bind(integrity)
|
||||
.bind(identity.size)
|
||||
.bind(identity.kind)
|
||||
.bind(kind)
|
||||
.bind(&kind_name)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?,
|
||||
None => sqlx::query_scalar(
|
||||
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind)
|
||||
VALUES (?, ?, ?, NULL, ?, ?)
|
||||
"INSERT INTO content (uuid, candidate_uuid, sampled_hash, integrity_hash, size, kind, kind_name)
|
||||
VALUES (?, ?, ?, NULL, ?, ?, ?)
|
||||
ON CONFLICT (sampled_hash) WHERE integrity_hash IS NULL DO UPDATE SET
|
||||
size = COALESCE(excluded.size, content.size),
|
||||
kind = COALESCE(excluded.kind, content.kind)
|
||||
kind = COALESCE(excluded.kind, content.kind),
|
||||
kind_name = COALESCE(excluded.kind_name, content.kind_name)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(content_uuid)
|
||||
.bind(candidate_uuid)
|
||||
.bind(&identity.sampled_hash)
|
||||
.bind(identity.size)
|
||||
.bind(identity.kind)
|
||||
.bind(kind)
|
||||
.bind(&kind_name)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?,
|
||||
};
|
||||
@@ -1323,7 +1359,13 @@ async fn bind_content(
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
|
||||
let Some((old_id, old_uuid, _, old_integrity)) = previous else {
|
||||
let Some(PreviousContent {
|
||||
id: old_id,
|
||||
uuid: old_uuid,
|
||||
integrity_hash: old_integrity,
|
||||
..
|
||||
}) = previous
|
||||
else {
|
||||
return Ok(content_id);
|
||||
};
|
||||
if old_id == content_id {
|
||||
|
||||
@@ -922,6 +922,37 @@ pub async fn mark_content_unreadable(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Name the kind of every content row whose files carry one of these
|
||||
/// extensions and that no extension kind has named yet.
|
||||
///
|
||||
/// This is the reidentification an extension's load runs over rows that
|
||||
/// were identified before it existed: one statement per declared kind, by
|
||||
/// extension, since the bytes are not read again. A row another extension
|
||||
/// already named keeps that name; magic-only claims wait for the next
|
||||
/// identity pass over those files. Returns the rows named.
|
||||
pub async fn name_content_kind_by_extension(
|
||||
pool: &sqlx::SqlitePool,
|
||||
kind_name: &str,
|
||||
kind: i64,
|
||||
extensions: &[String],
|
||||
) -> Result<u64> {
|
||||
if extensions.is_empty() {
|
||||
return Ok(0);
|
||||
}
|
||||
let sql = format!(
|
||||
"UPDATE content SET kind_name = ?, kind = ? WHERE kind_name IS NULL AND id IN (
|
||||
SELECT r.content_id FROM record r
|
||||
JOIN facet_file f ON f.record_uuid = r.uuid
|
||||
WHERE r.content_id IS NOT NULL AND lower(f.extension) IN ({}))",
|
||||
vec!["?"; extensions.len()].join(", ")
|
||||
);
|
||||
let mut query = sqlx::query(&sql).bind(kind_name).bind(kind);
|
||||
for ext in extensions {
|
||||
query = query.bind(ext.to_lowercase());
|
||||
}
|
||||
Ok(query.execute(pool).await?.rows_affected())
|
||||
}
|
||||
|
||||
/// One copy of some bytes: a record, and where it is.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ContentCopy {
|
||||
|
||||
@@ -62,8 +62,8 @@ pub use error::{Error, Result};
|
||||
pub use file::{
|
||||
content_of, copies_of_content, count_files_needing_content, count_files_needing_verification,
|
||||
duplicate_copies, files_needing_content, files_needing_verification, filesystem_schema,
|
||||
mark_content_unreadable, ContentCopy, FileKind, FileWrite, Ledger, Observation, PendingContent,
|
||||
PendingVerification, Resolution, SubtreeRename, Watermark,
|
||||
mark_content_unreadable, name_content_kind_by_extension, ContentCopy, FileKind, FileWrite,
|
||||
Ledger, Observation, PendingContent, PendingVerification, Resolution, SubtreeRename, Watermark,
|
||||
};
|
||||
pub use migrate::SCHEMA_VERSION;
|
||||
pub use read::{FsEntry, TitleMatches};
|
||||
|
||||
@@ -28,7 +28,11 @@ use crate::error::{Error, Result};
|
||||
/// 1. `content` rebuilt so a row's integrity hash belongs to every record on
|
||||
/// it: candidate rows keyed by sampled hash, confirmed rows by integrity
|
||||
/// hash, `candidate_uuid` beside `uuid`.
|
||||
pub const SCHEMA_VERSION: i64 = 1;
|
||||
/// 2. `content.kind_name` for the extension kind that identified the bytes,
|
||||
/// nullable and never backfilled, and an index over the lowercase file
|
||||
/// extension so a reidentification pass by extension does not walk the
|
||||
/// facet table.
|
||||
pub const SCHEMA_VERSION: i64 = 2;
|
||||
|
||||
/// The version a store carries. `0` for a store written before versions
|
||||
/// existed.
|
||||
@@ -76,6 +80,7 @@ pub(crate) async fn run(pool: &SqlitePool) -> Result<()> {
|
||||
async fn apply(pool: &SqlitePool, next: i64) -> Result<()> {
|
||||
match next {
|
||||
1 => content_rows_own_their_hash(pool).await,
|
||||
2 => content_rows_name_their_kind(pool).await,
|
||||
_ => Err(Error::Other(format!(
|
||||
"no migration defined for schema version {next}"
|
||||
))),
|
||||
@@ -103,6 +108,47 @@ const CONTENT_V1_INDEXES: [&str; 4] = [
|
||||
"CREATE INDEX idx_content_candidate_uuid ON content(candidate_uuid)",
|
||||
];
|
||||
|
||||
/// The index a reidentification pass by extension needs. Lowercase because
|
||||
/// the walk writes the extension as the name carries it and every lookup
|
||||
/// compares lowercase. Created here for a store that predates it and by the
|
||||
/// open path for a fresh one; `IF NOT EXISTS` makes both idempotent.
|
||||
pub const FACET_FILE_EXTENSION_INDEX: &str =
|
||||
"CREATE INDEX IF NOT EXISTS idx_facet_file_extension ON facet_file(lower(extension))";
|
||||
|
||||
/// Migration 2: `content.kind_name`, nullable, with no backfill. Before this
|
||||
/// no row carried a kind at all; the kind was derived from the file name on
|
||||
/// every read. Rows identified from here on get `kind` and `kind_name` at
|
||||
/// the content identity phase, and an extension that loads with kinds runs
|
||||
/// one reidentification update by extension over the rows it can name.
|
||||
async fn content_rows_name_their_kind(pool: &SqlitePool) -> Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let mut tx = conn.begin().await?;
|
||||
let present: Option<i64> =
|
||||
sqlx::query_scalar("SELECT 1 FROM pragma_table_info('content') WHERE name = 'kind_name'")
|
||||
.fetch_optional(&mut *tx)
|
||||
.await?;
|
||||
if present.is_none() {
|
||||
sqlx::query("ALTER TABLE content ADD COLUMN kind_name TEXT")
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
let has_facet_file: Option<i64> = sqlx::query_scalar(
|
||||
"SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'facet_file'",
|
||||
)
|
||||
.fetch_optional(&mut *tx)
|
||||
.await?;
|
||||
if has_facet_file.is_some() {
|
||||
sqlx::query(FACET_FILE_EXTENSION_INDEX)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
sqlx::query("PRAGMA user_version = 2")
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Migration 1: rebuild `content` so that no record carries an integrity
|
||||
/// hash its own bytes were never read for.
|
||||
///
|
||||
|
||||
@@ -53,6 +53,9 @@ pub struct FsEntry {
|
||||
pub sampled_hash: Option<String>,
|
||||
pub integrity_hash: Option<String>,
|
||||
pub content_kind: Option<i64>,
|
||||
/// The extension kind id the identity phase stored, loaded extension or
|
||||
/// not; `content_kind` is then its parent.
|
||||
pub content_kind_name: Option<String>,
|
||||
pub content_error: Option<String>,
|
||||
}
|
||||
|
||||
@@ -80,7 +83,8 @@ macro_rules! entry_columns {
|
||||
COALESCE(f.is_hidden, 0) AS is_hidden, f.extension AS extension, \
|
||||
f.link_target AS link_target, f.inode AS inode, f.mode AS mode, f.uid AS uid, f.gid AS gid, \
|
||||
f.content_error AS content_error, c.uuid AS content_uuid, \
|
||||
c.sampled_hash AS sampled_hash, c.integrity_hash AS integrity_hash, c.kind AS content_kind"
|
||||
c.sampled_hash AS sampled_hash, c.integrity_hash AS integrity_hash, c.kind AS content_kind, \
|
||||
c.kind_name AS content_kind_name"
|
||||
};
|
||||
}
|
||||
|
||||
@@ -130,6 +134,7 @@ struct EntryRow {
|
||||
sampled_hash: Option<String>,
|
||||
integrity_hash: Option<String>,
|
||||
content_kind: Option<i64>,
|
||||
content_kind_name: Option<String>,
|
||||
}
|
||||
|
||||
fn entry_from_row(row: EntryRow) -> Option<FsEntry> {
|
||||
@@ -155,6 +160,7 @@ fn entry_from_row(row: EntryRow) -> Option<FsEntry> {
|
||||
sampled_hash: row.sampled_hash,
|
||||
integrity_hash: row.integrity_hash,
|
||||
content_kind: row.content_kind,
|
||||
content_kind_name: row.content_kind_name,
|
||||
content_error: row.content_error,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -95,7 +95,8 @@ CREATE TABLE IF NOT EXISTS content (
|
||||
sampled_hash TEXT,
|
||||
integrity_hash TEXT UNIQUE,
|
||||
size INTEGER,
|
||||
kind INTEGER
|
||||
kind INTEGER,
|
||||
kind_name TEXT
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_content_candidate ON content(sampled_hash)
|
||||
WHERE integrity_hash IS NULL;
|
||||
@@ -238,7 +239,13 @@ pub struct ContentIdentity {
|
||||
/// Full tier — hash over every byte.
|
||||
pub integrity_hash: Option<String>,
|
||||
pub size: Option<i64>,
|
||||
/// The built-in kind, or an extension kind's parent, as a `ContentKind`
|
||||
/// discriminant.
|
||||
pub kind: Option<i64>,
|
||||
/// The extension kind id (`<extension id>:<name>`) when an extension's
|
||||
/// kind identified the bytes. Kept after the extension unloads, so a
|
||||
/// reader can fall back to `kind` until it returns.
|
||||
pub kind_name: Option<String>,
|
||||
}
|
||||
|
||||
/// Facet table name for a model. Namespaced so a model can never collide with a
|
||||
|
||||
@@ -37,6 +37,13 @@ async fn open_pool(db_path: &Path, create: bool) -> Result<SqlitePool> {
|
||||
/// A pool that can only read. No journal-mode pragma runs: the store's own
|
||||
/// files are already WAL, and a delivered artifact may legitimately carry a
|
||||
/// rollback journal that a read-only connection could not convert anyway.
|
||||
///
|
||||
/// A replica keeps its owner's schema version, and an owner on an older
|
||||
/// build has no `content.kind_name`. The entry readers name that column, so
|
||||
/// each connection to a store below version 2 gets a temporary view named
|
||||
/// `content` that adds it as `NULL`; an unqualified name resolves to the
|
||||
/// temp schema first, so every reader sees the current shape and the file
|
||||
/// is untouched. The owner's upgrade replaces the replica wholesale.
|
||||
async fn open_pool_read_only(db_path: &Path) -> Result<SqlitePool> {
|
||||
let options = SqliteConnectOptions::from_str(&format!("sqlite:{}", db_path.display()))
|
||||
.map_err(|e| Error::Other(format!("invalid database path: {e}")))?
|
||||
@@ -46,6 +53,22 @@ async fn open_pool_read_only(db_path: &Path) -> Result<SqlitePool> {
|
||||
|
||||
Ok(SqlitePoolOptions::new()
|
||||
.max_connections(4)
|
||||
.after_connect(|conn, _meta| {
|
||||
Box::pin(async move {
|
||||
let version: i64 = sqlx::query_scalar("PRAGMA user_version")
|
||||
.fetch_one(&mut *conn)
|
||||
.await?;
|
||||
if version < 2 {
|
||||
sqlx::query(
|
||||
"CREATE TEMP VIEW IF NOT EXISTS content AS \
|
||||
SELECT *, NULL AS kind_name FROM main.content",
|
||||
)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
})
|
||||
.connect_with(options)
|
||||
.await?)
|
||||
}
|
||||
@@ -103,6 +126,11 @@ impl SourceManager {
|
||||
for sql in &generate_ddl(schema) {
|
||||
sqlx::query(sql).execute(pool).await?;
|
||||
}
|
||||
if schema.models.contains_key("file") {
|
||||
sqlx::query(crate::migrate::FACET_FILE_EXTENSION_INDEX)
|
||||
.execute(pool)
|
||||
.await?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
@@ -471,3 +471,164 @@ async fn migration_1_timing_on_a_million_records() {
|
||||
assert_eq!(count, rows);
|
||||
println!("migration 1 over {rows} content rows and records: {elapsed:?}");
|
||||
}
|
||||
|
||||
/// Turn a fresh store into one written at schema version 1: the content
|
||||
/// table without `kind_name`, no extension index, `user_version` 1.
|
||||
async fn downgrade_to_v1(path: &Path) -> SqlitePool {
|
||||
let pool = raw(path).await;
|
||||
for sql in [
|
||||
"DROP TABLE content",
|
||||
migrate::CONTENT_V1,
|
||||
"CREATE UNIQUE INDEX idx_content_candidate ON content(sampled_hash) WHERE integrity_hash IS NULL",
|
||||
"CREATE INDEX idx_content_sampled ON content(sampled_hash)",
|
||||
"CREATE INDEX idx_content_uuid ON content(uuid)",
|
||||
"CREATE INDEX idx_content_candidate_uuid ON content(candidate_uuid)",
|
||||
"DROP INDEX IF EXISTS idx_facet_file_extension",
|
||||
"PRAGMA user_version = 1",
|
||||
] {
|
||||
sqlx::query(sql).execute(&pool).await.expect(sql);
|
||||
}
|
||||
pool
|
||||
}
|
||||
|
||||
async fn insert_v1_content(pool: &SqlitePool, id: i64, sampled: &str, kind: Option<i64>) {
|
||||
sqlx::query(
|
||||
"INSERT INTO content (id, uuid, candidate_uuid, sampled_hash, size, kind) VALUES (?, ?, ?, ?, 10, ?)",
|
||||
)
|
||||
.bind(id)
|
||||
.bind(uuid_for(sampled))
|
||||
.bind(uuid_for(sampled))
|
||||
.bind(sampled)
|
||||
.bind(kind)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("content row");
|
||||
}
|
||||
|
||||
async fn insert_file_facet(pool: &SqlitePool, record: Uuid, extension: &str) {
|
||||
sqlx::query("INSERT INTO facet_file (record_uuid, size, extension) VALUES (?, 10, ?)")
|
||||
.bind(record)
|
||||
.bind(extension)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("facet row");
|
||||
}
|
||||
|
||||
async fn kind_columns(pool: &SqlitePool) -> Vec<(i64, Option<i64>, Option<String>)> {
|
||||
sqlx::query_as("SELECT id, kind, kind_name FROM content ORDER BY id")
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("kind columns")
|
||||
}
|
||||
|
||||
/// Migration 2 on a version 1 store: the column arrives empty, the
|
||||
/// extension index exists, every row and its kind survive, and the
|
||||
/// reidentification statement names rows by extension without touching a
|
||||
/// row already named.
|
||||
#[tokio::test]
|
||||
async fn migration_2_adds_kind_name_without_a_backfill() {
|
||||
let fixture = Fixture::new().await;
|
||||
let pool = downgrade_to_v1(&fixture.db_path()).await;
|
||||
insert_v1_content(&pool, 1, "s1", Some(1)).await;
|
||||
insert_v1_content(&pool, 2, "s2", None).await;
|
||||
insert_v1_content(&pool, 3, "s3", Some(4)).await;
|
||||
let cr2 = insert_record(&pool, "shot.CR2", Some(1)).await;
|
||||
let nef = insert_record(&pool, "shot.nef", Some(2)).await;
|
||||
let pdf = insert_record(&pool, "paper.pdf", Some(3)).await;
|
||||
insert_file_facet(&pool, cr2, "CR2").await;
|
||||
insert_file_facet(&pool, nef, "nef").await;
|
||||
insert_file_facet(&pool, pdf, "pdf").await;
|
||||
pool.close().await;
|
||||
|
||||
let db = fixture
|
||||
.manager
|
||||
.open("source-1")
|
||||
.await
|
||||
.expect("open migrates");
|
||||
assert_eq!(db.schema_version(), 2);
|
||||
assert_eq!(
|
||||
migrate::version(db.pool()).await.expect("version"),
|
||||
2,
|
||||
"the file carries the new version"
|
||||
);
|
||||
assert_eq!(
|
||||
kind_columns(db.pool()).await,
|
||||
vec![(1, Some(1), None), (2, None, None), (3, Some(4), None)],
|
||||
"rows keep their kind and gain an empty kind_name"
|
||||
);
|
||||
let index: Option<String> = sqlx::query_scalar(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'index' AND name = 'idx_facet_file_extension'",
|
||||
)
|
||||
.fetch_optional(db.pool())
|
||||
.await
|
||||
.expect("index lookup");
|
||||
assert_eq!(index.as_deref(), Some("idx_facet_file_extension"));
|
||||
|
||||
let named = sd_store::name_content_kind_by_extension(
|
||||
db.pool(),
|
||||
"photos:raw",
|
||||
1,
|
||||
&["cr2".to_string(), "nef".to_string()],
|
||||
)
|
||||
.await
|
||||
.expect("reidentify");
|
||||
assert_eq!(named, 2, "the uppercase CR2 and the nef row are both named");
|
||||
let again =
|
||||
sd_store::name_content_kind_by_extension(db.pool(), "other:raw", 1, &["cr2".to_string()])
|
||||
.await
|
||||
.expect("reidentify again");
|
||||
assert_eq!(again, 0, "a named row keeps its name");
|
||||
assert_eq!(
|
||||
kind_columns(db.pool()).await,
|
||||
vec![
|
||||
(1, Some(1), Some("photos:raw".into())),
|
||||
(2, Some(1), Some("photos:raw".into())),
|
||||
(3, Some(4), None),
|
||||
]
|
||||
);
|
||||
let entry = db.entry_by_uuid(nef).await.expect("read").expect("row");
|
||||
assert_eq!(entry.content_kind_name.as_deref(), Some("photos:raw"));
|
||||
assert_eq!(entry.content_kind, Some(1));
|
||||
|
||||
// Reopening is a no-op.
|
||||
drop(db);
|
||||
let reopened = fixture.manager.open("source-1").await.expect("reopen");
|
||||
assert_eq!(reopened.schema_version(), 2);
|
||||
assert_eq!(kind_columns(reopened.pool()).await.len(), 3);
|
||||
}
|
||||
|
||||
/// A replica still at version 1 is read through the current entry shape:
|
||||
/// its rows report no kind name rather than failing on a column the owner
|
||||
/// has not written yet.
|
||||
#[tokio::test]
|
||||
async fn a_version_1_replica_reads_with_an_empty_kind_name() {
|
||||
let fixture = Fixture::new().await;
|
||||
let pool = downgrade_to_v1(&fixture.db_path()).await;
|
||||
insert_v1_content(&pool, 1, "s1", Some(1)).await;
|
||||
let a = insert_record(&pool, "a.cr2", Some(1)).await;
|
||||
insert_file_facet(&pool, a, "cr2").await;
|
||||
let replica_path = fixture.dir.path().join("replica.db");
|
||||
sqlx::query("VACUUM INTO ?")
|
||||
.bind(replica_path.to_string_lossy().to_string())
|
||||
.execute(&pool)
|
||||
.await
|
||||
.expect("vacuum into");
|
||||
pool.close().await;
|
||||
|
||||
let replica = SourceManager::open_file_read_only(&replica_path)
|
||||
.await
|
||||
.expect("replica opens");
|
||||
assert_eq!(replica.schema_version(), 1);
|
||||
let entry = replica.entry_by_uuid(a).await.expect("read").expect("row");
|
||||
assert_eq!(entry.content_kind, Some(1));
|
||||
assert_eq!(entry.content_kind_name, None);
|
||||
let (entries, _) = sd_store::read::all_entries_page(replica.pool(), 0, 10)
|
||||
.await
|
||||
.expect("page");
|
||||
assert_eq!(entries.len(), 1);
|
||||
assert_eq!(
|
||||
migrate::version(replica.pool()).await.expect("version"),
|
||||
1,
|
||||
"nothing migrated the replica"
|
||||
);
|
||||
}
|
||||
@@ -857,3 +857,76 @@ async fn duplicate_discovery_groups_by_sampled_hash_and_splits_by_integrity() {
|
||||
"the pair is reported under the confirmed uuid"
|
||||
);
|
||||
}
|
||||
|
||||
/// The kind the identity phase read from the bytes travels with the record
|
||||
/// to the confirmed row a verification pass moves it to, since that pass
|
||||
/// carries no kind of its own.
|
||||
#[tokio::test]
|
||||
async fn a_full_read_keeps_the_kind_the_sampled_pass_stored() {
|
||||
let fixture = Fixture::new().await;
|
||||
let db = fixture.open().await;
|
||||
db.begin_sync().await.expect("epoch");
|
||||
let note = db
|
||||
.upsert("note", "note-1", &json!({ "title": "A" }))
|
||||
.await
|
||||
.expect("note");
|
||||
|
||||
db.set_content_identity(
|
||||
note,
|
||||
&ContentIdentity {
|
||||
sampled_hash: Some("sampled-1".to_string()),
|
||||
kind: Some(1),
|
||||
kind_name: Some("photos:raw".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("candidate");
|
||||
let confirmed = db
|
||||
.set_content_identity(note, &confirmed("sampled-1", "integrity-1"))
|
||||
.await
|
||||
.expect("confirmed");
|
||||
|
||||
let (kind, kind_name): (Option<i64>, Option<String>) =
|
||||
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
|
||||
.bind(confirmed)
|
||||
.fetch_one(db.pool())
|
||||
.await
|
||||
.expect("confirmed row");
|
||||
assert_eq!(kind, Some(1));
|
||||
assert_eq!(kind_name.as_deref(), Some("photos:raw"));
|
||||
|
||||
// A later identity pass that names the kind itself still wins.
|
||||
db.set_content_identity(
|
||||
note,
|
||||
&ContentIdentity {
|
||||
sampled_hash: Some("sampled-1".to_string()),
|
||||
integrity_hash: Some("integrity-1".to_string()),
|
||||
kind: Some(7),
|
||||
kind_name: Some("other:thing".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("renamed");
|
||||
let (kind, kind_name): (Option<i64>, Option<String>) =
|
||||
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
|
||||
.bind(confirmed)
|
||||
.fetch_one(db.pool())
|
||||
.await
|
||||
.expect("confirmed row");
|
||||
assert_eq!((kind, kind_name.as_deref()), (Some(7), Some("other:thing")));
|
||||
|
||||
// Changed bytes with no kind of their own inherit nothing.
|
||||
let changed = db
|
||||
.set_content_identity(note, &sampled("sampled-2"))
|
||||
.await
|
||||
.expect("rehashed");
|
||||
let (kind, kind_name): (Option<i64>, Option<String>) =
|
||||
sqlx::query_as("SELECT kind, kind_name FROM content WHERE id = ?")
|
||||
.bind(changed)
|
||||
.fetch_one(db.pool())
|
||||
.await
|
||||
.expect("new row");
|
||||
assert_eq!((kind, kind_name), (None, None));
|
||||
}
|
||||
@@ -237,6 +237,7 @@ async fn content_tag_reaches_every_copy() {
|
||||
integrity_hash: None,
|
||||
size: Some(42),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
};
|
||||
db.set_content_identity(original, &identity)
|
||||
.await
|
||||
@@ -302,6 +303,7 @@ async fn late_binding_fills_the_content_key() {
|
||||
integrity_hash: None,
|
||||
size: Some(42),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
};
|
||||
db.set_content_identity(original, &identity)
|
||||
.await
|
||||
@@ -346,6 +348,7 @@ async fn a_content_tag_applied_before_verification_reaches_every_copy_afterwards
|
||||
integrity_hash: None,
|
||||
size: Some(42),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
};
|
||||
db.set_content_identity(original, &guess)
|
||||
.await
|
||||
@@ -477,6 +480,7 @@ async fn identical_claims_collapse_across_stores() {
|
||||
integrity_hash: None,
|
||||
size: Some(7),
|
||||
kind: None,
|
||||
kind_name: None,
|
||||
};
|
||||
|
||||
for (db, device, external) in [(&a, device_a, "copy-a"), (&b, device_b, "copy-b")] {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Extension File Kinds and Previews
|
||||
|
||||
> Status: proposed 2026-10-07; nothing built
|
||||
> Status: K1 and K2 built 2026-10-07 (SPAC-35); K3 to K5 open
|
||||
> Captured: 2026-10-07 (SPAC-31), from the tree at `b938526`
|
||||
> Owns: how an extension's manifest declares file kinds, how identification
|
||||
> assigns them, how the client renders a preview for them, and what happens
|
||||
@@ -268,15 +268,29 @@ without the `wasm` feature behaves as today.
|
||||
|
||||
| Phase | Work | Exit proof | Size |
|
||||
| --- | --- | --- | --- |
|
||||
| K1 Manifest and registry | `kinds` parsed and validated; layered `FileTypeRegistry::current()`; five `builtin()` call sites and the context field moved; `load_all` sorted; conflicts and contested claims recorded; `extensions.list` reports kinds and conflicts; ts-client regenerated | Unit tests: a manifest with a bad parent or a redefining claim is rejected with the reason; two fixtures claiming `.xyz` load with one conflict in the list output; `identify_by_extension` on `foo.cr2` returns the `raw` type with parent `image` while the fixture is loaded and `image` after `unload` | 2 days |
|
||||
| K2 Persisted kinds | Store schema v2 `kind_name` and `idx_facet_file_extension`; content identity phase writes `kind` and `kind_name` and checks extension-kind magic; `File.content_kind_name`; reidentification pass on load | Acceptance test (wasm group): index a folder with `.cr2` files under a fixture extension, read the store, both columns set; unload the extension, re-read through `files.list`, `content_kind_name` still `raw`, `content_kind` is `image`; a `.cr2` whose bytes carry no TIFF header and whose extension is also claimed by a second fixture with matching magic gets the second kind | 2 days |
|
||||
| K1 Manifest and registry (done 2026-10-07, SPAC-35) | `kinds` parsed and validated; layered `FileTypeRegistry::current()`; five `builtin()` call sites and the context field moved; `load_all` sorted; conflicts and contested claims recorded; `extensions.list` reports kinds and conflicts; ts-client regenerated | Unit tests: a manifest with a bad parent or a redefining claim is rejected with the reason; two fixtures claiming `.xyz` load with one conflict in the list output; `identify_by_extension` on `foo.cr2` returns the `raw` type with parent `image` while the fixture is loaded and `image` after `unload` | 2 days |
|
||||
| K2 Persisted kinds (done 2026-10-07, SPAC-35) | Store schema v2 `kind_name` and `idx_facet_file_extension`; content identity phase writes `kind` and `kind_name` and checks extension-kind magic; `File.content_kind_name`; reidentification pass on load | Acceptance test (wasm group): index a folder with `.cr2` files under a fixture extension, read the store, both columns set; unload the extension, re-read through `files.list`, `content_kind_name` still `raw`, `content_kind` is `image`; a `.cr2` whose bytes carry no TIFF header and whose extension is also claimed by a second fixture with matching magic gets the second kind | 2 days |
|
||||
| K3 Client registry | `renderers.ts`, `useExtensionKinds`, `ContentRenderer` becomes a lookup; `getContentKindName`; inspector and file kinds page show the name | `tsc` clean; Playwright or desktop recording of Quick Preview on a `raw` fixture rendering through the image renderer, then falling back to the same renderer with the extension removed from the data dir and the daemon restarted | 1.5 days |
|
||||
| K4 Bundle previews | `ui_manifest.json` `file_viewers` parsed for `id` and `bundle`; `/extension/:id/*path` route; `BundleRenderer` with the `mount` contract; a tiny fixture bundle in `extensions/test-extension/ui/` | Acceptance: the fixture declares a kind with `preview.viewer`; Quick Preview mounts the bundle and the bundle's DOM shows the file name; deleting the bundle file makes the preview fall back to the parent renderer with one logged warning | 2 days |
|
||||
| K5 Photos | Photos' manifest declares `raw` (and `heic` where the built-in table lacks a magic pattern) with `preview.renderer: image`; `ui_manifest.json` `file_viewers` gains `id` and `bundle`, with `raw` pointing at it once a viewer exists | Photos loads with no conflicts on a daemon with the built-in table; `sd-cli op extensions.list` shows its kinds | 0.5 day |
|
||||
| K5 Photos (manifest half done with K1) | Photos' manifest declares `raw` with `preview.renderer: image` (`heic` already has a built-in magic pattern, so it stays built in); `ui_manifest.json` `file_viewers` gains `id` and `bundle`, with `raw` pointing at it once a viewer exists | Photos loads with no conflicts on a daemon with the built-in table; `sd-cli op extensions.list` shows its kinds | 0.5 day |
|
||||
|
||||
K1 and K2 are core; K3 and K4 are client plus one server route; K5 is
|
||||
manifest edits. K3 can start after K1. Eight days, two PRs (K1+K2, K3 to K5).
|
||||
|
||||
As built (K1 and K2, SPAC-35), three details differ from the text above:
|
||||
|
||||
- `content.kind` is written for built-in kinds too, closing the first open
|
||||
question: every row the identity phase touches from now on carries its
|
||||
kind, and only rows identified before that derive one at read time.
|
||||
- The reidentification pass runs when a store opens as well as when an
|
||||
extension with kinds loads, because stores open lazily after the
|
||||
extensions load at startup. `idx_facet_file_extension` is over
|
||||
`lower(extension)` since the walk stores the extension as the name
|
||||
carries it.
|
||||
- A replica below schema version 2 is read through a per-connection
|
||||
temporary view that adds `kind_name` as `NULL`, so an older owner's
|
||||
database still answers the entry readers (`crates/store/src/source.rs`).
|
||||
|
||||
## Decisions for Jamie
|
||||
|
||||
1. Parent is always built in, and a kind is one level deep. The alternative,
|
||||
|
||||
@@ -68,5 +68,63 @@
|
||||
},
|
||||
"description": "ResNet50 trained on Places365 dataset"
|
||||
}
|
||||
],
|
||||
"kinds": [
|
||||
{
|
||||
"name": "raw",
|
||||
"display_name": "RAW photo",
|
||||
"parent": "image",
|
||||
"extensions": [
|
||||
"cr2",
|
||||
"cr3",
|
||||
"nef",
|
||||
"arw",
|
||||
"dng",
|
||||
"raf",
|
||||
"orf",
|
||||
"rw2",
|
||||
"pef",
|
||||
"dcr",
|
||||
"srw"
|
||||
],
|
||||
"mime_types": [
|
||||
"image/x-canon-cr2",
|
||||
"image/x-canon-cr3",
|
||||
"image/x-nikon-nef",
|
||||
"image/x-sony-arw",
|
||||
"image/x-adobe-dng",
|
||||
"image/x-fuji-raf",
|
||||
"image/x-olympus-orf",
|
||||
"image/x-panasonic-rw2",
|
||||
"image/x-pentax-pef",
|
||||
"image/x-kodak-dcr",
|
||||
"image/x-samsung-srw"
|
||||
],
|
||||
"magic": [
|
||||
{
|
||||
"pattern": "49 49 2A 00",
|
||||
"offset": 0
|
||||
},
|
||||
{
|
||||
"pattern": "4D 4D 00 2A",
|
||||
"offset": 0
|
||||
},
|
||||
{
|
||||
"pattern": "46 55 4A 49 46 49 4C 4D",
|
||||
"offset": 0
|
||||
},
|
||||
{
|
||||
"pattern": "49 49 52 4F",
|
||||
"offset": 0
|
||||
},
|
||||
{
|
||||
"pattern": "49 49 55 00",
|
||||
"offset": 0
|
||||
}
|
||||
],
|
||||
"preview": {
|
||||
"renderer": "image"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -11,5 +11,24 @@
|
||||
"write_sidecars": [
|
||||
"digest"
|
||||
]
|
||||
}
|
||||
},
|
||||
"kinds": [
|
||||
{
|
||||
"name": "fake",
|
||||
"display_name": "Fake file",
|
||||
"parent": "text",
|
||||
"extensions": [
|
||||
"fake"
|
||||
],
|
||||
"magic": [
|
||||
{
|
||||
"pattern": "46 41 4B 45",
|
||||
"offset": 0
|
||||
}
|
||||
],
|
||||
"preview": {
|
||||
"renderer": "text"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1257,7 +1257,7 @@ strategy: string | null; estimated_files: number | null; estimated_bytes: number
|
||||
|
||||
export type ExtensionCase = "lower" | "keep";
|
||||
|
||||
export type ExtensionInfo = { id: string; name: string; version: string; jobs: ExtensionJobInfo[] };
|
||||
export type ExtensionInfo = { id: string; name: string; version: string; jobs: ExtensionJobInfo[]; kinds: ExtensionKindInfo[] };
|
||||
|
||||
export type ExtensionJobInfo = {
|
||||
/**
|
||||
@@ -1269,6 +1269,22 @@ name: string;
|
||||
*/
|
||||
full_name: string; resumable: boolean };
|
||||
|
||||
/**
|
||||
* A file kind an extension declares, as the client resolves previews and
|
||||
* icons against it. The kinds of every loaded extension are the client's
|
||||
* only source of extension kinds, which is what lets a stored kind name
|
||||
* fall back to its parent once the extension is gone.
|
||||
*/
|
||||
export type ExtensionKindInfo = {
|
||||
/**
|
||||
* `<extension id>:<name>`, the value `File.content_kind_name` carries
|
||||
*/
|
||||
id: string; name: string; display_name: string; parent: ContentKind;
|
||||
/**
|
||||
* Every extension the manifest claims, contested ones included
|
||||
*/
|
||||
extensions: string[]; preview: PreviewSpec | null };
|
||||
|
||||
/**
|
||||
* A dependency Spacedrive knows how to discover and use.
|
||||
*/
|
||||
@@ -1338,7 +1354,13 @@ created_at: string; modified_at: string; accessed_at: string | null;
|
||||
/**
|
||||
* Additional computed fields
|
||||
*/
|
||||
content_kind: ContentKind; is_local: boolean;
|
||||
content_kind: ContentKind;
|
||||
/**
|
||||
* The extension kind id (`<extension id>:<name>`) the content identity
|
||||
* phase stored, whether or not that extension is loaded now;
|
||||
* `content_kind` is then its parent. `None` for a built-in kind.
|
||||
*/
|
||||
content_kind_name?: string | null; is_local: boolean;
|
||||
/**
|
||||
* Video duration (for grid display optimization)
|
||||
*/
|
||||
@@ -2387,6 +2409,26 @@ export type Keep =
|
||||
*/
|
||||
{ kind: "these"; paths: SdPath[] };
|
||||
|
||||
/**
|
||||
* A file extension two loaded extensions both claimed. The kind loaded
|
||||
* first holds the claim; the other kind's claim on that extension is
|
||||
* dropped from the lookup table and its patterns stay available to the
|
||||
* content identity phase as a tie-breaker.
|
||||
*/
|
||||
export type KindConflict = {
|
||||
/**
|
||||
* The file extension, lowercase, no dot.
|
||||
*/
|
||||
extension: string;
|
||||
/**
|
||||
* The kind id whose claim was dropped.
|
||||
*/
|
||||
kind: string;
|
||||
/**
|
||||
* The kind id that holds the extension.
|
||||
*/
|
||||
claimed_by: string };
|
||||
|
||||
/**
|
||||
* Latency metrics snapshot
|
||||
*/
|
||||
@@ -2915,7 +2957,12 @@ export type ListExtensionsOutput = {
|
||||
/**
|
||||
* Whether this build can load extensions at all
|
||||
*/
|
||||
supported: boolean; extensions: ExtensionInfo[] };
|
||||
supported: boolean; extensions: ExtensionInfo[];
|
||||
/**
|
||||
* File extensions two loaded extensions both claimed, resolved by load
|
||||
* order: the kind loaded first holds each one.
|
||||
*/
|
||||
conflicts: KindConflict[] };
|
||||
|
||||
export type ListLibrariesInput = {
|
||||
/**
|
||||
@@ -3683,6 +3730,12 @@ export type PortRange = { from: number; to: number };
|
||||
*/
|
||||
export type PreferencesOutput = { theme: string; language: string };
|
||||
|
||||
/**
|
||||
* How the client previews a kind: a built-in renderer by name, or a
|
||||
* `ui_manifest.json` `file_viewers` entry by id.
|
||||
*/
|
||||
export type PreviewSpec = { renderer: string } | { viewer: string };
|
||||
|
||||
export type Process = (ProcessStatus) & { id: string };
|
||||
|
||||
export type ProcessLeasesInput = Record<string, never>;
|
||||
|
||||
Reference in new issue
Block a user