diff --git a/docs/architecture/distributed-data-sync.md b/docs/architecture/distributed-data-sync.md index 8b2b65b7b..21a572cc2 100644 --- a/docs/architecture/distributed-data-sync.md +++ b/docs/architecture/distributed-data-sync.md @@ -14,14 +14,16 @@ mod sync { client_uuid: String, // client that created change timestamp: uhlc::Timestamp, // unique hybrid logical clock timestamp resource: SyncResource, // the CRDT resource - transport: SyncTransport, // method of data transport + transport: SyncTransport, // method of data transport (msg or binary) } + // we can now impl specfic CRDT traits to given resources enum SyncResource { FilePath(dyn Replicate), - File(dyn OperationalTransform + OperationalMerge), + File(dyn OperationalTransform), Tag(dyn OperationalTransform), TagOnFile(dyn LastWriteWin), + Jobs(dyn Replicate + OperationalTransform) } } ``` @@ -30,11 +32,11 @@ mod sync { ## Data Types Data is divided into several kinds, Shared, Relational and Owned. -- **Shared data** - Can be created and modified by any client. Has a UUID. +- **Shared data** - Can be created and modified by any client. Has a `uuid`. *Sync Method:* `Operational transform*` - > Shared resources could be,`files`, `tags`, `comments`, `albums`, `jobs`, `labels`. Since these can be created, updated or deleted by any client at any time. + > Shared resources could be,`files`, `tags`, `comments`, `albums` and `labels`. Since these can be created, updated or deleted by any client at any time. - **Relational data** - Can be created and modified by any client. Links two UUIDs by local IDs. @@ -42,15 +44,15 @@ Data is divided into several kinds, Shared, Relational and Owned. > Any many-to-many tables do not store UUIDs, we have to handle this data specifically. Querying for the resources local IDs before creating or deleting the relation. -- **Owned data** - Can only be modified by the client that created it. Has a UUID. +- **Owned data** - Can only be modified by the client that created it. Has a `client_id` and `uuid`. *Sync Method:* `Replicate` - > Owned resources would be `file_paths` & `media_data`, since a client is the single source of truth for this data. This means we can perform conflict free synchronization. + > Owned resources would be `file_paths`, `jobs`, `locations` and `media_data`, since a client is the single source of truth for this data. This means we can perform conflict free synchronization. - **Offline data** - Not synchronized at all. - > For example `logs` `pending_operations` `_migrations`. These are static and not part of this system. + > For example `logs`, `pending_operations` and `_migrations`. These are static and not part of this system. @@ -86,6 +88,8 @@ If a client has not been seen in X amount of time, other clients will not persis + + ## Clock With realtime synchronization it is important to maintain the true order of events, we can timestamp each operation, but have to account for time drift; there is no way to guarantee two machines have synchronized system clocks. @@ -111,17 +115,19 @@ Sync happens in the following order: Owned data → Bulk shared data → Shared data → Relational data + + +### Types of CRDT: + ```rust -enum Crdt { - OperationalTransform, - OperationalMerge, - LastWriteWin, - Replicate, -} +trait OperationalTransform; + +trait LastWriteWin; + +trait Replicate; ``` - **Operational Transform** - Update Shared resources at a property level. Operations stored in `pending_operations` table. -- **Operational Merge** - The newer resource is merged with an older resource at a property level, where oldest takes priority. Used specifically for the `files` resource, which is Shared data but is sometimes synced in bulk. - **Last Write Win** - The most recent event will always be applied, used for many-to-many datasets. - **Replicate** - Used exclusively for Owned data, clients will replicate with no questions asked. @@ -196,6 +202,109 @@ This is intended for the `files` resource. It requires Shared data behaviour as As `files` are created in abundance (hundreds of thousands at a time), it would be inefficient to record these changes in the `pending_operations` table. But we are also unable to sync in the same way as Owned data due to the possibility of conflicts. We handle this by using `SyncMethod::Merge`, simply merging the data where the oldest resource properties are prioritized. + + + + + +## Combining CRDTs + +Combining CRDT types allow for some tailored functionality for particular resources. + +Looking at the `jobs` resource let look how `OperationalTransform + Replicate` might work. + +Jobs are unique in that they have frequent updates to some properties and + +```rust +impl OperationalTransform for Job { + pub fn create () {} + pub fn update () {} + pub fn delete () {} +} + +impl Replicate for Job { + +} +``` + + + + + + + +## Creating Sync Events + +We have a simple Rust syntax for creating sync events in the core. + +```rust +aysnc fn my_core_function(&ctx: CoreContext) -> Result<()> { + let mut file = File::get_unique(1).await?; + + ctx.sync.operation(file.id, + SyncResource::File( + Operation::Update( + FileUpdate::HasThumbnail(true) + ) + ) + ); + + Ok(()) +} +``` + +Then inside the `sync` function we send the event to the + +```rust + impl SyncEngine { + pub fn operation(&self, uuid: &str, sync_resource: SyncResource) { + self.perform_operation( + uuid.clone(), + SyncTransport::Message(sync_resource) + ); + } + } +``` + +Files also impempent `OperationalMerge` would use + + + + + +## Ingesting Sync Events + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/architecture/extensions.md b/docs/architecture/extensions.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/architecture/jobs.md b/docs/architecture/jobs.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/architecture/rust-typescript-messaging.md b/docs/architecture/rust-typescript-messaging.md new file mode 100644 index 000000000..e69de29bb diff --git a/docs/architecture/virtual-filesystem.md b/docs/architecture/virtual-filesystem.md index b16ca0bbd..116bc038c 100644 --- a/docs/architecture/virtual-filesystem.md +++ b/docs/architecture/virtual-filesystem.md @@ -38,7 +38,7 @@ struct File { - `partial_checksum ` - A SHA256 checksum generated from 5 samples of 10,000 bytes throughout the file data, including the begining and end. This is used to identify a file as *likely* unique in under 100µs. -> It is impossible to have a unique constraint at a database level for the `partial_checksum` however we can asyncronously resolve conflicts by querying for duplicates and generating full checksums at a later date. +> ~~It is impossible to have a unique constraint at a database level for the `partial_checksum` however we can asyncronously resolve conflicts by querying for duplicates and generating full checksums at a later date.~~ > > For synchronization of this resource we can tolorate temporary duplicates, any client can calculate that two files resources are duplicate and merge them into a single resource. In turn, triggering a shared data merge operation, whereby the older record is prioritsed at a property level during the merge.