From 6cd53648a2ac8ebe4fc514ad4eac92728137e4d2 Mon Sep 17 00:00:00 2001 From: Pawit Pornkitprasan Date: Fri, 5 Nov 2021 06:58:24 +0700 Subject: [PATCH] docs: improve description of CAOS and CABS (#1438) * docs: improve description of CAOS and CABS * improve formatting * update encryption algorithm name --- .../docs/Advanced/Architecture/_index.md | 43 ++++++++++++++++--- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/site/content/docs/Advanced/Architecture/_index.md b/site/content/docs/Advanced/Architecture/_index.md index b278c6ebd..02e5a7cda 100644 --- a/site/content/docs/Advanced/Architecture/_index.md +++ b/site/content/docs/Advanced/Architecture/_index.md @@ -34,11 +34,15 @@ Block ID is generated by applying [cryptographic hash function](https://en.wikip A key property of block identifiers is that two identical block of data will produce exactly the same identifiers, thus resulting in natural de-duplication of data. -After hashing, the block data is encrypted using algorithm such as `AES256-CTR` or `SALSA20`. To make uploads to cloud storage more efficient and cheaper, multiple smaller blocks are combined into larger **Packs** of 20-40MB each. +After hashing, the block data is encrypted using algorithm such as `AES256-GCM-HMAC-SHA256` or `CHACHA20-POLY1305-HMAC-SHA256`. To make uploads to cloud storage more efficient and cheaper, multiple smaller blocks are combined into larger **Packs** of 20-40MB each. Pack files in blob storage have random names and don't reveal anything about their contents or structure. Their sizes are also generally unrelated to content due to splitting and merging. -To help efficiently find a block in the blob storage, CABS maintains an index that maps block ID to the blob file name, offset within the file and length. +To help efficiently find a block in the blob storage, CABS maintains an index that maps block ID to the blob file name, offset within the file and length. In addition, to make data recovery possible in case the index files got corrupted, a local index is also stored at the end of each pack. -Pack files in blob storage have random names and don't reveal anything about their contents or structure. Their sizes are also generally unrelated to content due to splitting and merging. +File names in the BLOB layer have prefixes to help quickly identify its type: + +* `p` represents packs containing data (e.g. `pb4cf8ca179d71478fb8d4b00b79a9a72`) +* `q` represents packs containing metadata (e.g. `q7a9939814e8aba1fdda2d87965f324d3`) +* `x` represents indices (e.g. `xn0_20db7984bd71c4042cea471a61fbcea1`) CABS is not meant to be used directly, instead it's a building block for object storage (CAOS) and manifest storage layers (LAMS) described below. @@ -54,17 +58,42 @@ Object IDs can also have an optional single-letter prefix `g..z` that helps quic * `k` represents directory listing (e.g. `kfed1b0498dc54d07cd69f272fb347ca3`) * `m` represents manifest block (e.g. `m0bf4da00801bd8c6ecfb66cffa67f32c`) -* `h` represents hash-cache (e.g. `h2e88080490a83c4b1cb344d861a3f537`) +* `x` represents indirect JSON content (e.g. `xac47f7ce280fdd81f04c670fec2353dc`) -To represent objects larger than the size of a single CABS block, Kopia links together multiple blocks via special indirect JSON content. Such blocks are distinguished from regular blocks by the `I` prefix. For example very large hash-cache object might have an identifier such as `Ih746f0a60f744d0a69e397a6128356331` and JSON content: +Objects without prefixes are stored in the `p` pack while objects with prefixes are stored in the `q` metadata pack by the CABS layer. + +To represent objects larger than the size of a single CABS block, Kopia links together multiple blocks via special indirect JSON content. Such blocks are distinguished from regular blocks the `I` virtual prefix. Indirection can be applied to any object type, including metadata such as directory listing. However, when applied to a data block, an `x` prefix is added to ensure than the indrect JSON is stored in a metadata pack instead of a data pack by the CABS layer. + +For example large file object might have an identifier such as `Ixac47f7ce280fdd81f04c670fec2353dc` and JSON content: ```json {"stream":"kopia:indirect","entries":[ - {"l":9076851,"o":"2fc5ac219af279579366d87029d682ea"}, - {"s":9076851,"l":6138016,"o":"107572665e69114d22cd43547ac1b33d"} + {"l":2617867,"o":"e510796ba6ffd15649ea400b67ef6159"}, + {"s":2617867,"l":2751278,"o":"5c090744e0cad69d0d1aecd4ffd69f69"}, + {"s":5369145,"l":2476164,"o":"1f780f1a968fcdaa9e3a51961ce1cd8a"}, + {"s":7845309,"l":2640451,"o":"c03399d1d2fb13c48c062d1243deef91"} ]} ``` +The [content list](/docs/reference/command-line/advanced/content-list/) command can be used to list CABS blocks in the repository: + +```shell +$ kopia content list +``` + +The [content show](/docs/reference/command-line/advanced/content-show/) command can be used to show the content of a specific CABS block: + +```shell +# Show the content of a directory listing +$ kopia content show kfed1b0498dc54d07cd69f272fb347ca3 +# Show the indirect JSON content of a large file object +$ kopia content show xac47f7ce280fdd81f04c670fec2353dc +# Show the actual content of a large file object +$ kopia content show Ixac47f7ce280fdd81f04c670fec2353dc +``` + +Note that the example above shows that `I` is a virtual prefix. The actual CABS block does not content the `I` prefix but referencing the block with the `I` prefix tells Kopia to interprete the block as an indirect JSON object and return the resolved content instead of the raw content. + The API for CAOS can be found in https://godoc.org/github.com/kopia/kopia/repo/object ### Label-Addressable Manifest Storage (LAMS)