From a022ab78f354c9c871f331f8267fa9614dfd91f4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Fri, 23 Jul 2021 15:17:18 +0000 Subject: [PATCH] rewrite with stronger spaces emphasis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/_index.md | 29 +- docs/extensions/storage/namespaces.md | 8 +- docs/extensions/storage/releasing.md | 31 - docs/extensions/storage/spaces.md | 28 + .../{architecture.md => spacesprovider.md} | 41 +- docs/extensions/storage/spacesregistry.md | 21 + .../storage/static/spacesprovider.drawio.svg | 352 +++++++++++ .../storage/static/spacesregistry.drawio.svg | 595 +++++++++--------- .../storage/static/storage.drawio.svg | 434 +++++++++++++ .../{storages.md => storagedrivers.md} | 12 +- docs/extensions/storage/terminology.md | 52 +- docs/extensions/storage/updating.md | 19 - docs/extensions/storage/users.md | 14 +- docs/ocis/deployment/ocis_keycloak.md | 2 +- docs/ocis/deployment/ocis_traefik.md | 2 +- 15 files changed, 1177 insertions(+), 463 deletions(-) delete mode 100644 docs/extensions/storage/releasing.md create mode 100644 docs/extensions/storage/spaces.md rename docs/extensions/storage/{architecture.md => spacesprovider.md} (77%) create mode 100644 docs/extensions/storage/spacesregistry.md create mode 100644 docs/extensions/storage/static/spacesprovider.drawio.svg create mode 100644 docs/extensions/storage/static/storage.drawio.svg rename docs/extensions/storage/{storages.md => storagedrivers.md} (97%) delete mode 100644 docs/extensions/storage/updating.md diff --git a/docs/extensions/storage/_index.md b/docs/extensions/storage/_index.md index 1f8bcf5e5..1d5023ed3 100644 --- a/docs/extensions/storage/_index.md +++ b/docs/extensions/storage/_index.md @@ -8,29 +8,12 @@ geekdocFilePath: _index.md geekdocCollapseSection: true --- -## Abstract +## Overview -This service provides an oCIS extension that wraps [reva](https://github.com/cs3org/reva/) and adds an opinionated configuration to it. +The storage extension wraps [reva](https://github.com/cs3org/reva/) and adds an opinionated configuration to provide two core services for the oCIS platform: +1. A [*Spaces Registry*]({{< ref "./spacesregistry.md" >}}) that acts as a dictionary for storage *Spaces* and their metadata +2. A [*Spaces Provider*]({{< ref "./spacesprovider.md" >}}) that organizes *Resources* in storage *Spaces* and persists them in an underlying *Storage System* -## Architecture Overview +*Clients* will use the *Spaces Registry* to poll or get notified about changes in all *Spaces* a user has access to. Every *Space* has a dedicated `/dav/spaces/` WebDAV endpoint that is served by a *Spaces Provider* which uses a specific reva storage driver to wrap an underlying *Storage System*. -The below diagram shows the oCIS services and the contained reva services within as dashed boxes. In general: -1. A request comes in at the proxy and is authenticated using OIDC. -2. It is forwarded to the oCIS frontend which handles ocs and ocdav requests by talking to the reva gateway using the CS3 API. -3. The gateway acts as a facade to the actual CS3 services: storage providers, user providers, group providers and sharing providers. - -{{< svg src="extensions/storage/static/overview.drawio.svg" >}} - -The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider: -1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`. -2. The gateway will verify the JWT signature of the `x-access-token` or try to authenticate the request itself, e.g. using a public link token. - -{{< hint warning >}} -The bottom part is lighter because we will deprecate it in favor of using only the CS3 user and group providers after moving some account functionality into reva and glauth. The metadata storage is not registered in the reva gateway to seperate metadata necessary for running the service from data that is being served directly. -{{< /hint >}} - -## Endpoints and references - -In order to reason about the request flow, two aspects in the architecture need to be understood well: -1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints? -2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based? +{{< svg src="extensions/storage/static/storage.drawio.svg" >}} diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index 3ca0f9981..eb68887c7 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -12,7 +12,7 @@ In ownCloud 10 all paths are considered relative to the users home. The CS3 API {{< svg src="extensions/storage/static/namespaces.drawio.svg" >}} -The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The following table lists the various namespaces, paths and id based references: +The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a storage [*space*]({{< ref "./spaces" >}}). The following table lists the various namespaces, paths and id based references: | oc10 namespace | CS3 global namespace | storage provider | reference | content | |--------------------------------------------------|----------------------------------------|------------------|-----------|---------| @@ -32,13 +32,13 @@ In the global CS3 namespaces we plan to move `/home/Shares`, which currently lis ## ownCloud namespaces -In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the [webdav]({{< ref "./architecture#webdav" >}}) and [ocs]({{< ref "./architecture#sharing" >}}) sections for more details end examples. +In contrast to the global namespace of CS3, ownCloud always presented a user specific namespace on all endpoints. It will always list the users private files under `/`. Shares can be mounted at an arbitrary location in the users private spaces. See the [webdav]({{< ref "./spacesprovider#webdav" >}}) and [ocs]({{< ref "./spacesprovider#sharing" >}}) sections for more details end examples. With the spaces concept we are planning to introduce a global namespace to the ownCloud webdav endpoints. This will push the users private space down in the hierarchy: it will move from `/webdav` to `/webdav/home` or `/webdav/users/`. The related [migration stages]({{< ref "../../ocis/migration.md" >}}) are subject to change. ## CS3 global namespaces -The *CS3 global namespace* in oCIS is configured in the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}). oCIS uses these defaults: +The *CS3 global namespace* in oCIS is configured in the storage [*spaces registry*]({{< ref "./spacesregistry" >}}). oCIS uses these defaults: | global namespace | description | |-|-| @@ -48,7 +48,7 @@ The *CS3 global namespace* in oCIS is configured in the [*storage space registry | `/public/` | a virtual folder listing public shares | | `/spaces/` | *TODO: project or group spaces* | -Technically, the `/home` namespace is not necessary: the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. +Technically, the `/home` namespace is not necessary: the storage [*spaces registry*]({{< ref "./spacesregistry" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. {{< hint warning >}} *@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. diff --git a/docs/extensions/storage/releasing.md b/docs/extensions/storage/releasing.md deleted file mode 100644 index b55369d32..000000000 --- a/docs/extensions/storage/releasing.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: "Releasing" -date: 2020-05-22T00:00:00+00:00 -weight: 60 -geekdocRepo: https://github.com/owncloud/ocis -geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: releasing.md ---- - -{{< toc >}} - -To release a new version of the storage submodule, you have to follow a few simple steps. - -## Preparation - -1. Before releasing, make sure that reva has been [updated to the desired version]({{< ref "updating" >}}) - -## Release -1. Check out master -{{< highlight txt >}} -git checkout master -git pull origin master -{{< / highlight >}} -2. Create a new tag (preferably signed) and replace the version number accordingly. Prefix the tag with the submodule `storage/v`. -{{< highlight txt >}} -git tag -s storage/vx.x.x -m "release vx.x.x" -git push origin storage/vx.x.x -{{< / highlight >}} -5. Wait for CI and check that the GitHub release was published. - -Congratulations, you just released the storage submodule! diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md new file mode 100644 index 000000000..f1342b407 --- /dev/null +++ b/docs/extensions/storage/spaces.md @@ -0,0 +1,28 @@ +--- +title: "Spaces" +date: 2018-05-02T00:00:00+00:00 +weight: 3 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: spaces.md +--- + +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Storage Spaces +A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), +a *quota*, *permissions* and is identified by a `storage space id`. + +{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} + +Examples would be every user's personal storage *space*, project storage *spaces* or group storage *spaces*. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A storage [*Spaces Registry*]({{< ref "./spacesregistry.md" >}}) then allows listing the capabilities of storage *spaces*, e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... + +Finally, a logical `storage space id` is not tied to a specific [*spaces provider*]({{< ref "./spacesprovider.md" >}}). If the [*storage driver*]({{< ref "./storagedrivers.md" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move storage *spaces* between [*spaces providers*]({{< ref "./spacesprovider.md" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. + +## Shares +*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like storage [*spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home or personal storage [*space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new storage [*space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file individual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* + diff --git a/docs/extensions/storage/architecture.md b/docs/extensions/storage/spacesprovider.md similarity index 77% rename from docs/extensions/storage/architecture.md rename to docs/extensions/storage/spacesprovider.md index ab8683234..2100a3fa4 100644 --- a/docs/extensions/storage/architecture.md +++ b/docs/extensions/storage/spacesprovider.md @@ -1,12 +1,25 @@ --- -title: "Architecture" +title: "Spaces Provider" date: 2018-05-02T00:00:00+00:00 -weight: 10 +weight: 6 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: architecture.md +geekdocFilePath: spacesprovider.md --- +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Spaces Provider +A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) +by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "./storagedrivers.md" >}}). + +{{< svg src="extensions/storage/static/spacesprovider.drawio.svg" >}} + + ## Frontend The oCIS frontend service starts all services that handle incoming HTTP requests: @@ -38,16 +51,17 @@ The ocdav service not only handles all WebDAV requests under `(remote.php/)(web) | `(remote.php/)webdav/users` | ocdav | storageprovider | `/users` | | | | `(remote.php/)dav/files/` | ocdav | storageprovider | `/users/` | | | | *Spaces concept also needs a new endpoint:* ||||| -| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) or a global endpoint | allow accessing spaces, listing is done by the graph api | +| `(remote.php/)dav/spaces//` | ocdav | storageregistry & storageprovider | bypass path based namespace and directly talk to the responsible storage provider using a relative path | [spaces concept](https://github.com/owncloud/ocis/pull/1827) needs to point to storage [*spaces*]({{< ref "./spaces.md" >}}) | allow accessing spaces, listing is done by the graph api | -The correct endpoint for a users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) in oc10 is `remote.php/dav/files/`. In oc10 All requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. +The correct endpoint for a users home storage [*space*]({{< ref "./spaces.md" >}}) in oc10 is `remote.php/dav/files/`. In oc10 all requests at this endpoint use a path based reference that is relative to the users home. In oCIS this can be configured and defaults to `/home` as well. Other API endpoints like ocs and the web UI still expect this to be the users home. In oc10 we originally had `remote.php/webdav` which would render the current users home [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}). The early versions (pre OC7) would jail all received shares into a `remote.php/webdav/shares` subfolder. The semantics for syncing such a folder are [not trivially predictable](https://github.com/owncloud/core/issues/5349), which is why we made shares [freely mountable](https://github.com/owncloud/core/pull/8026) anywhere in the users home. The current reva implementation jails shares into a `remote.php/webdav/Shares` folder for performance reasons. Obviously, this brings back the [special semantics for syncing](https://github.com/owncloud/product/issues/7). In the future we will follow [a different solution](https://github.com/owncloud/product/issues/302) and jail the received shares into a dedicated `/shares` space, on the same level as `/home` and `/spaces`. We will add a dedicated [API to list all *storage spaces*](https://github.com/owncloud/ocis/pull/1827) a user has access to and where they are mounted in the users *namespace*. {{< hint warning >}} +TODO rewrite this hint with `/dav/spaces` Existing folder sync pairs in legacy clients will break when moving the user home down in the path hierarchy like CernBox did. For legacy clients the `remote.php/webdav` endpoint will no longer list the users home directly, but instead present the different types of storage spaces: - `remote.php/webdav/home`: the users home is pushed down into a new `home` [*storage space*]({{< ref "./terminology.md#storage-spaces" >}}) @@ -55,11 +69,6 @@ For legacy clients the `remote.php/webdav` endpoint will no longer list the user - `remote.php/webdav/spaces`: other [*storage spaces*]({{< ref "./terminology.md#storage-spaces" >}}) the user has access to, e.g. group or project drives {{< /hint >}} -{{< hint warning >}} -An alternative would be to introduce a new `remote.php/dav/spaces` or `remote.php/dav/global` endpoint. However, `remote.php/dav` properly follows the WebDAV RFCs strictly. To ensure that all resources under that [*namespace*]({{< ref "./terminology.md#namespaces" >}}) are scoped to the user the URL would have to include the principal like `remote.php/dav/spaces/`, a precondition for e.g. WebDAV [RFC5397](https://tools.ietf.org/html/rfc5397). For a history lesson start at [Replace WebDAV with REST -owncloud/core#12504](https://github.com/owncloud/core/issues/12504#issuecomment-65218491) which spawned [Add extra layer in DAV to accomodate for other services like versions, trashbin, etc owncloud/core#12543](https://github.com/owncloud/core/issues/12543) -{{< /hint >}} - ### Sharing @@ -92,12 +101,12 @@ The user and public share provider implementations identify the file using the [ The OCM API takes an id based reference on the CS3 api, even if the OCM HTTP endpoint takes a path argument. *@jfd: Why? Does it not need the owner? It only stores the owner of the share, which is always the currently looged in user, when creating a share. Afterwards only the owner can update a share ... so collaborative management of shares is not possible. At least for OCM shares.* {{< /hint >}} -### User and Group provisioning -In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) +## REVA Storage Registry -{{< hint warning >}} -Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. -{{< /hint >}} +The reva *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): +It is used by the reva *gateway* +to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) +that should handle a [*reference*]({{< ref "#references" >}}). -We are currently working on adding [user management through the CS3 API](https://github.com/owncloud/ocis/pull/1930) to handle user and group provisioning (and deprovisioning). +{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} \ No newline at end of file diff --git a/docs/extensions/storage/spacesregistry.md b/docs/extensions/storage/spacesregistry.md new file mode 100644 index 000000000..d5be48f8a --- /dev/null +++ b/docs/extensions/storage/spacesregistry.md @@ -0,0 +1,21 @@ +--- +title: "Spaces Registry" +date: 2018-05-02T00:00:00+00:00 +weight: 9 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/extensions/storage +geekdocFilePath: spacesregistry.md +--- + +{{< hint warning >}} + +The current implementation in oCIS might not yet fully reflect this concept. Feel free to add links to ADRs, PRs and Issues in short warning boxes like this. + +{{< /hint >}} + +## Storage Space Registries + +A storage *spaces registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. + +{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} + diff --git a/docs/extensions/storage/static/spacesprovider.drawio.svg b/docs/extensions/storage/static/spacesprovider.drawio.svg new file mode 100644 index 000000000..d122c58f7 --- /dev/null +++ b/docs/extensions/storage/static/spacesprovider.drawio.svg @@ -0,0 +1,352 @@ + + + + + + + +
+
+
+ oCIS spaces provider +
+ [Software System] +
+
+
+ + + oCIS spaces provider... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + +
+
+
+ + reva gateway + +
+ [Component: golang] +
+
+
+ API facade for internal reva services +
+
+
+
+ + + reva gateway... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + +
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [WebDAV, OCS, OCM, tus] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Forwards to + +
+
+ [CS3, storage registry] +
+
+
+
+
+ + + Forwards to... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces provider +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces provider... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/static/spacesregistry.drawio.svg b/docs/extensions/storage/static/spacesregistry.drawio.svg index e3a771b07..f716b0bb0 100644 --- a/docs/extensions/storage/static/spacesregistry.drawio.svg +++ b/docs/extensions/storage/static/spacesregistry.drawio.svg @@ -1,136 +1,82 @@ - + - + -
+
- oCIS System + oCIS spaces registry
[Software System]
- - oCIS System... + + oCIS spaces registry... - - - + -
-
-
- - Einstein - -
- [Person] -
-
-
- End user -
-
-
-
- - - Einstein... - - - - - - - -
+
- Client + reva storage registry
- [Container: C++, Kotlin, Swift or Vue] + [Component: golang]

- A desktop, mobile or web Client + manages and caches storage space metadata
- - Client... + + reva storage registry... - + -
+
- Storage Space Registry + reva gateway
- [Container: golang, HTTP, libregraph] + [Component: golang]

- Manages spaces for users + API facade for internal reva services
- - Storage Space Registry... + + reva gateway... - + -
-
-
- - Storage Provider - -
- [Container: golang] -
-
-
- Persists storage spaces using reva -
-
-
-
- - - Storage Provider... - - - - - - - -
+
@@ -147,45 +93,285 @@
- + Storage System... - - - + + -
+
-
- - Moss - -
- [Person] -
-
-
- Administrator +
+
+
+ + Provisions and manages spaces in + +
+
+ [CS3] +
- - Moss... + + Provisions and manages spaces... - - + -
+
+
+
+ + reva frontend + +
+ [Component: golang] +
+
+
+ handles protocol translation +
+
+
+
+ + + reva frontend... + + + + + + + +
+
+
+ + oCIS proxy + +
+ [Component: golang] +
+
+
+ Routes requests to oc10 or ecis +
+
+
+
+ + + oCIS proxy... + + + + + + + + +
+
+
+
+
+ + Mints an internal JWT +
+ and torwards requests to + +
+
+ [libregraph] +
+
+
+
+
+ + + Mints an internal JWT... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, +
+ Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + + +
+
+
+
+
+ + polls or gets notified about changes in + +
+
+ [libregraph] +
+
+
+
+
+ + + polls or gets notified about c... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Lists spaces using + +
+
+ [CS3] +
+
+
+
+
+ + + Lists spaces using... + + + + + + + +
+
+
+

+ C4 Component diagram for an oCIS spaces registry +

+

+ An oCIS spaces provider manages resources in storage spaces by persisting them with a specific storage driver in a storage system. +

+

+ Date: 2021-07-22T12:40 +

+
+
+
+ + + C4 Component diagram for an oCIS spaces registry... + + + + + + + +
+
+
+ + reva storage provider + +
+ [Component: golang] +
+
+
+ hosts multiple storage spaces using a storage driver +
+
+
+
+ + + reva storage provider... + + + + + + + + +
@@ -202,226 +388,11 @@
- + Reads from and writes to... - - - - - -
-
-
-
-
- - Reads from and writes to - -
-
- [WebDAV, libregraph, CS3, tus] -
-
-
-
-
- - - Reads from and writes to... - - - - - - - - -
-
-
-
-
- - Manages the users Storage Spaces in - -
-
- [libregraph] -
-
-
-
-
- - - Manages the users Storage Spac... - - - - - - - - -
-
-
-
-
- - Manages resources with - -
-
- [Web UI or native clients] -
-
-
-
-
- - - Manages resources with... - - - - - - - - -
-
-
-
-
- - Registers itself at and -
- sends space root etag changes to - -
-
- [CS3, libregraph?, PUSH] -
-
-
-
-
- - - Registers itself at and... - - - - - - - - -
-
-
-
-
- - Manages organizational Storage Spaces in - -
-
- [WebDAV, libregraph, CS3, CLI] -
-
-
-
-
- - - Manages organizational Storage... - - - - - - - -
-
-
- - Identity Management System - -
- [Software System] -
-
-
- provides users and groups -
-
-
-
- - - Identity Management System... - - - - - - - - -
-
-
-
-
- - Authenticates users and searches recipients with - -
-
- [OpenID Connect, LDAP, REST] -
-
-
-
-
- - - Authenticates users and search... - - - - - - - -
-
-
-

- C4 Container diagram for the oCIS System -

-

- As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system -

-

- Date: 2021-07-22T16:43 -

-
-
-
- - - C4 Container diagram for the oCIS System... - - - diff --git a/docs/extensions/storage/static/storage.drawio.svg b/docs/extensions/storage/static/storage.drawio.svg new file mode 100644 index 000000000..fd6e759cf --- /dev/null +++ b/docs/extensions/storage/static/storage.drawio.svg @@ -0,0 +1,434 @@ + + + + + + + +
+
+
+ oCIS System +
+ [Software System] +
+
+
+ + + oCIS System... + + + + + + + + + +
+
+
+ + Einstein + +
+ [Person] +
+
+
+ End user +
+
+
+
+ + + Einstein... + + + + + + + +
+
+
+ + Client + +
+ [Container: C++, Kotlin, Swift or Vue] +
+
+
+ A desktop, mobile or web Client +
+
+
+
+ + + Client... + + + + + + + +
+
+
+ + Storage Space Registry + +
+ [Container: golang, HTTP, libregraph] +
+
+
+ Manages spaces for users +
+
+
+
+ + + Storage Space Registry... + + + + + + + +
+
+
+ + Storage Space Provider + +
+ [Container: golang] +
+
+
+ Persists storage spaces using reva +
+
+
+
+ + + Storage Space Provider... + + + + + + + +
+
+
+ + Storage System + +
+ [Software System] +
+
+
+ provides persistent storage +
+
+
+
+ + + Storage System... + + + + + + + + + +
+
+
+ + Moss + +
+ [Person] +
+
+
+ Administrator +
+
+
+
+ + + Moss... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [POSIX, S3] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Reads from and writes to + +
+
+ [WebDAV, libregraph, CS3, tus] +
+
+
+
+
+ + + Reads from and writes to... + + + + + + + + +
+
+
+
+
+ + Manages the users Storage Spaces in + +
+
+ [libregraph] +
+
+
+
+
+ + + Manages the users Storage Spac... + + + + + + + + +
+
+
+
+
+ + Manages resources with + +
+
+ [Web UI or native clients] +
+
+
+
+
+ + + Manages resources with... + + + + + + + + +
+
+
+
+
+ + Registers itself at and +
+ sends space root etag changes to + +
+
+ [CS3, libregraph?, PUSH] +
+
+
+
+
+ + + Registers itself at and... + + + + + + + + +
+
+
+
+
+ + Manages organizational Storage Spaces in + +
+
+ [WebDAV, libregraph, CS3, CLI] +
+
+
+
+
+ + + Manages organizational Storage... + + + + + + + +
+
+
+ + Identity Management System + +
+ [Software System] +
+
+
+ provides users and groups +
+
+
+
+ + + Identity Management System... + + + + + + + + +
+
+
+
+
+ + Authenticates users and searches recipients with + +
+
+ [OpenID Connect, LDAP, REST] +
+
+
+
+
+ + + Authenticates users and search... + + + + + + + +
+
+
+

+ C4 Container diagram for the oCIS System +

+

+ As a platform, the oCIS system may not only includes web, mobile and desktop clients but also the underlying storage system or an identity management system +

+

+ Date: 2021-07-22T16:43 +

+
+
+
+ + + C4 Container diagram for the oCIS System... + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/extensions/storage/storages.md b/docs/extensions/storage/storagedrivers.md similarity index 97% rename from docs/extensions/storage/storages.md rename to docs/extensions/storage/storagedrivers.md index 59927fd93..9a9d7fb4f 100644 --- a/docs/extensions/storage/storages.md +++ b/docs/extensions/storage/storagedrivers.md @@ -1,15 +1,18 @@ --- -title: "Storages" +title: "Storage drivers" date: 2020-04-27T18:46:00+01:00 -weight: 37 +weight: 12 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage geekdocFilePath: storages.md --- -## Storage commands +A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): -`storage` has multiple storage provider commands to preconfigure different default configurations for the reva *storage provider* service. While you could rerun `storage storage-oc` multiple times with different flags to get multiple instances we are giving the different commands the necessary default configuration to allow the `ocis` binary to simply start them and not deal with configuration. +It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.: +- eos file ids +- posix inodes or paths +- deconstructed filesystem nodes ## Storage providers @@ -25,7 +28,6 @@ A lot of different storage technologies exist, ranging from general purpose file Unfortunately, no POSIX filesystem natively supports all storage aspects that ownCloud 10 requires: - ### A hierarchical file tree An important aspect of a filesystem is organizing files and directories in a file hierarchy, or tree. It allows you to create, move and delete nodes. Beside the name a node also has well known metadata like size and mtime that are persisted in the tree as well. diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index 2b883b570..426c8448b 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -19,7 +19,6 @@ A *resource* is the basic building block that oCIS manages. It can be of [differ - a [*reference*]({{< ref "#references" >}}) which can point to a resource in another [*storage provider*]({{< ref "#storage-providers" >}}) ### References - A *reference* identifies a [*resource*]({{< ref "#resources" >}}). A [*CS3 reference*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.Reference) can carry a *path* and a [CS3 *resource id*](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId). The references come in two flavors: absolute and combined. Absolute references have either the *path* or the *resource id* set: - An absolute *path* MUST start with a `/`. The *resource id* MUST be empty. @@ -28,59 +27,12 @@ Combined references have both, *path* and *resource id* set: - the *resource id* identifies the root [*resource*]({{< ref "#resources" >}}) - the *path* is relative to that root. It MUST start with `.` - -### Storage Spaces -A *storage space* organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), -a *quota*, *permissions* and is identified by a `storage space id`. - -{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} - -Examples would be every user's personal storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... - -Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. - -### Shares -*To be clarified: we are aware that [*storage spaces*]({{< ref "#storage-spaces" >}}) may be too 'heavywheight' for ad hoc sharing with groups. That being said, there is no technical reason why group shares should not be treated like [*storage spaces*]({{< ref "#storage-spaces" >}}) that users can provision themselves. They would share the quota with the users home [*storage space*]({{< ref "#storage-spaces" >}}) and the share initiator would be the sole owner. Technically, the mechanism of treating a share like a new [*storage space*]({{< ref "#storage-spaces" >}}) would be the same. This obviously also extends to user shares and even file indvidual shares that would be wrapped in a virtual collection. It would also become possible to share collections of arbitrary files in a single storage space, e.g. the ten best pictures from a large album.* - - -### Storage Space Registries - -A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. - -{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} - - ## Technical concepts -### Storage Drivers - -A *storage driver* implements access to a [*storage system*]({{< ref "#storage-systems" >}}): - -It maps the *path* and *id* based CS3 *references* to an appropriate [*storage system*]({{< ref "#storage-systems" >}}) specific reference, e.g.: -- eos file ids -- posix inodes or paths -- deconstructed filesystem nodes - -### Storage Providers - -A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) -by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). - -{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} - -### Storage Registry - -A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): -It is used by the *gateway* -to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) -that should handle a [*reference*]({{< ref "#references" >}}). - -{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} - ### Storage Systems Every *storage system* has different native capabilities like id and path based lookups, recursive change time propagation, permissions, trash, versions, archival and more. -A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "#storage-drivers" >}}). -There migt be multiple [*storage drivers*]({{< ref "#storage-drivers" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. +A [*storage provider*]({{< ref "#storage-providers" >}}) makes the storage system available in the CS3 API by wrapping the capabilities as good as possible using a [*storage driver*]({{< ref "./storagedrivers.md" >}}). +There might be multiple [*storage drivers*]({{< ref "./storagedrivers.md" >}}) for a *storage system*, implementing different tradeoffs to match varying requirements. ### Gateways A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. diff --git a/docs/extensions/storage/updating.md b/docs/extensions/storage/updating.md deleted file mode 100644 index dd1920376..000000000 --- a/docs/extensions/storage/updating.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: "Updating reva" -date: 2020-05-22T00:00:00+00:00 -weight: 50 -geekdocRepo: https://github.com/owncloud/ocis -geekdocEditPath: edit/master/docs/extensions/storage -geekdocFilePath: updating.md ---- - -{{< toc >}} - -## Updating reva - -1. Run `go get github.com/cs3org/reva@master` in all repos that depend on reva -2. Create a changelog entry containing changes that were done in [reva](https://github.com/cs3org/reva/commits/master) -3. Create a Pull Request to ocis master with those changes -4. If test issues appear, you might need to adjust the tests -5. After the PR is merged, consider doing a [release of the storage submodule]({{< ref "releasing" >}}) - diff --git a/docs/extensions/storage/users.md b/docs/extensions/storage/users.md index 4a5e716fa..8e5d034ba 100644 --- a/docs/extensions/storage/users.md +++ b/docs/extensions/storage/users.md @@ -1,12 +1,24 @@ --- title: "Users" date: 2020-01-16T00:00:00+00:00 -weight: 35 +weight: 17 geekdocRepo: https://github.com/owncloud/ocis geekdocEditPath: edit/master/docs/extensions/storage geekdocFilePath: users.md --- +TODO add this to the storage overview? or is this a different part? That should be started as a separate service ? And documented elsewhere, eg. in the accounts? + +### User and Group provisioning + +In oc10 users are identified by a username, which cannot change, because it is used as a foreign key in several tables. For oCIS we are internally identifying users by a UUID, while using the username in the WebDAV and OCS APIs for backwards compatability. To distinguish this in the URLs we are using `` instead of ``. You may have encountered ``, which refers to a template that can be configured to build several path segments by filling in user properties, e.g. the first character of the username (`{{substr 0 1 .Username}}/{{.Username}}`), the identity provider (`{{.Id.Idp}}/{{.Username}}`) or the email (`{{.Mail}}`) + +{{< hint warning >}} +Make no mistake, the [OCS Provisioning API](https://doc.owncloud.com/server/developer_manual/core/apis/provisioning-api.html) uses `userid` while it actually is the username, because it is what you use to login. +{{< /hint >}} + +We are currently working on adding [user management through the CS3 API](https://github.com/owncloud/ocis/pull/1930) to handle user and group provisioning (and deprovisioning). + ### Demo driver This is a simple user driver for testing. It contains three users: diff --git a/docs/ocis/deployment/ocis_keycloak.md b/docs/ocis/deployment/ocis_keycloak.md index 46ef4ad06..4cc2be6cc 100644 --- a/docs/ocis/deployment/ocis_keycloak.md +++ b/docs/ocis/deployment/ocis_keycloak.md @@ -21,7 +21,7 @@ The docker stack consists 4 containers. One of them is Traefik, a proxy which is Keykloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) -The other container is oCIS itself running all extensions in one container. In this example oCIS uses [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}}) +The other container is oCIS itself running all extensions in one container. In this example oCIS uses the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}}) ## Server Deployment diff --git a/docs/ocis/deployment/ocis_traefik.md b/docs/ocis/deployment/ocis_traefik.md index 6bb308877..cb44362fe 100644 --- a/docs/ocis/deployment/ocis_traefik.md +++ b/docs/ocis/deployment/ocis_traefik.md @@ -18,7 +18,7 @@ geekdocFilePath: ocis_traefik.md The docker stack consists of two containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network. -The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storages#storage-drivers" >}}) +The other one is oCIS itself running all extensions in one container. In this example oCIS uses its internal IDP [LibreGraph Connect]({{< ref "../../extensions/idp" >}}) and the [oCIS storage driver]({{< ref "../../extensions/storage/storagedrivers" >}}) ## Server Deployment