From 2d51ea3ceef27511a0bc1c6a7d4413c848178ae5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Tue, 4 May 2021 15:05:20 +0000 Subject: [PATCH] add specs n docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/extensions/storage/namespaces.md | 4 +- docs/extensions/storage/terminology.md | 8 +- docs/ocis/_index.md | 22 ++ docs/ocis/static/idea.drawio.svg | 489 ++++++++++++++++++++++++ docs/ocis/storage_registry_discovery.md | 107 ++++++ 5 files changed, 624 insertions(+), 6 deletions(-) create mode 100644 docs/ocis/static/idea.drawio.svg create mode 100644 docs/ocis/storage_registry_discovery.md diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index 4b613b3b6e..3ca0f9981d 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -38,7 +38,7 @@ With the spaces concept we are planning to introduce a global namespace to the o ## CS3 global namespaces -The *CS3 global namespace* in oCIS is configured in the [*storage registry*]({{< ref "./terminology.md#storage-registries" >}}). oCIS uses these defaults: +The *CS3 global namespace* in oCIS is configured in the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}). oCIS uses these defaults: | global namespace | description | |-|-| @@ -48,7 +48,7 @@ The *CS3 global namespace* in oCIS is configured in the [*storage registry*]({{< | `/public/` | a virtual folder listing public shares | | `/spaces/` | *TODO: project or group spaces* | -Technically, the `/home` namespace is not necessary: the [*storage registry*]({{< ref "./terminology.md#storage-registries" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. +Technically, the `/home` namespace is not necessary: the [*storage space registry*]({{< ref "./terminology.md#storage-space-registries" >}}) knows the path to a users private space in the `/users` namespace and the gateway can forward the requests to the responsible storage provider. {{< hint warning >}} *@jfd: Why don't we use `/home/` instead of `/users/`. Then the paths would be consistent with most unix systems. diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index 4fc65ca580..0b3ce00a4e 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -23,7 +23,7 @@ A *reference* is a logical concept that identifies a [*resource*]({{< ref "#reso - a [CS3 *id* based reference](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ResourceId), uniquely identifying a [*resource*]({{< ref "#resources" >}}) in the [*namespace*]({{< ref "./namespaces.md" >}}) of a [*storage provider*]({{< ref "#storage-providers" >}}). It consists of a `storage provider id` and an `opaque id`. The `storage provider id` must NOT start with a `/`. {{< hint info >}} -The `/` is important because currently the static [*storage registry*]({{< ref "#storage-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. +The `/` is important because currently the static [*storage registry*]({{< ref "#storage-space-registries" >}}) uses a map to look up which [*storage provider*]({{< ref "#storage-providers" >}}) is responsible for the resource. Paths must be prefixed with `/` so there can be no collisions between paths and storage provider ids in the same map. {{< /hint >}} @@ -108,7 +108,7 @@ by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*stora By making [*storage providers*]({{< ref "#storage-providers" >}}) aware of [*storage spaces*]({{< ref "#storage-spaces" >}}) we can get rid of the current `enablehome` flag / hack in reva, which lead to the [spawn of `*home` drivers](https://github.com/cs3org/reva/tree/master/pkg/storage/fs). Furthermore, provisioning a new [*storage space*]({{< ref "#storage-space" >}}) becomes a generic operation, regardless of the need of provisioning a new user home or a new project space. {{< /hint >}} -## Storage Registries +## Storage Space Registries A *storage registry* manages the [*CS3 global namespace*]({{< ref "./namespaces.md#cs3-global-namespaces" >}}): It is used by the *gateway* @@ -119,7 +119,7 @@ that should handle a [*reference*]({{< ref "#references" >}}). {{< hint warning >}} **Proposed Change** -A *storage registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: +A *storage space registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: It is used by the *gateway* to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) that is currently serving a [*storage space*]({{< ref "#storage-space" >}}). @@ -137,7 +137,7 @@ a *quota* and *permissions*, identified by a `storage space id`. {{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} -Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage registry*]({{< ref "#storage-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... +Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... Finally, a logical `storage space id` is not tied to a specific [*storage provider*]({{< ref "#storage-providers" >}}). If the [*storage driver*]({{< ref "#storage-drivers" >}}) supports it, we can import existing files including their `file id`, which makes it possible to move [*storage spaces*]({{< ref "#storage-spaces" >}}) between [*storage providers*]({{< ref "#storage-providers" >}}) to implement storage classes, e.g. with or without archival, workflows, on SSDs or HDDs. diff --git a/docs/ocis/_index.md b/docs/ocis/_index.md index 7a49e7ef9e..953d808f84 100644 --- a/docs/ocis/_index.md +++ b/docs/ocis/_index.md @@ -13,6 +13,28 @@ geekdocFilePath: _index.md Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based [ownCloud server](https://owncloud.com/#server). +### The idea of federated storage + +To creata a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries. + +The below diagram shows the core conceps that are the foundation for the new architecture: +- End user devices can fetch the list of *storage spaces* a user has access to, by querying one or multiple *storage space registries*. The list contains a unique endpoint for every *storage space*. +- [*Storage space registries*]({{< ref "../extensions/storage/terminology#storage-space-registries" >}}) manage the list of storage spaces a user has access to. They may subscrible to *storage spaces* in order to receive notifications about changes on behalf of an end users mobile or desktop client. +- [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type. +- [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, sharing the load or there might be just one, hosting all types of *storage spaces*. + +{{< svg src="ocis/static/idea.drawio.svg" >}} + +As an example, Einstein might want to share something with Marie, who has an account at a different identity provider and uses a different storage space registry. The process makes use of [OpenID Connect (OIDC)](https://openid.net/specs/openid-connect-core-1_0.html) for authentication and would look something like this: + +To share something with Marie, Einstein would open `https://cloud.zurich.test`. His browser loads oCIS web and presents a login form that uses the [OpenID Connect Discovery](https://openid.net/specs/openid-connect-discovery-1_0.html#EmailSyntax) to look up the OIDC issuer. For `einstein@zurich.test` he will end up at `https://idp.zurich.test`, authenticate and get redirected back to `https://cloud.zurich.test`. Now, oCIS web will use a similar discovery to look up the *storage space registry* for the account, based on the email (or username). He will discover that `https://cloud.zurich.test` is also his *storage registry* that the web UI will use to load the list of *storage spaces* that are available to him. + +After locating a folder that he wants to share with Marie he enters her email `marie@paris.test` in the sharing dialog to grant her the editor role. This, in effect, creates a new *storage space* that is registered with the *storage space registry* at `https://cloud.zurich.test`. + +Einstein copies the URL in the browser (or an email with the same URL is sent automatically, or the storage registries use a backchannel mechanism). It contains the most specific `storage space id` and a path relative to it: `https://cloud.zurich.test/#/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path`. + +When Marie enters that URL she will be presented with a login form on the `https://cloud.zurich.test` instance, because the share was created on that domain. If `https://cloud.zurich.test` trusts her OpenID Connect identity provider `https://idp.paris.test` she can log in. This time, the *storage space registry* discovery will come up with `https://cloud.paris.test` though. Since that registry is different than the registry tied to `https://cloud.zurich.test` oCIS web can look up the *storage space* `716199a6-00c0-4fec-93d2-7e00150b1c84` and register the WebDAV URL `https://cloud.zurich.test/dav/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path` in Maries *storage space registry* at `https://cloud.paris.test`. When she accepts that share her clients will be able to sync the new *storage space* at `https://cloud.zurich.test`. + ### oCIS microservice runtime The oCIS runtime allows us to dynamically manage services running in a single process. We use [suture](https://github.com/thejerf/suture) to create a supervisor tree that starts each service in a dedicated goroutine. By default oCIS will start all built-in oCIS extensions in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A [go-micro](https://github.com/asim/go-micro/blob/master/registry/registry.go) based registry allows services in multiple nodes to form a distributed microservice architecture. diff --git a/docs/ocis/static/idea.drawio.svg b/docs/ocis/static/idea.drawio.svg new file mode 100644 index 0000000000..068b7318de --- /dev/null +++ b/docs/ocis/static/idea.drawio.svg @@ -0,0 +1,489 @@ + + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ a sea of storage spaces +
+
+
+ + + a sea of storage spaces + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+ einstein +
+
+
+ + + einst... + + + + + + + + +
+
+
+ marie +
+
+
+ + + marie + + + + + + + + +
+
+
+ richard +
+
+
+ + + richa... + + + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.zurich.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.paris.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ storage space +
+ registry at +
+ + https://cloud.princeton.test + +
+
+
+ + + storage sp... + + + + + + + +
+
+
+ oCIS = federated storage +
+
+
+ + + oCIS = federated storage + + + + + + + +
+
+
+ multiple +
+ storage registries +
+
+
+ + + multiple... + + + + + + + +
+
+
+ end user devices +
+
+
+ + + end user devices + + + + + + + +
+
+
+ storage provider +
+
+
+ + + storage provider + + + + + + storage space + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Viewer does not support full SVG 1.1 + + + + \ No newline at end of file diff --git a/docs/ocis/storage_registry_discovery.md b/docs/ocis/storage_registry_discovery.md new file mode 100644 index 0000000000..50c746a8fa --- /dev/null +++ b/docs/ocis/storage_registry_discovery.md @@ -0,0 +1,107 @@ +--- +title: "Storage Registry Discovery" +date: 2021-05-04T14:01:00+01:00 +weight: 40 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis +geekdocFilePath: storage_registry_discovery.md +--- + +## Introduction + +In order for an oCIS client to access oCIS storage spaces for an End-User, the client needs to know where the oCIS instance is. oCIS uses WebFinger [RFC7033](http://tools.ietf.org/html/rfc7033) to locate the oCIS instance for an End-User. + +This discovery is optional. If the client has another way of discovering the OpenID instance, e.g. when looging in with a username a static domain might be configured or the domain in the URL might be used. + +For guest accounts that do not have an OIDC issuer or whose IdP is not part of a trusted federation clients may fall back to a local IdP. + +## User Input using E-Mail Address Syntax + +To find the oCIS instance for the given user input in the form of an e-mail address `joe@example.com`, the WebFinger parameters are as follows: + +| WebFinger Parameter | Value | +|-|-| +| `resource` | `acct:joe@example.com` | +| `host` | `example.com` | +| `rel` | http://owncloud.com/specs/ocis/1.0/instance | + +Note that in this case, [the `acct:` scheme](http://tools.ietf.org/html/draft-ietf-appsawg-acct-uri-07) is prepended to the identifier. + +The client (relying party) would make the following WebFinger request to discover the oCIS instance location (with line wraps within lines for display purposes only): + +``` + GET /.well-known/webfinger + ?resource=acct%3Ajoe%40example.com + &rel=http%3A%2F%2Fowncloud.com%2Fspecs%2Focis%2F1.0%2Finstance + HTTP/1.1 + Host: example.com + + HTTP/1.1 200 OK + Content-Type: application/jrd+json + + { + "subject": "acct:joe@example.com", + "links": + [ + { + "rel": "http://owncloud.com/specs/ocis/1.0/instance", + "href": "https://cloud.example.com" + } + ] + } +``` + +{{< hint >}} +Note: the `example.com` domain is derived from the email. +{{< /hint >}} + +{{< hint danger >}} +The `https://cloud.example.com` domain above would point to the ocis instance. +TODO that ins ocis web ... not the registry ... hmmmm +maybe introduce an ocis provider which then has an `/.well-known/ocis-configuration`, similar to `/.well-known/openid-configuration`? +It would contain +- the ocis domain, e.g. `https://cloud.example.com` +- the web endpoint, e.g. `https://cloud.example.com` +- the registry / drives endpoint, e.g. `https://cloud.example.com/graph/v0.1/drives/me` see [Add draft of adr for spaces API. #1827](https://github.com/owncloud/ocis/pull/1827) + + +example: +``` +HTTP/1.1 200 OK + Content-Type: application/json + + { + "instance": "https://cloud.example.com/", + "graph_endpoint": "https://cloud.example.com/graph/", + "ocis_web_config": "https://cloud.example.com/web/config.json", + "issuer": "https://idp.example.com/", + } +``` + +`graph_endpoint` is the open-graph-api endpoint that is used to list storage spaces at e.g. `https://cloud.example.com/graph/v0.1/me/drives`. + +`ocis_web_config` points ocis web to the config for the instance. Maybe we can add more config in the `/.well-known/ocis-configuration` to replace the config.json? Is this the new status.php? How safe is it to expose all this info ...? + +The `issuer` could be used to detect the issuer that is used if no other issuer is found ... might be a fallback_issuer, but actually we may decide to skid the OIDC discovery and rely on this property. Maybe we need it if no IdP is present yet or the `/.well-known/openid-configuration` is not set up / reachable. + + +{{< /hint >}} + +## Obtaining oCIS Provider Configuration Information +Using the `instance` location discovered as described above or by other means, the oCIS Provider's configuration information can be retrieved. + +oCIS Providers supporting Discovery MUST make a JSON document available at the path formed by concatenating the string `/.well-known/openid-configuration` to the `instance`. The syntax and semantics of `.well-known` are defined in [RFC5785](http://tools.ietf.org/html/rfc5785) and apply to the `instance` value when it contains no path component. `ocis-configuration` MUST point to a JSON document compliant with this specification and MUST be returned using the `application/json` content type. + +### oCIS Provider Configuration Request + +An oCIS Provider Configuration Document MUST be queried using an HTTP GET request at the previously specified path. + +The client (relying party) would make the following request to the instance https://example.com to obtain its Configuration information, since the Issuer contains no path component: + + GET /.well-known/openid-configuration HTTP/1.1 + Host: example.com +If the Issuer value contains a path component, any terminating / MUST be removed before appending /.well-known/openid-configuration. The RP would make the following request to the Issuer https://example.com/issuer1 to obtain its Configuration information, since the Issuer contains a path component: + + GET /issuer1/.well-known/openid-configuration HTTP/1.1 + Host: example.com +Using path components enables supporting multiple issuers per host. This is required in some multi-tenant hosting configurations. This use of .well-known is for supporting multiple issuers per host; unlike its use in RFC 5785 [RFC5785], it does not provide general information about the host. \ No newline at end of file