From c0a4b1b6111fe0dc5f4c42da3571f0652a935698 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Mon, 19 Apr 2021 12:40:26 +0200 Subject: [PATCH 1/6] add cs3 api user management adr --- .../adr/0005-cs3-api-account-management.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 docs/ocis/adr/0005-cs3-api-account-management.md diff --git a/docs/ocis/adr/0005-cs3-api-account-management.md b/docs/ocis/adr/0005-cs3-api-account-management.md new file mode 100644 index 0000000000..be59318afe --- /dev/null +++ b/docs/ocis/adr/0005-cs3-api-account-management.md @@ -0,0 +1,211 @@ +--- +title: "5. Account Management through CS3 API" +date: 2021-04-12T15:00:00+01:00 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis/adr +geekdocFilePath: 0004-cs3api-user-management.md +--- + +# 5. Account Management via CS3 API + +* Status: proposed +* Deciders: @refs, @butonic, @micbar, @dragotin, @pmaier1 +* Date: 2021-04-12 + +Technical Story: [Write only management API for User and Group resources](https://github.com/cs3org/cs3apis/pull/119) + +## Context and Problem Statementf + +What would be a more effective way of using network resources and handle account management within the oCIS-Reva ecosystem? Separating account management at the service level is pragmatic and allows for fast iterations, but also steadily accumulates inconsistencies and bloats technical debt. + +## Decision Drivers + +* Reduce number of network calls. +* Reduce number of services (merge Account + GLAuth from ADR-0003). +* Formalize account management at the API level. + +## Considered Options + +* Account management delegated to vendors. +* Add account management to the CS3 API. + +## Decision Outcome + +Chosen option: "Add account management to the CS3 API". Making the API declare an API for account management will not only allow a deployment to fail fast (as in: the management node is not running) but would also centralize all management operations that should happen to be constrained within the Reva context. Constrained operations *SHOULD* be by definition more secure, or at least as secure as the rest of the system. + +### Positive Consequences + +* More resilient API. + * Because account management is considered a "first class citizen" changes are forced to go through a more exhaustive revision process. +* Removing Accounts from search users1. +* Replace the provisioning API in favor of the new Reva Admin node. + +(1) the current vendor implementation of searching a user (i.e: when sharing a resource) relies directly on the accounts service, since this is the only source of truth. Searching a user looks like: + +``` +┌────────────────────────────────────────┐ +│user search (no LDAP) │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ ┌ ─ ─ ─ ─ ─ ┐ │ +│ └──────────┘ go-micro │ +│ ▲ │ │ │ +│ │ Λ │ +│ ▼ │ ╱ ╲ │ │ +│ ┌──────────┐ ╱ ╲ │ +│ │ │ │ ╱ ╲ │ │ +│ │ ocs │◀──(1)───▶registry▏ │ +│ │ │ │ ╲ ╱ │ │ +│ └──────────┘ ╲ ╱ │ +│ ▲ │ ╲ ╱ │ │ +│ │ V │ +│ │ │ │ │ +│ │ │ +│ │ └ ─ ─ ─ ─ ─ ┘ │ +│ │ │ +│ │ │ +│ │ ┌──────────┐ │ +│ │ │ │ │ +│ └─────────────▶│ accounts │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ +│ │ +│(1) ocs requests a connection to the │ +│accounts service to the registry │ +│ │ +└────────────────────────────────────────┘ +``` + +Whereas, as a result of ADR-0003 and this ADR, we can simplify and improve this design: + +``` +┌─────────────────────────────────────────────┐ +│user search │ +│ │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ ocs │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │ +│ reva ▼ │ IDM │ +│ │ ┌──────────┐ │ ┌──────────┐ │ │ +│ │ │ │ │ │ │ +│ │ │ users │◀─────┼──▶│ GLAuth │ │ │ +│ │ │ │ │ │ │ +│ │ └──────────┘ │ └──────────┘ │ │ +│ │ │ +│ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ +│ │ +└─────────────────────────────────────────────┘ +``` + +And instead rely on the already existing Reva users provider. + + +## Pros and Cons of the Options + +### Account management delegated to vendors + +* Good, because it allows for fast iterations. +* Bad, because account management happens outside of the Reva process. This can potentially end up in invalid account creation / deletion / updates. + * An example with the existing Accounts service is that any client can fire CRUD accounts requests to the Accounts service as long as the client knows where the server is running and provides with an Authorization header (only required by the proxy). This request totally bypasses Reva middlewares and therefore any security measures that should be enforced by the entire system. +* Bad, because leaves teams the task of designing and implementing a way of dealing with account management. Ideally one schema should be provided / suggested. + +Creating an account using the first option looks currently is implemented in vendors as: + +``` +┌──────────────────────────────────────────────────┐ +│ creating a user (webui) │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ /api/v0/accounts/accounts-create │ +│ │ │ +│ │ │ +│ │ │ +│ ▼ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ accounts │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ +│ note that while doing CRUD operations changes │ +│ are instantly reflected for the IDP since out of │ +│ the box oCIS uses an accounts backend for │ +│ GLAuth. │ +└──────────────────────────────────────────────────┘ +``` + +As explained before, during this flow no Reva middlewares are ran. Creating an account will only use the embedded accounts js file alongside a minted jwt token (by the oCIS proxy) to communicate with the accounts service. + +### Add account management to the CS3 API + +* Good, because it solidifies what the CS3 API can or cannot do, and account management should be handled at the API level since ultimately accounts would contain a mix of required CS3 and vendor-specific attributes. +* Good, because it centralizes account management and constrains it within the Reva boundaries. +* Good, because there is a clear separation of concerns on what is accounts management logic. +* Good, because we already designed [a similar API for the accounts service](https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L42-L85) the only difference being we (vendors) [define their own messages](https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L252-L408). + * The API would fully include CRUD methods +* Bad, because development cycles are larger. + * an example flow will be: `update api > run prototool > publish language specific packages > update dependencies to fetch latest version of the package > utilize the new changes`. + +The new account management workflow will result in: +``` +┌───────────────────────────────────────────────────┐ +│creating a user (webui) │ +│ - maintain the same route for compatibility │ +│ │ +│ ┌──────────┐ │ +│ │ │ │ +│ │ proxy │ │ +│ │ │ │ +│ └──────────┘ │ +│ │ │ +│ │ │ +│ /api/v0/accounts/accounts-create │ +│ │ │ +│ │ │ +│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ │ +│ Reva │ IDM │ │ +│ │ │ │ │ │ +│ ▼ │ │ +│ │ ┌──────────┐ │ │ ┌──────────┐ │ +│ │ │ │ │ │ │ +│ │ │ admin │───────────┼──┼──▶│ GLAuth │ │ +│ │ │ │ │ │ │ +│ │ └──────────┘ │ │ └──────────┘ │ +│ │ │ +│ │ │ │ │ +│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ +│ │ +│ │ +│ │ +│ │ +│ (1) solidify driver interfaces with write │ +│ │ +│ (2) an example of a driver could be GLAuth │ +│ implementing the user management portion of the │ +│ GraphAPI │ +└───────────────────────────────────────────────────┘ +``` + +This flow allows Reva and oCIS Proxy to runs any middleware logic in order to validate a request. The communication between the proposed Admin api (CS3 API messages) with the IDM (GLAuth) are specific to the _drivers_. From efa214f6ef8e093754bcd87d54164ba7be5895f2 Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Mon, 19 Apr 2021 15:57:52 +0200 Subject: [PATCH 2/6] service registry adr --- docs/ocis/adr/0006-service-discovery.md | 58 +++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 docs/ocis/adr/0006-service-discovery.md diff --git a/docs/ocis/adr/0006-service-discovery.md b/docs/ocis/adr/0006-service-discovery.md new file mode 100644 index 0000000000..1364689618 --- /dev/null +++ b/docs/ocis/adr/0006-service-discovery.md @@ -0,0 +1,58 @@ +--- +title: "6. Service Discovery within oCIS and Reva" +date: 2021-04-19T13:00:00+01:00 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis/adr +geekdocFilePath: 0006-service-discovery.md +--- + +# 6. Service Discovery within oCIS and Reva + +* Status: proposed +* Deciders: @refs, @butonic, @micbar, @dragotin, @pmaier1 +* Date: 2021-04-19 + +Technical Story: [Introduce Named Services.](https://github.com/cs3org/reva/pull/1509) + +## Context and Problem Statement + +Reva relies heavily on config files. Implications of this approach are having to know a-priori where a service is running (host + port). We want to move away from hardcoded values and rely instead on named services for service discovery. Furthermore, we would like both platforms (Reva + oCIS) to have the same source of truth at any given time, not having one to notify the other whenever a service status changes. + +## Decision Drivers + +* Avoid a-priori knowledge of services. +* Ease of scalability. +* Always up-to-date knowledge of the running services on a given deployment (a service registry doesn't have to necessarily be running on the same machine / network) + +## Considered Options + +* Hardcoded tuples of hostname + port +* Dynamic service registration + +## Decision Outcome + +Chosen option: "Dynamic service registration". There were some drawbacks regarding this due to introducing go-micro to Reva was from start a problem, given how much of a bloat it would mean given the little usage of it. Instead we decided to define our very own [Registry interface](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/pkg/registry/registry.go#L22-L35) on Reva and extended the runtime arguments to [allow for injecting a registry](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/cmd/revad/runtime/option.go#L53-L58). + +### Positive Consequences + +* Having dynamic service registration delegates the entire lifecycle of finding a process to the service registry. +* Removing a-priori knowledge of hostname + port for services. +* Marrying go-micro's registry and a newly defined registry abstraction on Reva. +* We will embrace go-micro interfaces by defining a third, merger interface in order to marry go-micro registry and rega revistry. +* The ability to fetch a service node relying only on its name (i.e: com.owncloud.proxy) and not on a tuple hostname + port that we rely on being preconfigured during runtime. +* Conceptually speaking, a better framework to tie all the services together. Referring to services by names is less overall confusing than having to add a service name + where it is running. A registry is agnostic to "where is it running" because it, by definition, keeps track of this specific question, so when speaking about design or functionality, it will ease communication. + +## Pros and Cons of the Options + +### Hardcoded tuples of hostname + port + +* Good, because +* Bad, because it requires thorough planning of ports. + +### Dynamic service registration + +* Good, because it abstracts the use of service lookup away to registry logic from the admin or developer. +* Good, because it allows for, through interfaces, registry injection + * This means we can have a service registry that we extensively use in oCIS and inject its functionality onto Reva. +* Bad, because it's yet another abstraction. +* Bad, because most likely the entire logic of `/todo/pool.go` on Reva would have to be rethought, and that has some implications not only across Reva, but also oCIS, since we do use that same connection pooling package. From f78f6ceeaeaa3766a1a3629adc85f953025e2ebc Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Wed, 21 Apr 2021 13:02:52 +0200 Subject: [PATCH 3/6] fix typo --- docs/ocis/adr/0005-cs3-api-account-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/adr/0005-cs3-api-account-management.md b/docs/ocis/adr/0005-cs3-api-account-management.md index be59318afe..4f9a0ff4ad 100644 --- a/docs/ocis/adr/0005-cs3-api-account-management.md +++ b/docs/ocis/adr/0005-cs3-api-account-management.md @@ -14,7 +14,7 @@ geekdocFilePath: 0004-cs3api-user-management.md Technical Story: [Write only management API for User and Group resources](https://github.com/cs3org/cs3apis/pull/119) -## Context and Problem Statementf +## Context and Problem Statement What would be a more effective way of using network resources and handle account management within the oCIS-Reva ecosystem? Separating account management at the service level is pragmatic and allows for fast iterations, but also steadily accumulates inconsistencies and bloats technical debt. From 1782b27b511d2adba0d6ccba97529d6813a70dbe Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Mon, 26 Apr 2021 10:03:56 +0200 Subject: [PATCH 4/6] apply revie feedback --- docs/ocis/adr/0006-service-discovery.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/ocis/adr/0006-service-discovery.md b/docs/ocis/adr/0006-service-discovery.md index 1364689618..e8dc251304 100644 --- a/docs/ocis/adr/0006-service-discovery.md +++ b/docs/ocis/adr/0006-service-discovery.md @@ -16,7 +16,7 @@ Technical Story: [Introduce Named Services.](https://github.com/cs3org/reva/pull ## Context and Problem Statement -Reva relies heavily on config files. Implications of this approach are having to know a-priori where a service is running (host + port). We want to move away from hardcoded values and rely instead on named services for service discovery. Furthermore, we would like both platforms (Reva + oCIS) to have the same source of truth at any given time, not having one to notify the other whenever a service status changes. +Reva relies heavily on config files. A known implication of this approach are having to know a-priori where a service is running (host + port). We want to move away from hardcoded values and rely instead on named services for service discovery. Furthermore, we would like both platforms (Reva + oCIS) to have the same source of truth at any given time, not having one to notify the other whenever a service status changes. ## Decision Drivers @@ -31,14 +31,14 @@ Reva relies heavily on config files. Implications of this approach are having to ## Decision Outcome -Chosen option: "Dynamic service registration". There were some drawbacks regarding this due to introducing go-micro to Reva was from start a problem, given how much of a bloat it would mean given the little usage of it. Instead we decided to define our very own [Registry interface](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/pkg/registry/registry.go#L22-L35) on Reva and extended the runtime arguments to [allow for injecting a registry](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/cmd/revad/runtime/option.go#L53-L58). +Chosen option: "Dynamic service registration". There were some drawbacks regarding this due to introducing go-micro to Reva was from start an issue. Given the little usage of go-micro we need, we decided instead to define our very own [Registry interface](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/pkg/registry/registry.go#L22-L35) on Reva and extended the runtime arguments to [allow for injecting a registry](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/cmd/revad/runtime/option.go#L53-L58). ### Positive Consequences * Having dynamic service registration delegates the entire lifecycle of finding a process to the service registry. * Removing a-priori knowledge of hostname + port for services. * Marrying go-micro's registry and a newly defined registry abstraction on Reva. -* We will embrace go-micro interfaces by defining a third, merger interface in order to marry go-micro registry and rega revistry. +* We will embrace go-micro interfaces by defining a third merger interface in order to marry go-micro registry and rega revistry. * The ability to fetch a service node relying only on its name (i.e: com.owncloud.proxy) and not on a tuple hostname + port that we rely on being preconfigured during runtime. * Conceptually speaking, a better framework to tie all the services together. Referring to services by names is less overall confusing than having to add a service name + where it is running. A registry is agnostic to "where is it running" because it, by definition, keeps track of this specific question, so when speaking about design or functionality, it will ease communication. @@ -46,7 +46,8 @@ Chosen option: "Dynamic service registration". There were some drawbacks regardi ### Hardcoded tuples of hostname + port -* Good, because +* Good, because firewalls are easier to configure since IP are static. +* Good, because the mental model required is easier to grasp as IP addresses can be easily bundled. * Bad, because it requires thorough planning of ports. ### Dynamic service registration @@ -55,4 +56,4 @@ Chosen option: "Dynamic service registration". There were some drawbacks regardi * Good, because it allows for, through interfaces, registry injection * This means we can have a service registry that we extensively use in oCIS and inject its functionality onto Reva. * Bad, because it's yet another abstraction. -* Bad, because most likely the entire logic of `/todo/pool.go` on Reva would have to be rethought, and that has some implications not only across Reva, but also oCIS, since we do use that same connection pooling package. +* Bad, because firewalls are harder to configure with dynamic IPs.f From ae71239e9c332b8f0a935445779f13ce2bb1f1a9 Mon Sep 17 00:00:00 2001 From: Alex Unger <6905948+refs@users.noreply.github.com> Date: Mon, 26 Apr 2021 10:06:59 +0200 Subject: [PATCH 5/6] Update docs/ocis/adr/0005-cs3-api-account-management.md Co-authored-by: Michael Barz --- docs/ocis/adr/0005-cs3-api-account-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ocis/adr/0005-cs3-api-account-management.md b/docs/ocis/adr/0005-cs3-api-account-management.md index 4f9a0ff4ad..f6005cfc23 100644 --- a/docs/ocis/adr/0005-cs3-api-account-management.md +++ b/docs/ocis/adr/0005-cs3-api-account-management.md @@ -208,4 +208,4 @@ The new account management workflow will result in: └───────────────────────────────────────────────────┘ ``` -This flow allows Reva and oCIS Proxy to runs any middleware logic in order to validate a request. The communication between the proposed Admin api (CS3 API messages) with the IDM (GLAuth) are specific to the _drivers_. +This flow allows Reva and oCIS Proxy to run any middleware logic in order to validate a request. The communication between the proposed Admin api (CS3 API messages) and the IDM (GLAuth) are specific to the _drivers_. From 2816ec7f8bf24ce717e3522a0a2de381fcac16fc Mon Sep 17 00:00:00 2001 From: "A.Unger" Date: Mon, 26 Apr 2021 10:15:17 +0200 Subject: [PATCH 6/6] apply review feedback --- docs/ocis/adr/0005-cs3-api-account-management.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ocis/adr/0005-cs3-api-account-management.md b/docs/ocis/adr/0005-cs3-api-account-management.md index f6005cfc23..51dfce516b 100644 --- a/docs/ocis/adr/0005-cs3-api-account-management.md +++ b/docs/ocis/adr/0005-cs3-api-account-management.md @@ -200,9 +200,9 @@ The new account management workflow will result in: │ │ │ │ │ │ -│ (1) solidify driver interfaces with write │ │ │ -│ (2) an example of a driver could be GLAuth │ +│ │ +│ an example of a driver could be GLAuth │ │ implementing the user management portion of the │ │ GraphAPI │ └───────────────────────────────────────────────────┘