mirror/opencloud
1
0
Fork 0
mirror of https://github.com/opencloud-eu/opencloud.git synced 2026-05-16 04:03:58 -04:00
Code Issues Packages Projects Releases Wiki Activity

Remove the docs folder for now

Browse Source 
We'll bring back documentation once we decided on how to do it.
This commit is contained in:
André Duffeck
2025-01-20 15:04:42 +01:00
committed by Jörn Friedrich Dreyer
parent 6b36ce329d
commit 5c57982db0
258 changed files with 13 additions and 41644 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/.gitignore vendored
View File

@@ -1,5 +0,0 @@
hugo/
grpc_apis/
mutagen.yml.lock
helpers/output/*
services/**/_index.md

55
docs/Makefile
View File

@@ -1,55 +0,0 @@
SHELL := bash
include ../.bingo/Variables.mk
.PHONY: help
help:
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk -F'[:;##]' '{printf "\033[36m%-30s\033[0m %s\n", $$2, $$NF}'
.PHONY: docs-generate
docs-generate: ## run docs-generate for all oCIS services
@pushd helpers && go run .; popd
.PHONY: docs-init
docs-init:
@mkdir -p hugo/content/
@mkdir -p hugo/public/
@touch hugo/public/.nojekyll
@cd hugo && git init
@cd hugo && git config advice.detachedHead false
@cd hugo && git remote rm origin || true
@cd hugo && git remote add origin https://github.com/owncloud/owncloud.github.io
@cd hugo && git fetch --depth=1
@cd hugo && git checkout origin/main -f
@$(MAKE) -C hugo theme
.PHONY: docs-serve
docs-serve: docs-init docs-generate docs-copy ## serve docs with hugo
@bash -c "trap 'trap - SIGINT SIGTERM ERR; $(MAKE) --no-print-directory docs-sync-stop; exit 0' SIGINT SIGTERM ERR; $(MAKE) --no-print-directory docs-sync-start && $(MAKE) --no-print-directory hugo-serve"
.PHONY: test
test: $(HUGO)
@cd hugo && $(HUGO)
.PHONY: hugo-serve
hugo-serve: $(HUGO)
@cd hugo && $(HUGO) server
.PHONY: docs-copy
docs-copy: docs-init docs-sync-start docs-sync-stop
.PHONY: docs-sync-start
docs-sync-start: $(MUTAGEN)
@$(MUTAGEN) project terminate || true
@$(MUTAGEN) daemon stop || true
@$(MUTAGEN) project start
@$(MUTAGEN) project flush
.PHONY: docs-sync-stop
docs-sync-stop: $(MUTAGEN)
@$(MUTAGEN) project terminate
@$(MUTAGEN) daemon stop
.PHONY: clean
clean: ## clean up docs build artifacts
@rm -rf hugo

5
docs/README.md
View File

@@ -1,5 +0,0 @@
# Documentation
To contribute to documentation please see also: [Documentation](http://owncloud.dev/ocis/development/build-docs/)
Note that when running a helper like `make -C docs docs-generate` locally, you will see the generated output in e.g. docs/services/<service-name>. When merging a PR, the pipeline will generate the output in the master branch but finally moves it into the `docs` branch. You will therefore _not_ see changes in the docs folder of the master branch!

46
docs/_index.md
View File

@@ -1,46 +0,0 @@
---
title: ownCloud
---
{{< toc >}}
## Admin Documentation
Before you start reading, if you are interested in:
- learning ocis from an admin perspective,
- different deployment scenarios,
- deployment examples,
- detailed settings and more
we would recommend to continue with the [ownCloud Admin Documentation for Infinite Scale](https://doc.owncloud.com/ocis/next/).
## Developer Documentation
Welcome to our developer documentation. Here you find documentation with focus for *developers*:
- [oCIS]({{< ref "./ocis" >}}) server
- oCIS Services
- Clients like:
- [ownCloud Web](https://github.com/owncloud/web) - the new web frontend for oCIS and ownCloud
- [ownCloud Android app](https://github.com/owncloud/android)
- [ownCloud iOS app](https://github.com/owncloud/ios-app)
- [ownCloud Desktop Syncing Client](https://github.com/owncloud/client)
- Integrations
## We love open source
The oCIS server is Apache v2 licensed.
The lower storage layer of oCIS is defined by the CS3 APIs and implemented in the REVA project. Our goal is to develop the CS3 APIs to an open standard and collaborate on the open source REVA reference implementation for CS3 APIs.
You can also find all client sources on [GitHub](https://github.com/owncloud/).
## Join the oCIS Community
The [server repository](https://github.com/owncloud/ocis) on [GitHub](https://www.github.com) is a good entry point to the oCIS project. In addition to that there are also ownCloud projects for clients for [iOS](https://github.com/owncloud/ios-app), [Android](https://github.com/owncloud/android), the major [Desktop](https://github.com/owncloud/desktop) platforms and [ownCloud Web](https://github.com/owncloud/web).
To chat about development, [join our public chat](https://talk.owncloud.com/channel/ocis)
If you want to help and improve ownCloud or oCIS, start coding or open issues on GitHub in the related repository.
We are very happy to hear your feedback and ideas!

53
docs/apis/_index.md
View File

@@ -1,53 +0,0 @@
---
title: APIs
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
{{< toc-tree >}}
Infinite Scale provides a large set of different **application programming interfaces (APIs)**. Infinite Scale is built by microservices. That means many calls to "functions" in the code are remote calls.
Basically we have two different API "universes": [HTTP](http) and [gRPC](grpc_apis).
{{< columns >}} <!-- begin columns block -->
{{< figure src="/ocis/static/http-logo.png" width="70%" alt="Image sourced from https://commons.wikimedia.org/ free of licenses" >}}
<--->
{{< figure src="/ocis/static/grpc-logo.png" width="70%" alt="Image sourced from https://grpc.io/ under CC 4.0 BY license" >}}
{{< /columns >}}
For inter-service-communication we are using mostly gRPC calls because it has some advantages. In the future, clients may decide to use gRPC directly to make use of these advantages.
{{< figure src="/ocis/static/ocis-apis.drawio.svg" class="page-image">}}
## [HTTP](http)
HTTP APIs are mostly used for client <-> server communication. Modern applications are embracing a [RESTful](https://en.wikipedia.org/wiki/Representational_state_transfer) software architecture style. REST APIs are using the HTTP protocol to transfer data between clients and servers. All our clients talk to the Server using HTTP APIs. This has legacy reasons and is well-supported across many platforms and technologies. Infinite Scale uses an [HTTP API gateway](../services/proxy) to route client requests to the correct service.
### OpenAPI
It is best practise to define APIs and their behavior by a spec. We are using the OpenAPI standard for all new APIs. The [OpenAPI Specification](https://swagger.io/specification/), previously known as the Swagger Specification, is a specification for a machine-readable interface definition language for describing, producing, consuming and visualizing RESTful web services. Previously part of the Swagger framework, it became a separate project in 2016, overseen by the OpenAPI Initiative, an open-source collaboration project of the Linux Foundation. Swagger and some other tools can generate code, documentation and test cases from interface files.
### RFC
Some APIs have become a de facto standard and are additionally covered by an [RFC](https://en.wikipedia.org/wiki/Request_for_Comments).
## [gRPC](grpc_apis)
In gRPC, a client application can directly call methods on a server application on a different machine as if it was a local object. This makes it easier to create distributed applications based on microservices. In gRPC we can define a service and specify the methods that can be called remotely. A gRPC client has a stub that provides the same methods and types as the server.
Infinite Scale uses a [gRPC API Gateway](../services/gateway) to route the requests to the correct service.
### Protobuf
gRPC APIs are typically defined by [Protocol buffers](https://developers.google.com/protocol-buffers/docs/overview). The different client and server stubs are created from ``*.proto`` files by code generation tools.
## Versioning
There are different standards for API versioning: Through URL, through request parameter, through custom header and through content negotiation. Ocis uses the versioning by URL concept although this creates a big code footprint. The versioning follows [SemVer](https://semver.org). We update the major version number when breaking changes are needed. Clients can decide which major version they use through the request URL. The specific implementation is documented on each API.

63
docs/apis/grpc_apis/_index.md
View File

@@ -1,63 +0,0 @@
---
title: gRPC
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/grpc_apis/
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
{{< toc-tree >}}
## **R**emote &nbsp; **P**rocedure &nbsp; **C**alls
[gRPC](https://grpc.io) is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.
## Advantages of gRPC
{{< columns >}}
### {{< icon "gauge-high" >}} &nbsp; Performance
gRPC uses http/2 by default and is faster than REST. When using protocol buffers for encoding, the information comes on and off the wire much faster than JSON. Latency is an important factor in distributed systems. JSON encoding creates a noticeable factor of latency. For distributed systems and high data loads, gRPC can actually make an important difference. Other than that, gRPC supports multiple calls via the same channel and the connections are bidirectional. A single connection can transmit requests and responses at the same time. gRPC keeps connections open to reuse the same connection again which prevents latency and saves bandwidth.
<--->
### {{< icon "helmet-safety" >}} &nbsp; Robustness
gRPC empowers better relationships between clients and servers. The rules of communication are strictly enforced. That is not the case in REST calls, where the client and the server can send and receive anything they like and hopefully the other end understands what to do with it. In gRPC, to make changes to the communication, both client and server need to change accordingly. This prevents mistakes specially in microservice architectures.
{{< /columns >}}
{{< columns >}}
### {{< icon "magnifying-glass-plus" >}} &nbsp; Debuggability
gRPC requests are re-using the same context and can be tracked or traced across multiple service boundaries.
This helps to identify slow calls and see what is causing delays. It is possible to cancel requests which cancels
them on all involved services.
<--->
### {{< icon "boxes-stacked" >}} &nbsp; Microservices
gRPC has been evolving and has become the best option for communication between microservices because of its unmatched
performance and its polyglot nature. One of the biggest strengths of microservices is the freedom of programming
languages and technologies. By using gRPC we can leverage all the advantages of strictly enforced communication
standards combined with freedom of choice between different programming languages - whichever would fit best.
{{< /columns >}}
{{< hint type=info title="gRPC Advantages" >}}
- http/2
- protocol buffers
- reusable connections
- multi language support
{{< /hint >}}
## CS3 APIs
{{< figure src="/ocis/static/cs3org.png" >}}
The [CS3 APIs](https://github.com/cs3org/cs3apis) connect storages and application providers.
The CS3 APIs follow Google and Uber API design guidelines, specially on error handling and naming convention. You can read more about these
guidelines at https://cloud.google.com/apis/design/ and https://github.com/uber/prototool/blob/dev/style/README.md.
The CS3 APIs use [Protocol Buffers version 3 (proto3)](https://github.com/protocolbuffers/protobuf) as their
Interface Definition Language (IDL) to define the API interface and the structure of the payload messages.

11
docs/apis/http/_index.md
View File

@@ -1,11 +0,0 @@
---
title: Http
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
The [Hypertext Transfer Protocol (HTTP)](https://www.rfc-editor.org/rfc/rfc2616) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. HTTP is the foundation of data communication for the World Wide Web, where hypertext documents include hyperlinks to other resources that the user can easily access, for example by a mouse click or by tapping the screen in a web browser.
Development of HTTP was initiated by Tim Berners-Lee at CERN in 1989 and summarized in a simple document describing the behavior of a client and a server using the first HTTP protocol version that was named 0.9. That first version of HTTP protocol soon evolved into a more elaborated version that was the first draft toward a far future version 1.0

142
docs/apis/http/authorization.md
View File

@@ -1,142 +0,0 @@
---
title: Authorization
weight: 40
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/
geekdocFilePath: authorization.md
---
In its default configuration, Infinite Scale supports three authentication methods as outlined on the [OIDC official site](https://openid.net/specs/openid-connect-core-1_0.html#rfc.section.3):
1. Authorization Code Flow
2. Implicit Flow
3. Hybrid Flow
For detailed information on Infinite Scale's support for OpenID Connect (OIDC), please consult the [OIDC section](https://owncloud.dev/ocis/identity-provider/oidc).
To authenticate a client app using OIDC, both `client_id` and `client_secret` are essential. Infinite Scale does not offer dynamic registration. The required data for the default [ownCloud clients](https://doc.owncloud.com/server/next/admin_manual/configuration/user/oidc/oidc.html#client-ids-secrets-and-redirect-uris) can be found in the link and are availble for the following apps:
- Desktop
- Android
- iOS
While selecting an ownCloud client for authentication, take note of specific limitations such as the `Redirect URI`:
| Source | Redirect URI |
|------|--------|
|Android|oc://android.owncloud.com|
|iOS|oc://ios.owncloud.com|
|Desktop|http://127.0.0.1 <br> http://localhost |
In this example, the desktop app's `client_id` and `client_secret` are being used.
```bash
client_id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69
client_secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh
```
## Authorization Code Flow
1. Requesting authorization
To initiate the OIDC Code Flow, you can use tools like curl and a web browser.
The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard):
```plaintext
https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=code&redirect_uri=http://path-to-redirect-uri
```
After a successful authentication, the browser will redirect to a URL that looks like this:
```plaintext
http://path-to-redirect-uri?code=mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw&scope=openid%20profile%20email%20offline_access&session_state=32b08dd...&state=
```
For the next step extract the code from the URL.
In the above example,
the code is `mfWsjEL0mc8gx0ftF9LFkGb__uFykaBw`
2. Requesting an access token
The next step in the OIDC Code Flow involves an HTTP POST request
to the token endpoint of the **Infinite Scale Identity Server**.
```bash
curl -vk -X POST https://ocis.test/konnect/v1/token \
-d "grant_type=authorization_code" \
-d "code=3a3PTcO-WWXfN3l1mDN4u7G5PzWFxatU" \
-d "redirect_uri=http:path-to-redirect-uri" \
-d "client_id=client_id" \
-d "client_secret=client_secret"
```
Response looks like this:
```json
{
"access_token": "eyJhbGciOid...",
"token_type": "Bearer",
"id_token": "eyJhbGciOi...",
"refresh_token": "eyJhbGciOiJ...",
"expires_in": 300
}
```
3. Refreshing an access token
If the access token has expired, you can get a new one using the refresh token.
```bash
curl -vk -X POST https://ocis.test/konnect/v1/token \
-d "grant_type=refresh_token" \
-d "refresh_token=eyJhbGciOiJ..." \
-d "redirect_uri=http://path-to-redirect-uri" \
-d "client_id=client_id" \
-d "client_secret=client_secret"
```
Response looks like this:
```json
{
"access_token": "eyJhbGciOi...",
"token_type": "Bearer",
"expires_in": 300
}
```
## Implicit Code Flow
When using the implicit flow, tokens are provided in a URI fragment of the redirect URL.
Valid values for the `response_type` request parameter are:
- token
- id_token token
{{< hint type=warning title="Important Warning" >}}
If you are using the implicit flow, `nonce` parameter is required in the initial `/authorize` request.
`nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18=`
bash command to generate cryptographically random value
```bash
openssl rand -base64 32
```
{{< /hint >}}
The user should be directed to a URL to authenticate and give their consent (bypassing consent is against the standard):
```bash
https://ocis.test/signin/v1/identifier/_/authorize?client_id=client_id&scope=openid+profile+email+offline_access&response_type=id_token+token&redirect_uri=http://path-to-redirect-uri&nonce=pL3UkpAQPZ8bTMGYOmxHY/dQABin8yrqipZ7iN0PY18=
```
After a successful authentication, the browser will redirect to a URL that looks like this:
```bash
http://path-to-redirect-uri#access_token=eyJhbGciOiJQUzI...&expires_in=300&id_token=eyJhbGciOiJ...&scope=email%20openid%20profile&session_state=c8a1019f5e054d...&state=&token_type=Bearer
```
For the next step, extract the access_token from the URL.
```bash
access_token = 'eyJhbGciOiJQ...'
```
## Hybrid Flow
The Hybrid Flow in OpenID Connect melds features from both the Implicit and Authorization Code flows. It allows clients to directly retrieve certain tokens from the Authorization Endpoint, yet also offers the option to acquire additional tokens from the Token Endpoint.
The Authorization Server redirects back to the client with appropriate parameters in the response, based on the value of the response_type request parameter:
- code token
- code id_token
- code id_token token

72
docs/apis/http/graph/_index.md
View File

@@ -1,72 +0,0 @@
---
title: "LibreGraph"
date: 2018-05-02T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
The LibreGraph API is a REST Api which is inspired by the [Microsoft Graph API](https://developer.microsoft.com/en-us/graph). It tries to stay compliant with the Microsoft Graph API and aims to be the Next Generation Api in Infinite Scale where we want to support most of the features of the platform.
The [API specification](https://github.com/owncloud/libre-graph-api) is available in the OpenApi 3 standard and there are generated client and server [SDKs](https://github.com/owncloud/libre-graph-api#clients) available. You can browse the API with the [Swagger UI](https://owncloud.dev/libre-graph-api/).
## Calling the LibreGraph API
```sh
{HTTP method} https://ocis.url/graph/{version}/{resource}?{query-parameters}
```
The request component consists of:
| Component | Description |
|--------------------|-------------------------------------------------------------------------|
| {HTTP method} | The HTTP method which is used in the request. |
| {version} | The version of the LibreGraph API used by the client. |
| {resource} | The LibreGraph Resource which the client is referencing in the request. |
| {query-parameters} | Optional parameters for the request to customize the response. |
### HTTP methods
| Method | Description |
|--------|-------------------------------|
| GET | Read data from a resource. |
| POST | Create a new resource. |
| PATCH | Update an existing resource. |
| PUT | Replace an existing resource. |
| DELETE | Delete an existing resource. |
The methods `GET` and `DELETE` need no request body. The methods `POST`, `PATCH` and `PUT` require a request body, normally in JSON format to provide the needed values.
### Version
Infinite Scale currently provides the version `v1.0`.
### Resource
A resource could be an entity or a complex type and is usually defined by properties. Entities are always recognizable by an `Id` property. The URL contains the resource which you are interacting with e.g. `/me/drives` or `/groups/{group-id}`.
Each resource could possibly require different permissions. Usually you need permissions on a higher level for creating or updating an existing resource than for reading.
### Query parameters
Query parameters can be OData system query options, or other strings that a method accepts to customize its response.
You can use optional OData system query options to include more or fewer properties than the default response, filter the response for items that match a custom query, or provide additional parameters for a method.
For example, adding the following filter parameter restricts the drives returned to only those with the driveType property of `project`.
```shell
GET https://ocis.url/graph/v1.0/drives?$filter=driveType eq 'project'
```
For more information about OData query options please check the [API specification](https://github.com/owncloud/libre-graph-api) and the provided examples.
### Authorization
For development purposes the examples in the developer documentation use Basic Auth. It is disabled by default and should only be enabled by setting `PROXY_ENABLE_BASIC_AUTH` in [the proxy](../../../services/proxy/configuration/#environment-variables) for development or test instances.
To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer <token>'` header. A `<token>` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like [oidc-agent](https://github.com/indigo-dc/oidc-agent) should be used. The graph endpoints that support a preconfigured token can be found in the [API specification](https://github.com/owncloud/libre-graph-api)
## Resources
{{< toc-tree >}}

275
docs/apis/http/graph/groups.md
View File

@@ -1,275 +0,0 @@
---
title: Groups
weight: 40
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: groups.md
---
{{< toc >}}
## Groups API
The Groups API is implementing a subset of the functionality of the
[MS Graph Group resource](https://docs.microsoft.com/en-us/graph/api/resources/group?view=graph-rest-1.0)
The JSON representation of a Group as handled by the Groups API looks like this:
```
{
"displayName": "group",
"id": "f0d97060-da16-4b0d-9fa4-d1ec43afc5f1"
}
```
Our implementation currently supports two Attributes for a Group:
| Attribute | Description |
|---------------|-----------------------------------------------------------------------------------------------------------------------------|
| displayName | The groups name |
| id | An unique, stable readonly identifier for the group that stays the same for the whole lifetime of the Group, usually a UUID |
### Reading groups
#### `GET /groups`
Returns a list of all groups
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/groups' -u user:password
```
Response:
```
{
"value": [
{
"displayName": "group",
"id": "38580a2e-7018-42ed-aff6-b2af0b4e9790"
},
{
"displayName": "Example Users",
"id": "7a20f238-8a22-4458-902d-47674c317e5f"
}
]
}
```
#### `GET /groups?$expand=members`
Returns a list of all groups including its members
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/groups?$expand=members' -u user:password
```
Response:
```
{
"value": [
{
"displayName": "group",
"id": "38580a2e-7018-42ed-aff6-b2af0b4e9790",
"members": [
{
"displayName": "user1",
"id": "2e7b7e23-6c42-4d34-81b0-2bed34e51983",
"mail": "user1@example.org",
"onPremisesSamAccountName": "user1"
},
{
"displayName": "user2",
"id": "b45c9e35-0d95-4165-96bc-68bff4a316ed",
"mail": "user2@example.org",
"onPremisesSamAccountName": "user2"
}
]
},
{
"displayName": "Example Users",
"id": "7a20f238-8a22-4458-902d-47674c317e5f",
"members": [
{
"displayName": "user3",
"id": "026fbfef-79ef-4f5d-887b-9eaf42777239",
"mail": "user3@example.org",
"onPremisesSamAccountName": "user3"
}
]
}
]
}
```
#### `GET /groups/{groupid}`
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' -u user:password
```
Response:
```
{
"displayName": "Example Users",
"id": "7a20f238-8a22-4458-902d-47674c317e5f"
}
```
#### `GET /groups/{groupid}?$expand=members`
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f?$expand=members' -u user:password
```
Response:
```
{
"displayName": "Example Users",
"id": "7a20f238-8a22-4458-902d-47674c317e5f",
"members": [
{
"displayName": "user3",
"id": "026fbfef-79ef-4f5d-887b-9eaf42777239",
"mail": "user3@example.org",
"onPremisesSamAccountName": "user3"
}
]
}
```
### Getting Group Members
#### `GET /groups/{groupid}/members`
Returns a list of User objects that are members of a group.
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members' -u user:password
```
Response:
```
[
{
"displayName": "Test User",
"id": "c54b0588-7157-4521-bb52-c1c8ca84ea71",
"mail": "example@example.org",
"onPremisesSamAccountName": "example"
},
{
"displayName": "Albert Einstein",
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"mail": "einstein@example.org",
"onPremisesSamAccountName": "einstein"
}
]
```
### Creating / Updating Groups
#### `POST /groups`
Use this to create a new group.
h
##### Request Body
Note the missing `"id"` Attribute. It will be generated by the server:
```
{
"displayName": "Example Users"
}
```
##### Response
When successful, the response will return the new group including the newly allocated `"id"`:
```
{
"displayName": "Example Users",
"id": "7a20f238-8a22-4458-902d-47674c317e5f"
}
```
#### `DELETE /groups/{id}`
Example:
```
curl -k --request DELETE 'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f' -u user:password
```
When successful the API returns no response body and the HTTP status code 204 (No Content)
#### `PATCH /groups/{id}`
Updating attributes of a single group is supposed to be done with a patch request. This is however currently not fully
implemented for our write-enabled backends. The PATCH request can however be used to add multiple members to a group at once.
See below.
### Adding a single member to a group
#### `POST /groups/{id}/members/$ref`
The request body contains a single attribute "`@odata.id`" referencing the new member of the group by URI. Example:
```
curl -k --header "Content-Type: application/json" \
--request POST --data \
'{ "@odata.id": "https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51" }' \
'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/$ref' -u user:password
```
When successful the API returns no response body and the HTTP status code 204 (No Content)
### Adding multiple members in a single request
#### `PATCH /groups/{id}`
The request body contains the attribute `members@odata.bind` holding a list of URI references for the new members.
Example:
```
{
"members@odata.bind": [
"https://localhost:9200/graph/v1.0/users/4c510ada-c86b-4815-8820-42cdf82c3d51",
"https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71"
]
}
```
When successful the API returns no response body and the HTTP status code 204 (No Content)
### Removing a member
#### `DELETE /groups/{groupid}/members/{id}/$ref`
Example
```
curl -k --request DELETE \
'https://localhost:9200/graph/v1.0/groups/7a20f238-8a22-4458-902d-47674c317e5f/members/4c510ada-c86b-4815-8820-42cdf82c3d51/$ref' \
-u user:password
```
When successful the API returns no response body and the HTTP status code 204 (No Content)

184
docs/apis/http/graph/permissions.md
View File

@@ -1,184 +0,0 @@
---
title: Permissions
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: permissions.md
---
{{< toc >}}
## Permissions API
The Permissions API is implementing a subset of the functionality of the
[MS Graph Permission resource](https://learn.microsoft.com/en-us/graph/api/resources/permission?view=graph-rest-1.0).
### Example Permissions
The JSON representation of a Drive, as handled by the Spaces API, looks like this:
````json
{
"@libre.graph.permissions.roles.allowedValues": [
{
"id": "b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5",
"description": "Allows reading the shared file or folder",
"displayName": "Viewer",
"@libre.graph.weight": 1
},
{
"id": "fb6c3e19-e378-47e5-b277-9732f9de6e21",
"description": "Allows reading and writing the shared file or folder",
"displayName": "Editor",
"@libre.graph.weight": 2
},
{
"id": "312c0871-5ef7-4b3a-85b6-0e4074c64049",
"description": "Allows managing a space",
"displayName": "Manager",
"@libre.graph.weight": 3
},
{
"id": "4916f47e-66d5-49bb-9ac9-748ad00334b",
"description": "Allows creating new files",
"displayName": "File Drop",
"@libre.graph.weight": 4
}
],
"@libre.graph.permissions.actions.allowedValues": [
"libre.graph/driveItem/basic/read",
"libre.graph/driveItem/permissions/read",
"libre.graph/driveItem/upload/create",
"libre.graph/driveItem/standard/allTasks",
"libre.graph/driveItem/upload/create"
],
"value": [
{
"id": "67445fde-a647-4dd4-b015-fc5dafd2821d",
"link": {
"type": "view",
"webUrl": "https://cloud.example.org/s/fhGBMIkKFEHWysj"
}
},
{
"id": "34646ab6-be32-43c9-89e6-987e0c237e9b",
"roles": [
"b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5"
],
"grantedToV2": [
{
"user": {
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"displayName": "Albert Einstein"
}
}
]
},
{
"id": "81d5bad3-3eff-410a-a2ea-eda2d14d4474",
"roles": [
"b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5"
],
"grantedToV2": [
{
"user": {
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"displayName": "Albert Einstein"
}
}
]
},
{
"id": "b470677e-a7f5-4304-8ef5-f5056a21fff1",
"roles": [
"b1e2218d-eef8-4d4c-b82d-0f1a1b48f3b5"
],
"grantedToV2": [
{
"user": {
"id": "f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c",
"displayName": "Marie Skłodowska Curie"
}
}
]
},
{
"id": "453b02be-4ec2-4e7d-b576-09fc153de812",
"roles": [
"fb6c3e19-e378-47e5-b277-9732f9de6e21"
],
"grantedToV2": [
{
"user": {
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"displayName": "Albert Einstein"
}
}
],
"expirationDateTime": "2018-07-15T14:00:00.000Z"
},
{
"id": "86765c0d-3905-444a-9b07-76201f8cf7df",
"roles": [
"312c0871-5ef7-4b3a-85b6-0e4074c64049"
],
"grantedToV2": [
{
"group": {
"id": "167cbee2-0518-455a-bfb2-031fe0621e5d",
"displayName": "Philosophy Haters"
}
}
]
},
{
"id": "c42b5cbd-2d65-42cf-b0b6-fb6d2b762256",
"grantedToV2": [
{
"user": {
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"displayName": "Albert Einstein"
}
}
],
"@libre.graph.permissions.actions": [
"libre.graph/driveItem/basic/read",
"libre.graph/driveItem/path/update"
]
}
]
}
````
## Creating Share Invitation / Link
### Create a link share `POST /drives/{drive-id}/items/{item-id}/createLink`
https://owncloud.dev/libre-graph-api/#/drives.permissions/CreateLink
### Create a user/group share `POST /drives/{drive-id}/items/{item-id}/invite`
https://owncloud.dev/libre-graph-api/#/drives.permissions/Invite
## Reading Permissions
### List the effective sharing permissions on a driveitem `GET /drives/{drive-id}/items/{item-id}/permissions`
https://owncloud.dev/libre-graph-api/#/drives.permissions/ListPermissions
### List Get sharing permission for a file or folder `GET /drives/{drive-id}/items/{item-id}/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/drives.permissions/GetPermission
## Updating Permissions
### Updating sharing permission `POST /drives/{drive-id}/items/{item-id}/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/drives.permissions/UpdatePermission
### Set password of permission `POST /drives/{drive-id}/items/{item-id}/permissions/{perm-id}/setPassword`
https://owncloud.dev/libre-graph-api/#/drives.permissions/SetPermissionPassword
### Deleting permission `DELETE /drives/{drive-id}/items/{item-id}/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/drives.permissions/DeletePermission

38
docs/apis/http/graph/role.md
View File

@@ -1,38 +0,0 @@
---
title: Role
weight: 60
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: permissions.md
---
{{< toc >}}
## Role API
The Roles API is implementing a subset of the functionality of the
[MS Graph Role Management](https://learn.microsoft.com/en-us/graph/api/resources/rolemanagement?view=graph-rest-1.0).
## Role Management
### List roleDefinitions `GET /v1beta1/roleManagement/permissions/roleDefinitions`
https://owncloud.dev/libre-graph-api/#/roleManagement/ListPermissionRoleDefinitions
### Get unifiedRoleDefinition `GET /drives/{drive-id}/items/{item-id}/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/roleManagement/GetPermissionRoleDefinition
## Role Assignment
### Get appRoleAssignments of a user `GET /v1.0/users/{user-id}/appRoleAssignments`
https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.ListAppRoleAssignments
### Grant an appRoleAssignment to a user `POST /v1.0/users/{user-id}/appRoleAssignments`
https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.CreateAppRoleAssignments
### Delete the appRoleAssignment from a user `DELETE /v1.0/users/{user-id}/appRoleAssignments/{appRoleAssignment-id}`
https://owncloud.dev/libre-graph-api/#/user.appRoleAssignment/user.DeleteAppRoleAssignments

494
docs/apis/http/graph/spaces.md
View File

@@ -1,494 +0,0 @@
---
title: Spaces
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: spaces.md
---
{{< toc >}}
## Spaces API
The Spaces API is implementing a subset of the functionality of the
[MS Graph Drives resource](https://learn.microsoft.com/en-us/graph/api/resources/drive?view=graph-rest-1.0).
### Example Space
The JSON representation of a Drive, as handled by the Spaces API, looks like this:
````json
{
"driveAlias": "project/mars",
"driveType": "project",
"id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925",
"lastModifiedDateTime": "2023-01-24T21:19:26.417055+01:00",
"name": "Mars",
"owner": {
"user": {
"displayName": "",
"id": "89ad5ad2-5fdb-4877-b8c9-601a9670b925"
}
},
"quota": {
"remaining": 999853685,
"state": "normal",
"total": 1000000000,
"used": 146315
},
"root": {
"eTag": "\"910af0061161c42d8d1224df6c4a2527\"",
"id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925",
"permissions": [
{
"grantedToIdentities": [
{
"user": {
"displayName": "Admin",
"id": "some-admin-user-id-0000-000000000000"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925"
},
"special": [
{
"eTag": "\"f97829324f63ce778095334cfeb0097b\"",
"file": {
"mimeType": "image/jpeg"
},
"id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!40171bea-3263-47a8-80ef-0ca20c37f45a",
"lastModifiedDateTime": "2022-02-15T17:11:50.000000496+01:00",
"name": "Mars_iStock-MR1805_20161221.jpeg",
"size": 146250,
"specialFolder": {
"name": "image"
},
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/Mars_iStock-MR1805_20161221.jpeg"
},
{
"eTag": "\"ff38b31d8f109a4fbb98ab34499a3379\"",
"file": {
"mimeType": "text/markdown"
},
"id": "storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925!e2167612-7578-46e2-8ed7-971481037bc1",
"lastModifiedDateTime": "2023-01-24T21:10:23.661841+01:00",
"name": "readme.md",
"size": 65,
"specialFolder": {
"name": "readme"
},
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925%2189ad5ad2-5fdb-4877-b8c9-601a9670b925/.space/readme.md"
}
],
"webUrl": "https://localhost:9200/f/storage-users-1$89ad5ad2-5fdb-4877-b8c9-601a9670b925"
}
````
## Creating Spaces
### Create a single space `POST /drives`
https://owncloud.dev/libre-graph-api/#/drives/CreateDrive
### Create a space item (Enable sync) `POST /drives/{drive-id}/root/children`
https://owncloud.dev/libre-graph-api/#/drives.root/CreateDriveItem
## Reading Spaces
```shell
GET https://ocis.url/graph/{version}/{me/}drives/?{query-parameters}
```
| Component | Description |
|--------------------|------------------------------------------------------------------------------------------------------------------------|
| {version} | The version of the LibreGraph API used by the client. |
| {/me} | The `me` component of the part is optional. If used, you only see spaces where the acting user is a regular member of. |
| {query-parameters} | Optional parameters for the request to customize the response. |
### List all spaces `GET /drives`
Returns a list of all available spaces, even ones where the acting user is not a regular member of. You need elevated permissions to do list all spaces. If you don't have the elevated permissions, the result is the same like `GET /me/drives`.
{{< hint type=info title="Multiple Administration Personas" >}}
The ownCloud spaces concept draws a strict line between users which can work with the content of a space and others who have the permission to manage the space. A user which is able to manage quota and space metadata does not necessarily need to be able to access the content of a space.
**Space Admin**\
There is a global user role "Space Admin" which grants users some global permissions to manage space quota and some space metadata. This Role enables the user also to disable, restore and delete spaces. He cannot manage space members.
**Space Manager**\
The "Space Manager" is a user which is a regular member of a space because he has been invited. In addition to being part of a space the user can also manage the memberships of the space.
{{< /hint >}}
### List My Spaces `GET /me/drives`
https://owncloud.dev/libre-graph-api/#/me.drives/ListMyDrives
## Modifying Spaces
Modify the properties of a space. You need elevated permissions to execute this request.
### Set the space quota to 5GB `PATCH /drives/{drive-id}`
To limit the quota of a space you need to set the `quota.total` value. The API response will give back all actual quota properties.
````json
{
"quota": {
"remaining": 5368709120,
"state": "normal",
"total": 5368709120,
"used": 0
}
}
````
| Attribute | Description |
|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| remaining | The remaining disk space in `bytes`. If the quota is not limited, this will show the total available disk space. |
| state | The state of the space in regards to quota usage. This can be used for visual indicators. It can be `normal`(<75%), `nearing`(between 75% and 89%), `critical`(between 90% and 99%) and `exceeded`(100%). |
| total | The space id. The value needs to be a space ID. |
| used | The used disk space in bytes. |
{{< tabs "set-space-quota" >}}
{{< tab "Request" >}}
```shell
curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \
-H 'Content-Type: application/json' \
--data-raw '{
"quota": {
"total": 5368709120
}
}'
```
{{< /tab >}}
{{< tab "Response - 200 OK" >}}
````json {hl_lines=[17]}
{
"description": "Marketing team resources",
"driveAlias": "project/marketing",
"driveType": "project",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"lastModifiedDateTime": "2023-01-18T17:13:48.385204589+01:00",
"name": "Marketing",
"owner": {
"user": {
"displayName": "",
"id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
},
"quota": {
"remaining": 5368709120,
"state": "normal",
"total": 5368709120,
"used": 0
},
"root": {
"eTag": "\"f91e56554fd9305db81a93778c0fae96\"",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"permissions": [
{
"grantedToIdentities": [
{
"user": {
"displayName": "Admin",
"id": "some-admin-user-id-0000-000000000000"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
},
"webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
````
{{< /tab >}}
{{< /tabs >}}
### Change the space name, subtitle and alias `PATCH /drives/{drive-id}`
You can change multiple space properties in one request as long as you submit a valid JSON body. Please be aware that some properties need different permissions.
{{< tabs "change-space-props" >}}
{{< tab "Request" >}}
```shell
curl -L -k -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \
-H 'Content-Type: application/json' \
--data-raw '{
"name": "Mars",
"description": "Mission to mars",
"driveAlias": "project/mission-to-mars"
}'
```
{{< /tab >}}
{{< tab "Response - 200 OK" >}}
````json {hl_lines=[2,3,7]}
{
"description": "Mission to mars",
"driveAlias": "project/mission-to-mars",
"driveType": "project",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00",
"name": "Mars",
"owner": {
"user": {
"displayName": "",
"id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
},
"quota": {
"remaining": 15,
"state": "normal",
"total": 15,
"used": 0
},
"root": {
"eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"permissions": [
{
"grantedToIdentities": [
{
"user": {
"displayName": "Admin",
"id": "some-admin-user-id-0000-000000000000"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
},
"webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
````
{{< /tab >}}
{{< /tabs >}}
## Disabling / Deleting Spaces
### Disable a space `DELETE /drives/{drive-id}`
This operation will make the space content unavailable for all space members. No data will be deleted.
{{< tabs "disable-space" >}}
{{< tab "Request" >}}
```shell
curl -L -k -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/'
```
{{< /tab >}}
{{< tab "Response - 204 No Content" >}}
This response has no body value.
A disabled space will appear in listings with a `root.deleted.state=trashed` property. The space description and the space image will not be readable anymore.
```json {hl_lines=[18,19,20]}
{
"description": "Marketing team resources",
"driveAlias": "project/marketing",
"driveType": "project",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00",
"name": "Marketing",
"owner": {
"user": {
"displayName": "",
"id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
},
"quota": {
"total": 15
},
"root": {
"deleted": {
"state": "trashed"
},
"eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"permissions": [
{
"grantedToIdentities": [
{
"user": {
"displayName": "Admin",
"id": "some-admin-user-id-0000-000000000000"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
},
"webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
```
{{< /tab >}}
{{< /tabs >}}
### Restore a space `PATCH /drives/{drive-id}`
This operation will make the space content available again to all members. No content will be changed.
To restore a space, the Header `Restore: T` needs to be set.
{{< tabs "restore-space" >}}
{{< tab "Request" >}}
```shell
curl -L -X PATCH 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff/' \
-H 'Restore: T' \
-H 'Content-Type: text/plain' \
--data-raw '{}'
```
{{< hint type=info title="Body value" >}}
This request needs an empty body (--data-raw '{}') to fulfil the standard libregraph specification even when the body is not needed.
{{< /hint >}}
{{< /tab >}}
{{< tab "Response - 200 OK" >}}
```json
{
"description": "Marketing team resources",
"driveAlias": "project/marketing",
"driveType": "project",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"lastModifiedDateTime": "2023-01-19T14:17:36.094283+01:00",
"name": "Marketing",
"owner": {
"user": {
"displayName": "",
"id": "535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
},
"quota": {
"remaining": 15,
"state": "normal",
"total": 15,
"used": 0
},
"root": {
"eTag": "\"f5fee4fdfeedd6f98956500779eee15e\"",
"id": "storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff",
"permissions": [
{
"grantedToIdentities": [
{
"user": {
"displayName": "Admin",
"id": "some-admin-user-id-0000-000000000000"
}
}
],
"roles": [
"manager"
]
}
],
"webDavUrl": "https://localhost:9200/dav/spaces/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
},
"webUrl": "https://localhost:9200/f/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff"
}
```
{{< /tab >}}
{{< /tabs >}}
### Permanently delete a space `DELETE /drives/{drive-id}`
This operation will delete a space and all its data permanently. This is restricted to spaces which are already disabled.
To delete a space, the Header `Purge: T` needs to be set.
{{< tabs "delete-space" >}}
{{< tab "Request" >}}
```shell {hl_lines=[2]}
curl -L -X DELETE 'https://localhost:9200/graph/v1.0/drives/storage-users-1$535aa42d-a3c7-4329-9eba-5ef48fcaa3ff' \
-H 'Purge: T'
```
{{< hint type=warning title="Data will be deleted" >}}
This request will delete a space and all its content permanently. This operation cannot be reverted.
{{< /hint >}}
{{< /tab >}}
{{< tab "Response - 204 No Content" >}}
This response has no body value.
{{< /tab >}}
{{< tab "Response - 400 Bad Request" >}}
The space to be deleted was not disabled before.
```json
{
"error": {
"code": "invalidRequest",
"innererror": {
"date": "2023-01-24T19:57:19Z",
"request-id": "f62af40f-bc18-475e-acd7-e9008d6bd326"
},
"message": "error: bad request: can't purge enabled space"
}
}
```
{{< /tab >}}
{{< /tabs >}}
## Sharing Space
### Add member to space `POST /drives/{drive-id}/root/invite`
https://owncloud.dev/libre-graph-api/#/drives.permissions/Invite
### Sharing space as a link `POST /drives/{drive-id}/root/createLink`
https://owncloud.dev/libre-graph-api/#/drives.root/CreateLinkSpaceRoot
## Reading Space Permissions
### Listing permissions of a space `GET /drives/{drive-id}/root/permissions`
https://owncloud.dev/libre-graph-api/#/drives.root/ListPermissionsSpaceRoot
## Modifying / Deleting Space Permissions
### Update permissions of a drive `PATCH /drives/{drive-id}/root/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/drives.root/UpdatePermissionSpaceRoot
### Set password of a link share `POST /drives/{drive-id}/root/permissions/{perm-id}/setPassword`
https://owncloud.dev/libre-graph-api/#/drives.root/SetPermissionPasswordSpaceRoot
### Removing acess to a space `DELETE /drives/{drive-id}/root/permissions/{perm-id}`
https://owncloud.dev/libre-graph-api/#/drives.root/DeletePermissionSpaceRoot

270
docs/apis/http/graph/users.md
View File

@@ -1,270 +0,0 @@
---
title: Users
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/graph
geekdocFilePath: users.md
---
{{< toc >}}
## Users API
The Users API is implementing a subset of the functionality of the
[MS Graph User resource](https://docs.microsoft.com/en-us/graph/api/resources/user?view=graph-rest-1.0)
The JSON representation of a User handled by the Users API looks like this:
```
{
"displayName": "Albert Einstein",
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"mail": "einstein@example.org",
"onPremisesSamAccountName": "einstein"
}
```
Our implementation currently supports only a limited set of Attributes of Users:
| Attribute | Description |
|---------------|---------------------------------------------------------------------------------------------------------------------------|
| displayName | The full name of the user, usually a combination of given name and last name |
| mail | The user's email address |
| onPremisesSamAccountName | The loginname/account name of the user |
| id | An unique, stable readonly identifier for the user that stays the same for the whole lifetime of the User, usually a UUID |
| passwordProfile | Contains the password of the users. This is only present when updating or creating users. It is never returned by the API |
### Reading users
#### `GET /me`
Returns the user object of the currently signed-in user
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/me' -u user:password
```
Response:
```
{
"displayName": "Albert Einstein",
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"mail": "einstein@example.org",
"onPremisesSamAccountName": "einstein"
}
```
#### `GET /users`
Returns a list of all users
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/users' -u user:password
```
Response:
```
{
"value": [
{
"displayName": "Albert Einstein",
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"mail": "einstein@example.org",
"onPremisesSamAccountName": "einstein"
},
{
"displayName": "Maurice Moss",
"id": "058bff95-6708-4fe5-91e4-9ea3d377588b",
"mail": "moss@example.org",
"onPremisesSamAccountName": "moss"
}
]
}
```
#### `GET /users?$expand=memberOf`
Returns a list of all users
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/users?$expand=memberOf' -u user:password
```
Response:
```
{
"value": [
{
"displayName": "Albert Einstein",
"id": "4c510ada-c86b-4815-8820-42cdf82c3d51",
"mail": "einstein@example.org",
"onPremisesSamAccountName": "einstein",
"memberOf": [
{
"displayName": "users",
"id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa"
},
{
"displayName": "sailing-lovers",
"id": "6040aa17-9c64-4fef-9bd0-77234d71bad0"
},
{
"displayName": "violin-haters",
"id": "dd58e5ec-842e-498b-8800-61f2ec6f911f"
},
{
"displayName": "physics-lovers",
"id": "262982c1-2362-4afa-bfdf-8cbfef64a06e"
}
],
},
{
"displayName": "Maurice Moss",
"id": "058bff95-6708-4fe5-91e4-9ea3d377588b",
"mail": "moss@example.org",
"onPremisesSamAccountName": "moss",
"memberOf": [
{
"displayName": "users",
"id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa"
}
],
}
]
}
```
#### `GET /users/{userid or accountname}`
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b' -u user:password
```
Response:
```
{
"displayName": "Maurice Moss",
"id": "058bff95-6708-4fe5-91e4-9ea3d377588b",
"mail": "moss@example.org",
"onPremisesSamAccountName": "moss"
}
```
#### `GET /users/{userid or accountname}?$expand=memberOf`
Example:
```
curl -k 'https://localhost:9200/graph/v1.0/users/058bff95-6708-4fe5-91e4-9ea3d377588b?$expand=memberOf' -u user:password
```
Response:
```
{
"displayName": "Maurice Moss",
"id": "058bff95-6708-4fe5-91e4-9ea3d377588b",
"mail": "moss@example.org",
"onPremisesSamAccountName": "moss",
"memberOf": [
{
"displayName": "users",
"id": "509a9dcd-bb37-4f4f-a01a-19dca27d9cfa"
}
],
}
```
### Creating / Updating Users
#### `POST /users`
Use this to create a new user.
##### Request Body
Note the missing `"id"` Attribute. It will be generated by the server:
```
{
"displayName": "Example User",
"mail": "example@example.org",
"onPremisesSamAccountName": "example",
"passwordProfile": {
"password": "ThePassword"
}
}
```
##### Response
When successful, the response will return the new user, without the password, but including the newly allocated `"id"`:
```
{
"displayName":"Example User",
"id":"c067b139-c91c-4e47-8be6-669156a0587b",
"mail":"example@example.org",
"onPremisesSamAccountName":"example"
}
```
#### `DELETE /users/{id}`
Example:
```
curl -k --request DELETE 'https://localhost:9200/graph/v1.0/users/c067b139-c91c-4e47-8be6-669156a0587b' -u user:password
```
When successful the API returns no response body and the HTTP status code 204 (No Content)
#### `PATCH /users/{id}`
Updating attributes of a single user can be done with a patch request. The Request Body contains the new values of the attributes
to be updated. E.g. to update the `displayName` Attribute:
```
curl -k --header "Content-Type: application/json" \
--request PATCH --data '{"displayName": "Test User" }' \
'https://localhost:9200/graph/v1.0/users/c54b0588-7157-4521-bb52-c1c8ca84ea71' -u user:password
```
Similar to creating a user via `POST`, the `PATCH` request will return the user object containing the new attribute values.
### Change password
#### `POST /me/changePassword`
Users can change their own password by sending a POST request to `/me/changePassword`
##### Request Body
```
{
"currentPassword": "current",
"newPassword": "new"
}
```
When successful the API returns no response body and the HTTP status code 204 (No Content)
```
curl -i -k --header "Content-Type: application/json" \
--request POST --data '{"currentPassword": "current", "newPassword": "new" }' \
'https://localhost:9200/graph/v1.0/me/changePassword' -u user:current
```

249
docs/apis/http/tus_upload.md
View File

@@ -1,249 +0,0 @@
---
title: "Resumable Upload"
date: 2023-10-10T00:00:00+00:00
weight: 21
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http
geekdocFilePath: tus_upload.md
geekdocCollapseSection: true
---
Infinite Scale supports the tus resumable-upload protocol, which is a robust, modular, and open protocol designed to resume large file uploads reliably over HTTP.
In situations where file uploads might be interrupted due to network issues, browser crashes, or other unforeseen interruptions,
tus ensures that uploads can be resumed from the point of failure without losing data.
This documentation shows some basic examples, refer [tus official site](https://tus.io/protocols/resumable-upload) for more details.
## Supported tus Features
The backend announces certain tus features to clients. WebDAV responses come with tus HTTP headers for the offical tus features, and additional, ownCloud specific features are announced via the capabilities endpoint (e.g. `https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json`).
The following snippet shows the relevant part of the server capabilities of Infinite Scale that concerns the tus upload:
```json
{
"ocs": {
"data": {
"capabilities": {
"files": {
"tus_support": {
"version": "1.0.0",
"resumable": "1.0.0",
"extension": "creation,creation-with-upload",
"max_chunk_size": 10000000,
"http_method_override": ""
}
}
}
}
}
}
}
```
| Parameter | Environment Variable | Default Value | Description |
| -------------- | ------------------------------ | ------------- | ------------------------------------------------------------------- |
| max_chunk_size | FRONTEND_UPLOAD_MAX_CHUNK_SIZE | 10000000 | Announces the max chunk sizes in bytes for uploads via the clients. |
## Upload in Chunks
### Create an Upload URL
The client must send a POST request against a known upload creation URL to request a new upload resource.
The filename has to be provided in base64-encoded format.
Example:
```shell
# base64 encoded filename 'tustest.txt' is 'dHVzdGVzdC50eHQ='
echo -n 'tustest.txt' | base64
```
{{< tabs "create-upload-url" >}}
{{< tab "Request" >}}
```shell
curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/8d72036d-14a5-490f-889e-414064156402$196ac304-7b88-44ce-a4db-c4becef0d2e0 \
-H "Authorization: Bearer eyJhbGciOiJQUzI..."\
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Length: 10" \
-H "Upload-Metadata: filename dHVzdGVzdC50eHQ="
```
{{< /tab >}}
{{< tab "Response - 201 Created" >}}
```
< HTTP/1.1 201 Created
< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location
< Content-Length: 0
< Content-Security-Policy: default-src 'none';
< Date: Mon, 16 Oct 2023 08:49:39 GMT
< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM
< Tus-Extension: creation,creation-with-upload,checksum,expiration
< Tus-Resumable: 1.0.0
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host localhost left intact
```
{{< /tab >}}
{{< /tabs >}}
The server will return a temporary upload URL in the location header of the response:
```
< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTMyNTc5LCJpYXQiOjE2OTc0NDYxNzksInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy8zYTU3ZWZlMS04MzE0LTQ4MGEtOWY5Ny04N2Q1YzBjYTJhMTgifQ.FbrlY7mdOfsbFgMrP8OtcHlCEq72a2ZVnPD2iBo9MfM
```
### Upload the First Chunk
Once a temporary upload URL has been created, a client can send a PATCH request to upload a file. The file content should be sent in the body of the request:
{{< tabs "upload-the-first-chunk" >}}
{{< tab "Request" >}}
```shell
curl -ks -XPATCH https://temporary-upload-url \
-H "Authorization: Bearer eyJhbGciOiJQUzI..." \
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Offset: 0" \
-H "Content-Type: application/offset+octet-stream" -d "01234"
```
{{< /tab >}}
{{< tab "Response - 204 No Content" >}}
```
< HTTP/1.1 204 No Content
< Date: Tue, 17 Oct 2023 04:10:52 GMT
< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd
< Tus-Resumable: 1.0.0
< Upload-Expires: 1697602157
< Upload-Offset: 5
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx
<
* Connection #0 to host localhost left intact
```
{{< /tab >}}
{{< /tabs >}}
### Upload Further Chunks
After the first chunk is uploaded, the second chunk can be uploaded by pointing `Upload-Offset` to exact position that was returned in the first response.
Upload process will not be marked as complete until the total uploaded content size matches the `Upload-Length` specified during the creation of the temporary URL.
{{< tabs "upload-the-second-chunk" >}}
{{< tab "Request" >}}
```shell
curl -ks -XPATCH https://temporary-upload-url \
-H "Authorization: Bearer eyJhbGciOiJQUzI..." \
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Offset: 5" \
-H "Content-Type: application/offset+octet-stream" -d "56789"
```
{{< /tab >}}
{{< tab "Response - 204 No Content" >}}
```
< HTTP/1.1 204 No Content
< Date: Tue, 17 Oct 2023 04:11:00 GMT
< Oc-Fileid: 8d72036d-14a5-490f-889e-414064156402$73bb5450-816b-4cae-90aa-1f96adc95bd4!84e319e4-de1d-4dd8-bbd0-e51d933cdbcd
< Tus-Resumable: 1.0.0
< Upload-Expires: 1697602157
< Upload-Offset: 10
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx
<
* Connection #0 to host localhost left intact
```
{{< /tab >}}
{{< /tabs >}}
{{< hint type=warning title="Important Warning" >}}
`Upload-Offset` header indicates the byte position in the target file where the server should start writing the upload content.
It ensures data integrity and order during the upload process.
{{< /hint >}}
## Creation with Upload
{{< tabs "creation-with-upload" >}}
{{< tab "Request" >}}
```shell
curl -ks -XPOST https://ocis.test/remote.php/dav/spaces/{space-id} \
-H "Authorization: Bearer eyJhbGciOiJQUzI..." \
-H "Tus-Resumable: 1.0.0" \
-H "Upload-Length: 14" \
-H "Content-Type: application/offset+octet-stream" \
-H "Upload-Metadata: filename dGVzdC50eHQ=" \
-H "Tus-Extension: creation-with-upload" \
-d "upload content"
```
{{< /tab >}}
{{< tab "Response - 201 Created" >}}
```shell
< HTTP/1.1 201 Created
< Access-Control-Allow-Headers: Tus-Resumable, Upload-Length, Upload-Metadata, If-Match
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Tus-Resumable, Upload-Offset, Location
< Content-Length: 0
< Content-Security-Policy: default-src 'none';
< Content-Type: text/plain
< Date: Mon, 16 Oct 2023 04:18:25 GMT
< Etag: "372c96743f68bc40e789124d30567371"
< Last-Modified: Mon, 16 Oct 2023 04:18:25 +0000
< Location: https://ocis.test/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjk3NTE2MzA1LCJpYXQiOjE2OTc0Mjk5MDUsInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy82NjlhODBlZi1hN2VjLTQwYTAtOGNmOS05MTgwNTVhYzlkZjAifQ.yq-ofJYnJ9FLML7Z_jki1FJQ7Ulbt9O_cmLe6V411A4
< Oc-Etag: "372c96743f68bc40e789124d30567371"
< Oc-Fileid: 44d3e1e0-6c01-4b94-9145-9d0068239fcd$446bdad4-4b27-41f1-afce-0881f202a214!d7c292a6-c395-4e92-bf07-2c1663aec8dd
< Oc-Perm: RDNVWZP
< Tus-Extension: creation,creation-with-upload,checksum,expiration
< Tus-Resumable: 1.0.0
< Upload-Expires: 1697516305
< Upload-Offset: 14
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: xxxxxxxxxxxxxxxxxxxxxx
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host localhost left intact
```
{{< /tab >}}
{{< /tabs >}}
{{< hint type=warning title="Important Warning" >}}
The `Upload-Length` header of the request has to contain the exact size of the upload content in byte.
{{< /hint >}}
## Supported Upload-Metadata
Upload-metadata key-value pairs aren't specified in the general tus docs. The following ones are supported in the ownCloud ecosystem:
| Parameter (key) | Example (value, MUST be Base64 encoded) | Description |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `name` OR `filename` (mandatory) | example.pdf | Filename |
| `mtime` (recommended) | 1701708712 | Modification time (Unix time format) |
| `checksum` (recommended) | SHA1 a330de5886e5a92d78fb3f8d59fe469857759e72 | Checksum, computed from the client |
| `type` OR `filetype` | application/pdf | MIME Type, sent by the web UI |
| `relativePath` | undefined | File path relative to the folder that is being uploaded, including the filename. Sent by the web UI |
| `spaceId` | 8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI |
| `spaceName` | Personal | Sent by the web UI |
| `driveAlias` | personal/admin | Sent by the web UI |
| `driveType` | personal | Sent by the web UI |
| `currentFolder` | / | Sent by the web UI |
| `currentFolderId` | 8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993!a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI |
| `uppyId` | uppy-example/pdf-1e-application/pdf-238300 | Sent by the web UI |
| `relativeFolder` | | File path relative to the folder that is being uploaded, without filename. Sent by the web UI. |
| `tusEndpoint` | https://ocis.ocis-traefik.latest.owncloud.works/remote.php/dav/spaces/8748cddf-66b7-4b85-91a7-e6d08d8e1639$a9778d63-21e7-4d92-9b47-1b81144b9993 | Sent by the web UI |
| `uploadId` | 71d5f878-a96c-4d7b-9627-658d782c93d7 | Sent by the web UI |
| `topLevelFolderId` | undefined | Sent by the web UI |
| `routeName` | files-spaces-generic | Sent by the web UI |
| `routeDriveAliasAndItem` | cGVyc29uYWwvYWRtaW4= | Sent by the web UI |
| `routeShareId` | | Share ID when uploading into a received folder share. Sent by the web UI |

547
docs/apis/http/webdav/_index.md
View File

@@ -1,547 +0,0 @@
---
title: "WebDAV"
date: 2023-07-02T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/apis/http/webdav
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
{{< toc >}}
**Web** **D**istributed **A**uthoring and **V**ersioning (WebDAV) consists of a set of methods, headers, and content-types extending HTTP/1.1 for the management of resources and -properties, creation and management of resource collections, URL namespace manipulation, and resource locking (collision avoidance). WebDAV is one of the central APIs that ownCloud uses for handling file resources, metadata and locks.
{{< hint type=info title="RFC" >}}
**WebDAV RFCs**
RFC 2518 was published in February 1999. [RFC 4918](https://datatracker.ietf.org/doc/html/rfc4918), published in June 2008 obsoletes RFC 2518 with minor revisions mostly due to interoperability experience.
{{< /hint >}}
## Calling the WebDAV API
### Request URI
```sh
{HTTP method} https://ocis.url/{webdav-base}/{resourceID}/{path}
```
The request URI consists of:
| Component | Description |
|---------------|--------------------------------------------------------------------------------------------------------|
| {HTTP method} | The HTTP method which is used in the request. |
| {webdav-base} | The WebDAV base path component. Possible options are |
| | `dav/spaces/` This is the default and optimized endpoint for all WebDAV requests. |
| | `remote.php/dav/spaces/`* |
| | `remote.php/webdav/`* |
| | `webdav/`* |
| | `dav/`* |
| {resourceID} | This resourceID is used as the WebDAV root element. All children are accessed by their relative paths. |
| {path} | The relative path to the WebDAV root. In most of the casese, this is the space root. |
\* these dav endpoints are implemented for legacy reasons and should not be used. Note: The legacy endpoints **do not take the resourceID as an argument.**
### HTTP methods
| Method | Description |
|-----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| PROPFIND | Retrieve properties as XML from a web resource. It is also overloaded to retrieve the collection structure (a.k.a. directory hierarchy) of a remote system. |
| PROPPATCH | Process instructions specified in the request body to set and/or remove properties defined on the resource identified by the request uri. |
| MKCOL | Create a WebDAV collection (folder) at the location specified by the request uri. |
| GET | Retrieve a WebDAV resource. |
| HEAD | Retrieve a WebDAV resource without reading the body. |
| PUT | A PUT performed on an existing resource replaces the GET response entity of the resource. |
| POST | Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. |
| PATCH | Not part of the WebDAV rfc and has no effect on a WebDAV resource. However, this method is used in the TUS protocol for uploading resources. |
| COPY | Creates a duplicate of the source resource identified by the Request-URI, in the destination resource identified by the URI in the Destination header. |
| MOVE | The MOVE operation on a non-collection resource is the logical equivalent of a copy (COPY), followed by consistency maintenance processing, followed by a delete of the source, where all three actions are performed in a single operation. | |
| DELETE | Delete the resource identified by the Request-URI. |
| LOCK | A LOCK request to an existing resource will create a lock on the resource identified by the Request-URI, provided the resource is not already locked with a conflicting lock. |
| UNLOCK | The UNLOCK method removes the lock identified by the lock token in the Lock-Token request header. The Request-URI must identify a resource within the scope of the lock. |
The methods `MKCOL`, `GET`, `HEAD`, `LOCK`, `COPY`, `MOVE`, `UNLOCK` and `DELETE` need no request body.
The methods `PROPFIND`, `PROPPATCH`, `PUT` require a request body, normally in XML format to provide the needed values.
{{< hint type=tip title="Tooling" >}}
**WebDAV is not REST**
The WebDAV protocol was created before the REST paradigm has become the de-facto standard for API design. WebDAV uses http methods which are not part of REST. Therefore all the tooling around API design and documentation is not usable (like OpenApi 3.0 / Swagger or others).
{{< /hint >}}
### Authentication
For development purposes the examples in the developer documentation use Basic Auth. It is disabled by default and should only be enabled by setting `PROXY_ENABLE_BASIC_AUTH` in [the proxy](../../../services/proxy/configuration/#environment-variables) for development or test instances.
To authenticate with a Bearer token or OpenID Connect access token replace the `-u user:password` Basic Auth option of curl with a `-H 'Authorization: Bearer <token>'` header. A `<token>` can be obtained by copying it from a request in the browser, although it will time out within minutes. To automatically refresh the OpenID Connect access token an ssh-agent like solution like [oidc-agent](https://github.com/indigo-dc/oidc-agent) should be used.
## Listing Properties
This method is used to list the properties of a resource in xml. This method can also be used to retrieve the listing of a WebDAV collection which means the content of a remote directory.
{{< tabs "list-properties" >}}
{{< tab "Curl" >}}
```shell
curl -L -X PROPFIND 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/' \
-H 'Depth: 1' \
-d '<?xml version="1.0"?>
<d:propfind xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:prop>
<oc:permissions />
<oc:favorite />
<oc:fileid />
<oc:owner-id />
<oc:owner-display-name />
<oc:share-types />
<oc:privatelink />
<d:getcontentlength />
<oc:size />
<d:getlastmodified />
<d:getetag />
<d:getcontenttype />
<d:resourcetype />
<oc:downloadURL />
</d:prop>
</d:propfind>'
```
{{< /tab >}}
{{< tab "HTTP" >}}
```shell
PROPFIND /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/ HTTP/1.1
Host: localhost:9200
Origin: https://localhost
Access-Control-Request-Method: PROPFIND
Depth: 1
Content-Type: application/xml
Authorization: Basic YWRtaW46YWRtaW4=
Content-Length: 436
<?xml version="1.0"?>
<d:propfind xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:prop>
<oc:permissions />
<oc:favorite />
<oc:fileid />
<oc:owner-id />
<oc:owner-display-name />
<oc:share-types />
<oc:privatelink />
<d:getcontentlength />
<oc:size />
<d:getlastmodified />
<d:getetag />
<d:getcontenttype />
<d:resourcetype />
<oc:downloadURL />
</d:prop>
</d:propfind>
```
{{< /tab >}}
{{< /tabs >}}
The request consists of a request body and an optional `Depth` Header.
{{< hint type=tip title="PROPFIND usage" >}}
**Metadata and Directory listings**
Clients can use the `PROPFIND` method to retrieve properties of resources (metadata) and to list the content of a directories.
{{< /hint >}}
### Response
{{< tabs "response list properties" >}}
{{< tab "207 - Multistatus" >}}
#### Multi Status Response
A Multi-Status response conveys information about multiple resources
in situations where multiple status codes might be appropriate. The
default Multi-Status response body is an application/xml
HTTP entity with a `multistatus` root element. Further elements
contain `200`, `300`, `400`, and `500` series status codes generated during
the method invocation.
Although `207` is used as the overall response status code, the
recipient needs to consult the contents of the multistatus response
body for further information about the success or failure of the
method execution. The response MAY be used in success, partial
success and also in failure situations.
The `multistatus` root element holds zero or more `response` elements
in any order, each with information about an individual resource.
#### Body
```xml
<d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
<d:response>
<d:href>/dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/</d:href>
<d:propstat>
<d:prop>
<oc:permissions>RDNVCKZP</oc:permissions>
<oc:favorite>0</oc:favorite>
<oc:fileid>storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000</oc:fileid>
<oc:id>storage-users-1$some-admin-user-id-0000-000000000000!some-admin-user-id-0000-000000000000</oc:id>
<oc:owner-id>admin</oc:owner-id>
<oc:owner-display-name>Admin</oc:owner-display-name>
<oc:privatelink>https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%21some-admin-user-id-0000-000000000000</oc:privatelink>
<oc:size>10364682</oc:size>
<d:getlastmodified>Mon, 04 Sep 2023 20:10:09 GMT</d:getlastmodified>
<d:getetag>"c4d3610dfe4fac9b44e1175cfc44b12b"</d:getetag>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
<d:propstat>
<d:prop>
<oc:checksums></oc:checksums>
<oc:share-types></oc:share-types>
<d:getcontentlength></d:getcontentlength>
<d:getcontenttype></d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/New%20file.txt</d:href>
<d:propstat>
<d:prop>
<oc:permissions>RDNVWZP</oc:permissions>
<oc:checksums>
<oc:checksum>SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab</oc:checksum>
</oc:checksums>
<oc:favorite>0</oc:favorite>
<oc:fileid>storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990</oc:fileid>
<oc:id>storage-users-1$some-admin-user-id-0000-000000000000!90cc3e73-0c6c-4346-9c4d-f529976d4990</oc:id>
<oc:owner-id>admin</oc:owner-id>
<oc:owner-display-name>Admin</oc:owner-display-name>
<oc:share-types>
<oc:share-type>0</oc:share-type>
<oc:share-type>1</oc:share-type>
<oc:share-type>3</oc:share-type>
</oc:share-types>
<oc:privatelink>https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%2190cc3e73-0c6c-4346-9c4d-f529976d4990</oc:privatelink>
<d:getcontentlength>5</d:getcontentlength>
<oc:size>5</oc:size>
<d:getlastmodified>Mon, 28 Aug 2023 20:45:03 GMT</d:getlastmodified>
<d:getetag>"75115347c74701a3be9c635ddebbf5c4"</d:getetag>
<d:getcontenttype>text/plain</d:getcontenttype>
<d:resourcetype></d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/dav/spaces/storage-users-1$some-admin-user-id-0000-000000000000/NewFolder/</d:href>
<d:propstat>
<d:prop>
<oc:permissions>RDNVCKZP</oc:permissions>
<oc:favorite>0</oc:favorite>
<oc:fileid>storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7</oc:fileid>
<oc:id>storage-users-1$some-admin-user-id-0000-000000000000!5c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7</oc:id>
<oc:owner-id>admin</oc:owner-id>
<oc:owner-display-name>Admin</oc:owner-display-name>
<oc:privatelink>https://localhost:9200/f/storage-users-1$some-admin-user-id-0000-000000000000%215c73ecd9-d9f4-44f4-b685-ca4cb40aa6b7</oc:privatelink>
<oc:size>0</oc:size>
<d:getlastmodified>Mon, 28 Aug 2023 20:45:10 GMT</d:getlastmodified>
<d:getetag>"e83367534cc595a45d706857fa5f03d8"</d:getetag>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
<d:propstat>
<d:prop>
<oc:checksums></oc:checksums>
<oc:share-types></oc:share-types>
<d:getcontentlength></d:getcontentlength>
<d:getcontenttype></d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
</d:multistatus>
```
{{< /tab >}}
{{< tab "400 - Bad Request" >}}
#### Body
```xml
<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
<s:message>Invalid Depth header value: 3</s:message>
</d:error>
```
This can occur if the request is malformed e.g. due to an invalid xml request body or an invalid depth header value.
{{< /tab >}}
{{< tab "404 - Not Found" >}}
#### Body
```xml
<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\NotFound</s:exception>
<s:message>Resource not found</s:message>
</d:error>
```
{{< /tab >}}
{{< /tabs >}}
### Request Body
The `PROPFIND` Request can include an XML request body containing a list of namespaced property names.
### Namespaces
When building the body of your DAV request, you will request properties that are available under a specific namespace URI. It is usual to declare prefixes for those namespace in the `d:propfind` element of the body.
Available namespaces:
| URI | Prefix |
|-------------------------------------------|--------|
| DAV: | d |
| http://sabredav.org/ns | s |
| http://owncloud.org/ns | oc |
| http://open-collaboration-services.org/ns | ocs |
| http://open-cloud-mesh.org/ns | ocm |
### Request Example with declared namespaces
```xml
<?xml version="1.0"?>
<d:propfind xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns">
</d:propfind>
```
### Supported WebDAV Properties
| Property | Desription | Example |
| ----------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `<d:getlastmodified />` | The latest modification time. | `Fri, 30 Dec 2022 14:22:43 GMT` |
| `<d:getetag />` | The file's etag. | `"c3a1ee4a0c28edc15b9635c3bf798013"` |
| `<d:getcontenttype />` | The mime type of the file. | `image/jpeg` |
| `<d:resourcetype />` | Specifies the nature of the resource. | `<d:collection />` for a folder |
| `<d:getcontentlength />` | The size if it is a file in bytes. | `5` bytes |
| `<d:lockdiscovery />` | Describes the active locks on a resource. | Detailed Example in [Locking]() |
| `<oc:id />` | The globally unique ID of the resource. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` |
| `<oc:fileid />` | The globally unique ID of the resource. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` |
| `<oc:downloadURL />` | Direct URL to download a file from. | Not implemented. |
| `<oc:permissions />` | Determines the actions a user can take on the resource. | The value is a string containing letters that clients can use to determine available actions. |
| | | `S`: Shared |
| | | `M`: Mounted |
| | | `D`: Deletable |
| | | `NV`: Updateable, Renameable, Moveable |
| | | `W`: Updateable (file) |
| | | `CK`: Creatable (folders only) |
| | | `Z`: Deniable |
| | | `P`: Trashbin Purgable |
| | | `X`: Securely Viewable |
| | | In the early stages this was indeed a list of permissions. Over time, more flags were added and the term permissions no longer really fits well. |
| `<oc:tags />` | List of user specified tags. | `<oc:tag>test</oc:tag>` |
| `<oc:favorite /> ` | The favorite state. | `0` for not favourited, `1` for favourited |
| `<oc:owner-id />` | The user id of the owner of a resource. Project spaces have no owner. | `einstein` |
| `<oc:owner-display-name />` | The display name of the owner of a resource. Project spaces have no owner. | `Albert Einstein` |
| `<oc:share-types />` | List of share types. | `0` = User Share |
| | | `1` = Group Share |
| | | `2` = Public Link |
| `<oc:checksums />` | | `<oc:checksum>`<br/>`SHA1:1c68ea370b40c06fcaf7f26c8b1dba9d9caf5dea MD5:2205e48de5f93c784733ffcca841d2b5 ADLER32:058801ab`<br /> `</oc:checksum>` |
| | | Due to a bug in the very early development of ownCloud, this value is not an array, but a string separated by whitespaces. |
| `<oc:size />` | Similar to `getcontentlength` but it also works for folders. | `10` bytes |
| `<oc:shareid />` | The ID of the share if the resource is part of such. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` |
| `<oc:shareroot />` | The root path of the shared resource if the resource is part of such. | `/shared-folder` |
| `<oc:remoteItemId />` | The ID of the shared resource if the resource is part of such. | `storage-1$27475553-7fb7-4689-b4cf-bbb635daff79!27475553-7fb7-4689-b4cf-bbb635daff79` |
| `<oc:public-link-item-type />` | The type of the resource if it's a public link. | `folder` |
| `<oc:public-link-permission />` | The share permissions of the resource if it's a public link. | `1` |
| `<oc:public-link-expiration />` | The expiration date of the public link. | `Tue, 14 May 2024 12:44:29 GMT` |
| `<oc:public-link-share-datetime />` | The date the public link was created. | `Tue, 14 May 2024 12:44:29 GMT` |
| `<oc:public-link-share-owner />` | The username of the user who created the public link. | `admin` |
| `<oc:trashbin-original-filename />` | The original name of the resource before it was deleted. | `some-file.txt` |
| `<oc:trashbin-original-location />` | The original location of the resource before it was deleted. | `some-file.txt` |
| `<oc:trashbin-delete-datetime />` | The date the resource was deleted. | `Tue, 14 May 2024 12:44:29 GMT` |
| `<oc:audio />` | Audio meta data if the resource contains such. | `<oc:artist>Metallica</oc:artist><oc:album>Metallica</oc:album><oc:title>Enter Sandman</oc:title>` |
| `<oc:location />` | Location meta data if the resource contains such. | `<oc:latitude>51.504106</oc:latitude><oc:longitude>-0.074575</oc:latitude>` |
### Request Headers
A client executing a `PROPFIND` request MUST submit a Depth Header value. In practice, support for infinite-depth requests MAY be disabled, due to the performance and security concerns associated with this behavior. Servers SHOULD treat a
request without a Depth header as if a `Depth: infinity` header was included. Infinite depth requests are disabled by default in ocis.
| Name | Value |
|-------------------------------------------|---------------------------------------------------------------------------------------|
| Depth | `0` = Only return the desired resource. |
| | `1` = Return the desired resource and all resources one level below in the hierarchy. |
| | `infinity` = Return all resources below the root. |
{{< hint type=caution title="Use the Depth header with caution" >}}
**Depth: infinity**
Using the `Depth: infinity` header value can cause heavy load on the server, depending on the size of the file tree.
The request can run into a timeout and the server performance could be affected for other users.
{{< /hint >}}
## Create a Directory
Clients create directories (WebDAV collections) by executing a `MKCOL` request at the location specified by the request url.
{{< tabs "create-folder" >}}
{{< tab "Curl" >}}
```shell
curl -L -X MKCOL 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/' \
-H 'Authorization: Basic YWRtaW46YWRtaW4='
```
{{< /tab >}}
{{< tab "HTTP" >}}
```shell
MKCOL /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/NewFolder/ HTTP/1.1
Host: localhost:9200
Authorization: Basic YWRtaW46YWRtaW4=
```
{{< /tab >}}
{{< /tabs >}}
### Response
{{< tabs "response create folder" >}}
{{< tab "201 - Created" >}}
This indicates that the Resource has been created successfully.
#### Body
The response has no body.
{{< /tab >}}
{{< tab "403 - Forbidden" >}}
#### Body
```xml
<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\Forbidden</s:exception>
<s:message></s:message>
</d:error>
```
{{< /tab >}}
{{< tab "405 - Method not allowed" >}}
#### Body
```xml
<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\MethodNotAllowed</s:exception>
<s:message>The resource you tried to create already exists</s:message>
</d:error>
```
{{< /tab >}}
{{< /tabs >}}
## Upload File
To upload files to the remote server, clients can use the `PUT` method to create or fully replace the content of the remote file.
### Request Headers
| Name | Usage |
|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `X-OC-Mtime` | Send the last modified <br> time of the file to the server in unixtime format. The server applies this mtime to the resource rather than the actual time. |
| `OC-Checksum` | Provide the checksum of the <br> file content to the server. <br> This is used to prevent corrupted data transfers. |
| `If-Match` | The If-Match request-header field is used with a method to make it <br> conditional. A client that has one or more entities previously <br> obtained from the resource can verify that one of those entities is <br> current by including a list of their associated entity tags in the <br> If-Match header field. |
{{< tabs "upload-file" >}}
{{< tab "Curl" >}}
```shell
curl -L -X PUT 'https://localhost:9200/dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt' \
-H 'X-OC-Mtime: 1692369418' \
-H 'OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef' \
-H 'If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f"' \
-H 'Content-Type: text/plain' \
-H 'Authorization: Basic YWRtaW46YWRtaW4=' \
-d '123'
```
{{< /tab >}}
{{< tab "HTTP" >}}
```shell
PUT /dav/spaces/storage-users-1%24some-admin-user-id-0000-000000000000/test.txt HTTP/1.1
Host: localhost:9200
X-OC-Mtime: 1692369418
OC-Checksum: SHA1:40bd001563085fc35165329ea1ff5c5ecbdbbeef
If-Match: "4436aef907f41f1ac7dfd1ac3d0d455f"
Content-Type: text/plain
Authorization: Basic YWRtaW46YWRtaW4=
Content-Length: 3
123
```
{{< /tab >}}
{{< /tabs >}}
### Response
{{< tabs "response upload file" >}}
{{< tab "201 - Created" >}}
This indicates that the Resource has been created successfully.
#### Body
The response has no body.
#### Headers
```yaml
Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f"
Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f
Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000
X-Oc-Mtime: accepted
```
{{< /tab >}}
{{< tab "204 - No Content" >}}
This indicates that the Resource has been updated successfully.
#### Body
The response has no body.
#### Headers
```yaml
Oc-Etag: "4436aef907f41f1ac7dfd1ac3d0d455f"
Oc-Fileid: storage-users-1$some-admin-user-id-0000-000000000000!07452b22-0ba9-4539-96e1-3511aff7fd2f
Last-Modified: Fri, 18 Aug 2023 14:36:58 +0000
X-Oc-Mtime: accepted
```
{{< /tab >}}
{{< tab "400 - Bad Request" >}}
This indicates that the checksum, which was sent by the client, does not match the computed one after all bytes have been received by the server.
#### Body
```xml
<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns">
<s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
<s:message>The computed checksum does not match the one received from the client.</s:message>
</d:error>
```
{{< /tab >}}
{{< tab "403 - Forbidden" >}}
The user cannot create files in that remote location.
{{< /tab >}}
{{< tab "404 - Not Found" >}}
The remote target space cannot be found.
{{< /tab >}}
{{< tab "409 - Conflict" >}}
This error can occur when the request cannot be executed due to a missing precondition. One example is a PUT into a non-existing remote folder. It can also happen when the client sends the wrong etag in the `If-Match` header.
{{< /tab >}}
{{< /tabs >}}

13
docs/architecture/_index.md
View File

@@ -1,13 +0,0 @@
---
title: Architecture
date: 2023-12-06T13:00:00+01:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
In the architecture part of the documentation we collect useful developer documentation on different aspects of the architecture. We are using mermaid.js to collaborate on the necessary diagrams.
*Pictures tell more than a thousand words.*

111
docs/architecture/collaborative-storage.md
View File

@@ -1,111 +0,0 @@
---
title: "Collaborative Storage"
date: 2023-11-09T12:35:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/storage-backends/
geekdocFilePath: collaborative-storage.md
---
{{< toc >}}
One of the envisioned design goals of Infinite Scale is to work with so called _collaborative storage_, which means that the file system it is running on is not exclusive for Infinite Scale, but can be manipulated in parallel through third party tools. Infinite Scale is expected to monitor the changes that happen independently and react in a consistent and user friendly way.
A real world example of that would be a third party "data producer" that submits data directly into a file system path, not going through Infinite Scale APIs.
This document outlines a few challenges and design concepts for collaborative storage. It is also the base "checklist" for custom storage provider implementations for certain storages, ie. for Ceph- or IBM Storage Scale which provide features that allow more sophisticated and efficient implementations of this goal.
# Storage Driver Components
This discusses a few components and sub functions of the storage driver that have relevance for the collaborative storage.
## Path Locations
What is called "the oCIS file system" is defined as the entire filetree underneath a special path in the local POSIX file system, which might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that adds and edits files in that tree.
Underneath the oCIS file system root, there is an collection of different folders containing oCIS specific data. Specific storage driver data is in the directory `storage/users`, organized by spaces.
(TODO: Check again how different storage drivers work together without overwriting data of each other)
## Spaces
Infinite Scale provides spaces as an additional organization organizational unit for data. Each space is a separate entity with its own attributes such as access patterns and quota.
A storage driver has to model the separation of spaces and provide a list of spaces in general and also a list of spaces a user can access. Furthermore, it needs to be able to create different types of spaces (Home- or Project space).
On POSIX, each space could for example be mapped to it's own directory in a special spaces folder under the oCIS root folder.
## ID to Path Lookup
Infinite Scale uses file IDs to efficiently identify files within a file tree. The lookup from a given ID to a path within the oCIS file tree is a very basic function that more or less defines the Infinite Scale performance. The functionality to for example query the file path for a given Inode number (which is the nearest equivalent for the Infinite Scale file ID) can not be done with standard POSIX system calls.
The interface defining the collaborative storage needs an abstraction for this particular function, returning the file id for a given path, and returning the path for given id.
## Change Notification
When a file is changed by a process outside of oCIS, this needs to be monitored by oCIS to quickly maintain internal caches and data structures as required.
The collaborative storage driver needs a way to achieve that. The easiest way for an POSIX based collaborative storage is inotify, that needs to be set up recursively on a file tree to record changes. Additional it is a challenge to destinguish between changes that were done from external activity and the ones that oCIS creates by its own file operations.
For GPFS, there is a subsystem called delivering that:
https://www.ibm.com/docs/en/storage-scale/5.1.9?topic=reference-clustered-watch-folder]
## ETag Propagation
ownCloud requires that changes which happen "down" in a tree, can be detected in the root element of the tree. That happens through the change of the ETag metadata of each file and/or directory. An ETag is a random, text based tag, that only has one requirement: It has to change its content if a resource further down in the file tree has changed either its content or its metadata. (See [this issue](https://github.com/owncloud/ocis/issues/3782) for further discussion about the ETag/CTag).
POSIX file systems do not maintain a change flag like the ETag by default. The file time stamps (atime, ctime, mtime) in general are not fine granular enough (only seconds for some file systems) and depend on the server time, which renders them useless in a distributed environment.
Infinite Scale needs to implement ETag propagation "up". For the collaborative storage, that needs to be combined with the change notification described above.
Certain file systems implement this functionality either independently from Infinite Scale (EOS) or at least support proper change notifications (Ceph, GPFS?).
## Metadata Management
Metadata are data "snippets" that are as tightly attached to files as ever possible. In best case, a rename of a file silently keeps the metadata as well. In POSIX, this can be achieved by extended file attributes with certain limitations.
## Quota
Each space has it's own quota, thus a storage driver implementation needs to consider that.
For GPFS for example, there is support for quota handling in the file system.
https://www.ibm.com/docs/en/gpfs/4.1.0.4?topic=interfaces-gpfs-quotactl-subroutine
Other systems store quota data in the metadata storage and implement propagation of used quota similar to the ETag propagation.
## User Management
With user management it is meant how to handle the users and groups within oCIS and how that reflects to the file system where data is stored.
### Exclusive Environment
In exclusive environments (aka. decomposedFS) all files of oCIS (ie. the entire oCIS filetree) belongs to a system user with the name `ocis` typically.
### Collaborative Storage
For collaborative storages, the approach described above does not longer work because users are supposed to be able to manipulate data in "their" file tree parts, and that is identified by ACLs and the owner of the files.
That requires a few prerequisites that have to be fulfilled:
1. oCIS as one "client" changing data and the system that allows to access the file tree directly have to use the same user provider, to ensure that each user that is available on a shell is also available in oCIS. That ensures that changes are authenticated through system ACLs and users. LDAP based authentication on the system via PAM and the same LDAP as source for the oCIS IDP should be a sufficient setup.
2. oCIS must be able to write as a "different" user than the ocis system user. That means that we somehow have to impersonate file changing ooperations and run these as the user that is authenticated in oCIS.
Example: There is a user ben. It has to have an entry in the LDAP that is used by IDP which oCIS is running "behind". With that, ben is able to authenticate through the IDP and work in the oCIS web app. The oCIS linux process will do writes and other changes impersonated as user ben.
For the access of data on the commandline, the logins to the linux system must be authenticated against the same LDAP - so that ben can authenticate on a terminal using username and password. With that, the user can interactively change data that belongs to user ben (simplified said).
To give permissions to groups, the linux group management must work accordingly. The same is true for file permissions.
## Trashbin
When a user deletes a file in oCIS it is moved to a so called trashbin that allows to restore the file if the deletion was accidentailly.
## Versions
When an existing file is changed, the former file state is to be preserved with data and metadata by oCIS. Some file system types provide this functionality via snapshots on partition or even file level. Other do not and have to implement that via a hidden directory keeping old file versions.

202
docs/architecture/efficient-stat-polling.md
View File

@@ -1,202 +0,0 @@
---
title: "Efficient Stat Polling"
date: 2020-03-03T10:31:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: efficient-stat-polling.md
---
The fallback sync mechanism uses the ETag to determine which part of a sync tree needs to be checked by recursively descending into folders whose ETag has changed. The ETag can be calculated using a `stat()` call in the filesystem and we are going to explore how many `stat()` calls are necessary and how the number might be reduced.
## ETag propagation
What does ETag propagation mean? Whenever a file changes its content or metadata the ETag or "entity tag" changes. In the early days of ownCloud it was decided to extend this behavior to folders as well, which is outside of any WebDAV RFC specification. Nevertheless, here we are, using the ETag to reflect changes, not only on WebDAV resources but also WebDAV collections. The server will propagate the ETag change up to the root of the tree.
{{<mermaid class="text-center">}}
graph TD
linkStyle default interpolate basis
subgraph final ETag propagation
ert3(( etag:N )) --- el3(( etag:O )) & er3(( etag:N ))
er3 --- erl3(( etag:O )) & err3(( etag:N ))
end
subgraph first ETag propagation
ert2(( etag:O )) --- el2(( etag:O )) & er2(( etag:N ))
er2 --- erl2(( etag:O )) & err2(( etag:N ))
end
subgraph initial file change
ert(( etag:O )) --- el(( etag:O )) & er(( etag:O ))
er --- erl(( etag:O )) & err(( etag:N ))
end
{{</mermaid>}}
The old `etag:O` is replaced by propagating the new `etag:N` up to the root, where the client will pick it up and explore the tree by comparing the old ETags known to him with the state of the current ETags on the server. This form of sync is called *state based sync*.
## Single user sync
To let the client detect changes in the drive (a tree of files and folders) of a user, we rely on the ETag of every node in the tree. The discovery phase starts at the root of the tree and checks if the ETag has changed since the last discovery:
- if it is still the same nothing has changed inside the tree
- if it changed the client will compare the ETag of all immediate children and recursively descend into every node that changed
This works, because the server side will propagate ETag changes in the tree up to the root.
{{<mermaid class="text-center">}}
graph TD
linkStyle default interpolate basis
ec( client ) -->|"stat()"|ert
subgraph
ert(( )) --- el(( )) & er(( ))
er --- erl(( )) & err(( ))
end
{{</mermaid>}}
## Multiple users
On an ocis server there is not one user but many. Each of them may have one or more clients running. In the worst case all of them polling the ETag of his home root node every 30 seconds.
Keep in mind that etags are only propagated inside each distinct tree. No sharing is considered yet.
{{<mermaid class="text-center">}}
graph TD
linkStyle default interpolate basis
ec( client ) -->|"stat()"|ert
subgraph
ert(( )) --- el(( )) & er(( ))
er --- erl(( )) & err(( ))
end
mc( client ) -->|"stat()"|mrt
subgraph
mrt(( )) --- ml(( )) & mr(( ))
mr --- mrl(( )) & mrr(( ))
end
fc( client ) -->|"stat()"|frt
subgraph
frt(( )) --- fl(( )) & fr(( ))
fr --- frl(( )) & frr(( ))
end
{{</mermaid>}}
## Sharing
*Storage providers* are responsible for persisting shares as close to the storage as possible.
One implementation may persist shares using ACLs, another might use custom extended attributes. The chosen implementation is storage specific and always a tradeoff between various requirements. Yet, the goal is to treat the storage provider as the single source of truth for all metadata.
If users can bypass the storage provider using e.g. `ssh` additional mechanisms needs to make sure no inconsistencies arise:
- the ETag must still be propagated in a tree, eg using inotify, a policy engine or workflows triggered by other means
- deleted files should land in the trash (e.g. `rm` could be wrapped to move files to trash)
- overwriting files should create a new version ... other than a fuse fs I see no way of providing this for normal posix filesystems. Other storage backends that use the s3 protocol might provide versions natively.
The storage provider is also responsible for keeps track of references e.g. using a shadow tree that users normally cannot see or representing them as symbolic links in the filesystem (Beware of symbolic link cycles. The clients are currently unaware of them and would flood the filesystem).
To prevent write amplification ETags must not propagate across references. When a file that was shared by einstein changes the ETag must not be propagated into any share recipients tree.
{{<mermaid class="text-center">}}
graph TD
linkStyle default interpolate basis
ec( einsteins client ) -->|"stat()"|ert
subgraph
ml --- mlr(( ))
mrt(( )) --- ml(( )) & mr(( ))
mr --- mrl(( )) & mrr(( ))
end
mlr -. reference .-> er
subgraph
ert(( )) --- el(( )) & er(( ))
er --- erl(( )) & err(( ))
end
mc( maries client ) -->|"stat()"|mrt
{{</mermaid>}}
But how can Marie's client detect the change?
We are trading writes for reads: the client needs to stat the own tree & all shares or entry points into other storage trees.
It would require client changes that depend on the server side actually having an endpoint that can efficiently list all entry points into storages a user has access to including their current etag.
But having to list n storages might become a bottleneck anyway, so we are going to have the gateway calculate a virtual root ETag for all entry points a user has access to and cache that.
## Server Side Stat Polling
Every client polls the virtual root ETag (every 30 sec). The gateway will cache the virtual root ETag of every storage for 30 sec as well. That way every storage provider is only stated once every 30 sec (can be throttled dynamically to adapt to storage io load).
{{<mermaid class="text-center">}}
graph TD
linkStyle default interpolate basis
ec( client ) -->|"stat()"|evr
subgraph gateway caching virtual etags
evr(( ))
mvr(( ))
fvr(( ))
end
evr --- ert
mvr --- mrt
fvr --- frt
subgraph
ert(( )) --- el(( )) & er(( ))
er --- erl(( )) & err(( ))
end
mc( client ) -->|"stat()"|mvr
subgraph
mrt(( )) --- ml(( )) & mr(( ))
ml --- mlm(( ))
mr --- mrl(( )) & mrr(( ))
end
mlm -.- er
mvr -.- er
fc( client ) -->|"stat()"|fvr
subgraph
frt(( )) --- fl(( )) & fr(( ))
fr --- frl(( )) & frr(( ))
end
{{</mermaid>}}
Since the active clients will poll the etag for all active users the gateway will have their ETag cached. This is where sharing comes into play: The gateway also needs to stat the ETag of all other entry points ... or mount points. That may increase the number of stat like requests to storage providers by an order of magnitude.
### Ram considerations
For a single machine using a local posix storage the linux kernel already caches the inodes that contain the metadata that is necessary to calculate the ETag (even extended attributes are supported). With 4k inodes 256 nodes take 1Mb of RAM, 1k inodes take 4Mb and 1M inodes take 4Gb to completely cache the file metadata. For distributed filesystems a dedicated cache might make sense to prevent hammering it with stat like requests to calculate ETags.
### Bandwidth considerations
The bandwidth for a single machine might be another bottleneck. Consider a propfind request with roughly 500 bytes and a response with roughly 800 bytes in size:
- At 100Mbit (~10Mb/s) you can receive 20 000 PROPFIND requests
- At 1000Mbit (~100Mb/s) you can receive 200 000 PROPFIND requests
- At 10Gbit (~1Gb/s) you can receive 2 000 000 PROPFIND requests
This can be scaled by adding more gateways and sharding users because these components are stateless.
## Share mount point polling cache
What can we do to reduce the number of stat calls to storage providers. Well, the gateway queries the share manager for all mounted shares of a user (or all entry points, not only the users own root/home). The share references contain the storage provider that contains the share. If every user has its own storage provider id the gateway could check in its own cache if the storage root etag has changed. It will be up-to-date because another client likely already polled for its etag.
This would reduce the number of necessary stat requests to active storages.
### Active share node cache invalidation
We can extend the lifetime of share ETag cache entries and only invalidate them when the root of the storage that contains them changes its ETag. That would reduce the number of stat requests to the number of active users.
### Push notifications
We can further enhance this by sending push notifications when the root of a storage changes. Which is becoming increasingly necessary for mobile devices anyway.

207
docs/architecture/posixfs-storage-driver.md
View File

@@ -1,207 +0,0 @@
---
title: "PosixFS Storage Driver"
date: 2024-05-27T14:31:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: posixfs-storage-driver.md
---
{{< toc >}}
The Posix FS Storage Driver is a new storage driver for Infinite Scale.
The scope of this document is to give a high level overview to the technical aspects of the Posix FS Storage Driver and guide the setup.
## Introduction
The Posix FS Storage Driver is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale.
This does not mean that Infinite Scale is trading any of its benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to caches and search indexes, and it also features the full spaces concept as before, just to name a few examples.
The architecture of Infinite Scale allows configuring different storage drivers for specific storage types and purposes on a space granularity. The Posix FS Storage Driver is an alternative to the default driver called Decomposed FS.
However, the clarity of the file structure in the underlying file system is not the only benefit of the Posix FS Storage Driver. This new technology allows users to manipulate the data directly in the file system, and any changes made to files outside of Infinite Scale are monitored and directly reflected in Infinite Scale. For example, a scanner could store its output directly to the Infinite Scale file system, which immediately gets picked up in Infinite Scale.
For the first time ever with feature rich open source file sync & share systems, users can either choose to work with their data through the clients of the system, its APIs or even directly in the underlying file system on the server.
That is another powerful vector for integration and enables a new spectrum of use cases across all domains.
## Technical Aspects
The Posix FS Storage Driver uses a few features of the underlying file system, which are mandatory and directly contributing to the performance of the system.
While the simplest form of Posix FS Storage Driver runs with default file systems of every modern Linux system which are directly mounted and thus support inotify, the full power of this unfolds with more capable file systems such as IBM Storage Scale or Ceph. These are recommended as reliable foundations for big installations of Infinite Scale.
This chapter describes some technical aspects of this storage driver.
### Path Locations
The file tree that is used as storage path for both data and metadata is located under the local path on the machine that is running Infinite Scale. That might either be a real local file system or a mounted net filesystem. It is expected that oCIS is the only consumer of that file tree, except what is expected behaviour with a collaborative file system, that works with files in that tree.
Underneath the Infinite Scale file system root, there is a collection of different folders containing Infinite Scale specific data storing personal spaces, project spaces and indexes.
### Metadata
Infinite Scale is highly dependent on the efficient usage of meta data which are attached to file resources, but also logical elements such as spaces.
Metadata is stored in extended attributes (as also supported by decompsed FS) which poses the benefit that metadata is always directly attached to the actual resources. As a result, care has to be taken that extended attributes are considered when working with the file tree however, e.g. when creating or restoring backups.
Note: The maximum number and size of extended attributes are limited depending on the filesystem and block size. See [GPFS Specifics](#gpfs-specifics) for more details on GPFS file systems.
All indexing and caching of metadata is implemented in higher system levels than the storage driver, and thus are not different to the components used with other storage drivers like the decomposed FS.
### Monitoring
To get information about changes such as new files added, files edited or removed, Infinite Scale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as for example in Spectrum Scale (see [GPFS Specifics](#gpfs-specifics) for more details on GPFS file systems).
Based on the information transmitted by the watching service, Infinite Scale is able to "register" new or changed files into its own caches and internal management structures. This enables Infinite Scale to deliver resource changes through the "traditional" channels such as APIs and clients.
Since the most important metadata is the file tree structure itself, it is impossible for the "split brain" situation between data and metadata to cause trouble.
### Automatic ETag Propagation
The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is, that if the content of a file changed the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space.
Infinite Scale uses a built in mechanism to maintain the ETag for each resource in the file meta data, and also propagates it automatically.
A sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc.
### Automatic Tree Size Propagation
Similar to the ETag propagation described before, Infinite Scale also tracks the accumulated tree size in all nodes of the file tree. A change to any file requires a re-calculation of the size attribute in all parent folders.
Infinite Scale would benefit from file systems with native tree size propagation.
### Quota
Each space has it's own quota, thus every storage driver implementation needs to consider that.
For example, IBM Spectrum Scale supports quota handling directly in the file system.
Other systems store quota data in the metadata storage and implement propagation of used quota similar to the tree size propagation.
### File ID Resolution
Infinite Scale uses an ID based approach to work with resources, rather than a file path based mechanism. The reason for that is, that ID based lookups can be done way more efficiently compared to tree traversals, just to name one reason.
The most important component of the ID is a unique file ID that identifies the resource within a space. Ideally the Inode of a file could be used here. However, some file systems re-use inodes which must be avoided. Infinite Scale thus does not use the file Inode, but generates a UUID instead.
ID based lookups utilize an ID cache which needs to be shared between all storageprovider and dataprovider instances. During startup a scan of the whole file tree is performed to detect and cache new entities.
In the future a powerful underlying file system could support Infinite Scale by providing an API that
1. Provides the ID for a given file path referenced resource
2. Provides the path for a given ID.
These two operations are very crucial for the performance of the entire system.
### User Management
With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which UID the manipulation happens is important.
There are a few possible ways for user management:
1. Changes can either be only accepted by the same user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by this user.
2. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be member of that group. The default umask in the directory used has to allow group writing all over the place.
3. Impersonation: Infinite Scale impersonates the user who owns the folder on the file system to mimic the access as the user.
All possibilities have pros and cons for operations.
One for all, it seems reasonable to use LDAP to manage users which is the base for the Infinite Scale IDP as well as the system login system via PAM.
### GID Based Space Access
The Posix FS Storage Driver supports GID based space access to support the problem that project spaces might have to be accessible by multiple users on disk. In order to enable this feature the `ocis` binary needs to have the `setgid` capability and `STORAGE_USERS_POSIX_USE_SPACE_GROUPS` needs to be set to `true`. Inifinite Scale will then use the space GID (the gid of the space root) for all file system access using the `setfsgid` syscall, i.e. all files and directories created by Infinite Scale will belong to the same group as the space root.
## Advanced Features
Depending on the capabilities of the underlying file system, the Posix FS Storage Driver can benefit from more advanced functionality described here.
### Versioning
If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale.
In the current state of the Posix FS Storage Driver, versioning is not supported.
### Trashbin
If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale.
If not available it will follow the [the Free Desktop Trash specificaton](https://specifications.freedesktop.org/trash-spec/trashspec-latest.html).
## Limitations
As of Q2/2024 the Posix FS Storage Driver is not officially supported and in technical preview state.
The tech preview comes with the following limitations:
1. Only inotify and GPFS file system change notification methods are supported
1. Versioning is not supported yet
1. The space/project folders in the filesystem are named after the UUID, not the real space name
1. No CephFS support yet
1. Postprocessing (ie. anti virus check) does not happen for file actions outside of Infinite Scale
## Setup
This describes the steps to use the Posix FS Storage Driver storage driver with Infinite Scale.
It is possible to use different storage drivers in the same Infinite Scale installation. For example it is possible to set up one space running on Posix FS Storage Driver while others run Decomposed FS.
### Prerequisites
To use the Posix FS Storage Driver, the following prerequisites have to be fulfilled:
1. There must be storage available to store meta data and blobs, available under a root path.
1. When using inotify, the storage must be local on the same machine. Network mounts do not work with inotify. `inotifywait` needs to be installed.
1. The storage root path must be writeable and executable by the same user Infinite Scale is running under.
1. An appropiate version of Infinite Scale is installed, version number 5.0.5 and later.
1. `nats-js-kv` as cache service
### Setup Configuration
This is an example configuration with environment variables that configures Infinite Scale to use Posix FS Storage Driver for all spaces it works with, ie. Personal and Project Spaces:
```
export STORAGE_USERS_DRIVER="posix"
export STORAGE_USERS_POSIX_ROOT="/home/kf/tmp/posix-storage"
export STORAGE_USERS_POSIX_WATCH_TYPE="inotifywait"
export STORAGE_USERS_ID_CACHE_STORE="nats-js-kv"
export STORAGE_USERS_ID_CACHE_STORE_NODES="localhost:9233"
# Optionally enable gid based space access
export STORAGE_USERS_POSIX_USE_SPACE_GROUPS="true"
```
## GPFS Specifics
When using GPFS as the underlying filesystem the machine running the according `storage-users` service needs to have the GPFS filesystem mounted locally. The mount path is given to ocis as the `STORAGE_USERS_POSIX_ROOT` path.
Other than that there a few other points to consider:
### Extended Attributes
As described above metadata is stored as extended attributes of the according entities and thus is suspect to their limitations. In GPFS extended attributes are first stored in the inode itself but can then also use an overflow block which is at least 64KB and up to the metadata block size. Inode and metadata block size should be chosen accordingly.
### FS Watcher
The Posix FS Storage Driver supports two different watchers for detecting changes to the filesystem. The watchfolder watcher is better tested and supported at that point.
#### GPFS File Audit Logging
The `gpfsfileauditlogging` watcher tails a GPFS file audit log and parses the JSON events to detect relevant changes.
```
export STORAGE_USERS_POSIX_WATCH_TYPE="gpfsfileauditlogging"
export STORAGE_USERS_POSIX_WATCH_PATH="/path/to/current/audit/log"
```
#### GPFS Watchfolder
The `gpfswatchfolder` watcher connects to a kafka cluster which is being filled with filesystem events by the GPFS watchfolder service.
```
export STORAGE_USERS_POSIX_WATCH_TYPE="gpfswatchfolder"
export STORAGE_USERS_POSIX_WATCH_PATH="fs1_audit" # the kafka topic to watch
export STORAGE_USERS_POSIX_WATCH_FOLDER_KAFKA_BROKERS="192.168.1.180:29092"
```

200
docs/architecture/protocol-changes.md
View File

@@ -1,200 +0,0 @@
---
title: "Protocol changes"
date: 2022-05-17T08:46:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: protocol-changes.md
---
The spaces concept allows clients to look up the space endpoints a user has access to and then do individual sync discoveries. Technically, we introduce an indirection that allows clients to rely on server provided URLs instead of hardcoded `/webdav` or `/dav/files/{username}` paths, that may change over time.
## Space discovery
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant Graph
participant SpaceA
participant SpaceB
links Client: {"web": "https://owncloud.dev/clients/web/", "RClone": "https://owncloud.dev/clients/rclone/"}
link Graph: Documentation @ https://owncloud.dev/extensions/graph/
Note left of Client: First, a clients looks<br/>up the spaces a user has access to
opt space lookup
Client->>+Graph: GET /me/drives
Graph-->>-Client: 200 OK JSON list of spaces, say A, B and C,<br/> each with a dedicated webDavURL, etag and quota
end
Note left of Client: Then it can do a parallel<br/>sync discovery on spaces<br/>whose etag changed
par Client to Space A
Client->>+SpaceA: PROPFIND {webDavURL for Space A}
SpaceA-->>-Client: 207 Multistatus PROPFIND response
and Client to Space B
Client->>+SpaceB: PROPFIND {webDavURL for space B}
SpaceB-->>-Client: 207 Multistatus PROPFIND response
end
{{</mermaid>}}
### New /dav/spaces/{spaceid} endpoint with spaceid and a relative path
The ocDAV service is responsible for translating ownCloud flavoured WebDAV into CS3 API calls.
**General view**
A PROPFIND finds its way to a storage provider like this:
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocDAV
participant StorageProvider
Note right of Client: {spaceid} identifies the space<br>{relative/path} is relative to the space root
Client->>+ocDAV: PROPFIND /dav/space/{spaceid}/{relative/path}
Note right of ocDAV: translate ownCloud flavoured webdav<br>into CS3 API requests
ocDAV->>+StorageProvider: ListContainer({spaceid}, path: {relative/path})
StorageProvider-->>-ocDAV: []ResourceInfo
ocDAV-->>-Client: 207 Multistatus
{{</mermaid>}}
While the above is a simplification to get an understanding of what needs to go where, there are several places where sharding can happen.
**Proxy can do user based routing**
The ocis proxy authenticates requests and can forward requests to different backends, depending on the logged-in user or cookies. For example multiple ocdav services can be configured to shard users based on username or affiliation.
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant proxy
participant ocDAV1 as ocDAV [a-k]
participant ocDAV2 as ocDAV [l-z]
Note right of Client: {spaceid} identifies the space<br>{relative/path} is relative to the space root
Client->>+proxy: PROPFIND /dav/space/{spaceid}/{relative/path}
alt username starting with a-k
proxy->>+ocDAV1: PROPFIND /dav/space/{spaceid}/{relative/path}
Note right of ocDAV1: translate ownCloud flavoured webdav<br>into CS3 API requests
ocDAV1-->>-Client: 207 Multistatus
else username starting with l-z
proxy->>+ocDAV2: PROPFIND /dav/space/{spaceid}/{relative/path}
ocDAV2-->>-Client: 207 Multistatus
end
{{</mermaid>}}
**Gateway can do path or storage provider id based routing**
The reva gateway acts as a facade to multiple storage providers that can be configured with the storage registry:
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant ocDAV
participant Gateway
participant StorageRegistry
participant StorageProvider1 as StorageProvider [a-k]
participant StorageProvider2 as StorageProvider [l-z]
Note right of ocDAV: translate ownCloud flavoured webdav<br>into CS3 API requests
ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path})
Note right of Gateway: find address of the storage provider<br>that is responsible for the space
Gateway->>+StorageRegistry: ListStorageProviders({spaceid})
StorageRegistry-->>-Gateway: []ProviderInfo
Note right of Gateway: forward request to<br>correct storage provider
alt username starting with a-k
Gateway->>+StorageProvider1: ListContainer({spaceid}, path: {relative/path})
StorageProvider1-->>-Gateway: []ResourceInfo
else username starting with l-z
Gateway->>+StorageProvider2: ListContainer({spaceid}, path: {relative/path})
StorageProvider2-->>-Gateway: []ResourceInfo
end
Gateway-->>-ocDAV: []ResourceInfo
{{</mermaid>}}
### Old /dav/files/{username} endpoint with username and a path relative to the users home
**PROPFIND request against old webdav endpoints**
To route a PROPFIND request against the old webdav endpoints like `/dav/files/username`, ocdav first has to build a CS3 namespace prefix, e.g. `/users/{{.Id.OpaqueId}}` to the users home.
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocDAV
participant Gateway
opt old /dav/files/{username} endpoint with username and a path relative to the users home
Note right of Client: translate ownCloud flavoured webdav<br>into CS3 API requests
Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path}
Note right of ocDAV: translate ownCloud flavoured webdav<br>into CS3 API requests
ocDAV->>+Gateway: GetUser({username})
Gateway-->>-ocDAV: User
Note right of ocDAV: build path prefix to user home
ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({path layout}, User), eg. /users/e/einstein
Note right of ocDAV: look up the space responsible for a path
ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path})
Gateway-->>-ocDAV: []StorageSpace
Note right of ocDAV: make actual request with space and relative path
ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path})
Gateway-->>-ocDAV: []ResourceInfo
ocDAV-->>-Client: 207 Multistatus
end
{{</mermaid>}}
**Handling legacy global namespace webdav endpoints**
The reason ocis uses a path based lookup instead of looking up the current users home using the user id and a space type filter is, because there are deployments that use a global namespace at the legacy `/webdav` endpoint. To support these use cases, the gateway allows looking up spaces using their mount path.
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocDAV
participant Gateway
Note right of Client: translate ownCloud flavoured webdav<br>into CS3 API requests
alt old /dav/files/{username} endpoint with username and a path relative to the users home
Client->>+ocDAV: PROPFIND /dav/files/{username}/{relative/path}
Note right of ocDAV: look up {username} in URL path
ocDAV->>+Gateway: GetUser({username})
Gateway-->>-ocDAV: User
Note right of ocDAV:build namespace prefix to user home
ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein
else legacy /webdav/ endpoint with a path relative to the users home
Client->>+ocDAV: PROPFIND /webdav/{relative/path}
Note right of ocDAV: use currently logged in user
ocDAV->>+ocDAV: ContextGetUser()
Note right of ocDAV: build namespace prefix to user home
ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout({namespace layout}, User), eg. /users/e/einstein
else legacy /webdav/ endpoint with a path relative to a global namespace
Client->>+ocDAV: PROPFIND /webdav/{relative/path}
Note right of ocDAV: omit namespace prefix by using empty layout template
ocDAV->>+ocDAV: {namespace/prefix} = ApplyLayout("/", u), always returns "/"
end
Note right of ocDAV: look up the space responsible for a path
ocDAV->>+Gateway: ListStorageSpaces(path: {namespace/prefix}/{relative/path})
Gateway-->>-ocDAV: []StorageSpace
Note right of ocDAV: make actual request with space and relative path
ocDAV->>+Gateway: ListContainer({spaceid}, path: {relative/path})
Gateway-->>-ocDAV: []ResourceInfo
ocDAV-->>-Client: 207 Multistatus
{{</mermaid>}}

10
docs/architecture/services-communication.md
View File

@@ -1,10 +0,0 @@
---
title: "Services Communication"
date: 2022-02-15T11:26:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: services-communication.md
---
{{< figure src="/ocis/static/ocis-services-communication.drawio.svg" >}}

121
docs/architecture/upload-processing.md
View File

@@ -1,121 +0,0 @@
---
title: "Upload processing"
date: 2022-07-06T12:47:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/architecture
geekdocFilePath: upload-processing.md
---
Uploads are handled by a dedicated service that uses TUS.io for resumable uploads. When all bytes have been transferred the upload is finalized by making the file available in file listings and for download.
The finalization may be asynchronous when mandatory workflow steps are involved.
## Legacy PUT upload
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocdav
participant storageprovider
participant dataprovider
Client->>+ocdav: PUT /dav/spaces/{spaceid}/newfile.bin
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol simple, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/simple/91cc9882-db71-4b37-b694-a522850fcee1
ocdav->>+dataprovider: PUT /data<br>X-Reva-Transfer: {jwt}
dataprovider-->>-ocdav: 201 Created
ocdav-->>-Client: 201 Created
{{</mermaid>}}
## TUS upload
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocdav
participant storageprovider
participant datagateway
participant dataprovider
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
Note over datagateway: unwrap the {jwt} target
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
Note over dataprovider: storage driver<br>handles request
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
{{</mermaid>}}
## TUS upload with async postprocessing
{{<mermaid class="text-center">}}
%%{init: {"sequence": { "showSequenceNumbers":true, "messageFontFamily":"courier", "messageFontWeight":"normal", "messageFontSize":"11"}}}%%
%% font weight is a css bug: https://github.com/mermaid-js/mermaid/issues/1976
%% edit this diagram by pasting it into eg. https://mermaid.live
sequenceDiagram
participant Client
participant ocdav
participant storageprovider
participant datagateway
participant dataprovider
participant nats
Client->>+ocdav: POST /dav/spaces/{spaceid}<br>Upload-Metadata: {base64 encoded filename etc}<br>TUS-Resumable: 1.0.0
ocdav->>+storageprovider: InitiateFileUpload
storageprovider-->>-ocdav: OK, Protocol tus, UploadEndpoint: /data, Token: {jwt}
Note right of ocdav: The {jwt} contains the internal actual target, eg.:<br>http://localhost:9158/data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160
ocdav-->>-Client: 201 Created<br>Location: /data/{jwt}<br>TUS-Resumable: 1.0.0
Client->>+datagateway: PATCH /data/{jwt}<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 0
Note over datagateway: unwrap the {jwt} target
datagateway->>+dataprovider: PATCH /data/tus/24d893f5-b942-4bc7-9fb0-28f49f980160<br>X-Reva-Transfer: {jwt}
Note over dataprovider: storage driver<br>handles request
dataprovider-)nats: emit all-bytes-received event
nats-)processing: all-bytes-received({uploadid}) event
Note over dataprovider: TODO: A lot of time may pass here, we could use<br> the `Prefer: respond-async` header to return early<br>with a 202 Accepted status and a Location header<br>to a websocket endpoint
alt success
processing-)nats: emit processing-finished({uploadid}) event
nats-)dataprovider: processing-finished({uploadid}) event
dataprovider-->>-datagateway: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 204 No Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
else failure
activate dataprovider
activate datagateway
processing-)nats: emit processing-aborted({uploadid}) event
nats-)dataprovider: processing-aborted({uploadid}) event
Note over dataprovider: FIXME: What HTTP status code should we report?<br>422 Unprocessable Content is just a proposal, see<br>https://httpwg.org/specs/rfc9110.html#status.422
dataprovider-->>-datagateway: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
datagateway-->>-Client: 422 Unprocessable Content<br>TUS-Resumable: 1.0.0<br>Upload-Offset: 363976
end
{{</mermaid>}}
## Async TUS upload with postprocessing
This might be a TUS extension or a misunderstanding on our side of what tus can do for us. Clients should send a `Prefer: respond-async` header to allow the server to return early when postprocessing might take longer. The PATCH requests can then return status `202 Accepted` and a `Location` header to a websocket that clients can use to track the processing / upload progress.
TODO there is a conflict with the TUS.io POST request with the creation extension, as that also returns a `Location` header which carries the upload URL. We would need another header to transport the websocket location. Maybe `Websocket-Location` or `Progress-Location`?

9
docs/clients/_index.md
View File

@@ -1,9 +0,0 @@
---
title: Clients
date: 2023-12-06T13:00:00+01:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/clients/
geekdocFilePath: _index.md
geekdocCollapseSection: true
---

25
docs/clients/rclone/_index.md
View File

@@ -1,25 +0,0 @@
---
title: Rclone
date: 2021-11-17T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/clients/rclone
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
## About Rclone
{{< hint ok >}}
Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It is used at the command line, in scripts or via its API.
Users call rclone "The Swiss army knife of cloud storage", and "Technology indistinguishable from magic".
{{< /hint >}}
Source: [Rclone project website](https://rclone.org/)
## Table of Contents
{{< toc-tree >}}

46
docs/clients/rclone/webdav-sync-basic-auth.md
View File

@@ -1,46 +0,0 @@
---
title: WebDAV with Basic Authentication
date: 2021-11-17T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/clients/rclone
geekdocFilePath: webdav-sync-basic-auth.md
geekdocCollapseSection: true
---
## WebDAV with Basic Authentication
{{< hint danger >}}
Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication e.g. by setting the environment variable `PROXY_ENABLE_BASIC_AUTH` to `true`.
Please consider to use [Rclone with OpenID Connect]({{< ref "webdav-sync-oidc.md" >}}) instead.
{{< /hint >}}
For the usage of a WebDAV remote with Rclone see also the [Rclone documentation](https://rclone.org/webdav/)
## Configure the WebDAV remote
First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same.
``` bash
export RCLONE_WEBDAV_VENDOR=owncloud
export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/
export RCLONE_WEBDAV_USER=einstein
export RCLONE_WEBDAV_PASS=$(rclone obscure relativity)
```
{{< hint info >}}
Please note that `RCLONE_WEBDAV_PASS` is not set to the actual password, but to the value returned by `rclone obscure <password>`.
{{< /hint >}}
We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder.
### Sync to the WebDAV remote
``` bash
rclone sync :local:/tmp :webdav:/test
```
If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`.

77
docs/clients/rclone/webdav-sync-oidc.md
View File

@@ -1,77 +0,0 @@
---
title: WebDAV with OpenID Connect
date: 2021-11-17T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/clients/rclone
geekdocFilePath: webdav-sync-oidc.md
geekdocCollapseSection: true
---
## WebDAV with OpenID Connect
Rclone itself is not able to open and maintain an OpenID Connect session. But it is able to still use OpenID Connect for authentication by leveraging a so called OIDC-agent.
### Setting up the OIDC-agent
You need to install the [OIDC-agent](https://github.com/indigo-dc/oidc-agent) from your OS' package repository (e.g. [Debian](https://github.com/indigo-dc/oidc-agent#debian-packages) or [MacOS](https://github.com/indigo-dc/oidc-agent#debian-packages)).
### Configuring the OIDC-agent
Run the following command to add a OpenID Connect profile to your OIDC-agent. It will open the login page of OpenID Connect identity provider where you need to log in if you don't have an active session.
``` bash
oidc-gen \
--client-id=oidc-agent \
--client-secret="" \
--pub \
--issuer https://ocis.owncloud.test \
--redirect-uri=http://localhost:12345 \
--scope max \
einstein-ocis-owncloud-test
```
If you have dynamic client registration enabled on your OpenID Connect identity provider, you can skip the `--client-id`, `--client-secret` and `--pub` options.
If you're using a dedicated OpenID Connect client for the OIDC-agent, we recommend a public one with the following two redirect URIs: `http://127.0.0.1:*` and `http://localhost:*`. Alternatively you also may use the already existing OIDC client of the ownCloud Desktop Client (`--client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69` and `--client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh`, no `--pub` set, request specific scope for offline access), e.g.:
``` bash
oidc-gen \
--client-id=xdXOt13JKxym1B1QcEncf2XDkLAexMBFwiT9j6EfhhHFJhs2KM9jbjTmf8JBXE69 \
--client-secret=UBntmLjC2yYCeHwsyj73Uwo9TAaecAetRwMw0xYcvNL9yRdLSUi0hUAHfvCHFeFh \
--issuer https://cloud.ocis.test \
--redirect-uri=http://localhost:12345 \
--scope="openid offline_access profile email" \
my-client
```
When using a self signed certificate you have to provide the certificate chain using `--cp /etc/ssl/certs/test.cert.pem`. In case oidc-gen cannot determine the flow try with `--flow=code`.
Please also note that the OIDC-agent will listen on your localhost interface on port 12345 for the time of the initial authentication. If that port is already occupied on your machine, you can easily change that by setting the `--redirect-uri` parameter to a different value.
After a successful login or an already existing session you will be redirected to success page of the OIDC-agent.
You will now be asked for a password for your account configuration, so that your OIDC session is secured and cannot be used by other people with access to your computer.
## Configure the WebDAV remote
First of all we need to set up our credentials and the WebDAV remote for Rclone. In this example we do this by setting environment variables. You might also set up a named remote or use command line options to achieve the same.
``` bash
export RCLONE_WEBDAV_VENDOR=owncloud
export RCLONE_WEBDAV_URL=https://ocis.owncloud.test/remote.php/webdav/
export RCLONE_WEBDAV_BEARER_TOKEN_COMMAND="oidc-token einstein-ocis-owncloud-test"
```
### Sync to the WebDAV remote
We now can use Rclone to sync the local folder `/tmp/test` to `/test` in your oCIS home folder.
``` bash
rclone sync :local:/tmp :webdav:/test
```
If your oCIS doesn't use valid SSL certificates, you may need to use `rclone --no-check-certificate sync ...`.

230
docs/helpers/README.md
View File

@@ -1,230 +0,0 @@
# Docs Helpers
* [Introduction](#introduction)
* [Output Generated](#output)
* [Admin Doc Process](#admin-doc-process)
* [Branching](#branching)
* [Service-Dependent Output](#service-dependent-output)
* [Generate Envvar Docs for Config Structs](#generate-envvar-docs-for-config-structs)
* [Deprecation Process](#deprecation-process)
* [Global Envvars](#global-envvars)
* [Extended Envvars](#extended-envvars)
* [General Extended Envvars Info](#general-extended-envvars-info)
* [Extract Extended Envvars](#extract-extended-envvars)
* [Generate Extended Envvar Docs](#generate-extended-envvar-docs)
* [Tasks for New Releases](#tasks-for-new-releases)
* [Backporting](#backporting)
## Introduction
`docs/helpers` contains a go program named `main.go` which creates docs by extracting information from the code using additional go programs. Individual steps (programs) can be called manually if needed. Note that not all programs are called automatically on purpose, see the [Tasks for New Releases](#tasks-for-new-releases) below. `main.go` is used by `make docs-generate` (or `make -C docs docs-generate` when running manually from the repos root) which is triggered by the CI or can be called manually. It calls the other required programs and has these main responsibilities for automatic runs:
- Generate docs for envvars in config structs including deprecations if there are any.
- Extract and generate docs for `extended` envvars that are not mentioned in config structs (aka "rogue" envvars).
- Extract and generate docs for `global` envvars which occur in multiple services.
- Create `docs/service/<service-name>/_index.md` from `service/<service-name>/README.md` files while keeping the existing `_index.md` if the service README.md has not been created so far. Also see the important note at [docs README](../README.md).
## Output Generated
- The generated yaml files can be found at: `docs/services/_includes` when running locally respectively in the `docs branch` after the CI has finished.
- The generated adoc files can be found at: `docs/services/_includes/adoc` when running locally respectively in the `docs branch` after the CI has finished.
- The file name for global envvars is named: `global_configvars.adoc`.
- The file name for extended envvars is named: `extended_configvars.adoc`.
- A file named `docs/helpers/env_vars.yaml` containing envvar changes gets updated if changes have been identified.
- A file named `docs/helpers/extended_vars.yaml` containing changes for extended envvars gets updated if changes have been identified. Note, if changes appear, **this file needs manual treatment** before committing, see [Extended Envvars](#extended-envvars) below.
## Admin Doc Process
Whenever a build from the [ocis admin](https://github.com/owncloud/docs-ocis) documentation or any other admin related documentation is triggered, files generated here in the ocis repo are included into the build process and added in a proper manner defined by the admin documentation. The updated documentation will then show up on the public [admin documentation](https://doc.owncloud.com/ocis/next/).
## Branching
The following is valid for envvars and yaml files related to the doc process:
* When filing a pull request in the ocis master branch relating to docs, CI runs `make docs-generate` and copies the result into the `docs` branch of ocis. This branch is then taken as base for owncloud.dev and as reference for the [admin docs](https://doc.owncloud.com/ocis/next/).
* When running `make docs-generate` _locally_, the same output is created as above but it stays in the same branch where the make command was issued.
In both cases, `make docs-generate` removes files in the target folder `_includes` to avoid remnants. All content is recreated.
On a side note (unrelated to the `docs` branch), [deployment examples](https://github.com/owncloud/ocis/tree/master/deployments/examples) have their own branch related to an ocis stable version to keep the state consistent, which is necessary for the admin documentation.
## Service-Dependent Output
For each service available, a file named like `<service name>_configvars.adoc` is created containing a:
* table on top defining deprecated envvars - if applicable
* table containing all envvars with their name, type, default value and description
The table with deprecations is always printed in the final adoc file even if there are none, but is rendered in the docs build process only if the `HasDeprecations` value is set. This value is automatically handed over via the adoc file. The template file can be found at `docs/templates/ADOC.tmpl`.
### Generate Envvar Docs for Config Structs
Generates docs from a template file, mainly extracting `"env"` and `"desc"` tags from the config structs.
Templates can be found in `docs/helpers` folder. (Same as this `README`.) Check `.tmpl` files
### Deprecation Process
For details on deprecation see the [deprecating-variables](https://github.com/owncloud/ocis/blob/master/docs/ocis/development/deprecating-variables.md) documentation.
## Global Envvars
Global envvars are gathered by checking if the envvar is available in more than one service. The table created is similar to the service-dependent envvar table but additionally contains a column with all service names where this envvar occurs. The output is rendered in list form where each item is clickable and automatically points to the corresponding service page. The template file can be found at `docs/templates/ADOC_global.tmpl`.
If global envvars do not appear in the list of globals, before checking if the code works, do a manual search in the ocis/services folder with `grep -rn OC_xxx` if the envvar in question appears at least twice. If the envvar only appears once, the helpers code works correct.
## Extended Envvars
### General Extended Envvars Info
"Extended" envvars are variables that need to be present *before* the core or services are starting up as they depend on the info provided like path for config files etc. Therefore they are _not_ bound to services like other envvars. Extended envvars are identified via `os.Getenv`, usually defined via a subfolder of `ocis-pkg`. The real envvar name name cant be automatically assigned and needs to be manually defined via the code in the `extended_vars.yaml` file.
It can happen that extended envvars are found but do not need to be published as they are for internal use only. Those envvars can be defined to be ignored for further processing.
**IMPORTANT:**
- **First Time Identification**\
Once an extended envvar has been identified, it is added to the `extended_vars.yaml` file found, but never changed or touched by the process anymore. There is one exception with respect to single/double quote usage. While you can (and will) manually define a text like: `"'/var/lib/ocis'"`, quotes are transformed by the process in the .yaml file to: `'''/var/lib/ocis'''`. There is no need to change this back, as the final step transforms this correctly for the adoc table.
- **Item Naming**\
An extended envvar may not have the right naming. It may appear as `name: _registryEnv`. In case, this envvar needs to be named properly like `name: MICRO_REGISTRY` which can only be done in close alignment with development.
- **Item Uniqueness**\
The identification, if an envvar is already present in the yaml file, is made via the `rawname` and the `path` identifier which includes the line number. **If there is a change in the source file shifting line numbers, new items will get added and old ones do not get touched.** Though technically ok, this can cause confusion to identify which items are correctly present or just added additionally just be cause code location has changed. If there are multiple occurrences of the same `rawname` value, check which one contains relevant data and set `do_ignore` to `false` and all others to `true`. When there are two identical blocks with different source references, mostly the one containing a proper `default_value` is the active one. Populate the false block with the envvar data to be used.
- **Fixing Items**\
If an item has been identified as additionally added because there was a change in the code location, it is mostly sufficient to just fix the line number in the `path` key of the existing/correct one and double check by removing the newly added ones. Then, re-run `make docs-generate`. If the fix was correct, no new items of the same will re-appear.
- **Remove Orphaned Items**\
To get rid of items with wrong line numbers, check `rawname` the `path` and correct the _existing ones_, especially the one containing the description and which is marked `do_ignore` false. Only items that have a real line number match need to be present, orphaned items can safely be removed. You can double-check valid items by creating a dummy branch, delete the `extended_vars.yaml` and run `make docs-generate` to regenerate the file having only items with valid path references. With that info, you can remove orphaned items from the live file. Note to be careful on judging only on `foundincode` set to false indicating an item not existing anymore. Fix all items first, when rerunning `make docs-generate`, this may change back to true!
- **Sort Ordering**\
Do not change the sort order of extended envvar blocks as they are automatically reordered alphabetically.
- **Mandatory Key Values**\
Because extended envvars do not have the same structural setup as "normal" envvars (like type, description or defaults), this info needs to be provided manually once - for each valid block. Any change of this info will be noticed during the next CI run, the corresponding adoc file generated, changes transported to the docs branch and published in the next admin docs build. See the following example with all keys listed and populated:
```yaml
rawname: registryAddressEnv
path: ocis-pkg/registry/registry.go:44
foundincode: true
name: MICRO_REGISTRY_ADDRESS
type: string
default_value: ""
description: The bind address of the internal go micro framework. Only change on
supervision of ownCloud Support.
do_ignore: false
```
### Extract Extended Envvars
The grep command parses the code, looking for `os.Getenv` and passes these contents to a yaml file along with the following information:
```golang
// Variable contains all information about one rogue envvar
type Variable struct {
// These field structs are automatically filled:
// RawName can be the name of the envvar or the name of its var
RawName string `yaml:"rawname"`
// Path to the envvar with linenumber
Path string `yaml:"path"`
// FoundInCode indicates if the variable is still found in the codebase.
FoundInCode bool `yaml:"foundincode"`
// Name is equal to RawName but will not be overwritten in consecutive runs
Name string `yaml:"name"`
// These field structs need manual filling:
// Type of the envvar
Type string `yaml:"type"`
// DefaultValue of the envvar
DefaultValue string `yaml:"default_value"`
// Description of what this envvar does
Description string `yaml:"description"`
// Do not export this envvar into the generated adoc table
Ignore bool `yaml:"do_ignore"`
// For simplicity ignored for now:
// DependendServices []Service `yaml:"dependend_services"`
}
```
This yaml file can later be manually edited to add descriptions, default values, etc.
**IMPORTANT**: `RawName`, `Path` and `FoundInCode` are automatically filled by the program. DO NOT EDIT THESE VALUES MANUALLY.
### Generate Extended Envvar Docs
The process further picks up the `yaml` file generated in the `Extract Rogue Envvars` step and renders it to an adoc file (a table is created) using a go template. The template file for this step can be found at `docs/templates/ADOC_extended.tmpl`.
## Doc Tasks for New Releases
**IMPORTANT**\
For a new ocis release, some tasks are necessary to be done **before** and **after** releasing. Follow the steps carefully to avoid issues. Most of the docs related tasks are not part of the CI. With each step finished successfully, the next step can be started. Sometimes, due to last minute changes, steps need to be redone!
**Backgroud Information**\
Admin docs rely on the existance of the following branches in the ocis repo. Note that the reference in the admin docs which ocis branch is accessed is defined in the `antora.yml` file via `attributes` existing in each branch.
* `docs`\
This reflects ocis master and is referenced from the admin docs from the master branch showing as `next` in the documentation.
* `docs-stable-x.y`\
This reflects a published ocis release and is referenced from the admin docs from the corresponding branch showing like `7.0`
Because of this, branching and parametrizing admin docs occurs **after** branching an ocis release with its necessary branches! If you branch admin docs before the required ocis branches are available, you must set the ocis source branch to import data from to `docs` and reconfigure afterwards.
**Notes**
* When docs relevant data will be generated in ocis, they will be written into the `docs/servcies/...` folder structure, but they are in the master branch. When merging, an automated process will move/copy them into the `docs`branch. When running make commands locally, the relocation is not done and files reside on the generated location!
* .adoc file generation
* Service related adoc files are autogenerated and saved in `/docs/services/<service-name>` but will be _relocated_ by each merge into the docs branch into a subfolder named `services/_include/adoc/<service-name.adoc>`.
* Release based envvar changes are saved during manual generation in `/docs/services/general-info/env-var-deltas/<filename>` but will be _copied_ on each merge into the doc branch into a subfolder named `services/_include/adoc/env-var-deltas/<filename>`.
* Exclude paths for Hugo
* The paths defined above for .adoc files are excluded from parsing by Hugo. The source of what to exclude is defined in the `config.yaml` file located in [owncloud.github.io](https://github.com/owncloud/owncloud.github.io/). Any folder containg .adoc files must be added there else `make -C docs docs-serve` or CI will fail reporting an adoc related parsing error. (We could also add an asciidoc parser to Hugo as alternative...)
### Task List
The following can be done at any time but it must be done *latest* when no envvar changes are made which is just before a new release gets finally tagged. The data generated **must** be part of the upcoming release and be merged before tagging/branching!
**Before Releasing** Generate required .adoc files necessary for the admin docs
* Run from the ocis root `make -C docs docs-generate`
* **Check for Extended Envvars**\
Check if there is a change in the `extended-envars.yaml` output. In this case, process [Extended Envvars](#extended-envvars). When done, re-run the make command and check if the output of `./docs/services/_includes/adoc/extended_configvars.adoc` matches the expectations.
* **Check for a changed 'env_vars.yaml' File**\
This file will most likely show changes and merging them is essential as base for added/removed or deprecated envvars. This file will get additions/updates only, but things never get automatically deleted.
* **Mandatory for a new release**, remove all envvars from the `env_vars.yaml` file manually that have been removed from the code, either with or without deprecation.
Commit the changes, create a PR and merge it. Next steps are based on this!
* Create added/removed and deprecated envvar .adoc files between versions containing includable tables pulled by the admin docs.
* Parametrize variables in `docs/helpers/changed_envvars.py` according your needs necessary to generate proper output.
* Create a branch and run from the ocis root `python3 docs/helpers/changed_envvars.py`\
This will create three files that need manual treatment before committing/merging. Note that this script accesses sources from github and not locally, therefore an actual `env_vars.yaml` in github is essential.
* Check the output of all three files and fix/delete any envvar in `env_vars.yaml` if required. If fixes have been done, the changed `env_vars.yaml` file **MUST** be _merged_ first, you must rebase and rerun the python script.
* If all three files contain correct data, manually set the xrefs. This cant be done automatically. Use one of the existing `added` files as template to generate a consistent output.
* Merge the three files and rerun the python script to prove all has been done correctly. You should now see only changes in xrefs as they get overwritten by default data. If this is the case, you can safely drop the generated changes.
**After Releasing** Generate required .md files necessary for the dev docs
Similar to the Asciidoc files for the admin docs, Markdown files necessary for the dev docs need to be generated. The program for this is written in go and currently does not compare versions but only checks the introduction version. This is sufficient for devs as details necessary for migration documentation is done via the .adoc files in the admin docs.
* In `./docs/helpers` run: `go run . --help`\
This will give you an overview of available commands.
* Because `env_vars.yaml` has been cleaned up as part of the _before release_ tasks above, we can rely on its actuality for the branches to be compared.
* Create delta files for added, removed and deprecated envvars. To do so type:\
`go run . env-var-delta-table` and use as parameter the versions you want to compare. Example: `v5.0.0 v7.0.0`.
* List and check the files created in `./docs/helpers/output/env-deltas/`. The markdown files created contain a table with dev relevant data. Any other files created are not relevant and can safely be deleted.
* Create a branch and move the markdown files from `./docs/helpers/output/env-deltas/` to `./docs/services/general-info/env-var-deltas/`. The markdown files will be consumed by dev docs from this location.
* Commit all changes, create a PR and merge. Dev docs is now up-to-date.
## Backporting
The ocis repo contains branches which are necessary for the documentation. The `docs` branch is related to changes in master, necessary for owncloud.dev and the admin docs referencing master content when it comes to envvars and yaml files.
Cases for a backport can be a typo in an envvar description you want to have fixed in a stable branch too or a file was created after the stable branch was set up but needs to be available in that branch.
When a new stable ocis release (branch) is published, like `stable-5.0`, an additional branch (including CI) is set up manually by the dev team for referencing docs content like `docs-stable-5.0` - related to envvars and yaml files only - and added to the CI.
In case it is necessary to transport a change from master to a stable branch like `docs-stable-5.0`, you must backport the original changes that will create that file to the `stable-5.0` branch. The CI will then take care of creating the results in the target `docs-stable-5.0`.
If the change is expected to have a bigger impact on documenation, you can locally run `make -C docs docs-generate` in the respective branch containing the changes or independently in the `stable-x.y` branch after merging to see if there are additional actions necessary and changed files may need to get checked in.

221
docs/helpers/changed_envvars.py
View File

@@ -1,221 +0,0 @@
import yaml
import sys
import os
from datetime import date
from urllib.request import urlopen
## this python script generates based on defined variables adoc files for added, removed and deprecated
## envvars based on the env_vars.yaml that must exist in each referenced version.
## it is CRUCIAL that the version compared TO is actual - do required updates first!
## note that env_vars.yaml has been introduced with v6.0.0, comparing earlier is not possible
## note that we are always comparing from github sources and NOT local files
## when the files got created, you MUST do some post work manually like referencing services with xref:
## when running, files get recreated, existing content gets overwritten!!
## you MUST run this script from the local ocis repo root !!
## like: python3 docs/helpers/changed_envvars.py
## create a branch to prepare the changes
# CHANGE according your needs
# old is the base version to compare from
# new is the target version to compare to
# tagged versions must be of format: 'tags/v6.0.0'
# master is different, it must be: 'heads/master'
versionOld = 'tags/v7.0.0'
versionNew = 'heads/master'
# CHANGE according your needs
from_version = '7.0.0'
to_version = '7.1.0'
# CHANGE according your needs
# this will create files like 5.0.0-7.0.0-added and 5.0.0-7.0.0-removed
# this should match which versions you compare. master is ok if that is the base for a named release
nameComponent = '7.0.0-7.1.0'
# ADD new elements when a new version has been published so that it gets excluded
# array of version patterns to be excluded for added items. we dont need patch versions
excludePattern = ['pre5.0', '5.0', '6.0', '6.0.0', '6.0.1', '6.1.0', '6.7', '7.0', '7.0.0']
# DO NOT CHANGE
# this is the path the added/removed result is written to
adocWritePath = 'docs/services/general-info/env-var-deltas'
addedWith = {}
removedWith = {}
deprecatedWith = {}
def check_path():
# check which path the script started. we can do this because the target path must be present
# exit if not present
if not os.path.exists(adocWritePath):
print('Path not found: ' + adocWritePath)
sys.exit()
def get_sources(versionOld, versionNew):
# get the sources from github
git_bleft_dir = 'https://raw.githubusercontent.com/owncloud/ocis/refs/'
git_right_dir ='/docs/helpers/env_vars.yaml'
urlOld = git_bleft_dir + versionOld + git_right_dir
urlNew = git_bleft_dir + versionNew + git_right_dir
try:
fileOld = urlopen(urlOld).read().decode('utf-8')
fileNew = urlopen(urlNew).read().decode('utf-8')
return yaml.safe_load(fileOld), yaml.safe_load(fileNew)
except Exception as e:
print(e)
sys.exit()
def get_added(fileNew, excludePattern):
# create dict with added items
addedWith = {}
for key, value in fileNew.items():
if not fileNew[key]['introductionVersion'] in str(excludePattern):
addedWith[key] = value
return addedWith
def get_removed(fileOld, fileNew):
# create dict with removed items
removedWith = {}
for key, value in fileOld.items():
if not key in fileNew:
removedWith[key] = value
return removedWith
def get_deprecated(fileNew):
# create dict with deprecated items
deprecatedWith = {}
for key, value in fileNew.items():
if value['removalVersion']:
deprecatedWith[key] = value
return deprecatedWith
def create_adoc_start(type_text, from_version, to_version, creation_date, columns, closing):
# create the page/table header
# 'closing' contains variable column names dependen if added/removed ir deprecated
a = '''// # {ftype} Variables between oCIS {ffrom} and oCIS {fto}
// commenting the headline to make it better includable
// table created per {fdate}
// the table should be recreated/updated on source () changes
[width="100%",cols="{fcolumns}",options="header"]
|===
| Service | Variable | Description | {fclosing}
'''.format(ftype = type_text, ffrom = from_version, fto = to_version, fdate = creation_date, fcolumns = columns, fclosing = closing)
return a
def create_adoc_end():
# close the table
a = '''|===
'''
return a
def add_adoc_line_1(service, variable, description, value):
# add a table line for added/removed
# the dummy values are only here to have the same number of parameters as add_adoc_line_2
a = '''| {fservice}
| {fvariable}
| {fdescription}
| {fvalue}
'''.format(fservice = service, fvariable = variable, fdescription = description, fvalue = value)
return a
def add_adoc_line_2(service, variable, description, removalVersion, deprecationInfo):
# add a table line for deprecated, this has different columns
a = '''| {fservice}
| {fvariable}
| {fdescription}
| {fremovalVersion}
| {fdeprecationInfo}
'''.format(fservice = service, fvariable = variable, fdescription = description, fremovalVersion = removalVersion, fdeprecationInfo = deprecationInfo)
return a
def create_table(type_text, source_dict, from_version, to_version, date_today, type = False):
# get the table header
columns = '~,~,~,~' if type == False else '~,~,~,~,~'
closing = 'Default' if type == False else 'Removal Version | Deprecation Info'
a = create_adoc_start(type_text, from_version, to_version, date_today, columns, closing)
if not type:
# added and removed envvars
# first add all ocis_
for key, value in source_dict.items():
if key.startswith('OC_'):
a += add_adoc_line_1(
'xref:deployment/services/env-vars-special-scope.adoc[Special Scope Envvars]',
key,
value['description'],
value['defaultValue']
)
# then add all others
for key, value in source_dict.items():
if not key.startswith('OC_'):
a += add_adoc_line_1(
'xref:{s-path}/xxx.adoc[xxx]',
key,
value['description'],
value['defaultValue']
)
else:
# deprecated envvars
# first add all ocis_
for key, value in source_dict.items():
if key.startswith('OC_'):
a += add_adoc_line_2(
'xref:deployment/services/env-vars-special-scope.adoc[Special Scope Envvars]',
key,
value['description'],
value['removalVersion'],
value['deprecationInfo']
)
# then add all others
for key, value in source_dict.items():
if not key.startswith('OC_'):
a += add_adoc_line_2(
'xref:{s-path}/xxx.adoc[xxx]',
key,
value['description'],
value['removalVersion'],
value['deprecationInfo']
)
# finally close the table
a += create_adoc_end()
return a
def write_output(a, type_text):
# write the content to a file
try:
with open(adocWritePath + '/' + nameComponent + '-' + type_text + '.adoc', 'w') as file:
file.write(a)
except Exception as e:
print('Failed creating ' + type_text + ' file')
print(e)
sys.exit()
## here are the tasks in sequence
check_path()
fileOld, fileNew = get_sources(versionOld, versionNew)
addedWith = get_added(fileNew, excludePattern)
removedWith = get_removed(fileOld, fileNew)
deprecatedWith = get_deprecated(fileNew)
a = create_table('Added', addedWith, from_version, to_version, date.today().strftime('%Y.%m.%d'))
r = create_table('Removed', removedWith, from_version, to_version, date.today().strftime('%Y.%m.%d'))
d = create_table('Deprecated', deprecatedWith, from_version, to_version, date.today().strftime('%Y.%m.%d'), True)
write_output(a, 'added')
write_output(r, 'removed')
write_output(d, 'deprecated')
print('Success, see files created in: ' + adocWritePath)

79
docs/helpers/configenvextractor.go
View File

@@ -1,79 +0,0 @@
package main
import (
"fmt"
"log"
"os"
"os/exec"
"path"
"path/filepath"
"strings"
"text/template"
)
var targets = map[string]string{
"templates/adoc-generator.go.tmpl": "output/adoc/adoc-generator.go",
"templates/example-config-generator.go.tmpl": "output/exampleconfig/example-config-generator.go",
"templates/environment-variable-docs-generator.go.tmpl": "output/env/environment-variable-docs-generator.go",
"templates/envar-delta-table.go.tmpl": "output/env/envvar-delta-table.go",
}
// RenderTemplates does something with templates
func RenderTemplates() {
fmt.Println("Getting relevant packages")
paths, err := filepath.Glob("../../services/*/pkg/config/defaults/defaultconfig.go")
if err != nil {
log.Fatal(err)
}
replacer := strings.NewReplacer(
"../../", "github.com/opencloud-eu/opencloud/",
"/defaultconfig.go", "",
)
for i := range paths {
paths[i] = replacer.Replace(paths[i])
}
for template, output := range targets {
generateIntermediateCode(template, output, paths)
runIntermediateCode(output)
}
fmt.Println("Cleaning up")
err = os.RemoveAll("output")
if err != nil {
fmt.Println(err)
}
}
func generateIntermediateCode(templatePath string, intermediateCodePath string, paths []string) {
content, err := os.ReadFile(templatePath)
if err != nil {
log.Fatal(err)
}
fmt.Println("Generating intermediate go code for " + intermediateCodePath + " using template " + templatePath)
tpl := template.Must(template.New("").Parse(string(content)))
err = os.MkdirAll(path.Dir(intermediateCodePath), 0700)
if err != nil {
log.Fatal(err)
}
runner, err := os.Create(intermediateCodePath)
if err != nil {
log.Fatal(err)
}
err = tpl.Execute(runner, paths)
if err != nil {
log.Fatal(err)
}
}
func runIntermediateCode(intermediateCodePath string) {
fmt.Println("Running intermediate go code for " + intermediateCodePath)
defaultConfigPath := "/etc/opencloud"
defaultDataPath := "/var/lib/opencloud"
os.Setenv("OC_BASE_DATA_PATH", defaultDataPath)
os.Setenv("OC_CONFIG_DIR", defaultConfigPath)
out, err := exec.Command("go", "run", intermediateCodePath).CombinedOutput()
if err != nil {
log.Fatal(string(out), err)
}
fmt.Println(string(out))
}

122
docs/helpers/env-var-delta.go
View File

@@ -1,122 +0,0 @@
package main
import (
"fmt"
"github.com/rogpeppe/go-internal/semver"
"gopkg.in/yaml.v2"
"log"
"os"
"path/filepath"
"text/template"
)
const envVarYamlSource = "env_vars.yaml"
var envVarOutPutTemplates = map[string]string{
"added": "templates/env-vars-added.md.tmpl",
"removed": "templates/env-vars-removed.md.tmpl",
"deprecated": "templates/env-vars-deprecated.md.tmpl",
}
// ConfigField represents the env-var annotation in the code
type ConfigField struct {
Name string `yaml:"name"`
DefaultValue string `yaml:"defaultValue"`
Type string `yaml:"type"`
Description string `yaml:"description"`
IntroductionVersion string `yaml:"introductionVersion"`
DeprecationVersion string `yaml:"deprecationVersion"`
RemovalVersion string `yaml:"removalVersion"`
DeprecationInfo string `yaml:"deprecationInfo"`
}
type TemplateData struct {
StartVersion string
EndVersion string
DeltaFields []*ConfigField
}
// RenderEnvVarDeltaTable generates tables for env-var deltas
func RenderEnvVarDeltaTable(osArgs []string) {
if !semver.IsValid(osArgs[2]) {
log.Fatalf("Start version invalid semver: %s", osArgs[2])
}
if !semver.IsValid(osArgs[3]) {
log.Fatalf("Target version invalid semver: %s", osArgs[3])
}
if semver.Compare(osArgs[2], osArgs[3]) >= 0 {
log.Fatalf("Start version %s is not smaller than target version %s", osArgs[2], osArgs[3])
}
if semver.Compare(osArgs[2], "v5.0.0") < 0 {
log.Fatalf("This tool does not support versions prior v5.0.0, (given %s)", osArgs[2])
}
startVersion := osArgs[2]
endVersion := osArgs[3]
fmt.Printf("Generating tables for env-var deltas between version %s and %s...\n", startVersion, endVersion)
curdir, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
fullYamlPath := filepath.Join(curdir, envVarYamlSource)
configFields := make(map[string]*ConfigField)
variableList := map[string][]*ConfigField{
"added": {},
"removed": {},
"deprecated": {},
}
fmt.Printf("Reading existing variable definitions from %s\n", fullYamlPath)
yfile, err := os.ReadFile(fullYamlPath)
if err == nil {
err := yaml.Unmarshal(yfile, configFields)
if err != nil {
log.Fatal(err)
}
}
fmt.Printf("Success, found %d entries\n", len(configFields))
for _, field := range configFields {
if field.IntroductionVersion != "" &&
field.IntroductionVersion != "pre5.0" &&
!semver.IsValid(field.IntroductionVersion) &&
field.IntroductionVersion[0] != 'v' {
field.IntroductionVersion = "v" + field.IntroductionVersion
}
if field.IntroductionVersion != "pre5.0" && !semver.IsValid(field.IntroductionVersion) {
fmt.Printf("Invalid semver for field %s: %s\n", field.Name, field.IntroductionVersion)
os.Exit(1)
}
//fmt.Printf("Processing field %s dv: %s, iv: %s\n", field.Name, field.DeprecationVersion, field.IntroductionVersion)
if semver.IsValid(field.RemovalVersion) && semver.Compare(startVersion, field.RemovalVersion) < 0 && semver.Compare(endVersion, field.RemovalVersion) >= 0 {
variableList["removed"] = append(variableList["removed"], field)
}
if semver.IsValid(field.DeprecationVersion) && semver.Compare(startVersion, field.DeprecationVersion) <= 0 && semver.Compare(endVersion, field.DeprecationVersion) > 0 {
variableList["deprecated"] = append(variableList["deprecated"], field)
}
if semver.IsValid(field.IntroductionVersion) && semver.Compare(startVersion, field.IntroductionVersion) <= 0 && semver.Compare(endVersion, field.IntroductionVersion) >= 0 {
fmt.Printf("Adding field %s iv: %s\n", field.Name, field.IntroductionVersion)
variableList["added"] = append(variableList["added"], field)
}
}
for templateName, templatePath := range envVarOutPutTemplates {
content, err := os.ReadFile(templatePath)
if err != nil {
log.Fatal(err)
}
tpl := template.Must(template.New(templateName).Parse(string(content)))
err = os.MkdirAll("output/env-deltas", 0700)
if err != nil {
log.Fatal(err)
}
targetFile, err := os.Create(filepath.Join("output/env-deltas", fmt.Sprintf("%s-%s-%s.md", startVersion, endVersion, templateName)))
if err != nil {
log.Fatal(err)
}
err = tpl.Execute(targetFile, TemplateData{
StartVersion: startVersion,
EndVersion: endVersion,
DeltaFields: variableList[templateName],
})
if err != nil {
log.Fatal(err)
}
}
}

16614
docs/helpers/env_vars.yaml
View File

File diff suppressed because it is too large Load Diff

178
docs/helpers/extendedEnv.go
View File

@@ -1,178 +0,0 @@
package main
import (
"fmt"
"log"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"text/template"
"gopkg.in/yaml.v2"
)
const yamlSource = "extended_vars.yaml"
// ConfigVars is the main yaml source
type ConfigVars struct {
Variables []Variable `yaml:"variables"`
}
// Variable contains all information about one rogue envvar
type Variable struct {
// These field structs are automatically filled:
// RawName can be the name of the envvar or the name of its var
RawName string `yaml:"rawname"`
// Path to the envvar with linenumber
Path string `yaml:"path"`
// FoundInCode indicates if the variable is still found in the codebase. TODO: delete immediately?
FoundInCode bool `yaml:"foundincode"`
// Name is equal to RawName but will not be overwritten in consecutive runs
Name string `yaml:"name"`
// These field structs need manual filling:
// Type of the envvar
Type string `yaml:"type"`
// DefaultValue of the envvar
DefaultValue string `yaml:"default_value"`
// Description of what this envvar does
Description string `yaml:"description"`
// Ignore this envvar when creating docs?
Ignore bool `yaml:"do_ignore"`
// For simplicity ignored for now:
// DependendServices []Service `yaml:"dependend_services"`
}
// GetRogueEnvs extracts the rogue envs from the code
func GetRogueEnvs() {
curdir, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
fullYamlPath := filepath.Join(curdir, yamlSource)
re := regexp.MustCompile(`os.Getenv\(([^\)]+)\)`)
vars := &ConfigVars{}
fmt.Printf("Reading existing variable definitions from %s\n", fullYamlPath)
yfile, err := os.ReadFile(fullYamlPath)
if err == nil {
err := yaml.Unmarshal(yfile, &vars)
if err != nil {
log.Fatal(err)
}
}
if err := os.Chdir("../../"); err != nil {
log.Fatal(err)
}
fmt.Println("Gathering variable definitions from source")
out, err := exec.Command("sh", "-c", "grep -RHn os.Getenv --exclude-dir=vendor | grep -v extendedEnv.go |grep \\.go").Output()
if err != nil {
log.Fatal(err)
}
lines := strings.Split(string(out), "\n")
// find current vars
currentVars := make(map[string]Variable)
for _, l := range lines {
fmt.Printf("Parsing %s\n", l)
r := strings.SplitN(l, ":\t", 2)
if len(r) != 2 || r[0] == "" || r[1] == "" {
continue
}
res := re.FindAllSubmatch([]byte(r[1]), -1)
if len(res) < 1 {
fmt.Printf("Error envvar not matching pattern: %s", r[1])
continue
}
for _, m := range res {
path := r[0]
name := strings.Trim(string(m[1]), "\"")
currentVars[path+name] = Variable{
RawName: name,
Path: path,
FoundInCode: true,
Name: name,
}
}
}
// adjust existing vars
for i, v := range vars.Variables {
_, ok := currentVars[v.Path+v.RawName]
if !ok {
vars.Variables[i].FoundInCode = false
continue
}
vars.Variables[i].FoundInCode = true
delete(currentVars, v.Path+v.RawName)
}
// add new envvars
for _, v := range currentVars {
vars.Variables = append(vars.Variables, v)
}
less := func(i, j int) bool {
return vars.Variables[i].Name < vars.Variables[j].Name
}
sort.Slice(vars.Variables, less)
output, err := yaml.Marshal(vars)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Writing new variable definitions to %s\n", fullYamlPath)
err = os.WriteFile(fullYamlPath, output, 0666)
if err != nil {
log.Fatalf("could not write %s", fullYamlPath)
}
if err := os.Chdir(curdir); err != nil {
log.Fatal(err)
}
}
// RenderGlobalVarsTemplate renders the global vars template
func RenderGlobalVarsTemplate() {
curdir, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
fullYamlPath := filepath.Join(curdir, yamlSource)
content, err := os.ReadFile("../../docs/templates/ADOC_extended.tmpl")
if err != nil {
log.Fatal(err)
}
targetFolder := "../../docs/services/_includes/adoc/"
vars := &ConfigVars{}
fmt.Printf("Reading existing variable definitions from %s\n", fullYamlPath)
yfile, err := os.ReadFile(fullYamlPath)
if err != nil {
log.Fatal(err)
}
err = yaml.Unmarshal(yfile, &vars)
if err != nil {
log.Fatal(err)
}
targetFile, err := os.Create(filepath.Join(targetFolder, "extended_configvars.adoc"))
if err != nil {
log.Fatalf("Failed to create target file: %s", err)
}
defer targetFile.Close()
tpl := template.Must(template.New("").Parse(string(content)))
if err = tpl.Execute(targetFile, *vars); err != nil {
log.Fatalf("Failed to execute template: %s", err)
}
}

170
docs/helpers/extended_vars.yaml
View File

@@ -1,170 +0,0 @@
variables:
- rawname: CS3_GATEWAY
path: services/idp/pkg/backends/cs3/bootstrap/cs3.go:77
foundincode: true
name: CS3_GATEWAY
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: CS3_MACHINE_AUTH_API_KEY
path: services/idp/pkg/backends/cs3/bootstrap/cs3.go:78
foundincode: true
name: CS3_MACHINE_AUTH_API_KEY
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: DAYS
path: tests/qa-activity-report/generate-qa-activity-report.go:94
foundincode: true
name: DAYS
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: _registryRegisterIntervalEnv
path: ocis-pkg/registry/expiry.go:19
foundincode: true
name: EXPERIMENTAL_REGISTER_INTERVAL
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: _registryRegisterTTLEnv
path: ocis-pkg/registry/expiry.go:28
foundincode: true
name: EXPERIMENTAL_REGISTER_TTL
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: _serverMaxConnectionAgeEnv
path: ocis-pkg/service/grpc/keepalive.go:19
foundincode: true
name: GRPC_MAX_CONNECTION_AGE
type: Duration
default_value: "9223372036854775807"
description: Timeout for GRPC connections. After timeout, a new connection will
be established automatically. The default value is in ns and is about 2.5 mio
h.
do_ignore: false
- rawname: MICRO_LOG_LEVEL
path: ocis-pkg/log/log.go:34
foundincode: true
name: MICRO_LOG_LEVEL
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: MICRO_LOG_LEVEL
path: ocis-pkg/log/log.go:31
foundincode: false
name: MICRO_LOG_LEVEL
type: string
default_value: Error
description: Set the log level for the internal go micro framework. Only change
on supervision of ownCloud Support.
do_ignore: false
- rawname: MICRO_LOG_LEVEL
path: ocis-pkg/log/log.go:30
foundincode: true
name: MICRO_LOG_LEVEL
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: _registryEnv
path: ocis-pkg/registry/registry.go:87
foundincode: true
name: MICRO_REGISTRY
type: string
default_value: nats-js-kv
description: 'The Go micro registry type to use. Supported types are: ''memory''
and ''nats-js-kv'' (default). Only change on supervision of ownCloud Support.'
do_ignore: false
- rawname: _registryAddressEnv
path: ocis-pkg/registry/registry.go:91
foundincode: true
name: MICRO_REGISTRY_ADDRESS
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: _registryAddressEnv
path: ocis-pkg/natsjsregistry/registry.go:192
foundincode: true
name: MICRO_REGISTRY_ADDRESS
type: string
default_value: 127.0.0.1:9233
description: The bind address of the internal go micro framework. Only change on
supervision of ownCloud Support.
do_ignore: false
- rawname: _registryPasswordEnv
path: ocis-pkg/natsjsregistry/registry.go:220
foundincode: true
name: MICRO_REGISTRY_AUTH_PASSWORD
type: string
default_value: ""
description: Optional when using nats to authenticate with the nats cluster.
do_ignore: false
- rawname: _registryUsernameEnv
path: ocis-pkg/natsjsregistry/registry.go:220
foundincode: true
name: MICRO_REGISTRY_AUTH_USERNAME
type: string
default_value: ""
description: Optional when using nats to authenticate with the nats cluster.
do_ignore: false
- rawname: MONTH
path: tests/qa-activity-report/generate-qa-activity-report.go:92
foundincode: true
name: MONTH
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: OC_BASE_DATA_PATH
path: ocis-pkg/config/defaults/paths.go:23
foundincode: true
name: OC_BASE_DATA_PATH
type: string
default_value: ""
description: The base directory location used by several services and for user data.
See the General Info section in the documentation for more details on defaults.
Services can have, if available, an individual setting with an own environment
variable.
do_ignore: false
- rawname: OC_CONFIG_DIR
path: ocis-pkg/config/defaults/paths.go:56
foundincode: true
name: OC_CONFIG_DIR
type: string
default_value: ""
description: The default directory location for config files. See the General Info
section in the documentation for more details on defaults.
do_ignore: false
- rawname: RUN_CMD_TEST
path: internal/testenv/test.go:32
foundincode: true
name: RUN_CMD_TEST
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: YEAR
path: tests/qa-activity-report/generate-qa-activity-report.go:93
foundincode: true
name: YEAR
type: ""
default_value: ""
description: ""
do_ignore: true
- rawname: parts[0]
path: ocis-pkg/config/envdecode/envdecode.go:382
foundincode: true
name: parts[0]
type: ""
default_value: ""
description: false positive - code that extract envvars for config structs
do_ignore: true

45
docs/helpers/main.go
View File

@@ -1,45 +0,0 @@
package main
import (
"fmt"
"os"
)
func main() {
if len(os.Args) > 1 {
switch os.Args[1] {
case "templates":
RenderTemplates()
case "rogue":
GetRogueEnvs()
case "globals":
RenderGlobalVarsTemplate()
case "service-index":
GenerateServiceIndexMarkdowns()
case "env-var-delta-table":
// This step is not covered by the all or default case, because it needs explicit arguments
if len(os.Args) != 4 {
fmt.Println("Needs two arguments: env-var-delta-table <first-version> <second-version>")
fmt.Println("Example: env-var-delta-table v5.0.0 v6.0.0")
fmt.Println("Will not generate usable results for versions Prior to v5.0.0")
} else {
RenderEnvVarDeltaTable(os.Args)
}
case "all":
RenderTemplates()
GetRogueEnvs()
RenderGlobalVarsTemplate()
GenerateServiceIndexMarkdowns()
case "help":
fallthrough
default:
fmt.Printf("Usage: %s [templates|rogue|globals|service-index|env-var-delta-table|all|help]\n", os.Args[0])
}
} else {
// Left here, even though present in the switch case, for backwards compatibility
RenderTemplates()
GetRogueEnvs()
RenderGlobalVarsTemplate()
GenerateServiceIndexMarkdowns()
}
}

76
docs/helpers/markdowncreation.go
View File

@@ -1,76 +0,0 @@
package main
import (
"bytes"
"errors"
"fmt"
"log"
"os"
"path/filepath"
"text/template"
"time"
"github.com/opencloud-eu/opencloud/pkg/markdown"
)
var _configMarkdown = `{{< include file="services/_includes/%s-config-example.yaml" language="yaml" >}}
{{< include file="services/_includes/%s_configvars.md" >}}
`
// GenerateServiceIndexMarkdowns generates the _index.md files for the dev docu
func GenerateServiceIndexMarkdowns() {
paths, err := filepath.Glob("../../services/*/README.md")
if err != nil {
log.Fatal(err)
}
for _, p := range paths {
service := filepath.Base(filepath.Dir(p))
if err := generateMarkdown(p, service); err != nil {
fmt.Printf("error generating markdown for %s: %s\n", service, err)
}
}
}
func generateMarkdown(filepath string, servicename string) error {
f, err := os.ReadFile(filepath)
if err != nil {
return err
}
md := markdown.NewMD(f)
if len(md.Headings) == 0 || md.Headings[0].Level != 1 {
return errors.New("readme has invalid format")
}
// we don't need the main title, we add in our template
head := md.Headings[0]
md.Headings = md.Headings[1:]
md.Headings = append(md.Headings, markdown.Heading{
Level: 2,
Header: "Example Yaml Config",
Content: fmt.Sprintf(_configMarkdown, servicename, servicename),
})
tpl := template.Must(template.ParseFiles("templates/index.tmpl"))
b := bytes.NewBuffer(nil)
if err := tpl.Execute(b, map[string]interface{}{
"ServiceName": head.Header,
"CreationTime": time.Now().Format(time.RFC3339Nano),
"service": servicename,
"Abstract": head.Content,
"TocTree": md.TocString(),
"Content": md.String(),
}); err != nil {
return err
}
path := fmt.Sprintf("../../docs/services/%s", servicename)
if err := os.Mkdir(path, os.ModePerm); err != nil && !os.IsExist(err) {
return err
}
return os.WriteFile(path+"/_index.md", b.Bytes(), os.ModePerm)
}

272
docs/helpers/templates/adoc-generator.go.tmpl
View File

@@ -1,272 +0,0 @@
package main
import (
"fmt"
"log"
"os"
"path/filepath"
"reflect"
"regexp"
"strings"
"text/template"
"time"
"sort"
{{- range $key, $value := .}}
pkg{{$key}} "{{$value}}"
{{- end}}
)
// ConfigField is the representation of one configuration field
type ConfigField struct {
EnvVars []string
DefaultValue string
IntroductionVersion string
Type string
Description string
VersionInfo string
DeprecationLink string
}
// DeprecationField holds information about deprecation
type DeprecationField struct {
DeprecationVersion string
DeprecationInfo string
DeprecationReplacement string
RemovalVersion string
}
// EnvVar holds information about one envvar
type EnvVar struct {
Name string
IntroductionVersion string
DefaultValue string
Type string
Description string
Services []string
}
type templateData struct {
ExtensionName string
Fields []ConfigField
Deprecations []DeprecationField
HasDeprecations bool
Timestamp string
}
func main() {
fmt.Println("Generating adoc documentation for environment variables:")
adoc, err := os.ReadFile("../../docs/templates/ADOC.tmpl")
if err != nil {
log.Fatal(err)
}
dpr, err := os.ReadFile("../../docs/templates/ADOC_deprecation.tmpl")
if err != nil {
log.Fatal(err)
}
replacer := strings.NewReplacer(
"github.com/owncloud/ocis/v2/services/", "",
"/pkg/config/defaults", "",
)
var (
fields []ConfigField
deprecations []DeprecationField
adocFile *os.File
dprFile *os.File
)
adoctpl := template.Must(template.New("").Parse(string(adoc)))
dprtpl := template.Must(template.New("").Parse(string(dpr)))
m := map[string]interface{}{
{{- range $key, $value := .}}
"{{$value}}": *pkg{{$key}}.FullDefaultConfig(),
{{- end }}
}
targetFolder := "../../docs/services/_includes/adoc/"
all := make(map[string]EnvVar)
for pkg, conf := range m {
service := replacer.Replace(pkg)
timestamp := time.Now().Format("2006-01-02-15-04-05")
fields, deprecations = GetAnnotatedVariables(conf, timestamp)
var hasDeprecations bool
if len(deprecations) > 0 {
hasDeprecations = true
}
for _, f := range fields {
for _, e := range f.EnvVars {
if env, ok := all[e]; ok {
env.Services = append(env.Services, service)
sort.Slice(env.Services, func(i, j int) bool {
return env.Services[i] < env.Services[j]
})
all[e] = env
} else {
all[e] = EnvVar{
Name: e,
IntroductionVersion: replaceEnvVarPlaceHolder(f.IntroductionVersion),
Description: f.Description,
Type: f.Type,
DefaultValue: f.DefaultValue,
Services: []string{service},
}
}
}
}
if len(fields) > 0 || len(deprecations) > 0 {
fmt.Printf("... %s\n", pkg)
td := templateData{
ExtensionName: service,
Fields: fields,
Deprecations: deprecations,
HasDeprecations: hasDeprecations,
Timestamp: timestamp,
}
adocFile, err = os.Create(filepath.Join(targetFolder, service + "_configvars.adoc"))
if err != nil {
log.Fatalf("Failed to create target file: %s", err)
}
defer adocFile.Close()
if err := adoctpl.Execute(adocFile, td); err != nil {
log.Fatalf("Failed to execute template: %s", err)
}
dprFile, err = os.Create(filepath.Join(targetFolder, service + "_deprecation.adoc"))
if err != nil {
log.Fatalf("Failed to create target file: %s", err)
}
defer dprFile.Close()
if err := dprtpl.Execute(dprFile, td); err != nil {
log.Fatalf("Failed to execute template: %s", err)
}
}
}
// render global env vars
tmplValues := make([]map[string]interface{}, 0)
for _, env := range all {
if len(env.Services) > 1 {
tmplValues = append(tmplValues, map[string]interface{}{
"Name": env.Name,
"IntroductionVersion": replaceEnvVarPlaceHolder(env.IntroductionVersion),
"Services": env.Services,
"Description": env.Description,
"DefaultValue": env.DefaultValue,
"Type": env.Type,
})
}
}
// sort
sort.Slice(tmplValues, func(i, j int) bool {
return tmplValues[i]["Name"].(string) < tmplValues[j]["Name"].(string)
})
glc, err := os.ReadFile("../../docs/templates/ADOC_global.tmpl")
if err != nil {
log.Fatal(err)
}
gltpl := template.Must(template.New("").Parse(string(glc)))
glfile, err := os.Create(filepath.Join(targetFolder, "global_configvars.adoc"))
if err != nil {
log.Fatalf("Failed to create target file: %s", err)
}
if err := gltpl.Execute(glfile, tmplValues); err != nil {
log.Printf("Failed to execute template: %s", err)
}
fmt.Println("done")
}
func replaceEnvVarPlaceHolder(s string) string {
return strings.Replace(
strings.Replace(s, "%%NEXT%%", "next", -1),
"%%NEXT_PRODUCTION_VERSION%%",
"next-prod",
-1,
)
}
func GetAnnotatedVariables(s interface{}, timestamp string) ([]ConfigField, []DeprecationField) {
t := reflect.TypeOf(s)
v := reflect.ValueOf(s)
var fields []ConfigField
var deprecations []DeprecationField
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
value := v.Field(i)
switch value.Kind() {
default:
desc := field.Tag.Get("desc")
env, ok := field.Tag.Lookup("env")
deprecationLink := ""
if !ok {
continue
}
introductionVersion, _ := field.Tag.Lookup("introductionVersion")
deprecationVersion, _ := field.Tag.Lookup("deprecationVersion")
removalVersion, _ := field.Tag.Lookup("removalVersion")
deprecationInfo, _ := field.Tag.Lookup("deprecationInfo")
deprecationReplacement, _ := field.Tag.Lookup("deprecationReplacement")
if deprecationVersion != "" ||
removalVersion != "" ||
deprecationInfo != "" ||
deprecationReplacement != "" {
deprecationLink = fmt.Sprintf("xref:deprecation-note-%s[Deprecation Note]", timestamp)
}
v := fmt.Sprintf("%v", value.Interface())
td := strings.Split(env, ";")
// re := regexp.MustCompile(`^(https?:\/\/)`)
// v = re.ReplaceAllString(v,"\\$1")
re := regexp.MustCompile(`(https?:\/\/)`)
desc = re.ReplaceAllString(desc, "\\$1")
re = regexp.MustCompile(`(\|)`)
v = re.ReplaceAllString(v, "\\$1")
typeName := value.Type().Name()
if typeName == "" {
typeName = value.Type().String()
}
fields = append(fields,
ConfigField{
EnvVars: td,
IntroductionVersion: replaceEnvVarPlaceHolder(introductionVersion),
DefaultValue: v,
Description: desc,
Type: typeName,
DeprecationLink: deprecationLink,
})
if deprecationLink != "" {
deprecations = append(deprecations,
DeprecationField{
DeprecationVersion: replaceEnvVarPlaceHolder(deprecationVersion),
DeprecationInfo: deprecationInfo,
DeprecationReplacement: deprecationReplacement,
RemovalVersion: replaceEnvVarPlaceHolder(removalVersion),
})
}
case reflect.Ptr:
// PolicySelectors in the Proxy are being skipped atm
// they are not configurable via env vars, if that changes
// they are probably added to the Sanitize() function
// and this should not be an issue then
if !value.IsZero() && value.Elem().CanInterface() {
f, d := GetAnnotatedVariables(value.Elem().Interface(), timestamp)
fields = append(fields, f...)
deprecations = append(deprecations, d...)
}
case reflect.Struct:
f, d := GetAnnotatedVariables(value.Interface(), timestamp)
fields = append(fields, f...)
deprecations = append(deprecations, d...)
}
}
return fields, deprecations
}

7
docs/helpers/templates/env-vars-added.md.tmpl
View File

@@ -1,7 +0,0 @@
Added between Version {{ .StartVersion }} and {{ .EndVersion }}.
| Variable | Description |
| --- | --- |
{{- range $key, $value := .DeltaFields}}
| {{$value.Name}} | {{$value.Description}} |
{{- end}}

7
docs/helpers/templates/env-vars-deprecated.md.tmpl
View File

@@ -1,7 +0,0 @@
Deprecated between Version {{ .StartVersion }} and {{ .EndVersion }}.
| Variable | Description | Deprecation Info |
| --- | --- |
{{- range $key, $value := .DeltaFields}}
| {{$value.Name}} | {{$value.Description}} | {{$value.DeprecationInfo}} |
{{- end}}

7
docs/helpers/templates/env-vars-removed.md.tmpl
View File

@@ -1,7 +0,0 @@
Removed between Version {{ .StartVersion }} and {{ .EndVersion }}.
| Variable | Description | Deprecation Info |
| --- | --- |
{{- range $key, $value := .DeltaFields}}
| {{$value.Name}} | {{$value.Description}} | {{$value.DeprecationInfo}} |
{{- end}}

154
docs/helpers/templates/envar-delta-table.go.tmpl
View File

@@ -1,154 +0,0 @@
package main
import (
"fmt"
"gopkg.in/yaml.v2"
"html"
"reflect"
"strings"
"log"
"os"
"path/filepath"
{{- range $key, $value :=.}}
pkg{{$key}} "{{$value}}"
{{- end}}
)
const yamlSource = "env_vars.yaml"
type ConfigField struct {
Name string `yaml:"name"`
DefaultValue string `yaml:"defaultValue"`
Type string `yaml:"type"`
Description string `yaml:"description"`
IntroductionVersion string `yaml:"introductionVersion"`
DeprecationVersion string `yaml:"deprecationVersion"`
RemovalVersion string `yaml:"removalVersion"`
DeprecationInfo string `yaml:"deprecationInfo"`
}
func main() {
fmt.Println("Generating tables for env-var deltas...")
curdir, err := os.Getwd()
if err != nil {
log.Fatal(err)
}
fullYamlPath := filepath.Join(curdir, yamlSource)
var fields []ConfigField
configFields := make(map[string]*ConfigField)
fmt.Printf("Reading existing variable definitions from %s\n", fullYamlPath)
yfile, err := os.ReadFile(fullYamlPath)
if err == nil {
err := yaml.Unmarshal(yfile, configFields)
if err != nil {
log.Fatal(err)
}
}
m := map[string]interface{}{
{{- range $key, $value := .}}
"{{$value}}": *pkg{{$key}}.FullDefaultConfig(),
{{- end }}
}
for _, conf := range m {
fields = GetAnnotatedVariables(conf)
for _, field := range fields {
variants := strings.Split(field.Name, ";")
for _, variant := range variants {
if (configFields[variant] != nil && configFields[variant].Name == "") || configFields[variant] == nil {
configFields[variant] = &field
} else {
fmt.Printf("%v, duplicate key, merging\n", variant)
if strings.TrimSpace(configFields[variant].DefaultValue) != "" && configFields[variant].DefaultValue != field.DefaultValue {
configFields[variant].DefaultValue = field.DefaultValue
}
if strings.TrimSpace(configFields[variant].Description) != "" && configFields[variant].Description != field.Description {
configFields[variant].Description = field.Description
}
if strings.TrimSpace(configFields[variant].Type) != "" && configFields[variant].Type != field.Type {
configFields[variant].Type = field.Type
}
if strings.TrimSpace(configFields[variant].IntroductionVersion) != "" && configFields[variant].IntroductionVersion != field.IntroductionVersion {
configFields[variant].IntroductionVersion = field.IntroductionVersion
}
if strings.TrimSpace(configFields[variant].DeprecationVersion) != "" && configFields[variant].DeprecationVersion != field.DeprecationVersion {
configFields[variant].DeprecationVersion = field.DeprecationVersion
}
if strings.TrimSpace(configFields[variant].RemovalVersion) != "" && configFields[variant].RemovalVersion != field.RemovalVersion {
configFields[variant].RemovalVersion = field.RemovalVersion
}
if strings.TrimSpace(configFields[variant].Name) != "" && configFields[variant].Name != field.Name {
configFields[variant].Name = field.Name
}
if strings.TrimSpace(configFields[variant].DeprecationInfo) != "" && configFields[variant].DeprecationInfo != field.DeprecationInfo {
// there might be multiple superseeding DeprecationInformations, we might want to keep track of those, that's why we are not overwriting the field
configFields[variant].DeprecationInfo = configFields[variant].DeprecationInfo + " | " + field.DeprecationInfo
}
}
}
}
}
output, err := yaml.Marshal(configFields)
if err != nil {
log.Fatalf("Could not marshall variables: %v", err)
}
err = os.WriteFile(fullYamlPath, output, 0666)
if err != nil {
log.Fatalf("could not write %s", fullYamlPath)
}
if err := os.Chdir(curdir); err != nil {
log.Fatal(err)
}
}
func GetAnnotatedVariables(s interface{}) []ConfigField {
t := reflect.TypeOf(s)
v := reflect.ValueOf(s)
var fields []ConfigField
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
value := v.Field(i)
switch value.Kind() {
default:
desc := field.Tag.Get("desc")
env, ok := field.Tag.Lookup("env")
if !ok {
continue
}
introductionVersion := field.Tag.Get("introductionVersion")
deprecationVersion := field.Tag.Get("deprecationVersion")
removalVersion := field.Tag.Get("removalVersion")
deprecationInfo := field.Tag.Get("deprecationInfo")
v := fmt.Sprintf("%v", value.Interface())
typeName := value.Type().Name()
if typeName == "" {
typeName = value.Type().String()
}
//fields = append(fields, ConfigField{Name: strings.ReplaceAll(env, ";", "<br/>"), DefaultValue: html.EscapeString(strings.Replace(v, "|", "\\|", -1)), Description: desc, Type: typeName})
fields = append(fields, ConfigField{
Name: env,
DefaultValue: html.EscapeString(strings.Replace(v, "|", "\\|", -1)),
Description: desc,
Type: typeName,
IntroductionVersion: introductionVersion,
DeprecationVersion: deprecationVersion,
RemovalVersion: removalVersion,
DeprecationInfo: deprecationInfo,
})
case reflect.Ptr:
// PolicySelectors in the Proxy are being skipped atm
// they are not configurable via env vars, if that changes
// they are probably added to the Sanitize() function
// and this should not be an issue then
if !value.IsZero() && value.Elem().CanInterface() {
fields = append(fields, GetAnnotatedVariables(value.Elem().Interface())...)
}
case reflect.Struct:
fields = append(fields, GetAnnotatedVariables(value.Interface())...)
}
}
return fields
}

97
docs/helpers/templates/environment-variable-docs-generator.go.tmpl
View File

@@ -1,97 +0,0 @@
package main
import (
"fmt"
"html"
"log"
"os"
"path/filepath"
"reflect"
"strings"
"text/template"
{{- range $key, $value := .}}
pkg{{$key}} "{{$value}}"
{{- end}})
type ConfigField struct {
Name string
DefaultValue string
Type string
Description string
}
func main() {
fmt.Println("Generating documentation for environment variables:")
content, err := os.ReadFile("../../docs/templates/CONFIGURATION.tmpl")
if err != nil {
log.Fatal(err)
}
replacer := strings.NewReplacer(
"github.com/owncloud/ocis/v2/services/", "",
"/pkg/config/defaults", "",
)
var fields []ConfigField
var targetFile *os.File
tpl := template.Must(template.New("").Parse(string(content)))
m := map[string]interface{}{
{{- range $key, $value := .}}
"{{$value}}": *pkg{{$key}}.FullDefaultConfig(),
{{- end }}
}
targetFolder := "../../docs/services/_includes/"
for pkg, conf := range m {
fields = GetAnnotatedVariables(conf)
if len(fields) > 0 {
fmt.Printf("... %s\n", pkg)
targetFile, err = os.Create(filepath.Join(targetFolder, replacer.Replace(pkg) + "_configvars.md"))
if err != nil {
log.Fatalf("Failed to create target file: %s", err)
}
defer targetFile.Close()
if err := tpl.Execute(targetFile, fields); err != nil {
log.Fatalf("Failed to execute template: %s", err)
}
}
}
fmt.Println("done")
}
func GetAnnotatedVariables(s interface{}) []ConfigField {
t := reflect.TypeOf(s)
v := reflect.ValueOf(s)
var fields []ConfigField
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
value := v.Field(i)
switch value.Kind() {
default:
desc := field.Tag.Get("desc")
env, ok := field.Tag.Lookup("env")
if !ok {
continue
}
v := fmt.Sprintf("%v", value.Interface())
typeName := value.Type().Name()
if typeName == "" {
typeName = value.Type().String()
}
fields = append(fields, ConfigField{Name: strings.ReplaceAll(env, ";", "<br/>"), DefaultValue: html.EscapeString(strings.Replace(v, "|", "\\|", -1)), Description: desc, Type: typeName})
case reflect.Ptr:
// PolicySelectors in the Proxy are being skipped atm
// they are not configurable via env vars, if that changes
// they are probably added to the Sanitize() function
// and this should not be an issue then
if !value.IsZero() && value.Elem().CanInterface() {
fields = append(fields, GetAnnotatedVariables(value.Elem().Interface())...)
}
case reflect.Struct:
fields = append(fields, GetAnnotatedVariables(value.Interface())...)
}
}
return fields
}

50
docs/helpers/templates/example-config-generator.go.tmpl
View File

@@ -1,50 +0,0 @@
package main
import (
"fmt"
"log"
"os"
"path/filepath"
"strings"
"gopkg.in/yaml.v2"
{{- range $key, $value := .}}
pkg{{$key}} "{{$value}}"
{{- end}}
)
func main() {
replacer := strings.NewReplacer(
"github.com/owncloud/ocis/v2/services/", "",
"/pkg/config/defaults", "",
)
cfg := map[string]string{
{{- range $key, $value := .}}
replacer.Replace("{{$value}}"): func() string {
fmt.Println("Generating example YAML config for {{ $value -}}")
c := pkg{{$key}}.FullDefaultConfig()
yml, err := yaml.Marshal(c)
if err != nil {
log.Fatalf("Marshalling yaml for pkg0 failed: %s\n", err)
}
return fmt.Sprintf("# Autogenerated\n# Filename: %s-config-example.yaml\n\n%s", replacer.Replace("{{ $value }}"),string(yml))
}(),
{{- end}}
}
for pkg, yml := range cfg {
targetFolders := []string{
// TODO: comment in when it is clear how to commit this to the structure of the master|main branch
// filepath.Join("../../", pkg, "/config"),
"../../docs/services/_includes/",
}
for _, targetFolder := range targetFolders {
os.MkdirAll(targetFolder, 0700)
targetYamlFile, err := os.Create(filepath.Join(targetFolder, replacer.Replace(pkg) + "-config-example.yaml"))
if err != nil {
log.Fatalf("Failed to create target file for : %s", err)
}
defer targetYamlFile.Close()
targetYamlFile.WriteString(yml)
}
}
}

19
docs/helpers/templates/index.tmpl
View File

@@ -1,19 +0,0 @@
---
title: {{ .ServiceName }}
date: {{ .CreationTime }}
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/services/{{ .service }}
geekdocFilePath: README.md
geekdocCollapseSection: true
---
<!-- Do not edit this file, it is autogenerated. Edit the service README.md instead -->
## Abstract
{{ .Abstract }}
## Table of Contents
{{ .TocTree }}
{{ .Content }}

14
docs/mutagen.yml
View File

@@ -1,14 +0,0 @@
sync:
docs:
mode: one-way-replica
alpha: .
beta: ./hugo/content
flushOnCreate: true
ignore:
paths:
- .gitignore
- mutagen.yml
- mutagen.yml.lock
- hugo
- README.md
- Makefile

1
docs/ocis/.gitignore vendored
View File

@@ -1 +0,0 @@
configuration.md

67
docs/ocis/_index.md
View File

@@ -1,67 +0,0 @@
---
title: "oCIS - ownCloud Infinite Scale"
date: 2020-02-27T20:35:00+01:00
weight: -10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
{{< figure class="floatright" src="/media/is.png" width="70%" height="auto" >}}
## ownCloud Infinite Scale
Welcome to oCIS, the modern file-sync and share platform, which is based on our knowledge and experience with the PHP based [ownCloud server](https://owncloud.com/#server).
### The idea of federated storage
To create a truly federated storage architecture oCIS breaks down the old ownCloud 10 user specific namespace, which is assembled on the server side, and makes the individual parts accessible to clients as storage spaces and storage space registries.
The diagram below shows the core concepts that are the foundation for the new architecture:
- End user devices can fetch the list of *storage spaces* a user has access to, by querying one or multiple *storage space registries*. The list contains a unique endpoint for every *storage space*.
- [*Storage space registries*]({{< ref "./storage/terminology#storage-space-registries" >}}) manage the list of storage spaces a user has access to. They may subscribe to *storage spaces* in order to receive notifications about changes on behalf of an end users mobile or desktop client.
- [*Storage spaces*]({{< ref "./storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are contained in a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type.
- [*Storage providers*]({{< ref "./storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, either to shard the load, provide different levels of redundancy or support custom workflows. Or there might be just one, hosting all types of *storage spaces*.
{{< figure src="/ocis/static/idea.drawio.svg" >}}
As an example, Einstein might want to share something with Marie, who has an account at a different identity provider and uses a different storage space registry. The process makes use of [OpenID Connect (OIDC)](https://openid.net/specs/openid-connect-core-1_0.html) for authentication and would look something like this:
To share something with Marie, Einstein would open `https://cloud.zurich.test`. His browser loads oCIS web and presents a login form that uses the [OpenID Connect Discovery](https://openid.net/specs/openid-connect-discovery-1_0.html#EmailSyntax) to look up the OIDC issuer. For `einstein@zurich.test` he will end up at `https://idp.zurich.test`, authenticate and get redirected back to `https://cloud.zurich.test`. Now, oCIS web will use a similar discovery to look up the *storage space registry* for the account, based on the email (or username). He will discover that `https://cloud.zurich.test` is also his *storage registry* that the web UI will use to load the list of *storage spaces* that are available to him.
After locating a folder that he wants to share with Marie he enters her email `marie@paris.test` in the sharing dialog to grant her the editor role. This, in effect, creates a new *storage space* that is registered with the *storage space registry* at `https://cloud.zurich.test`.
Einstein copies the URL in the browser (or an email with the same URL is sent automatically, or the storage registries use a back-channel mechanism). It contains the most specific `storage space id` and a path relative to it: `https://cloud.zurich.test/#/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path`.
When Marie enters that URL she will be presented with a login form on the `https://cloud.zurich.test` instance, because the share was created on that domain. If `https://cloud.zurich.test` trusts her OpenID Connect identity provider `https://idp.paris.test` she can log in. This time, the *storage space registry* discovery will come up with `https://cloud.paris.test` though. Since that registry is different than the registry tied to `https://cloud.zurich.test` oCIS web can look up the *storage space* `716199a6-00c0-4fec-93d2-7e00150b1c84` and register the WebDAV URL `https://cloud.zurich.test/dav/spaces/716199a6-00c0-4fec-93d2-7e00150b1c84/a/rel/path` in Marie's *storage space registry* at `https://cloud.paris.test`. When she accepts that share her clients will be able to sync the new *storage space* at `https://cloud.zurich.test`.
### oCIS microservice runtime
The oCIS runtime allows us to dynamically manage services running in a single process. We use [suture](https://github.com/thejerf/suture) to create a supervisor tree that starts each service in a dedicated goroutine. By default oCIS will start all built-in oCIS services in a single process. Individual services can be moved to other nodes to scale-out and meet specific performance requirements. A [go-micro](https://github.com/asim/go-micro/blob/master/registry/registry.go) based registry allows services in multiple nodes to form a distributed microservice architecture.
### oCIS Services
Every oCIS service uses [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg), which implements the [go-micro](https://go-micro.dev/) interfaces for [servers](https://github.com/asim/go-micro/blob/v3.5.0/server/server.go#L17-L37) to register and [clients](https://github.com/asim/go-micro/blob/v3.5.0/client/client.go#L11-L23) to lookup nodes with a service [registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/registry.go).
We are following the [12 Factor](https://12factor.net/) methodology with oCIS. The uniformity of services also allows us to use the same command, logging and configuration mechanism. Configurations are forwarded from the
oCIS runtime to the individual extensions.
### go-micro
While the [go-micro](https://go-micro.dev/) framework provides abstractions as well as implementations for the different components in a microservice architecture, it uses a more developer focused runtime philosophy: It is used to download services from a repo, compile them on the fly and start them as individual processes. For oCIS we decided to use a more admin friendly runtime: You can download a single binary and start the contained oCIS services with a single `bin/ocis server`. This also makes packaging easier.
We use [ocis-pkg](https://github.com/owncloud/ocis/tree/master/ocis-pkg) to configure the default implementations for the go-micro [grpc server](https://github.com/asim/go-micro/tree/v3.5.0/plugins/server/grpc), [client](https://github.com/asim/go-micro/tree/v3.5.0/plugins/client/grpc) and [mdns registry](https://github.com/asim/go-micro/blob/v3.5.0/registry/mdns_registry.go), swapping them out as needed, e.g. to use the [kubernetes registry plugin](https://github.com/asim/go-micro/tree/v3.5.0/plugins/registry/kubernetes).
### REVA
A lot of embedded services in oCIS are built upon the [REVA](https://reva.link/) runtime. We decided to bundle some of the [CS3 services](https://github.com/cs3org/cs3apis) to logically group them. A [home storage provider](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L93-L108), which is dealing with [metadata](https://cs3org.github.io/cs3apis/#cs3.storage.provider.v1beta1.ProviderAPI), and the corresponding [data provider](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/storagehome.go#L109-L123), which is dealing with [up and download](https://cs3org.github.io/cs3apis/#cs3.gateway.v1beta1.FileUploadProtocol), are one example. The [frontend](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go) with the [oc flavoured webdav](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L132-L138), [ocs handlers](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L139-L148) and a [data-gateway](https://github.com/owncloud/ocis/blob/v1.2.0/storage/pkg/command/frontend.go#L126-L131) are another.
### Protocol driven development
Interacting with oCIS involves a multitude af APIs. The server and all clients rely on [OpenID Connect](https://openid.net/connect/) for authentication. The [embedded LibreGraph Connect](https://owncloud.dev/services/idp/) can be replaced with any other OpenID Connect Identity Provider. Clients use the [WebDAV](http://webdav.org/) based [oc sync protocol](https://github.com/cernbox/smashbox/blob/master/protocol/protocol.md) to manage files and folders, [ocs to manage shares](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html) and [TUS](https://tus.io/protocols/resumable-upload.html) to upload files in a resumable way. On the server side [REVA](https://reva.link/) is the reference implementation of the [CS3 apis](https://github.com/cs3org/cs3apis) which is defined using [protobuf](https://developers.google.com/protocol-buffers/). By embedding [libregraph/idm](https://github.com/libregraph/idm), oCIS provides a [LDAP](https://tools.ietf.org/html/rfc2849) interface to make accounts, including guests available to firewalls and other systems. In the future, we are looking into [the Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/overview?view=graph-rest-1.0), which is based on [odata](http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part1-protocol.html), as a well-defined REST/JSON dialect for the existing endpoints.
### Acceptance test suite
We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), which originated in ownCloud 10 and continues to grow. A detailed description can be found in the developer docs for [testing]({{< ref "development/testing" >}}).
### Architecture Overview
Running `bin/ocis server` will start the following services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed.
{{< figure src="/ocis/static/architecture-overview.drawio.svg" >}}

50
docs/ocis/adr/0001-introduce-accounts-service.md
View File

@@ -1,50 +0,0 @@
---
title: "1. Introduce an accounts service"
weight: 1
date: 2020-06-15T20:21:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0001-introduce-accounts-service.md
---
* Status: superseded by [ADR-0003]({{< ref "0003-external-user-management.md" >}})
* Deciders: [@butonic](https://github.com/butonic), [@felixboehm](https://github.com/felixboehm), [@micbar](https://github.com/micbar), [@pmaier1](https://github.com/pmaier1)
* Date: [2020-06-15](https://github.com/owncloud/ocis-accounts/pull/34/commits/2fd05e2b6fe2a47c687bd0c0bc5e1b5c48a585b2)
Technical Story: [persist accounts](https://github.com/owncloud/ocis-accounts/pull/34)
## Context and Problem Statement
To attach metadata like shares to users ownCloud relies on persistent, non-reassignable, unique identifiers for users (and files). Email und username can change when a user changes his name. But even the OIDC sub+iss combination may change when the IdP changes. While there is [an account porting protocol](https://openid.net/specs/openid-connect-account-porting-1_0.html) that describes how a relying party (RP) such as ownCloud should behave, it still requires the RP to maintain its own user identifiers.
## Decision Drivers
* oCIS should be a single binary that can run out of the box without external dependencies like an LDAP server.
* Time: we want to build a release candidate asap.
* Firewalls need access to guests, typically via LDAP.
* Not all external LDAPs are writeable for us to provision Guest accounts.
* We see multiple LDAP servers in deployments. Being able to handle them is important and should be covered by using OIDC + being able to query multiple LDAP servers.
## Considered Options
* Accounts service wraps LDAP
* [GLAuth](https://github.com/glauth/glauth) wraps accounts service
## Decision Outcome
Chosen option: "GLAuth wraps accounts service", because we need write access to provision guest accounts and GLAuth currently has no write support.
### Positive Consequences
* We can build a self-contained user management in the accounts service and can adjust it to our requirements.
* We do not rely on an LDAP server which would only be possible by implementing write support in the LDAP libraries used by GLAuth (hard to estimate effort, when will that be merged upstream).
### Negative Consequences
* We need to spend time on implementing user management
## Pros and Cons of the Options
### Accounts service wraps LDAP
* Bad, because not all external LDAPs are writeable for us to provision Guest accounts.

36
docs/ocis/adr/0002-persist-accounts-using-cs3-storage.md
View File

@@ -1,36 +0,0 @@
---
title: "2. Persist accounts in a CS3 storage"
weight: 2
date: 2020-08-21T20:21:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0002-persist-accounts-using-cs3-storage.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@felixboehm](https://github.com/felixboehm)
* Date: 2020-08-21
Technical Story: [File system based indexing](https://github.com/owncloud/ocis-accounts/pull/92)
## Context and Problem Statement
To set up High Availability (HA) or a geo-replicated setup we need to persist accounts in a distributed way. To efficiently query the accounts by email or username, and not only by id, they need to be indexed. Unfortunately, the [bleve](https://github.com/blevesearch/bleve) index we currently store locally on disk cannot be shared by multiple instances, preventing a scale out deployment.
## Considered Options
* Look into distributed bleve
* Persist users in a CS3 storage
## Decision Outcome
Chosen option: "Persist users in a CS3 storage", because we have one service less running and can rely on the filesystem for geo-replication and HA.
### Positive Consequences
* We can store accounts on the storage using the CS3 API, pushing geo-distribution to the storage layer.
* Backups of users and storage can be implemented without inconsistencies between using snapshots.
### Negative Consequences
* We need to spend time on implementing a reverse index based on files, and symlinks.

115
docs/ocis/adr/0003-external-user-management.md
View File

@@ -1,115 +0,0 @@
---
title: "3. Use external User Management"
weight: 3
date: 2020-12-09T20:21:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0003-external-user-management.md
---
* Status: superseded by [17. Allow read only external User Management]({{< ref "0017-allow-read-only-external-user-management.md" >}})
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1)
* Date: 2022-02-08
Technical Story: [Skip account-service by talking to CS3 user-api](https://github.com/owncloud/ocis/pull/1020)
## Context and Problem Statement
To attach metadata like shares to users ownCloud relies on persistent, non-reassignable, unique identifiers for users (and files). Email and username can change when a user changes his name. But even the OIDC sub+iss combination may change when the IdP changes. While there is [an account porting protocol](https://openid.net/specs/openid-connect-account-porting-1_0.html) that describes how a relying party (RP) such as ownCloud should behave, it still requires the RP to maintain its own user identifiers.
## Decision Drivers
* oCIS should be a single binary that can run out of the box without external dependencies like an LDAP server.
* Time: we want to build a release candidate asap.
* oCIS should be able to be easily integrated with standard user management components
## Considered Options
* Accounts service wraps LDAP
* [GLAuth](https://github.com/glauth/glauth) wraps accounts service
## Decision Outcome
Chosen option: "Move accounts functionality to GLAuth and name it accounts", by moving the existing accounts service file based persistence to GLAuth and use it as a drop in replacement for an LDAP server. The reverse index and web UI existing in the accounts service will move as well in order to make GLAuth a standalone, small scale user management with write capabilities.
### Product summary
- GLAuth is a drop in user management for small scale deployments that do not rely on an actual LDAP server.
- oCIS admins can either use the web UI to manage users in GLAuth or use existing tools in their IDM.
- We hide the complexity by embedding an OpenID Provider, an LDAP server and a user management web UI.
### Resulting deployment options
- Use internal user management
- Recommended for small scale use cases and simple deployments
- Users, groups and roles are stored and managed within GLAuth
- Use external user management
- Recommended for mid and large scale use cases
- Users, groups and roles are stored and managed within an external LDAP / AD / IDM
- Separate oCIS and LDAP admin: oCIS admin relies on the LDAP admin to manage users
- User permissions for roles are always managed in oCIS (settings service) because they are specific to oCIS
### Resulting technical implications
- Make the file based reverse index a standalone library
- Contribute to GLAuth
- Add ms graph based rest API to manage users, groups and roles (the LDAP lib is currently readonly)
- Add web UI to glauth that uses the ms graph based rest API to manage users
- Add a backend that uses the file based reverse index, currently living in the oCIS accounts service
- Move fallback mechanism from ocis/glauth service to upstream GLAuth to support multiple LDAP servers
- Make it a chain to support more than two LDAP servers
- Document the implications for merging result sets when searching for recipients
- At least one writeable backend is needed to support creating guest accounts
- Make all services currently using the accounts service talk to the CS3 userprovider
- To support multiple LDAP servers we need to move the fallback mechanism in ocis/glauth service to upstream GLAuth
- The current CS3 API for user management should be enriched with pagination, field mask and a query language as needed
- properly register an [auxiliary LDAP schema that adds an ownCloudUUID attribute to users and groups](https://github.com/owncloud/ocis/blob/c8668e8cb171860c70fec29e5ae945bca44f1fb7/deployments/examples/cs3_users_ocis/config/ldap/ldif/10_owncloud_schema.ldif)
### Positive Consequences
* The accounts service (which is our drop in LDAP solution) can be dropped. The CS3 userprovider service becomes the only service dealing with users.
* No sync
### Negative Consequences
* If users want to store users in their IDM and at the same time guests in a separate user management we need to implement GLAuth backends that support more than one LDAP server.
## Pros and Cons of the Options
### GLAuth wraps accounts service
Currently, the accounts service is the source of truth and we use it to implement user management.
* Good, because it solves the problem of storing and looking up an owncloud UUID for a user (and group)
* Good, because we can manage users out of the box
* Good, because we can persist accounts in a CS3 storage provider
* Bad, because it maintains a separate user repository: it needs to either learn or sync users.
### Move accounts functionality to GLAuth and name it accounts
We should use an existing LDAP server and make GLAuth a drop in replacement for it.
* Good, because we can use an existing user repository (an LDAP server), no need to sync or learn users.
* Good, because admins can rely on existing user management tools.
* Good, because we would have a clear separation of concerns:
- users reside in whatever repository, typically an LDAP server
- could be an existing LDAP server or AD
- could be our embeddable drop in glauth server
- we use a service to wrap the LDAP server with other APIs:
- ms graph API - ODATA based restful API,
- [SCIM](http://www.simplecloud.info/) - designed to manage user identities, supported by some IDPs,
- the current accounts API (which is a protobuf spec following the ms graph API)
- our account management UI can use the ms graph based API service which can have different backends
- an existing LDAP server
- our drop in glauth server (which might serve the ms graph based API itself)
- the CS3 API + a future guest provisioning API + a future CS3 user provisioning API (or [generic space provisioning](https://github.com/cs3org/cs3apis/pull/95))
- all oCIS services can use the service registry to look up the accounts service that provides an internal API
- could be the CS3 user provider (and API)
- could be the internal protobuf accounts API
- introduce a new guest provisioning API to CS3 which properly captures our requirement to have them in the user repository
- guests need to be made available to the firewall
- storages like EOS that integrate with the os for acl based file permissions need a numeric user and group id
* Good, because we can use the CS3 user provider with the existing LDAP / rest driver.
* Bad, because oCIS admins may not have the rights to manage role assignments. (But this is handled at a different department.)
* Bad, because oCIS admins may not have the rights to disable users if an external LDAP is used instead of the drop in GLAuth.
## Links
* supersedes [ADR-0001]({{< ref "0001-introduce-accounts-service.md" >}})
* superseded by [17. Allow read only external User Management]({{< ref "0017-allow-read-only-external-user-management.md" >}})

80
docs/ocis/adr/0004-support-hot-migration.md
View File

@@ -1,80 +0,0 @@
---
title: "4. Support Hot Migration"
weight: 4
date: 2020-12-09T20:21:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0004-support-hot-migration.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1)
* Date: 2021-03-16
Technical Story: \[description | ticket/issue URL\]
## Context and Problem Statement
Migration is one of the most important topics of the oCIS story. We need to provide a concept how to migrate from oC10 to oCIS.
## Decision Drivers
- Do not lose file blob or metadata.
- To prevent a sync surge from clients the etag for files should be migrated.
- To prevent internal links from breaking or pointing to wrong files the file id of existing files needs to be migrated.
- To prevent data loss trash and version blobs should be migrated.
- Existing shares like public links and federated shares must remain functional after the migration.
- To prevent internal shares the share type, permissions and expiry needs to be migrated.
- To prevent public links from breaking the url token, permissions, expiry and password needs to be migrated.
- *What about federated shares?*
- *What about additional share permissions, e.g. comment on office files?*
- Legacy clients need to keep working
- To keep existing clients working the `remote.php/webdav` and `dav/files/<username>` webdav endpoints as well as the ocs API need to be available.
- *What about [app passwords/tokens](https://doc.owncloud.com/server/user_manual/personal_settings/security.html#app-passwords-tokens)?*
## Considered Options
1. Cold Migration: migrate data while systems are not online, so no user interaction happens in between.
2. Hot Migration: one or both systems are online during migration.
## Decision Outcome
Chosen option: "\[option 1\]", because \[justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)\].
### Positive Consequences
- \[e.g., improvement of quality attribute satisfaction, follow-up decisions required, …\]
- …
### Negative Consequences
- \[e.g., compromising quality attribute, follow-up decisions required, …\]
- …
## Pros and Cons of the Options
### Cold Migration
The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. This can happen user by user, where every user export would contain the file blobs, their metadata, trash, versions, shares and all metadata that belongs to the user's storage. To prevent group shares from breaking, users in the same groups must be migrated in batch. Depending on the actual group shares in an instance this may effectively require a complete migration in a single batch.
- Good, because oCIS can be tested in a staging system without writing to the production system.
- Good, because file layout on disk can be changed to support new storage driver capabilities.
- Bad, because the export and import might require significant amounts of storage.
- Bad, because a rollback to the state before the migration might cause data loss of the changes that happened in between.
- Bad, because the cold migration can mean significant downtime.
### Hot Migration
The migration happens in subsequent stages while the service is online.
- Good, because the admin can migrate users from old to new backend in a controlled way.
- Good, because users and admins can learn to trust the new system.
- Good, because there can be preparations even long before the migrations happens in parallel on the oC10 codebase, i.e. addition of metadata that is needed while the system operates.
- Good, because the downtime of the system can be fairly small.
- Bad, because it is more complex and might drag on for a long time.
## Links
- [Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/1377) because the share manager for oCIS should store share information on the storage system. And [storage provider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)](https://github.com/cs3org/cs3apis/issues/93) finally: [eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/543)

210
docs/ocis/adr/0005-cs3-api-account-management.md
View File

@@ -1,210 +0,0 @@
---
title: "5. Account Management through CS3 API"
weight: 5
date: 2021-04-12T15:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0005-cs3-api-account-management.md
---
* Status: proposed
* Deciders: [@refs](https://github.com/refs), [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@pmaier1](https://github.com/pmaier1)
* Date: 2021-04-12
Technical Story: [Write only management API for User and Group resources](https://github.com/cs3org/cs3apis/pull/119)
## Context and Problem Statement
What would be a more effective way of using network resources and handle account management within the oCIS-Reva ecosystem? Separating account management at the service level is pragmatic and allows for fast iterations, but also steadily accumulates inconsistencies and bloats technical debt.
## Decision Drivers
* Reduce number of network calls.
* Reduce number of services (merge Account + GLAuth from ADR-0003).
* Formalize account management at the API level.
## Considered Options
* Account management delegated to vendors.
* Add account management to the CS3 API.
## Decision Outcome
Chosen option: "Add account management to the CS3 API". Making the API declare an API for account management will not only allow a deployment to fail fast (as in: the management node is not running) but would also centralize all management operations that should happen to be constrained within the Reva context. Constrained operations *SHOULD* be by definition more secure, or at least as secure as the rest of the system.
### Positive Consequences
* More resilient API.
* Because account management is considered a "first class citizen" changes are forced to go through a more exhaustive revision process.
* Removing Accounts from search users<sup>1</sup>.
* Replace the provisioning API in favor of the new Reva Admin node.
(1) the current vendor implementation of searching a user (i.e: when sharing a resource) relies directly on the accounts service, since this is the only source of truth. Searching a user looks like:
```
┌────────────────────────────────────────┐
│user search (no LDAP) │
│ │
│ ┌──────────┐ │
│ │ │ │
│ │ proxy │ │
│ │ │ ┌ ─ ─ ─ ─ ─ ┐ │
│ └──────────┘ go-micro │
│ ▲ │ │ │
│ │ Λ │
│ ▼ │ ╱ ╲ │ │
│ ┌──────────┐ ╱ ╲ │
│ │ │ │ ╱ ╲ │ │
│ │ ocs │◀──(1)───▶registry▏ │
│ │ │ │ ╲ ╱ │ │
│ └──────────┘ ╲ ╱ │
│ ▲ │ ╲ ╱ │ │
│ │ V │
│ │ │ │ │
│ │ │
│ │ └ ─ ─ ─ ─ ─ ┘ │
│ │ │
│ │ │
│ │ ┌──────────┐ │
│ │ │ │ │
│ └─────────────▶│ accounts │ │
│ │ │ │
│ └──────────┘ │
│ │
│ │
│(1) ocs requests a connection to the │
│accounts service to the registry │
│ │
└────────────────────────────────────────┘
```
Whereas, as a result of ADR-0003 and this ADR, we can simplify and improve this design:
```
┌─────────────────────────────────────────────┐
│user search │
│ │
│ │
│ ┌──────────┐ │
│ │ │ │
│ │ proxy │ │
│ │ │ │
│ └──────────┘ │
│ │ │
│ ▼ │
│ ┌──────────┐ │
│ │ │ │
│ │ ocs │ │
│ │ │ │
│ └──────────┘ │
│ │ │
│ │ │
│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ │
│ reva ▼ │ IDM │
│ │ ┌──────────┐ │ ┌──────────┐ │ │
│ │ │ │ │ │ │
│ │ │ users │◀─────┼──▶│ GLAuth │ │ │
│ │ │ │ │ │ │
│ │ └──────────┘ │ └──────────┘ │ │
│ │ │
│ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │
│ │
└─────────────────────────────────────────────┘
```
And instead rely on the already existing Reva users provider.
## Pros and Cons of the Options
### Account management delegated to vendors
* Good, because it allows for fast iterations.
* Bad, because account management happens outside of the Reva process. This can potentially end up in invalid account creation / deletion / updates.
* An example with the existing Accounts service is that any client can fire CRUD accounts requests to the Accounts service as long as the client knows where the server is running and provides with an Authorization header (only required by the proxy). This request totally bypasses Reva middlewares and therefore any security measures that should be enforced by the entire system.
* Bad, because leaves teams the task of designing and implementing a way of dealing with account management. Ideally one schema should be provided / suggested.
Creating an account using the first option looks currently is implemented in vendors as:
```
┌──────────────────────────────────────────────────┐
│ creating a user (webui) │
│ │
│ ┌──────────┐ │
│ │ │ │
│ │ proxy │ │
│ │ │ │
│ └──────────┘ │
│ │ │
│ │ │
│ /api/v0/accounts/accounts-create │
│ │ │
│ │ │
│ │ │
│ ▼ │
│ ┌──────────┐ │
│ │ │ │
│ │ accounts │ │
│ │ │ │
│ └──────────┘ │
│ │
│ note that while doing CRUD operations changes │
│ are instantly reflected for the IDP since out of │
│ the box oCIS uses an accounts backend for │
│ GLAuth. │
└──────────────────────────────────────────────────┘
```
As explained before, during this flow no Reva middlewares are run. Creating an account will only use the embedded accounts js file alongside a minted jwt token (by the oCIS proxy) to communicate with the accounts service.
### Add account management to the CS3 API
* Good, because it solidifies what the CS3 API can or cannot do, and account management should be handled at the API level since ultimately accounts would contain a mix of required CS3 and vendor-specific attributes.
* Good, because it centralizes account management and constrains it within the Reva boundaries.
* Good, because there is a clear separation of concerns on what is accounts management logic.
* Good, because we already designed [a similar API for the accounts service](https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L42-L85) the only difference being we (vendors) [define their own messages](https://github.com/owncloud/ocis/blob/master/accounts/pkg/proto/v0/accounts.proto#L252-L408).
* The API would fully include CRUD methods
* Bad, because development cycles are larger.
* an example flow will be: `update api > run prototool > publish language specific packages > update dependencies to fetch latest version of the package > utilize the new changes`.
The new account management workflow will result in:
```
┌───────────────────────────────────────────────────┐
│creating a user (webui) │
│ - maintain the same route for compatibility │
│ │
│ ┌──────────┐ │
│ │ │ │
│ │ proxy │ │
│ │ │ │
│ └──────────┘ │
│ │ │
│ │ │
│ /api/v0/accounts/accounts-create │
│ │ │
│ │ │
│ ┌ ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Reva │ IDM │ │
│ │ │ │ │ │
│ ▼ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ │ │ │ │ │
│ │ │ admin │───────────┼──┼──▶│ GLAuth │ │
│ │ │ │ │ │ │
│ │ └──────────┘ │ │ └──────────┘ │
│ │ │
│ │ │ │ │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │
│ │
│ │
│ │
│ │
│ │
│ │
│ an example of a driver could be GLAuth │
│ implementing the user management portion of the │
│ GraphAPI │
└───────────────────────────────────────────────────┘
```
This flow allows Reva and oCIS Proxy to run any middleware logic in order to validate a request. The communication between the proposed Admin api (CS3 API messages) and the IDM (GLAuth) are specific to the _drivers_.

58
docs/ocis/adr/0006-service-discovery.md
View File

@@ -1,58 +0,0 @@
---
title: "6. Service Discovery within oCIS and Reva"
weight: 6
date: 2021-04-19T13:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0006-service-discovery.md
---
* Status: accepted
* Deciders: [@refs](https://github.com/refs), [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@pmaier1](https://github.com/pmaier1)
* Date: 2021-04-19
Technical Story: [Introduce Named Services.](https://github.com/cs3org/reva/pull/1509)
## Context and Problem Statement
Reva relies heavily on config files. A known implication of this approach are having to know a-priori where a service is running (host + port). We want to move away from hardcoded values and rely instead on named services for service discovery. Furthermore, we would like both platforms (Reva + oCIS) to have the same source of truth at any given time, not having one to notify the other whenever a service status changes.
## Decision Drivers
* Avoid a-priori knowledge of services.
* Ease of scalability.
* Always up-to-date knowledge of the running services on a given deployment (a service registry doesn't have to necessarily be running on the same machine / network)
## Considered Options
* Hardcoded tuples of hostname + port
* Dynamic service registration
## Decision Outcome
Chosen option: "Dynamic service registration". There were some drawbacks regarding this due to introducing go-micro to Reva was from start an issue. Given the little usage of go-micro we need, we decided instead to define our very own [Registry interface](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/pkg/registry/registry.go#L22-L35) on Reva and extended the runtime arguments to [allow for injecting a registry](https://github.com/refs/reva/blob/58d013a7509d1941834e1bc814e9a9fa8bff00b1/cmd/revad/runtime/option.go#L53-L58).
### Positive Consequences
* Having dynamic service registration delegates the entire lifecycle of finding a process to the service registry.
* Removing a-priori knowledge of hostname + port for services.
* Marrying go-micro's registry and a newly defined registry abstraction on Reva.
* We will embrace go-micro interfaces by defining a third merger interface in order to marry go-micro registry and reva registry.
* The ability to fetch a service node relying only on its name (i.e: com.owncloud.proxy) and not on a tuple hostname + port that we rely on being preconfigured during runtime.
* Conceptually speaking, a better framework to tie all the services together. Referring to services by names is less overall confusing than having to add a service name + where it is running. A registry is agnostic to "where is it running" because it, by definition, keeps track of this specific question, so when speaking about design or functionality, it will ease communication.
## Pros and Cons of the Options
### Hardcoded tuples of hostname + port
* Good, because firewalls are easier to configure since IP are static.
* Good, because the mental model required is easier to grasp as IP addresses can be easily bundled.
* Bad, because it requires thorough planning of ports.
### Dynamic service registration
* Good, because it abstracts the use of service lookup away to registry logic from the admin or developer.
* Good, because it allows for, through interfaces, registry injection
* This means we can have a service registry that we extensively use in oCIS and inject its functionality onto Reva.
* Bad, because it's yet another abstraction.
* Bad, because firewalls are harder to configure with dynamic IPs.f

156
docs/ocis/adr/0007-api-for-spaces.md
View File

@@ -1,156 +0,0 @@
---
title: "7. Open Graph API for oCIS File Spaces"
weight: 7
date: 2021-05-03T09:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0007-api-for-spaces.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1)
* Date: 2021-03-19
Technical Story: API to enable the concept of [Spaces](https://github.com/owncloud/enterprise/issues/3863)
## Context and Problem Statement
As one of the building blocks for Spaces in oCIS we plan to add an API that returns information about available spaces. This ADR discusses the API design oriented on the Microsoft Graph API.
> Note: The term "spaces" is used here in the context of "a space where files can be saved", similar to a directory. It is not to be confused with space in the sense of free file space for example.
The purpose of this new API is to give clients a very simple way to query the dynamic list of spaces, that the user has access to. Clients can provide a better user experience with that.
This API is supposed to be queried often, to give clients a condensed view of the available spaces for a user, but also their eTags and cTags. Hence the clients do not have to perform a PROPFIND for every space separately.
This API would even allow providing (WebDAV-) endpoints depending on the kind and version of the client asking for it.
## Decision Drivers
- Make it easy to work with a dynamic list of spaces of a user for the clients.
- No longer the need to make assumptions about WebDAV- and other routes in clients.
- More meta data available about spaces for a better user experience.
- Part of the bigger spaces plan.
- Important to consider in client migration scenarios, i.e. in CERN.
## Considered Options
1. [Microsoft Graph API](https://developer.microsoft.com/en-us/graph) inspired API that provides the requested information.
## Decision Outcome
This the DRAFT for the API.
### API to Get Info about Spaces
ownCloud servers provide an API to query for available spaces of an user.
See the openAPI Specification for the [Libre Graph API](https://owncloud.dev/libre-graph-api/).
Most important, the API returns the WebDAV endpoint for each space. With that, clients do not have to make assumptions about WebDAV routes any more.
See [Drive item in Microsoft Graph API](https://docs.microsoft.com/en-us/graph/api/resources/onedrive?view=graph-rest-1.0) for an overview of `drive` and `driveItem` resources. The concrete list of drives / spaces a user has access to can be obtained on multiple endpoints.
### Get "Home folder"
Retrieve information about the home space of a user. Note: The user has access to more spaces. This call only returns the home space to provide API parity with the Graph API.
API Call: `/me/drive`: Returns the information about the users home folder.
### Get All Spaces of a User
Retrieve a list of available spaces of a user. This includes all spaces the user has access to at that moment, also the home space.
API Call: `/me/drives`: Returns a list of spaces.
There is also `/drives`, returning the list of spaces the user has access to. This endpoint is used to access any space by id using `/drives/{drive-id}`.
### Common Reply
The reply to both calls is either one or a list of [Drive representation objects](https://docs.microsoft.com/de-de/graph/api/resources/drive?view=graph-rest-1.0):
```
{
"id": "string",
"createdDateTime": "string (timestamp)",
"description": "string",
"driveType": "personal | projectSpaces | shares",
"lastModifiedDateTime": "string (timestamp)",
"name": "string",
"owner": { "@odata.type": "microsoft.graph.identitySet" },
"quota": { "@odata.type": "microsoft.graph.quota" },
"root": { "@odata.type": "microsoft.graph.driveItem" },
"webUrl": "url"
}
```
The meaning of the objects in Open Graph API context are:
1. **id** - a persistent and unique ID identifying the space, called Storage Space ID.
2. **driveType** - describing the type of the space.
3. **owner** - an owner object to whom the space belongs
4. **quota** - quota information about this space
5. **root** - the root driveItem object.
6. **webUrl** - The URL to make this space visible in the browser.
The following *driveType* values are available in the first step, but might be enhanced later:
* **personal**: The users home space
* **projectSpaces**: The project spaces available for the user (*)
* **shares**: The share jail, contains all shares for the user (*)
Other space types such as backup, hidden etc. can be added later as requested.
> Note: The *projectSpaces* and *shares* space are virtual spaces. They only contain other spaces, and no regular resources.
The (*) marked types are not defined in the official MS API. They are prefixed with `oc` to avoid namespace clashes.
The `root` object equals a [driveItem](https://docs.microsoft.com/de-de/graph/api/resources/driveitem?view=graph-rest-1.0) and contains information about the root resource (directory) of the space.
This is an example object as it can be expected as `root` element. It is not complete, as not all elements will be implemented so far.
```
{
"cTag": "string (etag)",
"webDavUrl": "string",
/* inherited from baseItem */
"id": "string (identifier)",
"createdBy": {"@odata.type": "microsoft.graph.identitySet"},
"createdDateTime": "String (timestamp)",
"eTag": "string",
"lastModifiedBy": {"@odata.type": "microsoft.graph.identitySet"},
"lastModifiedDateTime": "String (timestamp)",
"name": "string",
"webUrl": "string",
}
```
Meaningful fields of the root element in the context of the Open Graph API:
1. **id** - a persistent and unique ID identifying the root directory node.
2. **webDavUrl** - The webdav path of the top item of the space.
3. **eTag** - an identifier that changes automatically if the content *or* metadata of the node or the underlying resources changes.
4. **cTag** - an identifier that changes automatically if the content of the root node or of one of the underlying resources changes.
5. **webUrl** - The URL to make this space visible in the browser.
> Note: To indicate that only the metadata of a resource has changed, the eTag has changed, but the cTag not.
### Positive Consequences
- A well understood and mature API from Microsoft adopted to our needs.
- Prerequisite for Spaces in oCIS.
- Enables further steps in client development.
### Negative Consequences
- Migration impact on existing installations. Still to be investigated.
- Requires additional webdav endpoint that allows accessing an arbitrary storage space, either
- with an id: `/dav/spaces/<spaceid>/relative/path/to/file.ext`, or
- with a global path: `/dav/global/<accessible>/<mount>/<point>/relative/path/to/file.ext`, e.g. `/dav/global/projects/Golive 2021/Resources/slides.odt`
### Open Topics
- What are the WebDAV paths for Trashbin, Versions
+ option: additional entries in the reply struct
- The identitySet object used for "owner" and "coowner" require to implement the [https://docs.microsoft.com/de-de/graph/api/resources/identityset?view=graph-rest-1.0](IdentitySet) JSON object, which contains information that seems to be of limited benefit for oCIS. An alternative would be to implement a simpler identity object for oCIS and use that.

194
docs/ocis/adr/0008-configuration.md
View File

@@ -1,194 +0,0 @@
---
title: "8. Configuration"
weight: 8
date: 2021-05-03T15:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0008-configuration.md
---
* Status: proposed
* Deciders: [@refs](https://github.com/refs), [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@pmaier1](https://github.com/pmaier1)
* Date: 2021-05-03
## Context and Problem Statement
As per urfave/cli's doc:
>The precedence for flag value sources is as follows (highest to lowest):
>
>0. Command line flag value from user
>1. Environment variable (if specified)
>2. Configuration file (if specified)
>3. Default defined on the flag
An issue arises in point 2, in the sense that configuration file refers to a single file containing the value for the env variable. The CLI framework we use for flag parsing does not support merging config structs with CLI flags. This introduces an inconsistency with the framework: config structs are not supported, and we cannot hook to the lifecycle of the flags parsing to use a file as source and conform to these rules.
Because we solely rely on [structured configuration](https://github.com/owncloud/ocis/blob/master/ocis-pkg/config/config.go) we need a way to modify values in this struct using the provided means urfave/cli gives us (flags, env variables, config files and default value), but since we have different modes of operation (supervised Vs. unsupervised) we have to define a clear line.
### Decision Drivers
- Improve experience for the end user.
- Improve experience for developers.
- Sane defaults.
- Sane overrides.
### Considered Options
- Extend [FlagInputSourceExtension interface](https://github.com/urfave/cli/blob/master/altsrc/flag.go#L12-L17)
- Feature request: support for structured configuration (urfave/cli).
- Clearly defined boundaries of what can and cannot be done.
- Expose structured field values as CLI flags
- Drop support for structure configuration
- Adapt the "structured config files have the highest priority" within oCIS
### Decision Outcome
[STILL UNDECIDED]
#### Positive Consequences
[TBD, depends on Decision Outcome]
### Pros and Cons of the Options
#### Extend FlagInputSourceExtension interface
- Good, because we could still use Viper to load from config files here and apply values to the flags in the context.
- Bad, because urfave/cli team are [actively working on v3 of altsrc](https://github.com/urfave/cli/issues/1051#issuecomment-606311923) and we don't want to maintain yet another slice of the codebase.
notes: source is [FlagInputSourceExtension interface](https://github.com/urfave/cli/blob/master/altsrc/flag.go#L12-L17)
#### Feature request: support for structured configuration (urfave/cli).
- Good, because we could remove Viper off the codebase and solely rely on urfave/cli's native code.
- Bad, because there are no plans to support this upstream.
#### Clearly defined boundaries of what can and cannot be done.
- Good, because no changes to the codebase required (not drastic changes.)
- Bad, because we're limited by the framework
#### Expose structured field values as CLI flags
- Good, because it has been already taken into account on large projects (kubernetes) [here.](https://docs.google.com/document/d/1Dvct469xfjkgy3tjWMAKvRAJo4CmGH4cgSVGTDpay6A) in point 5.
- Bad, because it requires quite a bit<sup>1</sup> of custom logic.
- Bad, because how should these flags be present in the `-h` menu of a subcommand? Probably some code generation needed.
*[1] this is a big uncertainty.
#### Drop support for structure configuration
- Good, because it makes the integration with the cli framework easier to grasp.
- Good, because it is not encouraged by the 12factor app spec.
- Bad, because we already support if and users make active use of it. At least for development.
#### Adapt the "structured config files have the highest priority" within oCIS
- Good, because that would mean little structural changes to the codebase since the Viper config parsing logic already uses the `Before` hook to parse prior to the command's action executes.
### Notes
#### Use Cases and Expected Behaviors
##### Supervised (`ocis server` or `ocis run extension`)
![grafik](https://user-images.githubusercontent.com/6905948/116872568-62b1a780-ac16-11eb-9f29-030a651ee39b.png)
- Use a global config file (ocis.yaml) to configure an entire set of services: `> ocis --config-file /etc/ocis.yaml service`
- Use a global config file (ocis.yaml) to configure a single extension: `> ocis --config-file /etc/ocis/yaml proxy`
- When running in supervised mode, config files from extensions are NOT evaluated (only when running `ocis server`, runs with `ocis run extension` do parse individual config files)
- i.e: present config files: `ocis.yaml` and `proxy.yaml`; only the contents of `ocis.yaml` are loaded<sup>1</sup>.
- Flag parsing for subcommands are not allowed in this mode, since the runtime is in control. Configuration has to be done solely using config files.
*[1] see the development section for more on this topic.
###### Known Gotchas
- `> ocis --config-file /etc/ocis/ocis.yaml server` does not work. It currently only supports reading global config values from the predefined locations.
##### Unsupervised (`ocis proxy`)
![grafik](https://user-images.githubusercontent.com/6905948/116872534-54fc2200-ac16-11eb-8267-ffe7b03177b3.png)
- `ocis.yaml` is parsed first (since `proxy` is a subcommand of `ocis`)
- `proxy.yaml` is parsed if present, overriding values from `ocis.yaml` and any cli flag or env variable present.
#### Other known use cases
- Configure via env + some configuration files like WEB_UI_CONFIG or proxy routes
- Configure via flags + some configuration files like WEB_UI_CONFIG or proxy routes
- Configure via global (single file for all extensions) config file + some configuration files like WEB_UI_CONFIG or proxy routes
- configure via per extension config file + some configuration files like WEB_UI_CONFIG or proxy routes
Each individual use case DOES NOT mix sources (i.e: when using cli flags, do not use environment variables nor cli flags).
_Limitations on urfave/cli prevent us from providing structured configuration and framework support for cli flags + env variables._
#### Use Cases for Development
#### Config Loading
Sometimes is desired to decouple the main series of services from an individual instance. We want to use the runtime to startup all services, then do work only on a single service. To achieve that one could use `ocis server && ocis kill proxy && ocis run proxy`. This series of commands will 1. load all config from `ocis.yaml`, 2. kill the supervised proxy service and 3. start the same service with the contents from `proxy.yaml`.
#### Start an extension multiple times with different configs (in Supervised mode)
Flag parsing on subcommands in supervised mode is not yet allowed. The runtime will first parse the global `ocis.yaml` (if any) and run with the loaded configuration. This use case should provide support for having 2 different proxy config files and making use of the runtime start 2 proxy services, with different values.
For this to work, services started via `Service.Start` need to forward any args as flags:
```go
if err := client.Call("Service.Start", os.Args[2], &reply); err != nil {
log.Fatal(err)
}
```
This should provide with enough flexibility for interpreting different config sources as: `> bin/ocis run proxy --config-file /etc/ocis/unexpected/proxy.yaml`
#### Developing Considered Alternatives Further
Let's develop further the following concept: Adapt the "structured config files have the highest priority" within oCIS.
Of course it directly contradicts urfave/cli priorities. When a command finished parsing its cli args and env variables, only after that `Before` is called. This mean by the time we reach a command `Before` hook, flags have already been parsed and its values loaded to their respective destinations within the `Config` struct.
This should still not prevent a developer from using different config files for a single service. Let's analyze the following use case:
1. global config file present (ocis.yaml)
2. single proxy.yaml config file
3. another proxy.yaml config file
4. running under supervision mode
The outcome of the following set of commands should be having all bootstrapped services running + 2 proxies on different addresses:
```console
> ocis server
> ocis kill proxy
> ocis run proxy --config-file proxy.yaml
> ocis run proxy --config-file proxy2.yaml
```
This is a desired use case that is yet not supported due to lacking of flags forwarding.
#### Follow-up PR's
- Variadic runtime extensions to run (development mostly)
- Arg forwarding to command (when running in supervised mode, forward any --config-file flag to supervised subcommands)
- Ability to set `OC_URL` from a config file (this would require to extend the ocis-pkg/config/config.go file).
#### The case for `OC_URL`
`OC_URL` is a jack-of-all trades configuration. It is meant to ease up providing defaults and ensuring dependant services are well configured. It is an override to the following env vars:
```
OC_IDM_ADDRESS
PROXY_OIDC_ISSUER
STORAGE_OIDC_ISSUER
STORAGE_FRONTEND_PUBLIC_URL
STORAGE_LDAP_IDP
WEB_UI_CONFIG_SERVER
WEB_OIDC_AUTHORITY
OC_PUBLIC_URL
```
Because this functionality is only available as an env var, there is no current way to "normalize" its usage with a config file. That is, there is no way to individually set `OC_URL` via config file. This is clear technical debt, and should be added functionality.
#### State of the Art
- [Kubernetes proposal on this very same topic](https://docs.google.com/document/d/1Dvct469xfjkgy3tjWMAKvRAJo4CmGH4cgSVGTDpay6A)
- [Configuration \| Pulumi](https://www.pulumi.com/docs/intro/concepts/config/)
- Configuration can be altered via setters through the CLI.

71
docs/ocis/adr/0009-extension-template.md
View File

@@ -1,71 +0,0 @@
---
title: "9. Extension Template"
weight: 9
date: 2021-06-10T15:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0009-extension-template.md
---
* Status: proposed
* Deciders: [@c0rby](https://github.com/c0rby) <!-- optional -->
* Date: 2021-06-10
Technical Story: [description | ticket/issue URL] <!-- optional -->
## Context and Problem Statement
We want to accelerate and simplify extension development by removing the necessity to type or copy the boilerplate code. Can we provide a template or a similar mechanism to aid when developing new extensions?
## Decision Drivers <!-- optional -->
* The solution should be easily maintainable.
* It should always be up-to-date.
* The solution should be easy to use.
## Considered Options
* Use [boilr](https://github.com/tmrts/boilr)
* Create a template git repository.
* Use [ocis-hello](https://github.com/owncloud/ocis-hello/) as a "template"
## Decision Outcome
Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
### Positive Consequences: <!-- optional -->
* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
* …
### Negative consequences: <!-- optional -->
* [e.g., compromising quality attribute, follow-up decisions required, …]
* …
## Pros and Cons of the Options <!-- optional -->
### [boilr](https://github.com/tmrts/boilr)
We have a boilr template already. [boilr-ocis-extension](https://github.com/owncloud/boilr-ocis-extension/)
This approach is nice because it provides placeholders which can be filled during the generation of a new extension from the template. It also provides prompts for the placeholder values during generation.
* Good, because with the placeholders it is hard to miss values which should be changed
* Bad, because maintaining is more complex
### Template git repository
Create a git repository with an extension containing the boilerplate code.
* Good, because we can use the usual tools for QA and dependency scanning/updating.
* Good, because it doesn't require any additional tool.
### [ocis-hello](https://github.com/owncloud/ocis-hello/) as a "template"
We have the ocis-hello repository which acts as an example extension containing a grpc and http service and a web UI. It also demonstrates the usage of the settings service.
* Good, because it contains a bit more code than just the plain boilerplate
* Good, because the integration into oCIS is already tested for the Hello extension (e.g. with Proxy and Settings). This will ensure, that the example extension is up-to-date.
* Bad, because if you don't require all features you have to delete stuff

85
docs/ocis/adr/0010-policy-enforcement.md
View File

@@ -1,85 +0,0 @@
---
title: "10. Extension Policies"
weight: 10
date: 2021-06-30T14:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0010-policy-enforcement.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1), [@fschade](https://github.com/fschade)
* Date: 2021-06-30
## Context and Problem Statement
There should be a way to impose certain limitations in areas of the code that require licensing. This document researches an approach to achieve this goal, while limiting the scope to the enforcement side of it. The architecture for a policy system must be composed of 2 parts:
1. License creation and validation
2. Enforcement
It is desirable to keep both systems isolated, since the implementation of the latter has to be done within the constraints of the codebase. The alternative is running an enforcement service and have each and every single request evaluating whether the request is valid or not.
## Decision Drivers
- As a team, we want to have the licensing code concentrated in a central module
- We don't want to stop/start the extension whenever a policy is updated (hot reload). It must happen during runtime.
## Considered Options
1. Build the evaluation engine in-house.
2. Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go)
## Decision Outcome
Chosen option: option 2; Use third party libraries such as Open Policy Agent (a CNCF approved project written in Go)
### Positive Consequences
- OPA is production battle tested.
- Built around performance - policies evaluations are no longer than 1ms per request.
- Middleware friendly: we use gRPC clients all over our ecosystem; wrappers (or middlewares) is a viable way to solve this problem instead of a dedicated service or its own package.
- Community support.
- Kubernetes friendly.
- Supports envoy, kong, terraform, traefik, php, node and many more.
### Negative Consequences
- More vendor code inside the binary (larger attack surface, larger footprint [to be quantified] )
## Chosen option approach
Make use of [overloading Open Policy Agent's input](https://www.openpolicyagent.org/docs/latest/external-data/#option-2-overload-input) along with an external storage source (instead of an OPA service) in conjunction with go-micro's gRPC client wrappers (a.k.a. middlewares) to leverage policy rules evaluation.
### Terminology
New terms are defined to refer to new mental models:
- Policy: self-imposed limitation of a piece of software. i.e: "after 20 users limit the use of thumbnails".
- Checkers: in the context of a middleware, a checker is in charge of defining logical conditions that prevent requests (users) from doing an action.
- Policy file: a [rego file](https://www.openpolicyagent.org/docs/latest/policy-language/).
- Policy evaluation: the act of piecing together input (from a request), data (from an external storage) and policies in order to make a decision.
#### Temporary new Interfaces part of the PoC
- IStorage: provides means of extracting data from an external source (in case of the POC an etcd storage cluster).
### External data storages
However, for this to be usable it needs state. The Rego engine works with input and data, where data is essentially a database the input is tried against, in order to expand this poc to include functionality such as counters (i.e: give access to the thumbnails only to 50 users) we need an external storage, and consequentially, Rego needs to have an option to load data from an external storage. There is an entire chapter in the documentation regarding external data: https://www.openpolicyagent.org/docs/latest/external-data/. The most "natural" option (option 5) states:
> OPA includes functionality for reaching out to external servers during evaluation. This functionality handles those cases where there is too much data to synchronize into OPA, JWTs are ineffective, or policy requires information that must be as up-to-date as possible.
This is a natural option because it requires service-to-service communication, and by definition using microservices it should come "natural to us". Another approach is using JWT (which we already use) to encode the necessary data into the JWT and handing it over to rego as "data". The issue with this approach is that depending on the features of the licenses the JWT might grow and be filled with noise and redundancy (this is, unless a new token is issued for licensing purposes).
### Future ideas
[This proof of concept](https://github.com/owncloud/ocis/pull/2236) is very rigid in the sense that the `IStorage` interface only has one implementation that ties it to etcd, meaning running an oCIS cluster without an etcd service will result in a crash. This is by far ideal and less coupled implementations should be done. There is the case of using the storage metadata as a source to store data necessary to the policies, or even using the go-micro store as a kv store to achieve the exact same, since it already runs as its own service. The implementation of this is trivial and left out of the POC since it requires more time than the allotted for this task.
#### Message Broker
This problem perfectly encompasses the use of a message broker, where services such as OCS will emit messages to a bus and only listeners react to them. In this case the following applies:
![message broker](https://i.imgur.com/sa1pANQ.jpg)
The necessary interfaces are provided to us by go-micro, only implementations are to be done.

320
docs/ocis/adr/0011-global-url-format.md
View File

@@ -1,320 +0,0 @@
---
title: "11. WebUI URL format"
weight: 11
date: 2021-07-07T14:55:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0011-global-url-format.md
---
* Status: accepted
* Deciders: [@refs](https://github.com/refs), [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1), [@fschade](https://github.com/fschade), [@tbsbdr](https://github.com/tbsbdr), [@kulmann](https://github.com/kulmann)
* Date: 2021-07-07
## Context and Problem Statement
When speaking about URLs we have to make a difference between browser URLs and API URLs. Only browser URLs are visible to end users and will be bookmarked. The currently existing and bookmarked ownCloud 10 URLs look something like this:
```
GET https://demo.owncloud.com/apps/files/?dir=/path/to/resource&fileid=5472225
303 Location: https://demo.owncloud.com/apps/files/?dir=/path/to/resource
```
When the URL contains a `fileid` parameter the server will look up the corresponding `dir`, overwriting whatever was set before the redirect. The `fileid` always takes precedence and the server is responsible for the lookup.
```
GET https://demo.owncloud.com/apps/files/?dir=/path/to/resource
```
The `dir` parameter is then used to make a WebDAV request against the `/dav/files` endpoint of the currently logged-in user:
```
PROPFIND https://demo.owncloud.com/remote.php/dav/files/demo/path/to/resource
```
The resulting PROPFIND response is used to render the file listing. All good so far.
For the new ocis web UI we want to clean up the user visible Browser URLs. They currently look like this:
```
https://demo.owncloud.com/#/files/list/all/path/to/resource
```
Currently, there is no `fileid` like parameter in the browser URL, making bookmarks of it fragile (they break when a bookmarked folder is renamed).
The oCIS web UI just takes the path and uses the `/webdav` endpoint of the currently logged-in user:
```
PROPFIND https://demo.owncloud.com/remote.php/webdav/path/to/resource
```
With the new ownCloud web client (owncloud/web)
needs to interpret them to make API calls. With this in mind, this is the current mapping on ownCloud Web with OC10 and OCIS backend:
| | Browser URL | API URL |
|------|----------------------------------------------------------------|----------------------------------------------------|
| OC10 + classic WebUI | `https://demo.owncloud.com/apps/files/?dir=/path/to/resource&fileid=5472225` | `https://demo.owncloud.com/remote.php/dav/files/demo/path/to/resource` |
| OC10 + OCIS WebUI| `https://web.owncloud.com/index.html#/files/list/all/path%2Fto%2Fresource` | `https://demo.owncloud.com/remote.php/webdav/path/to/resource` |
| OCIS | `https://demo.owncloud.com/#/files/list/all/path/to/resource` | `https://demo.owncloud.com/remote.php/webdav/path/to/resource` |
On an OC10 backend the `fileid` query parameter takes precedence over the `dir`. In fact if `dir` is invalid but `fileid` isn't, the resolution will succeed, as opposed to if the `fileid` is wrong (doesn't exist) and `dir` correct, resolution will fail altogether with a 404.
This ADR is limited to the scope of "how will a web client deal with the browser URL?". The API URLs will change with the spaces concept to `https://demo.owncloud.com/dav/spaces/<space_id>/relative/path/to/resource`. The Web UI can look up a space id and the mount path using the `/graph/v1.0/drives` API:
1. TODO for a given resource id as part of the URL the `https://demo.owncloud.com/v1.0/drive/items/123456A14B0A7750!359?$select=parentReference` can be used to retrieve the drive/space:
```
{
"parentReference": {
"driveId": "123456a14b0a7750",
"driveType": "personal",
"id": "123456A14B0A7750!357",
"path": "/drive/root:"
}
}
```
2. TODO to fetch the list of all spaces with their mount points we need an API endpoint that allows clients (not only the web ui) to 'sync' the list of storages a user has access to from the storage registry on the server side. This allows clients to directly talk to a storage provider on another instance, allowing true storage federation. The MS graph api has no notion of mount points, so we will need to add a `mountpath` *(or `mountpoint`? or `alias`?)* to our [`drive` resource properties in the libreGraph spec](https://github.com/owncloud/open-graph-api/blob/dc6da5359eee0345429080b5b59762fd8c57b121/api/openapi-spec/v0.0.yaml#L351-L384). Tracked in https://github.com/owncloud/open-graph-api/issues/6
{{< hint >}}
@jfd: The graph api returns a `path` in the `parentReference`, which is part of the `root` in a `drive` resource. But it contains a value in the namespace of the `graph` endpoint, e.g.: `/drive/root:/Bilder` for the `/Bilder` folder in the root of the currently logged-in users personal drive/space. Which is again relative to the drive. To give the clients a way to determine the mount point we need to add a new `mountpath/point/alias` property.
{{< /hint >}}
## Decision Drivers
* To reveal relevant context to the user URLs should either carry a path component or a meaningful alias
* To prevent bookmarks from breaking URLs should have an id component that can be used by the system to lookup the resource
## Considered Options
* Existing ownCloud 10 URLs
* ID based URLs
* Path based URLs
* Space based URLs
* Mixed Global URLs
* Configurable path component in URLs
## Decision Outcome
Chosen option: "Mixed global URLs", because it meets the requirement to contain a path and a stable identifier.
### Positive Consequences <!-- optional -->
* The path makes it "human readable"
* The URL can be bookmarked
* The bookmarked URLs remain stable even if the path changes
* All URLs can be shortened to hide any metadata like path, resource name and query parameters
### Negative Consequences <!-- optional -->
* the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint
## Pros and Cons of the Options
### Existing OwnCloud 10 URLs
The existing ownCloud 10 URLs look like this
| URL | comment |
|-----|---------|
| `https://<host>/apps/files/?dir=<path>&fileid=<fileid>` | pattern |
| `https://demo.owncloud.com/apps/files/?dir=/&fileid=18` | root of the currently logged in user |
| `https://demo.owncloud.com/index.php/apps/files/?dir=/path/to/resource&fileid=192` | sub folder `/path/to/resource` |
It contains a path and a `fileid` (which takes precedence).
* Good, because the `fileid` prevents bookmarks from breaking
* Good, because the `dir` reveals context in the form of a path
* Bad, because the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint
* Bad, because URLs still contain a long prefix `(/index.php)/apps/files`
* Bad, because the `fileid` needs to be accompanied by a `storageid` to allow efficient routing in ocis
* Bad, because if not configured properly an additional `/index.php` prefixes the route
* Bad, because power users cannot navigate by updating only the path in the URL, as the `fileid` takes precedence. They have to delete the `fileid` to navigate
### ID based URLs
MS OneDrive has URLs like this:
| URL | comment |
|-----|---------|
| `https://<host>/?id=<fileid>(&cid=<cid>)` | pattern, the `cid` is optional but added automatically |
| `https://onedrive.live.com/?id=root&cid=A12345A14B0A7750` | root of a personal drive |
| `https://onedrive.live.com/?id=A12345A14B0A7750%21359&cid=A12345A14B0A7750` | sub folder in a personal drive |
It contains only IDs but no folder names. The `fileid` is a URL encoded `<cid>!<numericid>`. Very similar to the CS3 `resourceid` which consists of `storageid` and `nodeid`.
* Good, because bookmarks cannot break
* Good, because URLs do not disclose unshared path segments
* Bad, because the web UI needs to look up the space id in a registry to build an API request for the `/dav/space` endpoint
* Bad, because URLs reveal no context to users
### Path based URLs
There is a customized ownCloud instance that uses path only based URLs:
| URL | comment |
|-----|---------|
| `https://<host>/apps/files/?dir=/&` | root of the currently logged in user |
| `https://demo.owncloud.com/apps/files/?dir=/&` | root of the currently logged in user |
| `https://demo.owncloud.com/apps/files/?dir=/path/to/resource&` | sub folder `/path/to/resource` |
* Good, because the URLs reveal the full path context to users
* Good, because power users can navigate by updating the path in the url
* Bad, because the web UI needs to look up the space id in a registry to build an API request for the `/dav/space` endpoint
* Bad, because the bookmarks break when someone renames a folder in the path
* Bad, because there is no id that can be used as a fallback lookup mechanism
* Bad, because URLs might leak too much context (parent folders of shared files)
### Space based URLs
| URL | comment |
|-----|---------|
| `https://<host>/#/s/<space_id>(/<relative/path>)(?id=<resource_id>)` | the pattern, relative `path` and `resource_id` are optional |
| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607` | root of a storage space, might be the currently logged in users home |
| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to/resource` | sub folder `/relative/path/to/resource` in the storage with id `b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`, works ***only*** if path still exists |
| `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to/resource?id=ba4c1820-df12-11eb-8dcd-ff21f12c1264:beb78dd6-df12-11eb-a05c-a395505126f6` | sub folder `/relative/path/to/resource` in the storage with id `b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`, lookup can fall back to the `id` |
{{< hint >}}
* `/#` is used by the current vue router.
* `/s` denotes that this is a space url.
* `<space_id>` and `<resource_id>` both consist of `<storage_id>:<node_id>`, but the `space_id` can be replaced with a shorter id or an alias. See further down below.
* `<relative/path>` takes precedence over the `<resource_id>`, both are optional
{{< /hint >}}
* Good, because the web UI does not need to look up the space id in a registry to build an API request for the `/dav/space` endpoint
* Good, because the URLs reveal a relevant path context to users
* Good, because everything after the `#` is not sent to the server, building the webdav request to list the folder is offloaded to the clients
* Good, because power users can navigate by updating the path in the url
* Bad, because the current ids are uuid based, leading to very long URLs where the path component nearly vanishes between two very long strings
* Bad, because the `#` in the URL is just a technical requirement
* Bad, because ocis web requires a `/#/files/s` at the root of the route to distinguish the files app from other apps
* Bad, while navigating using the WebUI, the URL has to be updated whenever we change spaces.
* Bad, because the technical `<space_id>` is meaningless to end users
With the above explained, let's see some use cases:
#### Example 1: UserA shares something from her Home folder with UserB
- open the browser and go to `demo.owncloud.com`
- the browser's url changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`. You're now in YOUR home folder / personal space.
- you create a new folder `/relative/path/to/resource` and navigate into `/relative/path/to`
- the URL now changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607/relative/path/to`
- You share `resource` with some else
- You navigate into `/relative/path/to/resource`
- now the URL would look like: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:3a9305da-df17-11eb-ab99-abe09d93e08a`
As you can see, even if you're the owner of `/relative/path/to/resource` and navigate into it, the URL changes due to a new space being entered. This ensures that while working in your home folder, copying URLs and giving them to the person you share the resource with, the receiver can still navigate within the new space.
In short terms, while navigating using the WebUI, the URL has to constantly change whenever we change spaces to reflect the most explicit one.
#### Example 2: UserA shares something from a Workspace
Assuming we only have one storage provider; a consequence of this, all storage spaces will start with the same storage_id.
- open the browser and go to `demo.owncloud.com`
- the browser's url changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607`. You're now in YOUR home folder / personal space.
- you have access to a workspace called `foo` (created by an admin)
- navigate into workspace `foo`
- the URL now changes to: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74`. You are now at the root of the workspace `foo`.
- because we only have one storage provider, the `space_id` section of the URL only updates the `node_id` part of it.
- had we had more than one storage provider, the `space_id` would depend on which storage provider contains the storage space.
- you create a folder `/relative/path/to/resource`
- you navigate into `/relative/path/to/resource`
- now the URL would look like: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/relative/path/to/resource`
- or a more robust url: `https://demo.owncloud.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/relative/path/to/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3`
#### Spaces Registry
A big drawback against this idea is that the length of the URL is increased by a lot, rendering them almost unreadable. Introducing a Spaces Registry (SR) would shorten them. Let's see how.
A URL without a SR would look like: `https://ocis.com/#/s/b78c2044-5b51-446f-82f6-907a664d089c:d342f9ce-df18-11eb-b319-1b6d9df4bc74/TEST?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3`
The same URL with a SR `https://ocis.com/#/s/workspaceFoo/TEST?id=b78c2044-5b51-446f-82f6-907a664d089c:04f1991c-df19-11eb-9cc7-3b09f04f9ca3`
Space Registry resolution can happen at the client side (i.e: the client keeps a list of space name -> space id [where space id = storageid + nodeid]; the client queries a SR) or server side. Server side is more resilient due to clients can have limited networking; for instance if they are running on a tight intranet.
### Mixed Global URLs
While ID based space URLs can be made more readable by shortening the IDs they only start to reveal context when an alias is used instead of the space id. These aliases however have to be unique identifiers. These aliases should live in namespaces like `/workspaces/marketing` and `/personal/marketing` to make phishing attacks harder (in this case a user that registered with the username `marketing`). But namespaced aliases is semantically equivalent to ... a path hierarchy.
When every space has a namespaced alias and a relative path we can build a global namespace:
| URL | comment |
|-----|---------|
| `https://<host>/files</namespaced/alias></relative/path/to/resource>?id=<resource_id>` | the pattern, `/files` might become optional |
| `https://demo.owncloud.com/files/personal/einstein/?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21607` | root of user `einstein` |
| `https://demo.owncloud.com/files/personal/einstein/relative/path/to/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `/relative/path/to/resource` |
| `https://demo.owncloud.com/files/shares/einstein/somesharename?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | shared URL for `/relative/path/to/resource` |
| `https://demo.owncloud.com/files/personal/einstein/marie is stupid/and richard as well/resource?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | sub folder `marie is stupid/and richard as well/resource` ... something einstein might not want to reveal |
| `https://demo.owncloud.com/files/shares/einstein/resource (2)?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has an appended counter to avoid a collision |
| `https://demo.owncloud.com/files/shares/einstein/mybestfriends?id=b78c2044-5b51-446f-82f6-907a664d089c:194b4a97-597c-4461-ab56-afd4f5a21608` | named link URL for `/marie is stupid/and richard as well/resource`, does not disclose the actual hierarchy, has a custom alias for the share |
| `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` |
| `https://demo.owncloud.com/files/public/kcZVYaXr7oZ66bg/relative/path/to/resource` | sub folder `/relative/path/to/resource` in public link with token `kcZVYaXr7oZ66bg` |
| `https://demo.owncloud.com/s/kcZVYaXr7oZ66bg/` | shortened link to a resource. This is needed to be able to copy a link to a resource without leaking any metadata. |
`</namespaced/alias></relative/path/to/resource>` is the global path in the CS3 api. The CS3 Storage Registry is responsible by managing the mount points.
In order to be able to copy and paste URLs all resources must be uniquely identifiable:
* Instead of `/home` the URL always has to reflect the user: `/personal/einstein`
* Workspaces can use `/workspaces/<alias>` or `/workspaces/<additional>/<classification>/<alias>` where the hierarchy is given by the organization
* Experiments can use `/experiments/<alias>`
* Research institutes could set up `/papers/<researchgroup>/<alias>`
* Trash could be accessed by prefixing the namespace alias with `/trash`? or using `/trash/<space_id>`
* Instead of a namespaced alias a storage space id could be used with a generic `/space/<space_id>` namespace
* An url shortener can create urls like `/s/<token>` which could be used as a stable link to a resource.
* Links for anonymous users will resolve to `/public/<token>`
The alias namespace hierarchy and depth can be pre-determined by the admin. Even if aliases change the `id` parameter prevents bookmarks from breaking. A user can decide to build a different hierarchy by using his own registry.
What about shares? Similar to `/home` it must reflect the user: `/shares/einstein` would list all shares *by* einstein for the currently logged-in user. The ui needs to apply the same URL rewriting as for space based URLs: when navigating into a share the URL has to switch from `/personal/einstein/relative/path/to/shared/resource` to `/shares/einstein/<unique and potentially namespaced alias for shared resource>`. When more than one `resource` was shared a name collision would occur. To prevent this we can use ids `/shares/einstein/id/<resource_id` or namespaced aliases `/shares/einstein/files/alias`. Similar to the `/trash` prefix we could treat `/shares` as a filter for the shared resources a user has access to, but that would disclose unshared path segments in personal spaces. We could make that a feature and let users create an alias for a shared resource, similar as for public links. Then they can decide if they want to disclose the full path in their personal space (or another workspace) or if they want to use an alias which is then accessed at `/shares/einstein/<alias>`. As a default we could take the alias at creation time from the filename. That way two shares to a resource with the same name, e.g.: `/personal/einstein/project AAA/foo` and `/personal/einstein/project BBB/foo` would lead to `/shares/einstein/foo` (a CS3 internal reference to `/personal/einstein/project AAA/foo`) and `/shares/einstein/foo (2)` (a CS3 internal reference to `/personal/einstein/project BBB/foo`). `foo (2)` would keep its name even when `foo` is deleted or renamed. Well an id as the alias might be better then, because users might rename these aliases, which would break URLs if they have been bookmarked. In any case this would make end user more aware of what they share AND it would allow them to choose an arbitrary context for the links they want to send out: personal internal share URLs.
With these different namespaces the `/files` part in the URL becomes obsolete, because the files application can be registered for multiple namespaces: `/personal`, `/workspaces`, `/shares`, `/trash` ...
* Good, because it contains a global path
* Good, because spaces with namespaced aliases can be bookmarked and copied into mails or chat without disclosing unshared path segments, as the space is supposed to be shared
* Good, because the UI can detect broken paths and notify the user to update his bookmark if the resource could be found by `id`
* Good, because the `/files` part might only be required for `id` only based lookup to let the web ui know which app is responsible for the route
* Good, because it turns shares into deliberately named spaces in `/shares/<owner>/<alias>`
* Good, because all urls can be shortened to hide any metadata like path, resource name and query parameters
* Bad, because the web UI needs to look up the space alias in a registry to build an API request for the `/dav/space` endpoint
### Configurable path component in URLs
Not every deployment may have the requirement to have the path in the URL. We could use id only based URLs, similar to onedrive and make showing paths configurable.
| URL | comment |
|-----|---------|
| `https://<host>/files?id=<resource_id>` | default id based navigation |
| `https://<host>/files</namespaced/alias></relative/path/to/resource>?id=<resource_id>` | optional path based navigation with fallback to id |
In contrast to ownCloud 10 path takes precedence and the user is warned when the fileid in his bookmark no longer matches the id on the server: sth. like "The path of the resource has changed, please verify and update your bookmark!"
When a file is selected the filename also becomes part of the URL so individual files can be bookmarked.
If navigation is id based we need to look up the path for the id so we can make a webdav request, or we need to implement the graph drives and driveItem resources.
The URL `https://<host>/files?id=<resource_id>̀` is sent to the server. It has to look up the correct path and redirect the request, including the path. But that would make all bookmarks contain tha path again, even if paths were configured to not be part of the URL.
The `/meta/<fileid>` webdav endpoint can be used to look up the path with property `meta-path-for-user`.
For now, we would use path based navigation with URLs like this:
```
https://<host>/files</namespaced/alias></relative/path/to/resource>?id=<resource_id>
```
This means that only the _resource path_ is part of the URL path. Any other parameter, e.g. file `id`, `page` or sort order must be given as URL parameters.
- [ ] To make lookup by id possible we need to implement the `/meta/<fileid>` endpoint so the sdk can use it to look up the path. We should not implement a redirect on the ocis server side because the same redirect logic would need to be added to oc10. Having it in ocis web is the right place.
- [ ] The old sharing links and oc10 urls still need to be redirected by ocis/reva as in oc10.
Public links would have the same format: `https://<host>/files?id=<resource_id>` The web UI has to detect if the user is logged in or not and adjust the ui accordingly.
{{< hint warning >}}
Since there is no difference between public and private files a logged-in user cannot see the public version of a link unless he logs out.
{{< /hint >}}

49
docs/ocis/adr/0012-tracing.md
View File

@@ -1,49 +0,0 @@
---
title: "12. Tracing"
weight: 12
date: 2021-08-17T12:56:53+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0012-tracing.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@mstingl](https://github.com/mstingl) [@pmaier1](https://github.com/pmaier1), [@fschade](https://github.com/fschade)
* Date: 2021-08-17
## Context and Problem Statement
At the time of this writing we are in a situation where our logs have too much verbosity, rendering impossible or rather difficult to debug an instance. For this reason we are giving some care to our traces by updating dependencies from OpenCensus to OpenTelemetry.
## Decision Drivers
- We don't want to rely only on logs to debug an instance.
- Logs are too verbose.
- Since we have micro-services, we want to holistically understand a request.
## Considered Options
- Trim down logs
- Use OpenCensus
- Migrate to OpenTelemetry
## Decision Outcome
Chosen option: option 3; Migrate to OpenTelemetry. OpenCensus is deprecated, and OpenTelemetry is the merger from OpenCensus and OpenTelemetry and the most recent up-to-date spec.
### Positive Consequences
- Fix the current state of the traces on Reva.
- Add more contextual information on a span for a given request.
- Per-request filtering with the `X-Request-Id` header.
- Group the supported tracing backends to support Jaeger only for simplicity.
## Chosen option approach
- A trace is a tree, and the proxy will create the root trace and propagate it downstream.
- The Root trace will log the request headers.
- The unit that ultimately does the work will log the result of the operation if success.
- The unit that ultimately does the work will change the state of the span to error if any occurred.
With these premises, this is by no means a fixed document and the more we learn about the usage of an instance the more context we can add to the traces.

96
docs/ocis/adr/0013-locking.md
View File

@@ -1,96 +0,0 @@
---
title: "13. Locking"
weight: 13
date: 2021-08-17T12:56:53+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0013-locking.md
---
- Status: accepted
- Deciders: [@hodyroff](https://github.com/hodyroff), [@pmaier1](https://github.com/pmaier1), [@jojowein](https://github.com/jojowein), [@dragotin](https://github.com/dragotin), [@micbar](https://github.com/micbar), [@tbsbdr](https://github.com/tbsbdr), [@wkloucek](https://github.com/wkloucek)
- Date: 2021-11-03
## Context and Problem Statement
At the time of this writing no locking mechanisms exists in oCIS / REVA for both directories and files. The CS3org WOPI server implements a file based locking in order to lock files. This ADR discusses if this approach is ok for the general availability of oCIS or if changes are needed.
## Decision Drivers
- Is the current situation acceptable for the GA
- Is locking needed or can we have oCIS / REVA without locking
## Considered Options
1. File based locking
2. No locking
3. CS3 API locking
## Decision Outcome
For the GA we chose option 2. Therefore we need to remove or disable the file based locking functionality of the CS3org WOPI server. The decision was taken because the current file based locking does not work on file-only shares. The current locking also does not guarantee exclusive access to a file since other parts of oCIS like the WebDAV API or other REVA services don't respect the locks.
After the GA we need to implement option 3.
## Pros and Cons of the Options
### File based locking
The CS3org WOPI server creates a `.sys.wopilock.<filename>.` and `.~lock.<filename>#` file when opening a file in write mode
**File based locking is good**, because:
- it is already implemented in the current CS3org WOPI server
**File based locking is bad**, because:
- lock files should be checked by all parties manipulating files (e.g. the WebDAV api)
- lock files can be deleted by everyone
- you can not lock files in a file-only share (you need a folder share to create a lock file besides the original file)
If we have file based locks, we can also sync them with e.g. the Desktop Client.
**Syncing lock files is good**: because
- native office applications can notice lock files by the WOPI server and vice versa (LibreOffice also creates `.lock.<filename>#` files)
**Syncing lock files is bad**, because:
- if lockfile is not deleted, no one can edit the file
- creating lock files in a folder shared with 2000000 users creates a lot of noise and pressure on the server (etag propagation, therefore oC Desktop sync client has an ignore rule for `.~lock.*` files)
### No locking
We remove or disable the file based locking of the CS3org WOPI server.
**No locking is good**, because:
- you don't need to release locks
- overwriting a file just creates a new version of it
**No locking is bad**, because:
- merging changes from different versions is a pain, since there is no way to calculate differences for most of the files (e.g. docx or xlsx files)
- no locking breaks the WOPI specs, as the CS3 WOPI server won't be capable to honor the WOPI Lock related operations
### CS3 API locking
- Add CS3 API for resource (files, directories) locking, unlocking and checking locks
- locking always with timeout
- lock creation is a "create-if-not-exists" operation
- locks need to have arbitrary metadata (e.g. the CS3 WOPI server is stateless by storing information on / in the locks)
- Implement WebDAV locking using the CS3 API
- Implement Locking in storage drivers
- Change CS3 WOPI server to use CS3 API locking mechanism
- Optional: manual lock / unlock in ownCloud Web (who is allowed to unlock locks of another user?)
**CS3 API locking is good**, because:
- you can lock files on the actual storage (if the storage supports that -> storage driver dependent)
- you can lock files in ownCloud 10 when using the ownCloudSQL storage driver in the migration deployment (but oC10 Collabora / OnlyOffice also need to implement locking, to fully leverage that)
- clients can get the lock information via the api without ignoring / hiding lock file changes
- clients can use the lock information to lock the file in their context (e.g. via some file explorer integration)
**CS3 API locking is bad**, because:
- it needs to be defined and implemented, currently not planned for the GA

63
docs/ocis/adr/0014-microservices-runtime.md
View File

@@ -1,63 +0,0 @@
---
title: "14. Microservices Runtime"
weight: 14
date: 2022-01-21T12:56:53+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0014-microservices-runtime.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@mstingl](https://github.com/mstingl) [@pmaier1](https://github.com/pmaier1), [@fschade](https://github.com/fschade)
* Date: 2022-01-21
## Context and Problem Statement
In an environment where shipping a single binary makes it easier for the end user to use oCIS, embedding a whole family of microservices within a package and running it leveraging the use of the Go language has plenty of value. In such environment, a runtime is necessary to orchestrate the services that run within it. Other solutions are hot right now, such as Kubernetes, but for a single deployment this entails orbital measures.
## Decision Drivers
- Start oCIS microservices with a single command (`ocis server`).
- Clear separation of concerns between services.
- Control the lifecycle of the running services.
- Services can be distributed across multiple machines and still be controllable somehow.
## Considered Options
1.The use of frameworks such as:
- asim/go-micro
- go-kit/kit
2. Build and synchronize all services in-house.
3. A hybrid solution between framework and in-house.
## Options
### go-kit/kit
Pros
- Large community behind
- The creator is a maintainer of Go, so the code quality is quite high.
Cons
- Too verbose. Ultimately too slow to make progress.
- Implementing a service would require defining interfaces and a lot of boilerplate.
### asim/go-micro
Pros
- Implementation based in swappable interfaces.
- Multiple implementations, either in-memory or through external services
- Production ready
- Good compromise between high and low level code.
## Decision Outcome
Number 3: A hybrid solution between framework and in-house.
### Design
{{< figure src="/ocis/static/runtime.drawio.svg" >}}
First of, every ocis service IS a go-micro service, and because go-micro makes use of urfave/cli, a service can be conveniently wrapped inside a subcommand. Writing a supervisor is then a choice. We do use a supervisor to ensure long-running processes and embrace the "let it crash" mentality. The piece we use for this end is called [Suture](https://github.com/thejerf/suture).
The code regarding the runtime can be found pretty isolated [here](https://github.com/owncloud/ocis/blob/d6adb7bee83b58aa3524951ed55872a5f3105568/ocis/pkg/runtime/service/service.go). The runtime itself runs as a service. This is done so messages can be sent to it using the oCIS single binary to control the lifecycle of its services.

100
docs/ocis/adr/0015-events.md
View File

@@ -1,100 +0,0 @@
---
title: "15. oCIS Event System"
weight: 15
date: 2022-02-01T12:56:53+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0015-events.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@c0rby](https://github.com/c0rby), [@wkloucek](https://github.com/wkloucek)
* Date: 2022-01-21
## Context and Problem Statement
### Overview
To be able to implement simple, flexible and independent inter service communication there is the idea to implement an event system in oCIS. A service can send out events which are received by one or more other services. The receiving service can cause different kinds of actions based on the event by utilizing the information that the event carries.
### Example: Email Notification
A simple example is the notification feature for oCIS: Users should receive an email when another user shares a file with them. The information, that the file was shared should go out as an event from a storage provider or share manager, carrying the information which file was shared to which receiver. A potential notification service that sends out the email listens to these kinds of events and sends the email out once on every received event of that specific type.
## Decision Drivers
* Events are supposed to decouple services and raise flexibility, also considering extensions that are not directly controlled by the ownCloud project.
* Events should bring flexibility in the implementation of sending and receiving services.
* Events should not obsolete other mechanisms to communicate, i.e. grpc calls.
* Sending an event has to be as little resource consuming for the sender as possible.
* Events are never user visible.
## Considered Options
1. Lightweight Events with Event Queue and "At most once" QoS
2. As 1., but with "At least once" QoS
## Options
### 1. Lightweight Events with Event Queue and "At most once" QoS
Reva will get a messaging service that is available to all services within oCIS and Reva. It is considered as one of the mandatory services of the oCIS system. If the messaging backend is not running, neither Reva nor oCIS can be considered healthy and should shut down.
All oCIS- and Reva-services can connect to the messaging bus and send so-called events. The sender gets an immediate return if handing the event to the message bus was successful or not.
The sender can not make any assumptions when the message is delivered to any receiving service. Depending on the QoS model (as proposed as alternatives in this ADR) it might even be not guaranteed that the event is delivered at all. Also, the sender can not know if zero, one or many services are listening to that event.
#### Event Data
Events are identified by their namespace and their respective name. The namespace is delimited by dots and starts with either "reva" or "ocis" or a future extension name. It is followed by the name of the sending service and an unique name of the event.
Example: `ocis.ocdav.delete` - an event with that name sent out if an WebDAV DELETE request arrived in the oCDav service.
An event can carry a payload which is encoded as json object. (See for example [NATS](https://docs.nats.io/using-nats/developer/sending/structure) ). There are no pre-defined members in that object, it is fully up to the sender which data will be included in the payload. Receivers must be robust to deal with changes.
#### Quality of Service
Events are sent with "At most once" quality of service. That means, if a receiver is not present at the moment of publishing it might not receive the event. That requires that the sender and the receiver must have functionality to back up the situation that events were missed. That adds more state to the services because they always need to behave like a [FISM](https://en.wikipedia.org/wiki/Finite-state_machine). Given that the event queue can be considered the backbone of the system, it is unlikely that it is not running.
#### Transactions
The described way of inter service communication with events is not transactional. It is not supposed to be, but only provides a lightweight, loosely coupled way to "inform".
If transactions are required, proper synchronous GRPC API calls should be used. Another way would be to build asynchronous flows with request- and reply events as in [saga pattern](https://microservices.io/patterns/data/saga.html). That is only recommended for special cases.
#### Pros
* Simple setup
* Flexible way of connecting services
* Stateless event queue
* "State of the art" pattern in microservices architectures
#### Cons
* Over engineering: Can we do without an extra message queue component?
* Messages might get lost, so that eventual consistency is endangered
* A service needs to hold more state to ensure consistency
* Message queue needs to be implemented in Reva
### 2. Lightweight Events with Event Queue and "At-least once" QoS
Exactly as described above, but with a higher service level quality.
#### Quality of Service
Events are sent with "At least once" quality of service. That means the events will remain in the queue until they are received by all receivers. This puts more responsibility on the event bus and adds state to the events. Given that the event queue can be considered the backbone of the system, it is required to be running.
#### Pros
* Better service level: Messages do not get lost
* Simplifies the design of the microservices because the events are "fire-and-forget"
* Events would be idempotent. If a service goes down the events will stay in the queue until they are consumed
#### Cons
* Stateful event system with higher cost in terms of compute and storage
* The queue could become a bottleneck and needs to be scaled
## Decision Outcome
### Design

90
docs/ocis/adr/0016-files-metadata.md
View File

@@ -1,90 +0,0 @@
---
title: "16. Storage for Files Metadata"
weight: 16
date: 2022-03-02T00:00:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0016-files-metadata.md
---
* Status: superseded by [ADR-0024]({{< ref "0024-msgpack-metadata.md" >}})
* Deciders: [@butonic](https://github.com/butonic), [@dragotin](https://github.com/dragotin), [@micbar](https://github.com/micbar), [@c0rby](https://github.com/c0rby)
* Date: 2022-02-04
## Context and Problem Statement
In addition to the file content we need to store metadata which is attached to a file. Metadata describes additional properties of a file. These properties need to be stored as close as possible to the file content to avoid inconsistencies. Metadata are key to workflows and search. We consider them as an additional value which enhances the file content.
## Decision Drivers
* Metadata will become more important in the future
* Metadata are key to automated data processing
* Metadata storage should be as close as possible to the file content
* Metadata should be always in sync with the file content
## Considered Options
* Database
* Extended file attributes
* Metadata file next to the file content
* Linked metadata in separate file
## Decision Outcome
Chosen option: "Extended File Attributes", because we guarantee the consistency of data and have arbitrary simple storage mechanism.
### Positive Consequences
* Metadata is always attached to the file itself
* We can store arbitrary key/values
* No external dependencies are needed
### Negative consequences
* The storage inside extended file attributes has limits
* Changes to extended attributes are not atomic and need file locks
## Pros and Cons of the Options <!-- optional -->
### Database or Key-Value Store
Use a Database or an external key/value store to persist metadata.
* Good, because it scales well
* Good, because databases provide efficient lookup mechanisms
* Bad, because the file content and the metadata could run out of sync
* Bad, because a storage backup doesn't cover the file metadata
### Extended File Attributes
Extended File Attributes allow storing arbitrary properties. There are 4 namespaces `user`, `system`, `trusted` and `security`. We can safely use the `user` namespace. An example attribute name would be `user.ocis.owner.id`. The linux kernel has length limits on attribute names and values.
From Wikipedia on [Extended file attributes](https://en.wikipedia.org/wiki/Extended_file_attributes#Linux):
> The Linux kernel allows extended attribute to have names of up to 255 bytes and values of up to 64 KiB,[14] as do XFS and ReiserFS, but ext2/3/4 and btrfs impose much smaller limits, requiring all the attributes (names and values) of one file to fit in one “filesystem block” (usually 4 KiB). Per POSIX.1e,[citation needed] the names are required to start with one of security, system, trusted, and user plus a period. This defines the four namespaces of extended attributes.
* Good, because metadata is stored in the filesystem
* Good, because consistency is easy to maintain
* Good, because the data is attached to the file and survives file operations like copy and move
* Good, because a storage backup also covers the file metadata
* Bad, because we could hit the filesystem limit
* Bad, because changes to extended attributes are not atomic
### Metadata File
We could store metadata in a metadata file next to the file content which has a structured content format like .json, .yaml or .toml. That would give us more space to store bigger amounts of metadata.
* Good, because there are no size limits
* Good, because there is more freedom to the content format
* Good, because a storage backup also covers the file metadata
* Bad, because it doubles the amount of read / write operations
* Bad, because it needs additional measures against concurrent overwriting changes
### Link metadata with an id in the extended attributes
To link metadata to file content a single extended attribute with a file id (unique per storage space) is sufficient. This would also allow putting metadata in better suited storage systems like SQLite or a key value store.
* Good, because it avoids extended attribute limits
* Good, because the same mechanism could be used to look up files by id, when the underlying filesystem is an existing POSIX filesystem.
* Bad, because backup needs to cover the metadata as well. Could be mitigated by sharing metadata per space and doing space wide snapshots.
* Bad, because it is a bit more effort to access it to read or index it.

111
docs/ocis/adr/0017-allow-read-only-external-user-management.md
View File

@@ -1,111 +0,0 @@
---
title: "17. Allow read only external User Management"
weight: 17
date: 2022-02-08T10:53:00+01:00
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0017-allow-read-only-external-user-management.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@rhafer](https://github.com/rhafer)
* Date: 2022-02-08
## Context and Problem Statement
oCIS needs to be integrated with various external Authentication and Identity Management Systems.
Usually oCIS will have no administrative access to such a system and we will not be allowed to
reconfigure it to suit our needs (e.g. we will not be able to enhance the schema of an already existing
LDAP Directory). In most of the cases our access will be read-only.
Sidenote: There is a difference between users, identities and accounts: A user may have multiple
identities which he can authenticate with, e.g. his facebook, twitter, microsoft or google
identity. Multiple identities can be linked to an account in ocis, allowing to fall back to another
identity provider should one of them shut down. This also allows migrating from one identity
provider to another.
There are different cases where oCIS requires access to users:
1. While we settled on using OpenID Connect (OIDC) as the authentication protocol for oCIS, we
need to build a user object during authentication with at least an account UUID (to identify
the account) and the email or a name (for display purposes).
2. When searching for share recipients we need to be able to query existing users in the external
identity management system
3. When listing files we need to be able to look up a users display properties (username, email,
avatar...) based on the account UUID
oCIS internally relies on a stable and persistent identifier (e.g. a UUID) for accounts in order to
implement permissions and sharing. Unfortunately, some deployments are unable to deliver this kind
of stable identifier for users:
- In OIDC itself the only stable identifier that is guaranteed to be provided by the IDP is
combination of the sub and iss claims. IDPs can optionally return other claims, but we cannot
rely on a specific claim being present.
- When no other services (LDAP, SCIM, ...) are available that could be used look up a user UUID
## Decision Drivers
* oCIS should be a single binary that can run out of the box without external dependencies like an
LDAP server.
* Time: we want to build a release candidate asap.
* oCIS should be easy to integrate with standard external identity management systems
## Considered Options
There are two case to consider:
* External identity management system provides an OIDC IdP and an interface to query users
* External identity management system provides just an OIDC IdP with no possibility to query users
## Decision Outcome
It's not really possible single out any of the options for this ADR. In the end we will likely need
to support both scenarios.
### Positive Consequences: <!-- optional -->
* Very flexible integration with a wide range of external systems
### Negative consequences: <!-- optional -->
* configuration complexity, high support efforts
* Increasingly complex code in oCIS
## Pros and Cons of the Options <!-- optional -->
### External identity management system is read only and provides an interface to query users (e.g. Corporate Active Directory)
IdP sends sub & iss and mail or username claims, Identity Management System provides APIs (e.g.
LDAP, SCIM, REST ...) to lookup additional user information. All oCIS services use the CS3 API to
look up the account for the given email or username, where CS3 then uses a backend that relies on
the APIs provided by the IdM.
* Good, because we can rely on the external identity management
* Good, because ocis services only need to know about the CS3 user provider API, which acts as an
abstraction layer for different identity management systems
* Good, because there is only a single source of truth (the external IdM) and we don't need to
implement a synchronization mechanism to maintain an internal user database (we will likely need
some form of caching though, see below)
* Bad, because the identity management needs to provide a stable, persistent, non-reassignable user
identifier for an account, e.g. `owncloudUUID` or `ms-DS-ConsistencyGuid`
* Bad, because we need to implement tools that can change the account id when it did change anyway
* Bad, because without caching we will hammer the identity management system with lookup requests
### External identity management system is read only and does NOT provide an API to query users
Idp sends sub & iss and mail or username claims. We need to provision an internal account mapping,
creating a unique ID, upon the first login of a user to be able to look up user properties by account
id.
* Good, because this has very little external requirements
* Good, because we have accounts fully under our control
* Bad, because we have to provide the user lookup APIs
* Bad, because users will only a visible after the first login
* Bad, because our internal account mapping might get out of date when user attribute (e.g. name or
mail) change. At least until the next time that user logs in
## Links <!-- optional -->
* [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
* … <!-- numbers of links can vary -->
* supersedes [3. Use external User Management]({{< ref "0003-external-user-management.md" >}})

61
docs/ocis/adr/0018-file-search-api.md
View File

@@ -1,61 +0,0 @@
---
title: "18. File Search API"
date: 2022-03-18T09:00:00+01:00
weight: 18
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0018-file-search-api.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@c0rby](https://github.com/c0rby)
* Date: 2022-03-18
## Context and Problem Statement
The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale.
## Decision Drivers
* Have a simple yet powerful, scalable and performant way of finding files in oCIS
* Be able to construct intelligent searches based on metadata
* Allow the user to filter the search queries based on metadata
## Considered Options
* [Libre Graph API](#libre-graph-api)
* [WebDAV API](#webdav-api)
## Decision Outcome
Chosen option: [WebDAV API](#webdav-api) because the current WebUI is compatible with that API. We may use the GraphAPI later in a second iteration.
### Positive Consequences
* The existing Clients can continue to use the well-known API
* There are existing API tests which cover the basic behavior
### Negative consequences
* We have no server side result filtering capabilities
## Pros and Cons of the Options
### Libre Graph API
* Good, because we try to switch most of our HTTP requests to Libre Graph
* Good, because the Graph API supports scopes, sorting and query language
* Good, because it supports server side result filtering
* Bad, because there are currently no clients which support that
### WebDAV API
* Good, because WebDAV is a well-known and widely adopted Standard
* Good, because existing Clients continue to work without extra efforts
* Bad, because the syntax is limited
* Bad, because we cannot do server side result filtering
## Links
* [Search Indexing](0019-file-search-index.md)
* [Search Query Language](0020-file-search-query-language.md)

80
docs/ocis/adr/0019-file-search-index.md
View File

@@ -1,80 +0,0 @@
---
title: "19. File Search Index"
date: 2022-03-18T09:00:00+01:00
weight: 19
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0019-file-search-index.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@c0rby](https://github.com/c0rby)
* Date: 2022-03-18
## Context and Problem Statement
The ability to find files based on certain search terms is a key requirement for a system that provides the ability to store unstructured data on a large scale.
More sophisticated search capabilities are expected and can be implemented, especially based on metadata.
To trigger the indexing of a file, the search service listens to create, update and delete events on the internal event bus of oCIS.
The events need to contain a valid reference that defines the file space and file id of the file in question. The event only must be sent when the file operation (update, creation, removal) is finished.
Sharing adds more complexity because the index also needs to react to create, delete and modify shares events. Sharing should not duplicate the indexed data, especially within spaces or group shares.
## Decision Drivers
* Have a simple yet powerful, scalable and performant way of finding files in oCIS
* Be able to construct intelligent searches based on metadata
* Allow the user to filter the search queries based on metadata
* Basic File Search needs to be implemented out of the box without external dependencies
* The Search Indexing Service should be replaceable with more sophisticated technologies like Elasticsearch
* Make use of the spaces architecture to shard search indexes by space
* The Search Indexing Service needs to deal with multiple users accessing the same resources due to shares
* The Search Service should be compatible with different search indexing technologies
## Considered Options
* [Bleve Search](#bleve-search)
* [Elastic Search](#elastic-search)
## Decision Outcome
Chosen option: Bleve Search, because we can fulfill the MVP and include it into the single binary.
### Positive Consequences
* Basic File Search works out of the box
* We do not need heavy external dependencies which need to be deployed alongside
### Negative consequences
* We need to be aware of the scaling limits
* We need to find a way to work with shares and spaces
* It has a limited query language
## Pros and Cons of the Options
### Bleve Search
* Good, because it is written in GoLang and can be bundled into the single oCIS binary
* Good, because it is a lightweight but powerful solution which could fulfill a lot of use cases
* Bad, because we do not know exactly how we can represent shares in the index without duplicating data
* Bad, because it is a single process
* Bad, because the query language is limited
### Elastic Search
* Good, because it has become an industry standard
* Good, because it supports a rich query language
* Good, because it has built in cluster support and scales well
* Good, because it has a permission system and supports multiple users and groups to access the same resource
* Bad, because it is a heavy setup and needs extra effort and knowledge
## Links
* [Search API](0018-file-search-api.md)
* [Search Query Language](0020-file-search-query-language.md)
* [Bleve Search on GitHub](https://github.com/blevesearch/bleve)
* [ElasticSearch](https://www.elastic.co/elastic-stack/)

110
docs/ocis/adr/0020-file-search-query-language.md
View File

@@ -1,110 +0,0 @@
---
title: "20. File Search Query Language"
date: 2022-06-23T09:00:00+01:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0020-file-search-query-language.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin), [@c0rby](https://github.com/c0rby), [@kulmann](https://github.com/kulmann), [@felix-schwarz](https://github.com/felix-schwarz), [@JammingBen](https://github.com/JammingBen)
* Date: 2023-06-23
## Context and Problem Statement
From the users perspective, the interface to search is just a single form field where the user enters one or more search terms. The minimum expectation is that the search returns file names and links to files that:
* have a file name that contains at least one of the search terms
* contain at least one of the search terms in the file contents
* have metadata that is equal or contains one of the search terms
## Decision Drivers
* The standard user should not be bothered by a query syntax
* The power user should also be able to narrow his search with an efficient and flexible syntax
* We need to consider different backend technologies which we need to access through an abstraction layer
* Using different indexing systems should lead to a slightly different feature set without changing the syntax completely
## Considered Options
* [KQL - Keyword Query Language](#keyword-query-language)
* [Simple Query](#simplified-query)
* [Lucene Query Language](#lucene-query-language)
* [Solr Query Language](#solr-query-language)
* [Elasticsearch Query Language](#elasticsearch-query-language)
## Decision Outcome
Chosen option: [KQL - Keyword Query Language](#keyword-query-language), because it enables advanced search across all platforms.
### Positive Consequences
* We can use the same query language in all clients
### Negative consequences
* We need to build and maintain a backend connector
## Pros and Cons of the Options
### Keyword Query Language
The Keyword Query Language (KQL) is used by Microsoft Share Point and other Microsoft Services. It uses very simple query elements, property restrictions and operators.
* Good, because we can fulfill all our current needs
* Good, because it is very similar to the used query language in iOS
* Good, because it supports date time keywords like "today", "this week" and more
* Good, because it can be easily extended to use "shortcuts" for eg. document types like `:presentation` which combine multiple mime types.
* Good, because it is successfully implemented and used in similar use cases
* Good, because it gives our clients the freedom to always use the same query language across all platforms
* Good, because Microsoft Graph API is using it, we will have an easy transition in the future
* Bad, because we need to build and maintain a connector to different search backends (bleve, elasticsearch or others)
### Simplified Query
Implement a very simple search approach: Return all files which contain at least one of the keywords in their name, path, alias or selected metadata.
* Good, because that covers 80% of the users needs
* Good, because it is very straightforward
* Good, because it is a suitable solution for GA
* Bad, because it is below the industry standard
* Bad, because it only provides one search query
### Lucene Query Language
The Lucene Query Parser syntax supports advanced queries like term, phrase, wildcard, fuzzy search, proximity search, regular expressions, boosting, boolean operators and grouping. It is a well known query syntax used by the Apache Lucene Project. Popular Platforms like Wikipedia are using Lucene or Solr, which is the successor of Lucene
* Good, because it is a well documented and powerful syntax
* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility
* Bad, because there is no powerful and well tested query parser for golang available
* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle"
### Solr Query Language
Solr is highly reliable, scalable and fault-tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites.
* Good, because it is a well documented and powerful syntax
* Good, because it is very close to the Elasticsearch and the Lucene syntax which enhances compatibility
* Good, because it has a strong community with large resources and knowledge
* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle"
### Elasticsearch Query Language
Elasticsearch provides a full Query DSL (Domain Specific Language) based on JSON to define queries. Think of the Query DSL as an AST (Abstract Syntax Tree) of queries, consisting of two types of clauses. It is able to combine multiple query types into compound queries. It is also a successor of Solr.
* Good, because it is a well documented and powerful syntax
* Good, because it is very close to the Elasticsearch and the Solr syntax which enhances compatibility
* Good, because there is a stable and well tested go client which brings a query builder
* Good, because it could be used as the query language which supports different search backends by just implementing what is needed for our use-case
* Bad, because it adds complexity and fulfilling all the different query use-cases can be an "uphill battle"
## Links
* [Search API](0018-file-search-api.md)
* [Search Indexing](0019-file-search-index.md)
* [KQL](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference)
* [Apache Lucene](https://lucene.apache.org/)
* [Apache Solr](https://solr.apache.org/)
* [Elastic Search](https://solr.apache.org/)
* [Elastic Search for go](https://github.com/elastic/go-elasticsearch)

95
docs/ocis/adr/0021-service-accounts.md
View File

@@ -1,95 +0,0 @@
---
title: "21. Service accounts"
date: 2023-01-18T16:07:00+01:00
weight: 21
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0021-service-accounts.md
---
* Status: proposed
* Deciders: [@butonic](https://github.com/butonic), [@c0rby](https://github.com/c0rby)
* Date: 2023-01-18
## Context and Problem Statement
There are three levels of security checks in a microservice web application that uses OpenID Connect:
1. **scope claims** limit the possible operations to what the user (or admin on behalf of the organization) consented to
2. **service authorization** limit the possible operations to what specific services are allowed to do, on behalf of users or even without them
3. **permission checks** limit the possible operations to the relationships between subject, permission and resource allow
This ADR deals with a prerequisite for service authorization: service accounts.
Some services need access to file content without a user being logged in. We currently pass the owner or manager
of a space in events which allows the search service to impersonate that user to extract metadata from the changed resource.
There are two problems with this:
1. The service could get all permissions of the user and gain write permission
2. There is a race condition where the user in the event might no longer have read permission, causing the index to go stale
The race condition will become more of an issue when we start working on a workflow engine.
How can we grant services the least amount of permissions required for their purpose?
## Decision Drivers
* It should be possible to represent this as servicePrincipals in the libregraph API, similar to the [MS Graph servicePrincipal](https://learn.microsoft.com/en-us/graph/api/resources/serviceprincipal?view=graph-rest-1.0).
* Services should check permissions using the oCIS permissions or reva auth service, we don't want to introduce a new mechanism for this
## Considered Options
* [Service Accounts](#service-accounts)
* [Impersonate Space-Owners](#impersonate-space-owners)
## Decision Outcome
Chosen option: [Service Accounts](#service-accounts)
### Consequences
* Good, because it allows provisioning permissions for services
* Good, because it uses existing CS3 concepts
* Good, because it uses the existing permissions service
* Good, because it can be mapped to libre graph permissions
* Bad, because we have to make the reva auth manager aware of CS3 [`USER_TYPE_SERVICE`](https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserType)
* Bad, because we have to provision and manage service accounts on init
* Bad, because external APIs may need to filter out service accounts
* Bad, because we need to persist service accounts in addition to normal user accounts
## Pros and Cons of the Options
### Service Accounts
Make the reva auth manager and registry aware of CS3 users of type [`USER_TYPE_SERVICE`](https://cs3org.github.io/cs3apis/#cs3.identity.user.v1beta1.UserType). Then we can provision service accounts at oCIS initialization and use the permissions service to check permissions.
When assigning permissions we use the permission constraints to define the scope of permissions, see [Permission Checks](#permission-checks) for more details.
To authenticate service accounts the static reva auth registry needs to be configured with a new auth provider for type `service`. The actual provider can use a plain JSON file or JSONCS3 that is provisioned once with `ocis init`. TODO Furthermore, the user provider needs to be able to return users for service accounts.
* Good, because we could replace machine auth with specific service accounts and no longer have to distribute a shared secret everywhere
* Bad, because we don't know if a there are places in the code that try to look up a user with USER_TYPE_SERVICE at the cs3 users service ... they might not exist there ... or do we have to implement a userregistry, similar to the authregistry?
* Bad, because we have to provision and manage service accounts on init
* Bad, because we have to write code to manage service accounts or at least filter them out in the admin ui
### Impersonate Space-Owners
We could implement a new auth manager that can authenticate space owners, a CS3 user type we introduced for project spaces which 'have no owner', only one or more managers.
* Good, because it reuses the space owner user type
* Bad, because the space owner always has write permission
* Bad, because we don't know if a there are places in the code that try to look up a user with USER_TYPE_SPACE_OWNER at the cs3 users service ... they might not exist there ... or do we have to implement a userregistry, similar to the authregistry?
* Bad, because it feels like another hack and does not protect against compromised services that try to execute operations that the user did not consent to.
## Links
* [MS Graph servicePrincipal](https://learn.microsoft.com/en-us/graph/api/resources/serviceprincipal?view=graph-rest-1.0)
* [reva auth managers](https://reva.link/docs/config/packages/auth/manager/) - lacks docs for `auth_machine`, to be found [in the code](https://github.com/cs3org/reva/blob/edge/pkg/auth/manager/machine/machine.go)
## Permission checks
When checking permissions we do not check for global permissions but for the concrete permission. Global permissions describe permissions that are used when assigning permissions, e.g. the index service account has the read permission constrained to tenant. The concrete permission check always contains a resource and a specific permission like `Resource.Read` or `Space.Delete`. That we currently check if a user has the `delete-all-spaces` permission is wrong. It should instead check if the user has the permission `Space.Delete` on a specific space. The permissions service can implement the check by taking the permission constraint into account.
Another example would be a `Resource.Read` check for a specific resource. Normal users like the demo users Einstein and Marie would have the permission `Resource.ReadWrite` with the constraint ALL (which limits them to all files they own and that have been shared with them). The permissions service can return true. Service accounts like the indexer would have `Resource.Read` with the constraint TENANT and thus be granted read access to all resources.
In the storage drive implementation we can check the ACLs first (which would allow service accounts that are known to the underlying storage system, e.g. EOS to access the resource) and then make a call to the permissions service. At least for the Read Resource permission. Other permission checks can be introduced as needed.
The permission names and constraints are different from the MS Graph API. Giving permission like [`Files.ReadWrite.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#user-permissions) a different meaning, depending on the type of user (for normal users it means all files they have access to, for service accounts it means all files in the organization) is a source of confusion which only gets worse when there are two different UUIDs for this.

87
docs/ocis/adr/0022-sharing-and-space-management-api.md
View File

@@ -1,87 +0,0 @@
---
title: "22. Sharing and Space Management API"
date: 2023-09-08T02:29:00+01:00
weight: 22
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0022-sharing-and-space-management-api.md
---
* Status: accepted
* Deciders: [@JammingBen](https://github.com/JammingBen) [@butonic](https://github.com/butonic) [@theonering](https://github.com/theonering) [@kobergj](https://github.com/kobergj) [@micbar](https://github.com/micbar)
* Date: 2023-08-08
Technical Story: [Public issue](https://github.com/owncloud/ocis/issues/6993)
## Context and Problem Statement
In the early days of the rewrite of ownCloud it was an important goal to keep all important APIs compatible with ownCloud 10. Infinite Scale embraced that goal until version 1.0.0.
After that first release, the focus changed.
Infinite Scale started the spaces feature which brings a whole new set of APIs and concepts. We made the conscious decision to keep the sharing API as it was, live with its shortcomings and create workarounds to support spaces. We have come a long way so far. Now we need to move on. The Web Client has made the decision to drop the support of ownCloud 10 and keep version 7.0 alive for ownCloud 10 to keep the easy migration path intact.
The desktop and mobile client platforms were suffering from poor support from the server and can now move forward with a new API implementation. By using openApi 3 and all the needed tooling around it developing the LibreGraph specification, documentaion and SDKs, we now feel confident to move on.
## Decision Drivers
* The Path based nature of the OCS API lacks spaces support
* The permissions bitmask is no longer working when using sharing roles
* We want to support server announced sharing roles which are different per instance or scope
* We need to get rid of the currently hardcoded sharing roles in our clients
* New sharing roles and permissions are needed to support secure view and other new features
* Space Memberships are not shares and need to have different semantics
* Elevation of permissions in subfolders or full denials should be possible without creating a new share
* Third party integrations need generated SDKs in different languages to speed up the development
## Considered Options
* [New OCS Api Version](#new-ocs-api-version)
* [Sharing via LibreGraph](#sharing-via-libregraph)
## Decision Outcome
Chosen option: "[LibreGraph](#sharing-via-libregraph)"
### Positive Consequences:
* We can create a new clean API which fits the spaces concept
* LibreGraph embraces OData which is a known API pattern
* Sharing will be integrated in the existing SDKs and documentation
* Removing the OCS Api reduces complexity
* Removing the OCS Api makes the clients codebases smaller and removes manually maintained parts of the SDKs
* The extra error handling for the OCS API can be dropped from our clients
### Negative Consequences:
* We need to deprecate and remove the OCS API
* Existing third party integrations need to do some refactoring
## Pros and Cons of the Options
### New OCS Api Version
To overcome the limitations of the OCS 2.0 API we could create a new major version with the spaces concept in mind. This would give us the opportunity to create a new openApi Spec.
* Good, because the workarounds from version 2.0 could be dropped
* Bad, because we would need to deprecate the version 2.0
* Bad, because we would need to maintain a separate specification / repository
* Bad, because it would create the need to use two different SDKs in our clients
* Bad, because we would need to implement query parameters and filters on our own
* Bad, because sharing information could not be included in the spaces API via queries or filters
### Sharing via LibreGraph
Integrate Sharing into the [LibreGraph API](https://github.com/owncloud/libre-graph-api) by using the already existing toolchain and documentation flows.
* Good, because that reduces the number of SDKs
* Good, because it reduces the number of APIs
* Good, because spaces and shares can be used together in queries and filters
* Good, because we would use the existing OData pattern
* Bad, because we need to deprecate the OCS API
## Links <!-- optional -->
* [LibreGraph API](https://github.com/owncloud/libre-graph-api)
* [OData](https://www.odata.org/documentation/)
* [OpenAPI Standard](https://www.openapis.org/)

79
docs/ocis/adr/0023-index-and-store-metadata.md
View File

@@ -1,79 +0,0 @@
---
title: "23. Index and store metadata"
date: 2023-10-17T15:15:00+01:00
weight: 23
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0023-index-and-store-metadata.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@theonering](https://github.com/dschmidt), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin)
* Date: 2023-10-17
## Context and Problem Statement
ownCloud Infinite Scale is supposed to become a data platform and as such it needs to provide access to metadata.
Currently only metadata common to all file types (filesize, mime-type, ...) is stored in the index and the metadata storage.
We want to make other file type specific metadata available to consumers of our internal and external APIs.
Simple examples would be audio metadata like artist, album and title or exif metadata in images.
## Decision Drivers <!-- optional -->
## Considered Options
* [Store subset of extracted metadata required for graph api](#store-subset-of-extracted-metadata-required-for-graph-api)
* [Store subset of extracted metadata specified by another standard](#store-subset-of-extracted-metadata-specified-by-another-standard)
* [Store everything from extractors](#store-everything-from-extractors)
## Decision Outcome
Chosen option: "[store only subset of extracted metadata required for graph api](#store-subset-of-extracted-metadata-required-for-graph-api)", because Graph API is a simple common denominator and we want to avoid putting the complexity of mapping non-standardized data from potentially different extractors in several areas of the code base. Storage and index keys are determined by facet and property name, e.g. `audio.artist` for the artist in a music file. Storage keys are additionally prefixed with `libre.graph.`, i.e. `libre.graph.audio.artist`.
Handling Graph API specific metadata is a first step towards handling metadata. More generic and extensible handling of arbitrary metadata can be added later.
### Positive Consequences:
* Graph API endpoint implementation is trivial
* Documented public api and stored data are the same
* Reasonable complexity for the initial implementation
### Negative Consequences:
* Graph API is limited, so not *all* available metadata can be accessed
* Switching the internal format and adding more metadata later will require re-indexing
## Pros and Cons of the Options <!-- optional -->
### Store Subset of Extracted Metadata Required for Graph API
Use Graph API facets and properties for determining the subset of stored metadata and the storage key.
The index key for the `artist` property of the `audio` facet is `audio.artist`, the storage key is additionally prefixed with `libre.graph.`.
* Good, because central mapping of values happens consistently and only once in a central place
- it happens in the extractor (integration) which likely knows best how to map metadata to standard properties
* Good, because when multiple extractors share a common set of provided values, applications can rely on the mapping and the complexity is kept low
* Bad, because not all metadata is available, not everything can be searched
* Good, because Graph API already chose a reasonable subset of most interesting properties
### Store Subset of Extracted Metadata Specified by Another Standard
There are a bunch of metadata standards but none of them is really universal. There is always something that is only supported in one or the other standard. Tika for example extracts audio metadata using a mixture of Dublin Core and XMP Dynamic Media keys.
- Bad, because it makes implementing a new extractor integration harder
- Bad, because it makes using the stored data more complicated than a simple standard like discussed above
### Store Everything from Extractors
- Good, because all metadata is available and searchable
- Good, because consuming applications can decide how to map data
- Good, because extractor implementation becomes more trivial
- Bad, because all applications become dependent on the extractor and need to handle different extractors on their own
## Links <!-- optional -->
* https://github.com/owncloud/libre-graph-api/pull/120 / https://learn.microsoft.com/de-de/graph/api/resources/audio?view=graph-rest-1.0
* https://github.com/owncloud/libre-graph-api/pull/122 / https://learn.microsoft.com/en-us/graph/api/resources/photo?view=graph-rest-1.0
* https://github.com/owncloud/libre-graph-api/pull/123 / https://learn.microsoft.com/en-us/graph/api/resources/geoCoordinates?view=graph-rest-1.0
* https://developer.adobe.com/xmp/docs/XMPNamespaces/xmpDM/
* https://www.dublincore.org/schemas/

83
docs/ocis/adr/0024-msgpack-metadata.md
View File

@@ -1,83 +0,0 @@
---
title: "24. Messagepack metadata"
date: 2024-02-09T14:57:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0024-msgpack-metadata.md
---
* Status: accepted
* Deciders: [@butonic](https://github.com/butonic), [@aduffeck](https://github.com/aduffeck), [@micbar](https://github.com/micbar), [@dragotin](https://github.com/dragotin)
* Date: [2023-03-15](https://github.com/cs3org/reva/pull/3711/commits/204253eee9dbb8e7fa93a01f3f94a2d28ce40a06)
## Context and Problem Statement
File metadata management is an important aspect for oCIS as a data platform. While using extended attributes to store metadata allows attaching the metadata to the actual file it causes a significant amount of syscalls that outweigh the benefits. Furthermore, filesystems are subject to different limitations in the number of extended attributes or the value size that is available.
## Decision Drivers <!-- optional -->
Performance of reading extended attributes suffers from the syscall overhead when listing and reading all attributes. Getting rid of limitations imposed by the filesystem used to store decomposedfs metadata.
## Considered Options
Going back to the original [ADR-0016 Storage for Files Metadata]({{< ref "0016-files-metadata.md" >}}) we decided to use a dedicated file for metadata storage next to the decomposedfs file representing the node. Several options for the data format were considered:
* Use JSON files to store metadata
* Use INI files to store metadata
* Use msgpack files to store metadata
* Use protobuf messages to store metadata
## Decision Outcome
Chosen option: "[msgpack files](#msgpack-files)", because we want to stay with a self describing binary format. This is a performance tradeoff that is faster and more efficient than text based formats and more flexible but less efficient than protobuf.
Note: directory listings are still read from the storage and remain uncached.
### Positive Consequences:
* Way less syscalls
* Node metadata can easily be cached, avoiding all trips to the storage until a file changes.
### Negative Consequences:
* We need to migrate existing metadata
* We need to build tooling that allows manipulating metadata similar to `setfattr` and `getfattr`.
## Pros and Cons of the Options <!-- optional -->
### Ini files
* Good, human readable
* Good, self describing
* Good, widely used and well understood
* Good, suited for key value like content - exactly what we need for extended attributes
* Bad, slower and less efficient than binary formats
### JSON files
* Good, human readable
* Good, self describing
* Good, widely used and well understood
* Good, could be used for more than just key value
* Bad, slower and less efficient than binary formats
### Msgpack files
* Good, self describing
* Good, efficient because it is binary encoded
* Good, could be used for more than just key value
* Bad, not human readable - requires tooling to manipulate safely
### protobuf files
* Good, very efficient because it is binary encoded
* Good, could be used for more than just key value
* Bad, not human readable
* Bad, not self describing - requires tooling to evolve the messages
## Links <!-- optional -->
* supersedes [ADR-0016 Storage for Files Metadata]({{< ref "0016-files-metadata.md" >}})
* [The need for speed — Experimenting with message serialization](https://medium.com/@hugovs/the-need-for-speed-experimenting-with-message-serialization-93d7562b16e4)

76
docs/ocis/adr/0025-distributed-search-index.md
View File

@@ -1,76 +0,0 @@
---
title: "25. Distributed Search Index"
date: 2024-02-09T16:27:00+01:00
weight: 25
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0025-distributed-search-index.md
---
* Status: draft
* Deciders: [@butonic](https://github.com/butonic), [@fschade](https://github.com/fschade), [@aduffeck](https://github.com/aduffeck)
* Date: 2024-02-09
## Context and Problem Statement
Search is currently implemented with [blevesearch](https://github.com/blevesearch/bleve), which internally uses bbolt. bbolt writes to a local file, which prevents scaling out the service.
The initial implementation used a single blevesearch index for all spaces. While this makes querying all spaces easy because the results do not need to be aggregated from multiple indexes, the single node becomes a bottleneck when answering search queries. Furthermore, indexing is also part of the search service and has to share the resources.
## Decision Drivers <!-- optional -->
* Indexing should be decoupled from the search service
* The search service should be able to scale horizontally
* The solution needs to be embeddable in the single binary
## Considered Options
* one index per space
* [elasticsearch](https://github.com/elastic/elasticsearch) (java)
* [dgraph](https://github.com/dgraph-io/dgraph) (go)
* [manticore](https://github.com/manticoresoftware/manticoresearch/) (C++)
* [meilisearch](https://github.com/meilisearch/meilisearch) (Rust)
## Decision Outcome
Chosen option: *???*
### Positive Consequences:
* TODO
### Negative Consequences:
* TODO
## Pros and Cons of the Options <!-- optional -->
### one index per space
Instead of using a single index (current implementation) or a distributed search index like elasticsearch the search service should aggregate queries from dedicated indexes per space. The api to a space index provider should be able to take multiple space ids in the request, similar to how a storage provider can handle multiple spaces. When treating spaces and the corresponding search index to belong together we can also treat them as a single unit for backup and restore. In federated deployments we can send the search queries to all search providers / spaces that the user has access to.
How a search provider is implemented then depends on the requirements. For a single node deployment bleve might be fine, for a kubernetes deployment a dedicated service might be the better fit.
### elasticsearch
* Good, commercial support available at https://www.elastic.co/de/pricing
* Good, industry standard
* Bad, nobody seems to like it
* Bad, not embeddable (Java)
### dgraph
* Good, commercial support available at https://dgraph.io/pricing
* Good, embeddable? (go) - TODO verify
### manticore
* Good, commercial support available at https://manticoresearch.com/services/
* Bad, not embeddable (C++)
### meilisearch
* Good, commercial support available at https://www.meilisearch.com/pricing
* Bad, not embeddable (Rust)
## Links <!-- optional -->
* supersedes [ADR-0019 File Search Index]({{< ref "0019-file-search-index.md" >}})

93
docs/ocis/adr/0026-application-based-user-settings.md
View File

@@ -1,93 +0,0 @@
---
title: "26. Application based user settings"
date: 2024-02-09T17:30:00+01:00
weight: 26
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0026-application-based-user-settings.md
---
* Status: draft
* Deciders: [@butonic](https://github.com/butonic), [@fschade](https://github.com/fschade), [@kulmann](https://github.com/kulmann)
* Date: 2024-02-09
## Context and Problem Statement
To share user settings across devices applications want to store user specific settings on the server. The ePUB app wants to remember which page the user is on. The iOS app wants to rember search queries. The Caldav app needs a space to store data.
## Decision Drivers <!-- optional -->
## Considered Options
* OCS provisioning API
* settings service
* libregraph API
## Decision Outcome
Chosen option: *???*
### Positive Consequences:
* TODO
### Negative Consequences:
* TODO
## Pros and Cons of the Options <!-- optional -->
### OCS provisioning API
Nextcloud added a `/ocs/v2.php/apps/provisioning_api/api/v1/config/users/{appId}/{configKey}` endpoint
* Bad, legacy API we want to get rid of
### settings service
- Bad, yet another API. Always uses POST requests.
### libregraph API
The MS Graph API has [a special approot driveItem](https://learn.microsoft.com/en-us/graph/api/drive-get-specialfolder?view=graph-rest-1.0&tabs=http) that apps can use to store arbitrary files. See also:
[Using an App Folder to store user content without access to all files](https://learn.microsoft.com/en-us/onedrive/developer/rest-api/concepts/special-folders-appfolder?view=odsp-graph-online) and a blog post with the section [Store data in the application’s personal folder](https://blog.mastykarz.nl/easiest-store-user-settings-microsoft-365-app/#store-data-in-the-applications-personal-folder).
It basically uses the `/me/drive/special/approot:/{filename}` endpoint to
```http
PUT https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json:/content
content-type: text/plain
authorization: Bearer abc
{"key": "value"}
```
or
```http
GET https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json:/content
authorization: Bearer abc
```
On single page apps you need two requests:
```http
GET https://graph.microsoft.com/v1.0/me/drive/special/approot:/settings.json?select=@microsoft.graph.downloadUrl
authorization: Bearer abc
```
followed by
```http
GET <url from the response['@microsoft.graph.downloadUrl'] property>
```
Currently, applications have no dedicated tokens that we could use to derive the `appid` from. All apps should have an `appid` and [be discoverable under](https://learn.microsoft.com/en-us/graph/api/application-list?view=graph-rest-1.0&tabs=http)
```http
GET /applications
```
In any case for libregraph we could introduce a `LIBRE_GRAPH_APPID` header to make these requests possible rather soon.
Then we can decide if we want to store these files in the users personal drive, or if we create a space for every app that then uses the userid as a folder that contains all the files for the user.
- Good, because clients can remain in libregraph API land
- Bad, we currently have no application tokens
## Links <!-- optional -->

94
docs/ocis/adr/0027-new-share-jail.md
View File

@@ -1,94 +0,0 @@
---
title: "27. New Share Jail"
date: 2024-02-21T15:19:00+01:00
weight: 27
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0027-new-share-jail.md
---
* Status: draft
* Deciders: [@butonic](https://github.com/butonic), [@rhafer](https://github.com/rhafer), [@dragotin](https://github.com/dragotin)
* Date: 2024-02-21
## Context and Problem Statement
The oCIS share jail is a space that contains all accepted / synced shares of a user. In contrast to a personal or project space that contains actual resources, the share jail space only contains references pointing to shared resources. The root directory only consists of mountpoints that actually represent resources in other spaces. On the WebDAV API clients expect an `oc:fileid` property to identify resources in other API endpoints, eg. the libregraph `/me/sharedWithMe` endpoint.
Currently, we construct the `oc:fileid` from the pattern `{shareproviderid}${sharespaceid}!{sharemountid}`. `{shareproviderid}`and `{sharespaceid}` are both hardcoded to `a0ca6a90-a365-4782-871e-d44447bbc668`. The `{sharemountid}` itself uses the pattern `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`.
Since a resource can be shared to the same user in multiple ways (a group share and a user share) we deduplicate the two shares and only show one mountpoint in the share jail root. This is where this solution starts to fall apart:
* When accepting, mounting or syncing a share we implicitly have to accept all shares
* Each share has a different `{shareid}`, so we currently look up the oldest share and use it to build the `oc:fileid`
* Consequently, when the oldest share is revoked the `oc:fileid` changes.
We need to build the `oc:fileid` from a more stable pattern.
### Shareid
The WebDAV PROPFIND response also contains a `oc:shareid` which currently is derived from the path when the spaceid matches the share jail. The jsoncs3 implementation of the share manager currently is the only one using the `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}` pattern, where `{shareid}` is a uuid that is generated when creating the share.
Again, the problem is that a resource can be shared multiple times.
## Decision Drivers <!-- optional -->
* We need to change the `oc:fileid` pattern without breaking clients.
* We need to be able to correlate files from WebDAV and the Graph API.
## Considered Options
* [Share based id](#share-based-id)
* [Resource based id](#resource-based-id)
* [Permission based id](#permission-based-id)
* [Use graph for file metadata](#use-graph-for-file-metadata)
## Decision Outcome
Resource based id: it correctly reflects the semantic meaning of a mount point, by indirectly pointing to the resource, not the share. The permissions on the share have to be checked in the storageprovider itself, anyway. Switching to graph requires more effort and the transition can happen gradually ofter changing the `oc:fileid` pattern in the sharejail.
### Positive Consequences:
* We get rid of mixing share ids with fileids, preventing unexpected `oc:fileid` changes.
### Negative Consequences:
* We need to teach clients about a new share jail space that uses the new `oc:fileid` pattern. They may need to implement a migration strategy to switch from the old share jail space to a new share jail space by replacing the fileid in their internal database. The might be able to just switch over, because the only `oc:fileid` that changes is the one from the mountpoints. The other nodes in the subtree already use the resourceid of the shared resource.
* Clients relying on `oc:shareid` to correlate share jail entries in PROPFIND responses need to either deal with multiple `oc:shareid` as a resource can be shared multiple times, or we deprecate `oc:shareid` and only use the `oc:fileid`. *jfd: Who is using this? why? Please explain and add to the decision drivers above!*
* The graph api also needs to be able to list entries from the new share jail. *jfd: clients could use a filter to ask for the new share jail id*
## Pros and Cons of the Options <!-- optional -->
### Share based id
Follow the pattern `{shareproviderid}${sharespaceid}!{sharemountid}`, where `{sharemountid}` is `{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`.
Combined patter `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shareid}`.
`{shareproviderid}` and `{sharespaceid}`are both hardcodet to `a0ca6a90-a365-4782-871e-d44447bbc668` to route all id based requests for mountpoints to the share jail space.
+ Good, the `{shared-resource-providerid}` and `{shared-resource-spaceid}` are used to shard the shares per space.
- Bad, `oc:fileid` changes if the oldest received share to a resource is revoked.
### Resource based id
Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}`.
Hardcode `756e6cdf-5630-4b66-9380-55a85188e0f6` as a new `{sharespaceid}` to allow clients to detect the new share jail and change it at their own pace.
+ Good, stable `oc:fileid` that remains the same, regardless of permission changes.
+ Good, clients can detect the new share jail and deal with it on their terms.
### Permission based id
Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}:{permission-id}`.
- Bad, same instability as the share id
- Bad, we don't even have a permission id. We could construct one from the grantee, but this leads nowhere.
### Use graph for file metadata
Instead of using WebDAV to correlate files with shares fully embrace libregraph to manage file metadata.
Follow the pattern `{shareproviderid}${sharespaceid}!{shared-resource-providerid}:{shared-resource-spaceid}:{shared-resource-opaqueid}`.
WebDAV can be stripped of any ownCloud specific properties and will only be used for file up and download.
- Bad, more effort
+ Good, clean way of representing mountpoints and the shared resource in one `driveItem` that can include the resource based id.
+ Good, pagination, sorting and filtering cleanly specified
+ Good, WebDAV can be stripped down.
+ Good, Clients could get rid of WebDAV client and XML libs as all endpoints use JSON (all OCS endpoins return JSON when appending a `format=json` query parameter)
## Links <!-- optional -->

106
docs/ocis/adr/0028-activities-service.md
View File

@@ -1,106 +0,0 @@
---
title: "28. Activity Service"
date: 2024-05-16T15:00:00+01:00
weight: 28
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: 0028-activities-service.md
---
* Status: draft
* Deciders: [@kobergj](https://github.com/kobergj), [@fschade](https://github.com/fschade)
* Date: 2024-05-16
## Context and Problem Statement
The user should be able to see all activities for a resource.
Besides the current resource, the user should also be able to decide if he wants to include child resource activities or not.
## Decision Drivers <!-- optional -->
* The user should be able to see all activities for a resource.
* The user should be able to decide if he wants to include child resource activities.
* Activities should be stored space efficiently.
* Activities should be stored in a way that they can be queried efficiently.
* Activities should stay in place even if the resource is gone.
* Activities reflect the state at a given point in time and not the current state.
* The Service should only store a configurable number of activities per resource.
## Considered Options
### Activity store
* Use a go-micro store to store the individual activities.
* Use a time series database to store the activities.
* Use a graph database to store the activities.
* Use a relational database to store the activities.
* Use the file system to store the activities.
### Activity format
* Normalize the activities before storing them.
* Only store relevant data to get the related event from the event-history service when needed, e.g.,
```go
package pseudo_code
import (
"time"
)
type Activity struct {
ResourceId string
EventID string
Depth int64
Timestamp time.Time
}
```
* Store the activity in a human-readable way e.g. "resource A has been shared with user B."
* Store each activity only on the resource itself.
* Store each activity only on the resource itself and all its parents.
## Decision Outcome
* Activity store:
* Use a go-micro store to store the individual activities.
* Activity format:
* Store each activity only on the resource itself and all its parents.
* Only store event ids and get the related event from the event-history service when needed.
### Positive Consequences:
* Activity store (go-micro store):
* Reuse existing technology.
* We can use nats-js-kv store which already proved reliable in production.
* No need to introduce any kind of new technology, e.g., a time series database, a relational database.
* Activity Format:
* Having each activity stored on each resource (the resource itself and its parents)
makes it easy to retrieve the timeline of activities for a resource and its children.
* Only storing the event id and getting the related event from the event-history we benefit
from the event-history services capabilities to store and query events.
* Walking the resource tree from the resource to the root is a linear operation and can be done efficiently.
### Negative Consequences:
* Activity store:
* Other database types might be more efficient for storing activities.
* Using the go-micro-store only allows storing the activity in a key-value format.
* Activity Format:
* Storing only the event ids and getting the related data from the event-history service when needed
might introduce additional latency when querying activities.
* Adding each event-id to each resource parent leads to a lot of duplicated data.
## Pros and Cons of the Options <!-- optional -->
* Activity store:
* (PRO) Introducing a new database type might be more efficient for storing activities.
* (CON) Introducing a new database type brings extra complexity and maintenance overhead.
* (CON) Using the file system to store the activities might be inefficient and could be problematic especially in a distributed environment.
* Activity format:
* (PRO) Normalizing the activities before storing them might make it easier and more efficient to query them.
* (PRO) Storing each activity only on the resource itself is more space-efficient.
* (CON) Storing each activity only on the resource itself increases the complexity of querying activities.
* (CON) Storing each activity in a human-readable format is not space-efficient.
## Links <!-- optional -->
* [Story](https://github.com/owncloud/ocis/issues/8881)

15
docs/ocis/adr/_index.md
View File

@@ -1,15 +0,0 @@
---
title: "Architecture Decisions"
date: 2021-02-10T20:21:00+01:00
weight: 15
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/adr
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
oCIS is documenting architecture decisions using [Markdown Architectural Decision Records](https://adr.github.io/madr/) (MADR), following [Documenting Architecture Decisions by Michael Nygard](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions).
{{< toc >}}
To manage the records we use [butonic/adr-tools](https://github.com/butonic/adr-tools), a fork of the original [npryce/adr-tools](https://github.com/npryce/adr-tools), based on [a pull request that should have added MADR support](https://github.com/npryce/adr-tools/pull/43). It also supports a YAML header that is used by our Hugo based doc generation

229
docs/ocis/backup.md
View File

@@ -1,229 +0,0 @@
---
title: "Backup Considerations"
date: 2024-05-07T10:31:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis
geekdocFilePath: backup.md
---
This small guide aims to shed some light on the internal Infinite Scale data structure. You can refer to it when you are trying to optimize your backups or if you are just curious about how Infinite Scale stores its data.
Note, as a prerequisite backing up Infinite Scale, the instance has to be fully shut down for the time being.
## Ocis Data Structure
Ocis stores its data in a folder that can be configured via the environment variable `OC_BASE_DATA_PATH`. Without further configuration, services derive from that path when they store data, though individual settings for certain data types can be configured.
The default value for the `OC_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/var/lib/ocis` when using the docker container. Note: Configuration data is by default stored in `/etc/ocis/` in the container.).
Inside this folder, Infinite Scale will store all its data in separate subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files (blobs) are handled by a so called blobstore, which also stores here by default. Depending on the used blobstore, the blobs need to be backed up separately, for example if S3 is used. Note: See special case for the `config` folder in a docker container.
### Base Data Path Overview
Listing the contents of the folder will return the following:
```bash
~/.ocis/:tree -L 1
.
├── config
├── idm
├── idp
├── nats
├── proxy
├── search
├── storage
├── thumbnails
└── web
10 directories, 0 files
```
The following sections describe the content and background of the subdirectories to decide if a backup is required or recommended and its effect when it is not backed up.
### `config`
Contains basic Infinite Scale configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OC_CONFIG_DIR` environment variable, but for this document we will assume this variable is not set and the default is used.)
```bash
~/.ocis/config/:tree
.
└── ocis.yaml
1 directory, 1 file
```
* `ocis.yaml`:\
BACKUP RECOMMENDED. Holds Infinite Scale configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore:
| Field Name | Envvar Name | Description | If not backed up |
| --- | --- | --- | --- |
| `idp.ldap.bind_password` | `OC_LDAP_BIND_PASSWORD` | Password for the idp | no logins possible |
| `idm.service_user_passwords.idp_password`| `IDM_IDPSVC_PASSWORD` | Same as above | no logins possible |
| `system_user_id` | `OC_SYSTEM_USER_ID` | The id of storage-system user | no logins possible |
| `idm.service_user_passwords.reva_password`| `IDM_REVASVC_PASSWORD` | The reva password | no logins possible |
| `auth_basic.auth_providers.ldap.bind_password` | `AUTH_BASIC_LDAP_BIND_PASSWORD` | Same as above | no logins possible |
| `users.drivers.ldap.bind_password` | `USERS_LDAP_BIND_PASSWORD` | Same as above | no logins possible |
| `groups.drivers.ldap.bind_password` | `GROUPS_LDAP_BIND_PASSWORD` | Same as above | no logins possible |
| `storage_users.mount_id` | `STORAGE_USERS_MOUNT_ID` | The mountid of the storage_users service | sharing data lost |
| `gateway.storage_registry.storage_users_mount_id` | `GATEWAY_STORAGE_USERS_MOUNT_ID` | Same as above | sharing data lost |
### `idm`
Note: This folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case.
Contains the data for the internal Infinite Scale identity management. See the [IDM README]({{< ref "../services/idm/_index.md" >}}) for more details.
```bash
~/.ocis/idm/:tree
.
├── ldap.crt
├── ldap.key
└── ocis.boltdb
1 directory, 3 files
```
* `ocis.boltdb`:\
BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, Infinite Scale will have no users, therefore also all data is lost.
* `ldap.crt`:\
BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up.
* `ldap.key`:\
BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to specify its path. Will be auto-generated if not backed up.
### `idp`
Note: This folder will not appear if you use an external idp. Refer to your idp's documentation for backup details in this case.
Contains the data for the internal Infinite Scale identity provider. See the [IDP README]({{< ref "../services/idp/_index.md" >}}) for more details.
```bash
~/.ocis/idp/:tree
.
├── encryption.key
├── private-key.pem
└── tmp
└── identifier-registration.yaml
2 directories, 3 files
```
* `encryption.key`:\
BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Not backing this up will force users to relogin.
* `private-key.pem`:\
BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin.
* `identifier-registration.yml`:\
BACKUP OPTIONAL. It holds configuration for oidc clients (web, desktop, ios, android). Will be recreated if not backed up.
### `nats`
Note: This folder will not appear if you use an external nats installation. In that case, data has to secured in alignment with the external installation.
Contains nats data for streams and stores. See the [NATS README]({{< ref "../services/nats/_index.md" >}}) for more details.
```bash
~/.ocis/nats/:tree -L 1
.
└── jetstream
```
* `jetstream`:\
BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The Infinite Scale functionality is not impacted if omitted.
### `proxy`
Contains proxy service data. See the [PROXY README]({{< ref "../services/proxy/_index.md" >}}) for more details.
```bash
~/.ocis/proxy/:tree
.
├── server.crt
└── server.key
1 directory, 2 files
```
* `server.crt`:\
BACKUP OPTIONAL. This is the certificate for the http services. Use `PROXY_TRANSPORT_TLS_CERT` to specify its path.
* `server.key`:\
BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_TRANSPORT_TLS_KEY` to specify its path.
### `search`
Contains the search index. See the [SEARCH README]({{< ref "../services/search/_index.md" >}}) for more details.
```bash
~/.ocis/search/:tree -L 1
.
└── bleve
2 directories, 0 files
```
* `bleve`:\
BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files.
### `storage`
Contains Infinite Scale meta (and blob) data, depending on the blobstore. See the [STORAGE-USERS README]({{< ref "../services/storage-users/_index.md" >}}) for more details.
```bash
~/.ocis/storage/:tree -L 1
.
├── metadata
├── ocm
└── users
4 directories, 0 files
```
* `metadata`:\
BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OC_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings.
* `ocm`:\
BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up.
* `users`:\
BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty Infinite Scale instance, which is fully functional accepting new data. Old data is lost.
### `thumbnails`
Contains thumbnails data. See the [THUMBNAILS README]({{< ref "../services/thumbnails/_index.md" >}}) for more details.
```bash
~/.ocis/thumbnails/:tree -L 1
.
└── files
```
* `files`:\
OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service.
### `web`
Contains web assets such as custom logos, themes etc. See the [WEB README]({{< ref "../services/web/_index.md" >}}) for more details.
```bash
~/.ocis/web/:tree -L 1
.
└── assets
2 directories, 0 files
```
* `assets`:\
BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assets. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded.
### `external services`
When using an external idp/idm/nats or blobstore, its data needs to be backed up separately. Refer to your idp/idm/nats/blobstore documentation for backup details.
## Backup Consistency Command
Infinite Scale now allows checking an existing backup for consistency. Use the command:
```bash
ocis backup consistency -p "<path-to-base-folder>"
```
`path-to-base-folder` needs to be replaced with the path to the storage providers base path. Should be same as the `STORAGE_USERS_OCIS_ROOT`
Use the `-b s3ng` option when using an external (s3) blobstore. Note: When using this flag, the path to the blobstore must be configured via envvars or a yaml file to match the configuration of the original instance. Consistency checks for other blobstores than `ocis` and `s3ng` are not supported at the moment.

170
docs/ocis/config.md
View File

@@ -1,170 +0,0 @@
---
title: "Configuration"
date: "2021-11-09T00:03:16+0100"
weight: 2
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis
geekdocFilePath: config.md
---
{{< toc >}}
## Configuration Framework
In order to simplify deployments and development the configuration model from oCIS aims to be simple yet flexible.
## Overview of the approach
{{< figure src="/ocis/static/ocis-config-redesign.drawio.svg" >}}
## In-depth configuration
Since we include a set of predefined extensions within the single binary, configuring an extension can be done in a variety of ways. Since we work with complex types, having as many cli per config value scales poorly, so we limited the options to config files and environment variables.
The hierarchy is clear enough, leaving us with:
_(each element above overwrites its precedent)_
1. env variables
2. extension config
3. ocis config
This is manifested in the previous diagram. We can then speak about "configuration file arithmetics", where resulting config transformations happen through a series of steps. An administrator must be aware of these sources, since mis-managing them can be a source of confusion, having undesired transformations on config files believed not to be applied.
## Flows
Let's explore the various flows with examples and workflows.
### Examples
Let's explore with examples this approach.
#### Expected loading locations
- docker images: `/etc/ocis/`
- binary releases: `$HOME/.ocis/config/`
followed by the `<extension name>.yaml`, e.g. `proxy.yaml` for the extension configuration. You also can put an `ocis.yaml` config file to the expected loading location to use a single config file.
You can set another directory as config path in the environment variable `OC_CONFIG_DIR`. It will then pick the same file names, but from the folder you configured.
#### Only config files
The following config files are present in the default loading locations:
_ocis.yaml_
```yaml
proxy:
http:
addr: localhost:1111
log:
pretty: false
color: false
level: info
log:
pretty: true
color: true
level: info
```
_proxy.yaml_
```yaml
http:
addr: localhost:3333
```
Note that the extension files will overwrite values from the main `ocis.yaml`, causing `ocis server` to run with the following configuration:
```yaml
proxy:
http:
addr: localhost:3333
log:
pretty: true
color: true
level: info
```
#### Using ENV variables
The logging configuration if defined in the main ocis.yaml is inherited by all extensions. It can be, however, overwritten by a single extension file if desired. The same example can be used to demonstrate environment values overwrites. With the same set of config files now we have the following command `PROXY_HTTP_ADDR=localhost:5555 ocis server`, now the resulting config looks like:
```yaml
proxy:
http:
addr: localhost:5555
log:
pretty: true
color: true
level: info
```
#### Substitute ENV variables in configuration files
Environment variables can be used in the configurations files and will be replaced by oCIS when loading these.
Default values can be specified after a `|` character - see below.
```yaml
proxy:
http:
addr: ${PROXY_HTTP_ADDR|localhost:4321}
log:
pretty: true
color: true
level: info
```
### Workflows
Since one can run an extension using the runtime (supervised) or not (unsupervised), we ensure correct behavior in both modes, expecting the same outputs.
#### Supervised
You are using the supervised mode whenever you issue the `ocis server` command. We start the runtime on port `9250` (by default) that listens for commands regarding the lifecycle of the supervised extensions. When an extension runs supervised and is killed, the only way to provide / overwrite configuration values will be through an extension config file. This is due to the parent process has already started, and it already has its own environment.
#### Unsupervised
All the points from the priority section hold true. An unsupervised extension can be started with the format: `ocis [extension]` i.e: `ocis proxy`. First, `ocis.yaml` is parsed, then `proxy.yaml` followed by environment variables.
## Shared Values
When running in supervised mode (`ocis server`) it is beneficial to have common values for logging, so that the log output is correctly formatted, or everything is piped to the same file without duplicating config keys and values all over the place. This is possible using the global `log` config key:
_ocis.yaml_
```yaml
log:
level: error
color: true
pretty: true
file: /var/tmp/ocis_output.log
```
There is, however, the option for extensions to overwrite this global values by declaring their own logging directives:
_ocis.yaml_
```yaml
log:
level: info
color: false
pretty: false
```
One can go as far as to make the case of an extension overwriting its shared logging config that received from the main `ocis.yaml` file. Because things can get out of hands pretty fast we recommend not mixing logging configuration values and either use the same global logging values for all extensions.
{{< hint warning >}}
When overwriting a globally shared logging values, one *MUST* specify all values.
{{< /hint >}}
### Log config keys
```yaml
log:
level: [ error | warning | info | debug ]
color: [ true | false ]
pretty: [ true | false ]
file: [ path/to/log/file ] # MUST not be used with pretty = true
```
## Default config values (in yaml)
TBD. Needs to be generated and merged with the env mappings.

31
docs/ocis/deployment/_index.md
View File

@@ -1,31 +0,0 @@
---
title: "Deployment"
date: 2020-10-01T20:35:00+01:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: _index.md
geekdocCollapseSection: true
---
{{< toc >}}
## Deployments scenarios and examples
This section handles deployments and operations for admins and people who are interested in how versatile oCIS is. If you want to just try oCIS you may also follow [Getting started]({{< ref "../getting-started" >}}).
### Setup oCIS on your server
oCIS deployments are super simple, yet there are many configurations possible for advanced setups.
- [Basic oCIS setup]({{< ref "basic-remote-setup" >}}) - configure domain, certificates and port
- [oCIS setup with Keycloak as identity provider]({{< ref "ocis_keycloak" >}})
- [Flexible oCIS setup with WebOffice and Search capabilities]({{< ref "ocis_full" >}})
- [Parallel deployment of oC10 and oCIS]({{< ref "oc10_ocis_parallel" >}})
- [oCIS with the Hello extension example]({{< ref "ocis_hello" >}})
## Secure an oCIS instance
oCIS no longer has any default secrets in versions later than oCIS 1.20.0. Therefore you're no
longer able to start oCIS without generating / setting all needed secrets.
The recommended way is to use `ocis init` for that. It will generate a secure config file for you.

54
docs/ocis/deployment/basic-remote-setup.md
View File

@@ -1,54 +0,0 @@
---
title: "Basic Remote Setup"
date: 2020-02-27T20:35:00+01:00
weight: 16
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: basic-remote-setup.md
---
{{< toc >}}
The default configuration of the oCIS binary and the `owncloud/ocis` docker image assume, that you access oCIS on `localhost`. This enables you to do quick testing and development without any configuration.
If you need to access oCIS running in a docker container, on a VM or a remote machine via another hostname than `localhost`, you need to configure this hostname in oCIS. The same applies if you are not using hostnames but instead an IP (e.g. `192.168.178.25`).
## Start the oCIS fullstack server from binary
Initialize the oCIS configuration by running `./bin/ocis init`.
Upon first start of the oCIS fullstack server with `./bin/ocis server` it will generate a directory tree skeleton in `$HOME/.ocis`. If that is already existing it will not be overwritten as it contains all relevant data for oCIS.
For the following examples you need to have the oCIS binary in your current working directory, we assume it is named `ocis` and it needs to be marked as executable. See [Getting Started]({{< ref "../getting-started/#binaries" >}}) for where to get the binary from.
### Using automatically generated certificates
In order to run oCIS with automatically generated and self-signed certificates please execute following command. You need to replace `your-host` with an IP or hostname. Since you have only self-signed certificates you need to have `OC_INSECURE` set to `true`.
```bash
OC_INSECURE=true \
PROXY_HTTP_ADDR=0.0.0.0:9200 \
OC_URL=https://your-host:9200 \
./ocis server
```
### Using already present certificates
If you have your own certificates already in place, you may want to make oCIS use them:
```bash
OC_INSECURE=false \
PROXY_HTTP_ADDR=0.0.0.0:9200 \
OC_URL=https://your-host:9200 \
PROXY_TRANSPORT_TLS_KEY=./certs/your-host.key \
PROXY_TRANSPORT_TLS_CERT=./certs/your-host.crt \
./ocis server
```
If you generated these certificates on your own, you might need to set `OC_INSECURE` to `true`.
For more configuration options check the configuration section in [oCIS]({{< ref "../config" >}}) and the oCIS services.
## Start the oCIS fullstack server with Docker Compose
Please have a look at our other [deployment examples]({{< ref "./" >}}).

318
docs/ocis/deployment/bridge.md
View File

@@ -1,318 +0,0 @@
---
title: "Bridge"
date: 2020-02-27T20:35:00+01:00
weight: 30
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: bridge.md
---
{{< toc >}}
We are planning to build a bridge from ownCloud 10 to ocis. The idea is to have a reverse proxy in front of ownCloud 10 that will forward requests to ownCloud 10 or ocis-reva, depending on the migration status of the logged-in user.
This document is a work in progress of the current setup.
## Current status
Using ocis and the ownCloud 10 [graphapi app](https://github.com/owncloud/graphapi/) it is possible today to use an existing owncloud 10 instance as a user backend and storage backend for ocis.
## How to do it
### Install the owncloud 10 graphapi app
In an owncloud 10 apps folder
```
$ git clone git@github.com:owncloud/graphapi.git
$ cd graphapi
$ composer install
```
### Enable the graphapi app
```
occ a:e graphapi
```
No configuration necessary. You can test with `curl`:
```console
$ curl https://cloud.ocis.test/index.php/apps/graphapi/v1.0/users -u admin -s | jq
Enter host password for user 'admin':
{
"value": [
{
"id": "admin",
"displayName": "admin",
"mail": null
},
{
"id": "demo",
"displayName": "Demo",
"mail": null
},
...
],
"@odata.nextLink": "https://cloud.ocis.test/apps/graphapi/v1.0/users?$top=10&$skip=10"
}
```
{{< hint >}}
The MS graph api actually asks for `Bearer` auth, but in order to check users passwords during an LDAP bind we are exploiting ownClouds authentication implementation that will grant access when `Basic` auth is used. An LDAP Bind you may ask? Read on!
{{< /hint >}}
### Grab ocis!
```
$ git clone git@github.com:owncloud/ocis.git
$ cd ocis
$ make -C ocis build
```
This should give you an `ocis/bin/ocis` binary. Try listing the help with `ocis/bin/ocis --help`.
{{< hint >}}
You can check out a custom branch and build a custom binary which can then be used for the steps below.
{{< /hint >}}
### Start ocis glauth
We are going to use the built binary and ownCloud 10 graphapi app to turn ownCloud 10 into the datastore for an LDAP proxy.
#### configure it
While ocis can be configured using environment variables, e.g. for a docker compose setup we are going to use a more traditional config file here.
Create a config file for ocis in either `/etc/ocis`, `$HOME/.ocis` or `./.config`. You can use `.json`, `.yaml` or `.toml`. I will use toml here, because ... reasons.
```toml
[glauth.backend]
datastore = "owncloud" # switch to the owncloud datastore
servers = ["https://cloud.ocis.test/apps/graphapi/v1.0"] # the graph api endpoint to connect to
basedn = "dc=ocis,dc=test" # base dn to construct the LDAP dn. The user `admin` will become `cn=admin,dc=ocis,dc=test`
```
{{< hint >}}
There is a bug in the config merging for environment variables, cli flags and config files causing log settings not to be picked up from the config file when specifying `--extensions`. That is why I will
* configure most of the config in a file,
* adjust logging using `OC_LOG_*` environment variables and
* specify which extension to run using `ocis/bin/ocis server --extensions "comma, separated, list, of, extensions"`.
{{< /hint >}}
#### Run it!
For now, we only start the glauth extension:
```console
$ OC_LOG_PRETTY=true OC_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth"
```
#### Check it is up and running
You should now be able to list accounts from your ownCloud 10 oc_accounts table using:
```console
$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixaccount)'
```
Groups should work as well:
```console
$ ldapsearch -x -H ldap://127.0.0.1:9125 -b dc=ocis,dc=test -D "cn=admin,dc=ocis,dc=test" -W '(objectclass=posixgroup)'
```
{{< hint >}}
This is currently a readonly implementation and minimal to the use-case of authenticating users with an IDP.
{{< /hint >}}
### Start ocis storage-gateway, storage-authbasic and storage-userprovider
We are going to set up reva to authenticate users against our glauth LDAP proxy. This allows us to log in and use the reva cli. The ocis storage-gateway starts the reva gateway which will authenticate basic auth requests using the storage-authbasic service. Furthermore, users have to be available in the storage-userprovider to retrieve displayname, email address and other user metadata.
To configure LDAP to use our glauth we add this section to the config file:
```toml
[storage.reva.ldap]
idp = "https://ocis.ocis.test"
basedn = "dc=ocis,dc=test"
binddn = "cn=admin,dc=ocis,dc=test" # an admin user in your oc10
bindpassword = "secret"
userschema = { uid = "uid", displayname = "givenname" } # TODO make glauth return an ownclouduuid and displayname attribute
```
Now we can start all necessary services.
```console
$ OC_LOG_PRETTY=true OC_LOG_COLOR=true ocis/bin/ocis server --extensions "glauth, storage-gateway, storage-authbasic, storage-userprovider"
```
{{< hint warning >}}
Here I ran out of time. I tried to verify this step with the reva cli:
`cmd/reva/reva -insecure -host localhost:9142`
`login basic`
but it tries to create the user home, which cannot be disabled in a config file: https://github.com/owncloud/ocis/issues/2416#issuecomment-901197053
starting `STORAGE_GATEWAY_DISABLE_HOME_CREATION_ON_LOGIN=true OC_LOG_LEVEL=debug OC_LOG_PRETTY=true OC_LOG_COLOR=true ocis/bin/ocis server --extensions "storage-gateway, storage-authbasic, storage-userprovider"` let me login:
```console
✗ cmd/reva/reva -insecure -host localhost:9142
reva-cli v1.11.0-27-g95b1f2ee (rev-95b1f2ee)
Please use `exit` or `Ctrl-D` to exit this program.
>> login basic
username: jfd
password: OK
>> whoami
id:<idp:"https://ocis.ocis.test" opaque_id:"jfd" type:USER_TYPE_PRIMARY > username:"jfd" mail:"jfd@butonic.de" display_name:"J\303\266rn" uid_number:99 gid_number:99
>> exit
```
I hope https://github.com/owncloud/ocis/pull/2024 fixes the parsing order of things.
everything below this is outdated
... gotta run
{{< /hint >}}
### Start ocis storage-userprovider
```console
ocis/bin/ocis storage-userprovider --ldap-port 19126 --ldap-user-schema-uid uid --ldap-user-schema-displayName givenName --addr :19144
```
TODO clone `git clone git@github.com:cs3org/cs3apis.git`
query users using [grpcurl](https://github.com/fullstorydev/grpcurl)
```console
grpcurl -import-path ./cs3apis/ -proto ./cs3apis/cs3/identity/user/v1beta1/user_api.proto -plaintext localhost:19144 cs3.identity.user.v1beta1.UserAPI/FindUsers
ERROR:
Code: Unauthenticated
Message: auth: core access token not found
```
### Start ocis idp
#### Set environment variables
The built-in [libregraph/lico](https://github.com/libregraph/lico) needs environment variables to configure the LDAP server:
```console
export OC_URL=https://ocis.ocis.test
export IDP_LDAP_URI=ldap://127.0.0.1:9125
export IDP_LDAP_BASE_DN="dc=ocis,dc=test"
export IDP_LDAP_BIND_DN="cn=admin,dc=ocis,dc=test"
export IDP_LDAP_BIND_PASSWORD="its-a-secret"
export IDP_LDAP_SCOPE=sub
export IDP_LDAP_LOGIN_ATTRIBUTE=uid
export IDP_LDAP_NAME_ATTRIBUTE=givenName
```
Don't forget to use an existing user with admin permissions (only admins are allowed to list all users via the graph api) and the correct password.
#### Run it!
You can now bring up `ocis/bin/ocis idp` with:
```console
$ ocis/bin/ocis idp server --iss http://127.0.0.1:9130 --signing-kid gen1-2020-02-27
```
`ocis/bin/ocis idp` needs to know
- `--iss http://127.0.0.1:9130` the issuer, which must be a reachable http endpoint. For testing an ip works. For openid connect HTTPS is NOT optional. This URL is exposed in the `http://127.0.0.1:9130/.well-known/openid-configuration` endpoint and clients need to be able to connect to it, securely. We will change this when introducing the proxy.
- `--signing-kid gen1-2020-02-27` a signature key id, otherwise the jwks key has no name, which might cause problems with clients. a random key is ok, but it should change when the actual signing key changes.
{{< hint warning >}}
* TODO: the port in the `--iss` needs to be changed when hiding the idp behind the proxy
* TODO: the signing keys and encryption keys should be precreated so they are reused between restarts. Otherwise all client sessions will become invalid when restarting the IdP.
{{< /hint >}}
#### Check it is up and running
1. Try getting the configuration:
```console
$ curl http://127.0.0.1:9130/.well-known/openid-configuration
```
2. Check if the login works at http://127.0.0.1:9130/signin/v1/identifier
{{< hint >}}
If you later get a `Unable to find a key for (algorithm, kid):PS256, )` Error make sure you did set a `--signing-kid` when starting `ocis/bin/ocis idp` by checking it is present in http://127.0.0.1:9130/konnect/v1/jwks.json
{{< /hint >}}
### Start ocis proxy
{{< hint >}}
Everything below this hint is outdated. Next steps are roughly:
* directly after glauth start the `ocis storage-userporvider`?
- how to verify that works?
- https://github.com/fullstorydev/grpcurl
* start proxy
- the ocis ipd url can be changed to https
- when do we hide oc10 behind ocis? -> advanced bridge at the end? for now run it without touching the existing oc10 instance
* start web
- verify the login works, but how?
- TODO the login works, but then the capabilities requests will fail ... unless we make the proxy answer them by talking to oc10?
Other ideas:
* the owncloud backend in glauth also works with the user provisioning api ... no changes to a running production instance? db access could be done with a read only account as well...
{{< /hint >}}
### Start ocis-web
#### Run it!
Point `ocis-web` to your owncloud domain and tell it where to find the openid connect issuing authority:
```console
$ bin/web server --web-config-server https://cloud.example.com --oidc-authority https://192.168.1.100:9130 --oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration --oidc-client-id ocis
```
`ocis-web` needs to know
- `--web-config-server https://cloud.example.com` is ownCloud url with webdav and ocs endpoints (oc10 or ocis)
- `--oidc-authority https://192.168.1.100:9130` the openid connect issuing authority, in our case `oidc-idp`, running on port 9130
- `--oidc-metadata-url https://192.168.1.100:9130/.well-known/openid-configuration` the openid connect configuration endpoint, typically the issuer host with `.well-known/openid-configuration`, but there are cases when another endpoint is used, e.g. ping identity provides multiple endpoints to separate domains
- `--oidc-client-id ocis` the client id we will register later with `ocis-idp` in idp OIDC client settings
### Patch owncloud
While the UserSession in ownCloud 10 is currently used to test all available IAuthModule implementations, it immediately logs out the user when an exception occurs. However, existing owncloud 10 instances use the oauth2 app to create Bearer tokens for mobile and desktop clients.
To give the openidconnect app a chance to verify the tokens we need to change the code a bit. See https://github.com/owncloud/core/pull/37043 for a possible solution.
> Note: The PR is hot ... as in *younger than this list of steps*. And it messes with authentication. Use with caution.
### Install the owncloud 10 openidconnect app
In an owncloud 10 apps folder
```
$ git clone git@github.com:owncloud/openidconnect.git
$ cd openidconnect
$ composer install
```
After enabling the app configure it in `config/oidc.config.php`
```php
$CONFIG = [
'openid-connect' => [
'provider-url' => 'https://192.168.1.100:9130',
'client-id' => 'ocis',
'loginButtonName' => 'OpenId Connect @ Konnectd',
],
'debug' => true, // if using self-signed certificates
// allow the different domains access to the ocs and webdav endpoints:
'cors.allowed-domains' => [
'https://cloud.example.com',
'http://localhost:9100',
],
];
```
In the above configuration replace
- `provider-url` with the URL to your `ocis-idp` issuer
- `https://cloud.example.com` with the URL to your ownCloud 10 instance
- `http://localhost:9100` with the URL to your ownCloud Web instance
> Note: By default the openidconnect app will use the email of the user to match the user from the oidc userinfo endpoint with the ownCloud account. So make sure your users have a unique primary email.
## Next steps
Aside from the above todos these are the next steps
- tie it all together behind `ocis-proxy`
- create an `ocis bridge` command that runs all the ocis services in one step with a properly preconfigured idp OIDC client `ocis-idp` for `ownCloud Web` and the owncloud 10 `openidconnect` app, as well as a randomized `--signing-kid`.

113
docs/ocis/deployment/continuous_deployment.md
View File

@@ -1,113 +0,0 @@
---
title: "Continuous Deployment"
date: 2020-10-12T14:04:00+01:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: continuous_deployment.md
---
{{< toc >}}
We are continuously deploying the following deployment examples. Every example is deployed in two flavors:
- Master: reflects the current master branch state of oCIS and will be updated with every commit to master
- Rolling: reflects the latest rolling release of oCIS and will be updated with every rolling release
- Production: reflects the latest production release of oCIS and will be updated with every production release
The configuration for the continuous deployment can be found in the [oCIS repository](https://github.com/owncloud/ocis/tree/master/deployments/continuous-deployment-config).
# oCIS with Web Office
This deployment is based on our modular [ocis_full Example](ocis_full.md) and uses the default configuration with Collabora Online as the office suite, traefik reverse proxy, cloudimporter and the inbucket mail catching server to showcase the full feature set of oCIS.
Credentials:
- oCIS: see [default demo users]({{< ref "../getting-started#login-to-owncloud-web" >}})
## Master
- oCIS: [ocis.ocis.master.owncloud.works](https://ocis.ocis.master.owncloud.works)
- Mail: [mail.ocis.master.owncloud.works](https://mail.ocis.master.owncloud.works)
## Rolling Release
- oCIS: [ocis.ocis.rolling.owncloud.works](https://ocis.ocis.rolling.owncloud.works)
- Mail: [mail.ocis.rolling.owncloud.works](https://mail.ocis.rolling.owncloud.works)
## Production Release
- oCIS: [ocis.ocis.production.owncloud.works](https://ocis.ocis.production.owncloud.works)
- Mail: [mail.ocis.production.owncloud.works](https://mail.ocis.production.owncloud.works)
## Master with OnlyOffice
This example is using OnlyOffice as the office suite.
- oCIS: [ocis.ocis-onlyoffice.master.owncloud.works](https://ocis.ocis-onlyoffice.master.owncloud.works)
## Rolling Release with OnlyOffice
This example is using OnlyOffice as the office suite.
- oCIS: [ocis.ocis-onlyoffice.rolling.owncloud.works](https://ocis.ocis-onlyoffice.rolling.owncloud.works)
# oCIS and ownCloud Web with both most recent development versions
Credentials:
- oCIS: see [default demo users]({{< ref "../getting-started#login-to-owncloud-web" >}})
## Master branches
- oCIS: [ocis.ocis-web.master.owncloud.works](https://ocis.ocis-web.master.owncloud.works)
# oCIS with Keycloak
Credentials:
- oCIS: see [default demo users]({{< ref "../getting-started#login-to-owncloud-web" >}})
- Keycloak:
- username: admin
- password: admin
## Rolling Release
- oCIS: [ocis.ocis-keycloak.rolling.owncloud.works](https://ocis.ocis-keycloak.rolling.owncloud.works)
- Keycloak admin access: [keycloak.ocis-keycloak.rolling.owncloud.works](https://keycloak.ocis-keycloak.rolling.owncloud.works)
- Keycloak account management: [keycloak.ocis-keycloak.rolling.owncloud.works/realms/oCIS/account/#/](https://keycloak.ocis-keycloak.rolling.owncloud.works/realms/oCIS/account/#/)
# oCIS with S3 storage backend (MinIO)
This deployment is based on our modular [ocis_full Example](ocis_full.md), see the [oCIS with Web Office](#ocis-with-web-office) description for the feature set. In addition to that, we deployed a MinIO S3 storage backend. oCIS stores the data in the S3 server and the metadata on the local disk by using the `s3ng` storage driver.
The MinIO server provides a powerful Web UI for browser-based access to the storage which makes it possible to manage the data stored in the S3 server and understand how different policies and configurations affect the data.
Credentials:
- oCIS: see [default demo users]({{< ref "../getting-started/demo-users/" >}})
- MinIO:
- access key: ocis
- secret access key: ocis-secret-key
## Rolling Release
- oCIS: [ocis.ocis-s3.rolling.owncloud.works](https://ocis.ocis-s3.rolling.owncloud.works)
- MinIO: [minio.ocis-s3.rolling.owncloud.works](https://minio.ocis-s3.rolling.owncloud.works)
- Mail: [mail.ocis-s3.rolling.owncloud.works](https://mail.ocis-s3.rolling.owncloud.works)
# oCIS with LDAP for users and groups
Credentials:
- oCIS: see [default demo users]({{< ref "../getting-started/demo-users/" >}})
- LDAP admin:
- username: cn=admin,dc=owncloud,dc=com
- password: admin
## Rolling Release
- oCIS: [ocis.ocis-ldap.rolling.owncloud.works](https://ocis.ocis-ldap.rolling.owncloud.works)
- LDAP admin: [ldap.ocis-ldap.rolling.owncloud.works](https://ldap.ocis-ldap.rolling.owncloud.works)

73
docs/ocis/deployment/kubernetes.md
View File

@@ -1,73 +0,0 @@
---
title: "Kubernetes"
date: 2021-09-23T11:04:00+01:00
weight: 25
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: kubernetes.md
---
{{< toc >}}
## What is Kubernetes
Formally described as:
> Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.
_[source](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)_
Without getting too deep in definitions, and for the purpose of compactness, Kubernetes can be summarized as a way of managing containers that run applications to ensure that there is no downtime and an optimal usage of resources. It provides with a framework in which to run distributed systems.
Kubernetes provides you with:
- **Service discovery and load balancing**: Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
- **Storage orchestration**: Kubernetes allows you to automatically mount a storage system of your choice, such as local storages, public cloud providers, and more.
- **Automated rollouts and rollbacks**: You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers and adopt all their resources to the new container.
- **Automatic bin packing**: You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.
- **Self-healing**: Kubernetes restarts containers that fail, replaces containers, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.
- **Secret and configuration management**: Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.
_[extracted from k8s docs](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/#why-you-need-kubernetes-and-what-can-it-do)_
If that is still too abstract, [here is an ELI5 writeup](https://dev.to/miguelmota/comment/filh).
### References and further reads
- [Marcel Wunderlich's](https://github.com/Deaddy) [4 series articles](http://deaddy.net/introduction-to-kubernetes-pt-1.html) on Kubernetes clarifying its declarative nature, deep diving into ingress networking, storage and monitoring.
### How does oCIS fit in the Kubernetes model
oCIS was designed with running on Kubernetes in mind. We set up to adopt the [Twelve-Factor App](https://12factor.net/) principles regarding configuration, with almost every aspect of oCIS being modifiable via environment variables. This comes in handy when you especially have a look at how a helm chart's (we will introduce this concept shortly) [list of values](https://github.com/owncloud/ocis-charts/blob/d8735e3222d2050504303851d3461909c86fcc89/ocis/values.yaml) looks like.
## What is Minikube
[Minikube](https://minikube.sigs.k8s.io/docs/) lets you run a Kubernetes cluster locally. It is the most approachable way to test a deployment. It requires no extra configuration on any cloud platform, as everything runs on your local machine. For the purpose of these docs, this is the first approach we chose to run oCIS and will develop on how to set it up.
## What is `kubectl`
[kubectl](https://kubernetes.io/docs/tasks/tools/) is the command-line tool for Kubernetes. It allows users to run commands against a k8s cluster the user has access to. It supports for having multiple contexts for as many clusters as you have access to. In these docs we will setup 2 contexts, a minikube and a GCP context.
## What are Helm Charts, and why they are useful for oCIS
[Helm](https://helm.sh/) is the equivalent of a package manager for Kubernetes. It can be described as a layer on top of how you would write pods, deployments or any other k8s resource declaration.
### Installing Helm
[Follow the official installation guide](https://helm.sh/docs/intro/install/).
## Setting up Minikube
For a guide on how to set minikube up follow the [official minikube start guide](https://minikube.sigs.k8s.io/docs/start/) for your specific OS.
## oCIS charts
We have not yet published the oCIS Helm charts, therefore you need to clone the git repository manually. It currently also does not support to be run on Kind or Minikube clusters. For known issues and planned features, please have a look at the [GitHub issue tracker](https://github.com/owncloud/ocis-charts/issues).
Configuration options are described [here](https://github.com/owncloud/ocis-charts/tree/master/charts/ocis#configuration).
### Run oCIS
1. clone the charts: `git clone https://github.com/owncloud/ocis-charts.git /var/tmp/ocis-charts`
2. cd into the charts root: `cd /var/tmp/ocis-charts/charts/ocis`
3. install the package: `helm install ocis .` (you need to set configuration values in almost all cases)
4. verify the application is running in the cluster: `kubectl get pods`

42
docs/ocis/deployment/monitoring-tracing.md
View File

@@ -1,42 +0,0 @@
---
title: "Monitoring & Tracing"
date: 2020-02-27T20:35:00+01:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: monitoring-tracing.md
---
{{< toc >}}
Monitoring and tracing gives developers and admin insights into a complex system, in this case oCIS.
If you are a developer and want to trace during developing you should have a look at [example server setup]({{< ref "../development/tracing" >}}).
This documentation describes how to set up a long-running monitoring & tracing infrastructure for one or multiple oCIS servers or deployments. After reading this guide, you also should know everything needed to integrate oCIS into your existing monitoring and tracing infrastructure.
# Overview about the proposed solution
{{< figure src="/ocis/static/monitoring_tracing_overview.drawio.svg" >}}
## Monitoring & tracing clients
We assume that you already have oCIS deployed on one or multiple servers by using our deployment examples (see rectangle on the left). On these servers our monitoring & tracing clients, namely Telegraf and Jaeger agent, need to be added.
Telegraf will collect host metrics (CPU, RAM, network, processes, ...) and docker metrics (per container CPU, RAM, network, ...). Telegraf is also configured to scrape metrics from Prometheus metric endpoints which oCIS exposes, this is done by the Prometheus input plugin . The metrics from oCIS and all other metrics gathered will be exposed with the Prometheus output plugin and can therefore be scraped by our monitoring & tracing server.
Jaeger agent is being configured as target for traces in oCIS. It then will receive traces from all oCIS services, add some process tags to them and forward them to our Jaeger collector on our monitoring & tracing server.
For more information and how to deploy it, see [monitoring & tracing client](https://github.com/owncloud-devops/monitoring-tracing-client).
## Monitoring & tracing server
A live version of the monitoring and tracing server for our demo instances is available here: [Grafana](https://grafana.infra.owncloud.works), [Prometheus](https://prometheus.infra.owncloud.works) and [Jaeger Query](https://jaeger.infra.owncloud.works).
The monitoring & tracing server is considered as shared infrastructure and is normally used for different services. This means that oCIS is not the only software whose metrics and traces are available on the monitoring server. It is also possible that data of multiple oCIS instances are available on the monitoring server.
Metrics are scraped, stored and can be queried with Prometheus. For the visualization of these metrics Grafana is used. Because Prometheus is scraping the metrics from the oCIS server (pull model instead of a push model), the Prometheus server must have access to the exposed endpoint of the Telegraf Prometheus output plugin.
Jaeger collector receives traces sent by the Jaeger agent on the oCIS servers and persists them in ElasticSearch. From there the user can query and visualize the traces in Jaeger query or in Grafana. Because Jaeger agent is actively sending traces to the monitoring & tracing server, the server must be reachable from the oCIS server.
For more information and how to deploy it, see [monitoring & tracing server](https://github.com/owncloud-devops/monitoring-tracing-server).

168
docs/ocis/deployment/oc10_ocis_parallel.md
View File

@@ -1,168 +0,0 @@
---
title: "Parallel deployment of oC10 and oCIS"
date: 2020-10-12T14:04:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: oc10_ocis_parallel.md
---
{{< toc >}}
## Overview
- This setup reflects [stage 6 of the oC10 to oCIS migration plan]({{< ref "migration#stage-6-parallel-deployment" >}})
- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup
- OpenLDAP server with demo users
- LDAP admin interface to edit users
- Keycloak as OpenID Connect provider in federation with the LDAP server
- ownCloud 10 with MariaDB and Redis
- ownCloud 10 is configured to synchronize users from the LDAP server
- ownCloud 10 is used to use OpenID Connect for authentication with Keycloak
- oCIS running behind Traefik as reverse proxy
- oCIS is using the ownCloudSQL storage driver on the same files and same database as ownCloud 10
- oCIS is using Keycloak as OpenID Connect provider
- oCIS is using the LDAP server as user backend
- All requests to both oCIS and oC10 are routed through the oCIS proxy and will be routed based on an OIDC claim to one of them. Therefore admins can change on a user basis in the LDAP which backend is used.
[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/oc10_ocis_parallel)
## Server Deployment
### Requirements
- Linux server with docker and docker-compose installed
- four domains set up and pointing to your server
- cloud.\* for serving oCIS
- keycloak.\* for serving Keycloak
- ldap .\* for serving the LDAP management UI
- traefik.\* for serving the Traefik dashboard
See also [example server setup]({{< ref "preparing_server" >}})
### Install this example
- Clone oCIS repository
`git clone https://github.com/owncloud/ocis.git`
- Go to the deployment example
`cd ocis/deployment/examples/oc10_ocis_parallel`
- Open the `.env` file in a text editor
The file by default looks like this:
```bash
# If you're on a internet facing server please comment out following line.
# It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
INSECURE=true
### Traefik settings ###
TRAEFIK_LOG_LEVEL=
# Serve Traefik dashboard. Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
TRAEFIK_ACME_MAIL=
### shared oCIS / oC10 settings ###
# Domain of oCIS / oC10, where you can find the frontend. Defaults to "cloud.owncloud.test"
CLOUD_DOMAIN=
### oCIS settings ###
# oCIS version. Defaults to "latest"
OC_DOCKER_TAG=
# JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
OC_JWT_SECRET=
# JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
STORAGE_TRANSFER_SECRET=
# Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
OC_MACHINE_AUTH_API_KEY=
### oCIS settings ###
# oC10 version. Defaults to "latest"
OC10_DOCKER_TAG=
# client secret which the openidconnect app uses to authenticate to Keycloak. Defaults to "oc10-oidc-secret"
OC10_OIDC_CLIENT_SECRET=
# app which will be shown when opening the ownCloud 10 UI. Defaults to "files" but also could be set to "web"
OWNCLOUD_DEFAULT_APP=
# if set to "false" (default) links will be opened in the classic UI, if set to "true" ownCloud Web is used
OWNCLOUD_WEB_REWRITE_LINKS=
### LDAP settings ###
# password for the LDAP admin user "cn=admin,dc=owncloud,dc=com", defaults to "admin"
LDAP_ADMIN_PASSWORD=
# Domain of the LDAP management frontend. Defaults to "ldap.owncloud.test"
LDAP_MANAGER_DOMAIN=
### Keycloak ###
# Domain of Keycloak, where you can find the management and authentication frontend. Defaults to "keycloak.owncloud.test"
KEYCLOAK_DOMAIN=
# Realm which to be used with oC10 and oCIS. Defaults to "owncloud"
KEYCLOAK_REALM=
# Admin user login name. Defaults to "admin"
KEYCLOAK_ADMIN_USER=
# Admin user login password. Defaults to "admin"
KEYCLOAK_ADMIN_PASSWORD=
```
You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
Set your domain for the oC10 and oCIS frontend in `CLOUD_DOMAIN=`, e.g. `CLOUD_DOMAIN=cloud.owncloud.test`.
By default ownCloud 10 will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC10_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
You can switch the default application of ownCloud 10 by setting`OWNCLOUD_DEFAULT_APP=files` in oder to have the classic UI as frontend, which is also the default. If you prefer ownCloud Web as the default application in ownCloud 10 just set `OWNCLOUD_DEFAULT_APP=web`.
In oder to change the default link open action which defaults to the classic UI (`OWNCLOUD_WEB_REWRITE_LINKS=false`) you can set it to `OWNCLOUD_WEB_REWRITE_LINKS=true`. This will lead to links being opened in ownCloud Web.
The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep these examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`.
Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, e.g. `ldap.owncloud.test`.
Set your domain for the Keycloak administration panel and authentication endpoints to `KEYCLOAK_DOMAIN=` e.g. `KEYCLOAK_DOMAIN=keycloak.owncloud.test`.
Changing the used Keycloak realm can be done by setting `KEYCLOAK_REALM=`. This defaults to the ownCloud realm `KEYCLOAK_REALM=owncloud`. The ownCloud realm will be automatically imported on startup and includes our demo users.
You probably should secure your Keycloak admin account by setting `KEYCLOAK_ADMIN_USER=` and `KEYCLOAK_ADMIN_PASSWORD=` to values other than `admin`.
Now you have configured everything and can save the file.
- Start the docker stack
`docker-compose up -d`
- You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI.
## Local setup
For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this:
```
127.0.0.1 cloud.owncloud.test
127.0.0.1 keycloak.owncloud.test
127.0.0.1 ldap.owncloud.test
127.0.0.1 traefik.owncloud.test
```
After that you're ready to start the application stack:
`docker-compose up -d`
You now can visit the cloud, oC10 or oCIS depending on the user configuration. Marie defaults to oC10 and Richard and Einstein default to oCIS, but you can change the ownCloud selector at any time in the LDAP management UI.

4
docs/ocis/deployment/ocis_and_wopi_drawio.svg
View File

File diff suppressed because one or more lines are too long
Side by Side

Before

Width:  |  Height:  |  Size: 92 KiB

415
docs/ocis/deployment/ocis_full.md
View File

@@ -1,415 +0,0 @@
---
title: "Full modular oCIS with WebOffice"
date: 2024-06-25T00:00:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: opencloud_full.md
---
{{< toc >}}
## Overview
* oCIS, the collaboration service, Collabora or OnlyOffice running behind Traefik as reverse proxy
* Collabora or OnlyOffice enable you to edit office documents in your browser
* The collaboration server acts as a bridge to make the oCIS storage accessible to Collabora and OnlyOffice
* Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup
* The whole deployment acts as a modular toolkit to use different flavors of office suites and ocis features
[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_full)
## Easy Default
The Infinite Scale Team and product management are providing a default setup for oCIS.
### Goal:
- provide a good starting point for a production deployment
- minimal effort to get started with an opinionated setup
- keep it adjustable it to your needs.
### Default Components
- Infinite Scale
- Full Text Search
- Collabora Online Web Office
- Prepared for LetsEncrypt SSL certificates via Traefik Reverse Proxy
### Optional Components
- ClamAV Virusscanner
- Cloud Importer (Experimental)
- OnlyOffice as an alternative to Collabora
- S3 Storage config to connect to an S3 storage backend
- S3 Minio Server as a local S3 storage backend for debugging and development
### Important Note
If you deviate from the configuration setup and let the `collaboration` service run in its own container, you MUST
ensure the ocis configuration is shared as shown in the example deployment. This is because secrets generated
must be accessible for all services.
## Server Deployment
### Requirements
* Linux server with docker and docker-compose installed
* Three domains set up and pointing to your server
* ocis.* for serving oCIS
* collabora.* for serving Collabora
* onlyoffice.* for serving OnlyOffice
* wopiserver.* for serving the WOPI server
* traefik.* for serving the Traefik dashboard
* companion.* for serving the uppy companion app
See also [example server setup]({{< ref "preparing_server" >}})
### Install oCIS and Traefik
* Clone oCIS repository
`git clone https://github.com/owncloud/ocis.git --depth 1`
* Go to the deployment example
`cd ocis/deployments/examples/ocis_full`
* Open the `.env` file in a text editor.
The file by default looks like this:
```shell {linenos=table,hl_lines=[8,24,48,50,135,138]}
## Basic Settings ##
# Define the docker compose log driver used.
# Defaults to local
LOG_DRIVER=
# If you're on an internet facing server, comment out following line.
# It skips certificate validation for various parts of Infinite Scale and is
# needed when self signed certificates are used.
INSECURE=true
## Traefik Settings ##
# Note: Traefik is always enabled and can't be disabled.
# Serve Traefik dashboard.
# Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard.
# Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the traefik dashboard.
# Defaults to user "admin" and password "admin" (written as: "admin:admin").
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates.
# Needs only be changed if this is a public facing server.
TRAEFIK_ACME_MAIL=
# Set to the following for testing to check the certificate process:
# "https://acme-staging-v02.api.letsencrypt.org/directory"
# With staging configured, there will be an SSL error in the browser.
# When certificates are displayed and are emitted by # "Fake LE Intermediate X1",
# the process went well and the envvar can be reset to empty to get valid certificates.
TRAEFIK_ACME_CASERVER=
## Infinite Scale Settings ##
# Beside Traefik, this service must stay enabled.
# Disable only for testing purposes.
# Note: the leading colon is required to enable the service.
OCIS=:ocis.yml
# The oCIS container image.
# For production releases: "owncloud/ocis"
# For rolling releases: "owncloud/ocis-rolling"
# Defaults to production if not set otherwise
OC_DOCKER_IMAGE=owncloud/ocis-rolling
# The oCIS container version.
# Defaults to "latest" and points to the latest stable tag.
OC_DOCKER_TAG=
# Domain of oCIS, where you can find the frontend.
# Defaults to "ocis.owncloud.test"
OC_DOMAIN=
# oCIS admin user password. Defaults to "admin".
ADMIN_PASSWORD=
# Demo users should not be created on a production instance,
# because their passwords are public. Defaults to "false".
# Also see: https://doc.owncloud.com/ocis/latest/deployment/general/general-info.html#demo-users-and-groups
DEMO_USERS=
# Define the oCIS loglevel used.
# For more details see:
# https://doc.owncloud.com/ocis/latest/deployment/services/env-vars-special-scope.html
LOG_LEVEL=
# Define the kind of logging.
# The default log can be read by machines.
# Set this to true to make the log human readable.
# LOG_PRETTY=true
#
# Define the oCIS storage location. Set the paths for config and data to a local path.
# Note that especially the data directory can grow big.
# Leaving it default stores data in docker internal volumes.
# For more details see:
# https://doc.owncloud.com/ocis/next/deployment/general/general-info.html#default-paths
# OC_CONFIG_DIR=/your/local/ocis/config
# OC_DATA_DIR=/your/local/ocis/data
# S3 Storage configuration - optional
# Infinite Scale supports S3 storage as primary storage.
# Per default, S3 storage is disabled and the local filesystem is used.
# To enable S3 storage, uncomment the following line and configure the S3 storage.
# For more details see:
# https://doc.owncloud.com/ocis/next/deployment/storage/s3.html
# Note: the leading colon is required to enable the service.
#S3NG=:s3ng.yml
# Configure the S3 storage endpoint. Defaults to "http://minio:9000" for testing purposes.
S3NG_ENDPOINT=
# S3 region. Defaults to "default".
S3NG_REGION=
# S3 access key. Defaults to "ocis"
S3NG_ACCESS_KEY=
# S3 secret. Defaults to "ocis-secret-key"
S3NG_SECRET_KEY=
# S3 bucket. Defaults to "ocis"
S3NG_BUCKET=
#
# For testing purposes, add local minio S3 storage to the docker-compose file.
# The leading colon is required to enable the service.
#S3NG_MINIO=:minio.yml
# Minio domain. Defaults to "minio.owncloud.test".
MINIO_DOMAIN=
# Define SMPT settings if you would like to send Infinite Scale email notifications.
# For more details see:
# https://doc.owncloud.com/ocis/latest/deployment/services/s-list/notifications.html
# NOTE: when configuring Inbucket, these settings have no effect, see inbucket.yml for details.
# SMTP host to connect to.
SMTP_HOST=
# Port of the SMTP host to connect to.
SMTP_PORT=
# An eMail address that is used for sending Infinite Scale notification eMails
# like "ocis notifications <noreply@yourdomain.com>".
SMTP_SENDER=
# Username for the SMTP host to connect to.
SMTP_USERNAME=
# Password for the SMTP host to connect to.
SMTP_PASSWORD=
# Authentication method for the SMTP communication.
SMTP_AUTHENTICATION=
# Allow insecure connections to the SMTP server. Defaults to false.
SMTP_INSECURE=
## Default Enabled Services ##
### Apache Tika Content Analysis Toolkit ###
# Tika (search) is enabled by default, comment if not required.
# Note: the leading colon is required to enable the service.
TIKA=:tika.yml
# Set the desired docker image tag or digest.
# Defaults to "latest"
TIKA_IMAGE=
### Collabora Settings ###
# Collabora web office is default enabled, comment if not required.
# Note: the leading colon is required to enable the service.
COLLABORA=:collabora.yml
# Domain of Collabora, where you can find the frontend.
# Defaults to "collabora.owncloud.test"
COLLABORA_DOMAIN=
# Domain of the wopiserver which handles OnlyOffice.
# Defaults to "wopiserver.owncloud.test"
WOPISERVER_DOMAIN=
# Admin user for Collabora.
# Defaults to "admin".
# Collabora Admin Panel URL:
# https://{COLLABORA_DOMAIN}/browser/dist/admin/admin.html
COLLABORA_ADMIN_USER=
# Admin password for Collabora.
# Defaults to "admin".
COLLABORA_ADMIN_PASSWORD=
# Set to true to enable SSL for Collabora Online. Default is true if not specified.
COLLABORA_SSL_ENABLE=false
# If you're on an internet-facing server, enable SSL verification for Collabora Online.
# Please comment out the following line:
COLLABORA_SSL_VERIFICATION=false
...
```
#### Reverse Proxy and SSL
{{< hint type=important >}}
**Domains and SSL**\
Though it may sound strange, most of the setups are failing due to a misconfiguration regarding domains and SSL. Please make sure that you have set up the domains correctly and that they are pointing to your server. Also, make sure that you have set up the email address for the LetsEncrypt certificates in `TRAEFIK_ACME_MAIL=`.
{{< /hint >}}
You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
#### Infinite Scale Release and Version
By default oCIS will be started in the `latest` rolling version. Please note that this deployment does currently not work with the 5.x productions releases.
The oCIS "collaboration" service, which is required by this deployment, is not part of the 5.x releases.
If you want to use a specific version of oCIS, set the version to a dedicated tag like `OC_DOCKER_TAG=6.3.0`. The minimal required oCIS Version to run this deployment is 6.3.0. Available production versions can be found on [Docker Hub Production](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated) and available rolling releases can be found on [Docker Hub Rolling](https://hub.docker.com/r/owncloud/ocis-rolling/tags?page=1&ordering=last_updated)
{{< hint type=info title="oCIS Releases" >}}
You can read more about the different oCIS releases in the [oCIS Release Lifecycle](../release_roadmap.md).
{{< /hint >}}
Set your domain for the oCIS frontend in `OC_DOMAIN=`, e.g. `OC_DOMAIN=ocis.owncloud.test`.
Set the initial admin user password in `ADMIN_PASSWORD=`, it defaults to `admin`.
Web Office needs a public domain for the WOPI server to be set in `WOPISERVER_DOMAIN=`, where the office suite can work on the files via the WOPI protocol.
Now it's time to set up Collabora and you need to configure the domain of Collabora in `COLLABORA_DOMAIN=`.
If you want to use the Collabora admin panel you need to set the username and password for the administrator in `COLLABORA_ADMIN_USER=` and `COLLABORA_ADMIN_PASSWORD=`.
* Start the docker stack
`docker-compose up -d`
* You now can visit oCIS and are able to open an office document in your browser. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
## Local Setup
This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this:
```
127.0.0.1 ocis.owncloud.test
127.0.0.1 traefik.owncloud.test
127.0.0.1 collabora.owncloud.test
127.0.0.1 onlyoffice.owncloud.test
127.0.0.1 wopiserver.owncloud.test
127.0.0.1 mail.owncloud.test
127.0.0.1 companion.owncloud.test
127.0.0.1 minio.owncloud.test
```
After that, you're ready to start the application stack:
`docker-compose pull && docker-compose up -d`
Open https://collabora.owncloud.test in your browser and accept the invalid certificate warning.
Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You are now able to open an office document in your browser. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
## Additional services
### Clamav Virusscanner
You can add a Clamav Virusscanner to the stack. The service is disabled by default. To enable it, uncomment the `CLAMAV` line in the `.env` file.
```shell {linenos=table,hl_lines=[3]}
## Clamav Settings ##
# The leading colon is required to enable the service.
CLAMAV=:clamav.yml
```
After enabling that service, you can add the service to the stack with `docker-compose up -d` again.
### Traefik Dashboard
If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
```shell {linenos=table,hl_lines=[4,7,10]}
### Traefik Settings ###
# Serve Traefik dashboard.
# Defaults to "false".
TRAEFIK_DASHBOARD=true
# Domain of Traefik, where you can find the dashboard.
# Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the traefik dashboard.
# Defaults to user "admin" and password "admin" (written as: "admin:admin").
TRAEFIK_BASIC_AUTH_USERS=
```
### Cloud Importer
Cloud importer can provide an Upload Interface to your oCIS instance. It is a separate service that can be enabled in the `.env` file.
```shell {linenos=table,hl_lines=[3]}
## Uppy Companion Settings ##
# The leading colon is required to enable the service.
CLOUD_IMPORTER=:cloudimporter.yml
## The docker image to be used for uppy companion.
# owncloud has built a container with public link import support.
COMPANION_IMAGE=
# Domain of Uppy Companion. Defaults to "companion.owncloud.test".
COMPANION_DOMAIN=
# Provider settings, see https://uppy.io/docs/companion/#provideroptions for reference.
# Empty by default, which disables providers.
COMPANION_ONEDRIVE_KEY=
COMPANION_ONEDRIVE_SECRET=
```
After Enabling that servive by uncommenting the `CLOUD_IMPORTER` line, you can add the service to the stack with `docker-compose up -d` again.
### S3 Storage
You can use an S3 compatible Storage as the primary data store. The metadatata of the files will still be stored on the local filesystem.
{{<hint type="info">}}
The endpoint, region and keys for your S3 Server need to be provided by the service or company who operates it. Normally you can get these via web portal.
{{</hint>}}
```shell {linenos=table,hl_lines=[8,10,12,14,16,18]}
# S3 Storage configuration - optional
# Infinite Scale supports S3 storage as primary storage.
# Per default, S3 storage is disabled and the local filesystem is used.
# To enable S3 storage, uncomment the following line and configure the S3 storage.
# For more details see:
# https://doc.owncloud.com/ocis/next/deployment/storage/s3.html
# Note: the leading colon is required to enable the service.
# S3NG=:s3ng.yml
# Configure the S3 storage endpoint. Defaults to "http://minio:9000" for testing purposes.
S3NG_ENDPOINT=
# S3 region. Defaults to "default".
S3NG_REGION=
# S3 access key. Defaults to "ocis"
S3NG_ACCESS_KEY=
# S3 secret. Defaults to "ocis-secret-key"
S3NG_SECRET_KEY=
# S3 bucket. Defaults to "ocis"
S3NG_BUCKET=
```
#### Use a Local Minio S3 Storage Backend
For testing purposes, you can use a local minio S3 storage backend. To enable it, uncomment the `S3NG_MINIO` line in the `.env` file.
The frontend for the minio server is available at `http://minio.owncloud.test` and the access key is `ocis` and the secret key is `ocis-secret`.
## Local Setup for Web Development
In case you want to run ownCloud Web from a development branch together with this deployment example (e.g. for feature development for the app provider frontend) you can use this deployment example with the local setup and some additional steps as described below.
1. Clone the [ownCloud Web repository](https://github.com/owncloud/web) on your development machine.
2. Run `pnpm i && pnpm build:w` for `web`, so that it creates and continuously updates the `dist` folder for web.
3. Add the dist folder as read only volume to `volumes` section of the `ocis` service in the `docker-compose.yml` file:
```yaml
- /your/local/path/to/web/dist/:/web/dist:ro
```
Make sure to point to the `dist` folder inside your local copy of the web repository.
4. Set the oCIS environment variables `WEB_ASSET_CORE_PATH` and `WEB_ASSET_APPS_PATH` in the `environment` section of the `ocis` service, so that it uses your mounted dist folder for the web assets, instead of the assets that are embedded into oCIS.
```yaml
WEB_ASSET_CORE_PATH: "/web/dist"
WEB_ASSET_APPS_PATH: "/web/dist"
```
5. Start the deployment example as described above in the `Local setup` section.
For app provider frontend development in `web` you can find the source code in `web/packages/web-app-external`. Some parts of the integration live in `web/packages/web-app-files`.
## Using Podman
Podman doesn't have a "local" log driver. Also it's docker-compatibility socket does live in a different location, especially when running a rootless podman.
Using the following settings you can run the deployment with a recent podman version:
```bash
LOG_DRIVER=journald \
DOCKER_SOCKET_PATH=/run/user/1000/podman/podman.sock \
podman compose start
```

121
docs/ocis/deployment/ocis_hello.md
View File

@@ -1,121 +0,0 @@
---
title: "oCIS with Hello extension"
date: 2020-10-12T14:04:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: ocis_hello.md
---
{{< toc >}}
## Overview
- oCIS running behind Traefik as reverse proxy
- oCIS Hello extension runs beside the main oCIS stack and providing the Hello functionality
- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup
[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_hello)
The docker stack consists of 3 containers. One of them is Traefik, a proxy which is terminating SSL and forwards the requests to oCIS in the internal docker network.
The oCIS Hello extension is running in another container and enables you to use its functionality from within ownCloud Web.
## Server Deployment
### Requirements
- Linux server with docker and docker-compose installed
- two domains set up and pointing to your server
- ocis.\* for serving oCIS
- traefik.\* for serving the Traefik dashboard
See also [example server setup]({{< ref "preparing_server" >}})
### Install oCIS and Traefik
- Clone oCIS repository
`git clone https://github.com/owncloud/ocis.git`
- Go to the deployment example
`cd ocis/deployment/examples/ocis_hello`
- Open the `.env` file in a text editor.
The file by default looks like this:
```bash
# If you're on a internet facing server please comment out following line.
# It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
INSECURE=true
### Traefik settings ###
# Serve Traefik dashboard. Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
TRAEFIK_ACME_MAIL=
### oCIS settings ###
# oCIS version. Defaults to "latest"
OC_DOCKER_TAG=
# Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test"
OC_DOMAIN=
# oCIS admin user password. Defaults to "admin".
ADMIN_PASSWORD=
# The demo users should not be created on a production instance
# because their passwords are public. Defaults to "false".
DEMO_USERS=
### oCIS Hello settings ###
# oCIS Hello version. Defaults to "latest"
OC_HELLO_DOCKER_TAG=
```
You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
Set your domain for the oCIS frontend in `OC_DOMAIN=`, e.g. `OC_DOMAIN=ocis.owncloud.test`.
Set the initial admin user password in `ADMIN_PASSWORD=`, it defaults to `admin`.
By default the oCIS Hello extension will be started in the `latest` version. If you want to start a specific version of oCIS Hello set the version to `OC_HELLO_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis-hello/tags?page=1&ordering=last_updated).
Now you have configured everything and can save the file.
- Start the docker stack
`docker-compose up -d`
- You now can visit oCIS and are able to switch to the Hello extension by using the application switcher on the top right corner of ownCloud Web. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
## Local setup
For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this:
```
127.0.0.1 ocis.owncloud.test
127.0.0.1 traefik.owncloud.test
```
After that you're ready to start the application stack:
`docker-compose up -d`
Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You are now able to switch to the Hello extension by using the application switcher on the top right corner of ownCloud Web. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.

136
docs/ocis/deployment/ocis_keycloak.md
View File

@@ -1,136 +0,0 @@
---
title: "oCIS with Keycloak"
date: 2020-10-12T14:04:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: ocis_keycloak.md
---
{{< toc >}}
## Overview
* oCIS and Keycloak running behind Traefik as reverse proxy
* Keycloak acting as the IDP for oCIS
* Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup
[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_keycloak)
The docker stack consists 4 containers. One of them is Traefik, a proxy which is terminating ssl and forwards the requests to oCIS in the internal docker network. It
is also responsible for redirecting requests on the OIDC discovery endpoints (e.g. `.well-known/openid-configuration`) to the correct destination in Keycloak.
Keycloak add two containers: Keycloak itself and a PostgreSQL as database. Keycloak will be configured as oCIS' IDP instead of the internal IDP [LibreGraph Connect]({{< ref "../../services/idp" >}})
The other container is oCIS itself, running all services in one container. In this example oCIS uses the [oCIS storage driver]({{< ref "../storage/storagedrivers" >}})
## Server Deployment
### Requirements
* Linux server with docker and docker-compose installed
* Three domains set up and pointing to your server
- ocis.* for serving oCIS
- keycloak.* for serving Keycloak
- traefik.* for serving the Traefik dashboard
See also [example server setup]({{< ref "preparing_server" >}})
### Install oCIS and Traefik
* Clone oCIS repository
`git clone https://github.com/owncloud/ocis.git`
* Go to the deployment example
`cd ocis/deployments/examples/ocis_keycloak`
* Open the `.env` file in a text editor.
The file by default looks like this:
```bash
# If you're on a internet facing server please comment out following line.
# It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
INSECURE=true
### Traefik settings ###
# Serve Traefik dashboard. Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
TRAEFIK_ACME_MAIL=
### oCIS settings ###
# oCIS version. Defaults to "latest"
OC_DOCKER_TAG=
# Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test"
OC_DOMAIN=
# ownCloud Web openid connect client id. Defaults to "ocis-web"
OC_OIDC_CLIENT_ID=
### Keycloak ###
# Domain of Keycloak, where you can find the management and authentication frontend. Defaults to "keycloak.owncloud.test"
KEYCLOAK_DOMAIN=
# Realm which to be used with oCIS. Defaults to "oCIS"
KEYCLOAK_REALM=
# Admin user login name. Defaults to "admin"
KEYCLOAK_ADMIN_USER=
# Admin user login password. Defaults to "admin"
KEYCLOAK_ADMIN_PASSWORD=
```
You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
Set your domain for the oCIS frontend in `OC_DOMAIN=`, e.g. `OC_DOMAIN=ocis.owncloud.test`.
If you want to change the OIDC client id of th ownCloud Web frontend, you can do this by setting the name to `OC_OIDC_CLIENT_ID=`.
Set your domain for the Keycloak administration panel and authentication endpoints to `KEYCLOAK_DOMAIN=` e.g. `KEYCLOAK_DOMAIN=keycloak.owncloud.test`.
Changing the used Keycloak realm can be done by setting `KEYCLOAK_REALM=`. This defaults to the oCIS realm `KEYCLOAK_REALM=oCIS`. The oCIS realm will be automatically imported on startup and includes our demo users.
You probably should secure your Keycloak admin account by setting `KEYCLOAK_ADMIN_USER=` and `KEYCLOAK_ADMIN_PASSWORD=` to values other than `admin`.
Now you have configured everything and can save the file.
* Start the docker stack
`docker-compose up -d`
* You now can visit oCIS, Keycloak and Traefik dashboard on your configured domains. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
## Local setup
For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this:
```
127.0.0.1 ocis.owncloud.test
127.0.0.1 traefik.owncloud.test
127.0.0.1 keycloak.owncloud.test
```
After that you're ready to start the application stack:
`docker-compose up -d`
Open https://keycloak.owncloud.test in your browser and accept the invalid certificate warning.
Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You now can login to oCIS with the demo users. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.

134
docs/ocis/deployment/ocis_ldap.md
View File

@@ -1,134 +0,0 @@
---
title: "oCIS with LDAP"
date: 2020-10-12T14:04:00+01:00
weight: 24
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: ocis_ldap.md
---
{{< toc >}}
## Overview
- Traefik generating self-signed certificates for local setup or obtaining valid SSL certificates for a server setup
- OpenLDAP server with demo users
- LDAP admin interface to edit users
- oCIS running behind Traefik as reverse proxy
- oCIS is using the LDAP server as user backend
[Find this example on GitHub](https://github.com/owncloud/ocis/tree/master/deployments/examples/ocis_ldap)
## Server Deployment
### Requirements
- Linux server with docker and docker-compose installed
- four domains set up and pointing to your server
- ocis.\* for serving oCIS
- ldap .\* for serving the LDAP management UI
- traefik.\* for serving the Traefik dashboard
See also [example server setup]({{< ref "preparing_server" >}})
### Install this example
- Clone oCIS repository
`git clone https://github.com/owncloud/ocis.git`
- Go to the deployment example
`cd ocis/deployment/examples/ocis_ldap`
- Open the `.env` file in a text editor.
The file by default looks like this:
```bash
# If you're on a internet facing server please comment out following line.
# It skips certificate validation for various parts of oCIS and is needed if you use self signed certificates.
INSECURE=true
### Traefik settings ###
# Serve Traefik dashboard. Defaults to "false".
TRAEFIK_DASHBOARD=
# Domain of Traefik, where you can find the dashboard. Defaults to "traefik.owncloud.test"
TRAEFIK_DOMAIN=
# Basic authentication for the dashboard. Defaults to user "admin" and password "admin"
TRAEFIK_BASIC_AUTH_USERS=
# Email address for obtaining LetsEncrypt certificates, needs only be changed if this is a public facing server
TRAEFIK_ACME_MAIL=
### oCIS settings ###
# oCIS version. Defaults to "latest"
OC_DOCKER_TAG=
# Domain of oCIS, where you can find the frontend. Defaults to "ocis.owncloud.test"
OC_DOMAIN=
# JWT secret which is used for the storage provider. Must be changed in order to have a secure oCIS. Defaults to "Pive-Fumkiu4"
OC_JWT_SECRET=
# JWT secret which is used for uploads to create transfer tokens. Must be changed in order to have a secure oCIS. Defaults to "replace-me-with-a-transfer-secret"
STORAGE_TRANSFER_SECRET=
# Machine auth api key secret. Must be changed in order to have a secure oCIS. Defaults to "change-me-please"
OC_MACHINE_AUTH_API_KEY=
### LDAP server settings ###
# Password of LDAP user "cn=admin,dc=owncloud,dc=com". Defaults to "admin"
LDAP_ADMIN_PASSWORD=
### LDAP manager settings ###
# Domain of LDAP manager. Defaults to "ldap.owncloud.test"
LDAP_MANAGER_DOMAIN=
```
You are installing oCIS on a server and Traefik will obtain valid certificates for you so please remove `INSECURE=true` or set it to `false`.
If you want to use the Traefik dashboard, set TRAEFIK_DASHBOARD to `true` (default is `false` and therefore not active). If you activate it, you must set a domain for the Traefik dashboard in `TRAEFIK_DOMAIN=` e.g. `TRAEFIK_DOMAIN=traefik.owncloud.test`.
The Traefik dashboard is secured by basic auth. Default credentials are the user `admin` with the password `admin`. To set your own credentials, generate a htpasswd (e.g. by using [an online tool](https://htpasswdgenerator.de/) or a cli tool).
Traefik will issue certificates with LetsEncrypt and therefore you must set an email address in `TRAEFIK_ACME_MAIL=`.
By default oCIS will be started in the `latest` version. If you want to start a specific version of oCIS set the version to `OC_DOCKER_TAG=`. Available versions can be found on [Docker Hub](https://hub.docker.com/r/owncloud/ocis/tags?page=1&ordering=last_updated).
Set your domain for the oCIS frontend in `OC_DOMAIN=`, e.g. `OC_DOMAIN=cloud.owncloud.test`.
The OpenLDAP server in this example deployment has an admin users, which is also used as bind user in order to keep these examples simple. You can change the default password "admin" to a different one by setting it to `LDAP_ADMIN_PASSWORD=...`.
Set your domain for the LDAP manager UI in `LDAP_MANAGER_DOMAIN=`, e.g. `ldap.owncloud.test`.
Grant the oCIS Admin role to the admin user from your LDAP in `OC_ADMIN_USER_ID:`. You need to enter the uuid of LDAP user.
{{< hint type=tip title=Encoding >}}
In the .ldif file in this example, the admin user id is base64 encoded. You need to decode it to make it work.
{{< /hint >}}
Now you have configured everything and can save the file.
- Start the docker stack
`docker-compose up -d`
- You now can visit oCIS and Traefik dashboard on your configured domains. You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.
## Local setup
For a more simple local ocis setup see [Getting started]({{< ref "../getting-started" >}})
This docker stack can also be run locally. One downside is that Traefik can not obtain valid SSL certificates and therefore will create self-signed ones. This means that your browser will show scary warnings. Another downside is that you can not point DNS entries to your localhost. So you have to add static host entries to your computer.
On Linux and macOS you can add them to your `/etc/hosts` file and on Windows to `C:\Windows\System32\Drivers\etc\hosts` file like this:
```
127.0.0.1 cloud.owncloud.test
127.0.0.1 keycloak.owncloud.test
127.0.0.1 ldap.owncloud.test
127.0.0.1 traefik.owncloud.test
```
After that you're ready to start the application stack:
`docker-compose up -d`
Open https://ocis.owncloud.test in your browser and accept the invalid certificate warning. You now can login to oCIS with the default users, which also can be found here: [Getting started]({{< ref "../getting-started#login-to-ocis-web" >}}). You may need to wait some minutes until all services are fully ready, so make sure that you try to reload the pages from time to time.

67
docs/ocis/deployment/preparing_server.md
View File

@@ -1,67 +0,0 @@
---
title: "Preparing a server"
date: 2020-10-12T14:04:00+01:00
weight: 100
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: preparing_server.md
---
{{< toc >}}
## Example for Hetzner Cloud
* create server on Hetzner Cloud. Set labels "owner" and "for". Example for hcloud cli:
`hcloud server create --type cx21 --image ubuntu-20.04 --ssh-key admin --name ocis-server --label owner=admin --label for=testing`
* Configure DNS A-records for needed domains pointing on the servers ip address, for example in CloudFlare
* Access server via ssh as root
* Create a new user
`$ adduser --disabled-password --gecos "" admin`
* Add user to sudo group
`$ usermod -aG sudo admin`
* Install docker
```
apt update
apt install docker.io
```
* Add user to docker group
`usermod -aG docker admin`
* Install docker-compose via
`curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose`
(docker compose version 1.27.4 as of today)
* Make docker-compose executable
`chmod +x /usr/local/bin/docker-compose`
* Add users pub key to
```
mkdir /home/admin/.ssh
echo "<pubkey>" >> /home/admin/.ssh/authorized_keys
chown admin:admin -R /home/admin/.ssh
```
* Secure ssh daemon by editing `/etc/ssh/sshd_config`
```
PermitRootLogin no
ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM no
```
* restart sshd server to apply settings `systemctl restart sshd`
* Login as the user you created

72
docs/ocis/deployment/systemd.md
View File

@@ -1,72 +0,0 @@
---
title: "Systemd service"
date: 2020-09-27T06:00:00+01:00
weight: 16
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: systemd.md
---
{{< toc >}}
## Install the oCIS binary
Download the oCIS binary of your preferred version and for your CPU architecture and operating system from [download.owncloud.com](https://download.owncloud.com/ocis/ocis).
Rename the downloaded binary to `ocis` and move it to `/usr/bin/`. As a next step, you need to mark it as executable with `chmod +x /usr/bin/ocis`.
When you now run `ocis help` on your command line, you should see the available options for the oCIS command.
## Systemd service definition
Create the Systemd service definition for oCIS in the file `/etc/systemd/system/ocis.service` with following content:
```systemd
[Unit]
Description=OCIS server
[Service]
Type=simple
User=root
Group=root
EnvironmentFile=/etc/ocis/ocis.env
ExecStart=ocis server
Restart=always
[Install]
WantedBy=multi-user.target
```
{{< hint warning >}}
For reasons of simplicity we are using the root user and group to run oCIS which is not recommended. Please use a non-root user in production environments and modify the oCIS service definition accordingly.
{{< /hint >}}
In the service definition we referenced `/etc/ocis/ocis.env` as our file containing environment variables for the oCIS process.
In order to create the file we need first to create the folder `/etc/ocis/` and then we can add the actual `/etc/ocis/ocis.env` with following content:
```bash
OC_URL=https://some-hostname-or-ip:9200
PROXY_HTTP_ADDR=0.0.0.0:9200
OC_INSECURE=false
OC_LOG_LEVEL=error
OC_CONFIG_DIR=/etc/ocis
OC_BASE_DATA_PATH=/var/lib/ocis
```
Since we set `OC_CONFIG_DIR` to `/etc/ocis` you can also place configuration files in this directory.
Please change your `OC_URL` in order to reflect your actual deployment. If you are using self-signed certificates you need to set `OC_INSECURE=true` in `/etc/ocis/ocis.env`.
oCIS will store all data in `/var/lib/ocis`, because we configured it so by setting `OC_BASE_DATA_PATH`. Therefore you need to create that directory and make it accessible to the user, you use to start oCIS.
## Starting the oCIS service
Initialize the oCIS configuration by running `ocis init --config-path /etc/ocis`.
You can enable oCIS now by running `systemctl enable --now ocis`. It will ensure that oCIS also is restarted after a reboot of the host.
If you need to restart oCIS because of configuration changes in `/etc/ocis/ocis.env`, run `systemctl restart ocis`.
You can have a look at the logs of oCIS by issuing `journalctl -f -u ocis`.

144
docs/ocis/deployment/ubernauten.md
View File

@@ -1,144 +0,0 @@
---
title: "Installing ownCloud Infinite Scale at Ubernauten"
date: 2023-12-04T14:04:00+01:00
weight: 100
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/deployment
geekdocFilePath: ubernauten.md
---
{{< toc >}}
## Uberspace and ownCloud Infinite Scale in 50 seconds
This howto shows how to set up ownCloud Infinite Scale for a quick test. For convenience, we will use the free service from the Ubernauten -- Join us here: [Uberspace Registration Page](https://dashboard.uberspace.de/register?lang=en). They offer free of charge (for the first month) web hosting.
In this documentation, we are assuming you already have an account there and it is configured for SSH access. This guide is using "ocis.uber.space" as a domain at Uberspace, version 4.0.3 of Infinite Scale and the local user "owncloud". Make sure you adapt the example code and scripts mentioned in this page to your needs Both username and domain will be set when you sign in to Uberspace.
Installing ownCloud Infinite Scale on Ubernauten is pretty straigt-forward, you can do it in three steps that take less than a minute: Here's a short video that shows how fast the whole process actually is - thanks to Infinite Scale's cloud native architecture. [ownCloud Infinite Scale in 50 seconds](/ocis/deployment/ubernauten_media/001-OCIS-in-50-seconds-2023-10-17.mkv)
### Three Steps to your Infinite UberSpace
* Download the Infinite Scale binary and make it executable
```
curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis
chmod +x ocis
```
* Set some environment variables related to Uberspace (**Make sure you fill in YOUR domain!**)
```
uberspace web backend set / --http --port 9200
export OC_URL=https://ocis.uber.space
export PROXY_TLS=false
export PROXY_HTTP_ADDR=0.0.0.0:9200
export PROXY_LOG_LEVEL=debug
```
* Start the `ocis` binary, first with the parameter `init` for initial configuration. This will also give you your unique login password for the user `admin`. Once finished, call `ocis start`:
```
./ocis init
./ocis server
```
Wait a few seconds, now you can visit the url of your uberspace server and login:
{{< figure src="/ocis/deployment/ubernauten_media/login.png" width="70%" height="auto" >}}
To make it easier (and faster), here's the commands in a script called `ocis.install`:
```
#!/bin/bash
# This file is named ocis.install
# It downloads ocis, configures the environment varibles and starts
# ownCloud Infinite Scale on a ubernauten account. You can run it in your home directory
curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis
chmod +x ocis
uberspace web backend set / --http --port 9200
export OC_URL=https://ocis.uber.space
export PROXY_TLS=false
export PROXY_HTTP_ADDR=0.0.0.0:9200
export PROXY_LOG_LEVEL=debug
./ocis init
./ocis server
```
### Service Management with Supervisord
If you want `ocis` to run continuously, you need to configure `supervisord` (http://supervisord.org) which is the tool Uberspace is using for service management.
You can start and stop services with `supervisorctl`, it will (re)read configuration files it finds in your home directory, under `etc/services.d/`, in `.ini` files. The content of these files is very simple, you only have to enter three lines, here is the example for Infinite Scale in `/home/owncloud/etc/services.d/ocis.ini`.
```
[program:ocis]
command="/home/owncloud/ocis.start"
startsecs=60
```
`ocis.start` is a script that combines all of the commands above except for the download of the ocis binary. It looks like this:
```
#!/bin/bash
# This file is named ocis.start.
# It sets environment variables needed for uber.space needed for Infinite Scale
/usr/bin/uberspace web backend set / --http --port 9200 &
export OC_URL=https://ocis.uber.space
export PROXY_TLS=false
export PROXY_HTTP_ADDR=0.0.0.0:9200
export PROXY_LOG_LEVEL=debug
/home/owncloud/ocis server
```
There are four supervisorctl commands that you will find useful (many more can be found in its documentation). You can use `supervisorctl status` to check which services managed by supervisorctl are running, a `supervisorctl reread` will be necessary after you changed the `ini` files, an `update` is applying changes, and `supervisorctl stop` will stop a running service:
```
[owncloud@ocis ~]$ supervisorctl status
ocis RUNNING pid 9813, uptime 0:01:40
[owncloud@ocis ~]$ supervisorctl reread
No config updates to processes
[owncloud@ocis ~]$ supervisorctl update
```
You can find all information on Supervisord and `supervisorctl` on its website: [Running Supervisord](http://supervisord.org/running.html).
### Updating ownCloud Infinite Scale
Updating the ocis binary is simple: When a new version comes to life, just download the new `ocis` binary from the download server, replacing the old `ocis` executable on your uberspace server.
Make a backup of your data and make sure you have read and understood the release notes of your new version , especially the "breaking changes" section before starting the binary.
Don't worry, you can always go back to the older version you had installed, there's a long list of older versions available for download.
Mind that if you want to re-configure, re-install or start a new version of ocis, make sure you have stopped supervisorctl from restarting ocis. Enter `supervisorctl stop ocis`, followed by a `killall -9 ocis` to make sure no ocis version is still running.
### Wiping and Clean Restart from Scratch
This little script is removing your ocis installation (and **all of your data!**), replacing it with a new, clean ocis installation. Be careful and only use it for testing purposes. Specify your desired ocis version in the curl command.
```
#!/bin/bash
# This file is named ocis.reinstall
# It deletes the old ocis installation, fetches a new binary and starts ocis.
rm -rf .ocis
curl https://download.owncloud.com/ocis/ocis/stable/4.0.3/ocis-4.0.3-linux-amd64 --output ocis
chmod +x ocis
uberspace web backend set / --http --port 9200
export OC_URL=https://ocis.uber.space
export PROXY_TLS=false
export PROXY_HTTP_ADDR=0.0.0.0:9200
export PROXY_LOG_LEVEL=debug
./ocis init
./ocis server
```
### Troubleshooting
* SSL/TLS Certificates: Every Uberspace comes with its own HTTPS certificate via Let's Encrypt. See the [Uberspace - HTTPS](https://manual.uberspace.de/web-https/) documentation for more details.
* Error message about `jwt_secret`: If you get the following error message, then you probably forgot to run the `ocis init` command. If `ocis server` find an configuration hasn't been set up, it will complain like this:
```
[owncloud@ocis ~]$ ./ocis server
The jwt_secret has not been set properly in your config for ocis.
Make sure your /home/mfeilner/.ocis/config config contains the proper values
(e.g. by running ocis init or setting it manually in the config/corresponding
environment variable).
```

BIN
docs/ocis/deployment/ubernauten_media/001-OCIS-in-50-seconds-2023-10-17.mkv
View File

Binary file not shown.

BIN
docs/ocis/deployment/ubernauten_media/login.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 209 KiB

9
docs/ocis/development/_index.md
View File

@@ -1,9 +0,0 @@
---
title: "Development"
date: 2020-10-01T20:35:00+01:00
weight: 10
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: _index.md
geekdocCollapseSection: true
---

316
docs/ocis/development/beta-testplan.md
View File

@@ -1,316 +0,0 @@
---
title: "Beta testplan"
date: 2022-03-24T00:00:00+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: beta-testplan.md
---
# Beta Testing
This document is supposed to give you some ideas how and what to test on ocis. It's not meant to be an extensive list of all tests to be done, rather it should help you, as beta-tester, to get started and enable you to get creative and create your own test-cases. [Derive from these examples, be creative, do unusual and unconventional things, to try to break things](https://twitter.com/sempf/status/514473420277694465).
One option to create new test-cases and to stress the system is to examine what the [API acceptance-tests](https://owncloud.dev/ocis/development/testing/#testing-with-test-suite-natively-installed) or the [web-UI](#web) does, [examine the requests](#decode-https-traffic-with-wireshark) and do something a bit different with curl. This is also a good way to find out how APIs work that are not already fully documented.
Some cases have suggested setup steps, but feel free to use other setups. This can include:
- different deployment methods (e.g. running single binary, docker-container, docker-compose setup, [individual services in own docker containers](https://owncloud.dev/ocis/deployment/ocis_individual_services/))
- different identity managers (e.g. [different external LDAP](https://owncloud.dev/ocis/deployment/ocis_ldap/), internal IDM)
- different reverse proxies (e.g. [traefik](https://owncloud.dev/ocis/deployment/ocis_traefik/))
- different OpenIDConnect IDPs (e.g builtin IDP, [keycloak](https://owncloud.dev/ocis/deployment/ocis_keycloak/), AzureAD)
- for some functionalities you will need an [LDAP server](https://owncloud.dev/ocis/deployment/ocis_ldap/) where the IDP and oCIS both get the users from
- [keycloak example](https://owncloud.dev/ocis/deployment/ocis_keycloak/)
- [service configuration](https://owncloud.dev/extensions/idp/configuration/)
- different storage systems (decomposedFS on local POSIX (default), [decomposedFS on NFS](https://owncloud.dev/ocis/storage-backends/dcfsnfs/), [S3](https://owncloud.dev/ocis/deployment/ocis_s3/) )
It's a good idea to test ocis in the same environment where you are planning to use it later (with the LDAP server, storage system, etc. of your organisation).
# run oCIS
For a quick start, please have a look into the [getting started documentation of oCIS]({{< ref "../getting-started/#run-ocis" >}})
If you would like
to access oCIS remotely please refer to the [Basic Remote Setup]({{< ref "../deployment/basic-remote-setup" >}}) section.
## additional tips
- to allow basic auth (e.g. to easily access oCIS using curl commands) you have to set `PROXY_ENABLE_BASIC_AUTH=true` environment variable
- if you cannot use real SSL Certificates set `OC_INSECURE=true` as environment variable
# Testplan
## user / groups from LDAP
Prerequisite:
- connect ocis to your preferred LDAP server
- create users and groups in LDAP
- start ocis with basic auth
documentation resources:
- [configure ocis with LDAP](https://owncloud.dev/ocis/deployment/ocis_ldap/)
- [sharing API is compatible to ownCloud 10](https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html)
- [webDav operations](#webdav)
| Test Case | Expected Result | Comment |
|-------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------|
| share file / folder to a group | member of the group can access shared item | |
| share file / folder to a group, remove member from group in LDAP | removed member should not have access to the shared item | |
| share file / folder to a group with different permissions, as receiver try to violate the permissions | receiver should not be able to violate the permissions | |
| try to login with wrong credentials | login should not be possible | |
| set a quota in LDAP, upload files till the quota is exceeded | upload should work till quota is full, uploads should not work when quota is full | |
| try to access files / folders of other users | access should not be possible | |
| try to share with non-existing users and groups | sharing should not be possible | |
| try to share with user/groups-names that contain special characters | sharing should be possible, access shares with that user does not create any problems | |
## other sharing
should be tried in various ways and in different environments
documentation resources:
- [sharing API is compatible to ownCloud 10](https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html)
- [webDav operations](#webdav)
| Test Case | Expected Result | Comment |
|---------------------------------------------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------|
| share a file/folder with the same name from different users | receiver can accept and access both file/folders and distinguish them | |
| share a file/folder with the same name but different permissions from different users | receiver can access both file/folders according to the permissions | |
| share a file/folder with the same name but different locations from one user | receiver can accept and access both file/folders and distinguish them | |
| share a file/folder back to the sharer | sharing back should not be possible | |
| re-share a file/folder with different permissions | sharing with lower permissions is possible, but not with higher | |
| decline received share | shared resource should not be shown to the receiver | |
## parallel deployment
- [configure ocis with LDAP](https://owncloud.dev/ocis/deployment/ocis_ldap/)
- [setup oC10 and ocis in parallel](https://owncloud.dev/ocis/deployment/oc10_ocis_parallel/)
- create users and groups in LDAP
documentation resources:
- [sharing API is compatible to ownCloud 10](https://doc.owncloud.com/server/10.9/developer_manual/core/apis/ocs-share-api.html)
| Test Case | Expected Result | Comment |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------|---------|
| share file / folder to a group in one implementation (use different permissions), access the items with the other implementation, try to violate the permissions | receiver should not be able to violate the permissions | |
| share file / folder to a group, remove member from group in LDAP, try to access items with the removed member from both implementations | removed member should not have access to the shared item | |
## Spaces
Prerequisite:
- start ocis with basic auth
- variable declaration, for curl examples:
```shell
SERVER_URI=https://localhost:9200
GRAPH_API_PATH=graph/v1.0
SHARE_API_PATH=ocs/v2.php/apps/files_sharing/api/v1/shares
USER=admin
PASSWORD=admin
```
- create a new user `curl -k -u $USER:$PASSWORD "$SERVER_URI/$GRAPH_API_PATH/users" -X POST -d'{"displayName":"Example User","mail":"example@example.org","onPremisesSamAccountName":"example","passwordProfile":{"password":"ThePassword"}}'`
- give the user the "Space Admin" role
1. get the id of the user: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/users/<user-name> | jq '.id'`
2. assign role user to role: `curl -k -u $USER:$PASSWORD $SERVER_URI/api/v0/settings/assignments-add -d'{"account_uuid":"<user-id>","role_id":"2aadd357-682c-406b-8874-293091995fdd"}`
- create a space: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$GRAPH_API_PATH/drives" -X POST -d'{"Name":"Space for finance","driveType":"project", "description":"we need some space"}'`
- list spaces: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/me/drives`
- disable a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X DELETE`
- delete a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X DELETE -H'Purge: T'`
- restore a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X PATCH -d"{}" -H"Restore: true"`
- rename the space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X PATCH -d'{"name":"नेपालि नाम"}'`
- change description of the space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X PATCH -d'{"description":"this contains important data"}'`
- change quota of the space `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X PATCH -d'{"quota":{"total":<bytes>}}'`
- change image of space:
1. upload an image file to the `.spaces` folder: `curl -k -u $USER:$PASSWORD https://localhost:9200/dav/spaces/<space-id>/.space/<file-name> -T <path-of-local-image> -v`
2. note the id provided in the `Oc-Fileid` header
3. set the image as "special-folder": `curl -k -u $USER:$PASSWORD $SERVER_URI/$GRAPH_API_PATH/drives/<space-id> -X PATCH -d'{"special":[{"specialFolder":{"name":"image"},"id":"<oc-fileid>"}]}'`
- share a space: `curl -k -u $USER:$PASSWORD $SERVER_URI/$SHARE_API_PATH -d'space_ref=<space-id>&shareType=7&shareWith=<receiver-username>&role=<role>'`
- Roles:
- viewer
- editor
- manager
- share a resource within a space `curl -k -u $USER:$PASSWORD $SERVER_URI/$SHARE_API_PATH -d'shareType=0&shareWith=<receiver-username>&space_ref=<space-id>%2F<resource>&permissions=<permissions>`
- possible permissions
- 1 = read
- 2 = update
- 4 = create
- 8 = delete
- 15 = read/write
- 16 = share
- 31 = All permissions
- WebDAV
- root: `https://<server-uri>/dav/spaces/<space-id>`
- [example commands for operations](#webdav)
| Test Case | Expected Result | Comment |
|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|---------|
| create a space | space should exist | |
| create a space with special characters as a name & description | space should exist | |
| create a space, delete the space | space should not exist | |
| create a space, share the space with a user | space should be accessible | |
| create a space, share the space with a group | space should be accessible, space content is shared among all users | |
| create a space, share the space with a group, disable the space | space should not be accessible | |
| create a space, share the space with a user, disable the space, restore the space | space should be accessible | |
| create a space, disable the space, delete the space, restore the space | it should not be possible to restore the space | |
| create a space, disable the space, try to share the space | sharing the space should not be possible | |
| create a space, try to delete the space | it should not be possible to delete an enabled space | |
| create & share a space with a group with viewer role, do CRUD file/folder operations with WebDAV | space content is readable but neither space not content should not be writable | |
| create & share a space with a group with editor role, do CRUD file/folder operations with WebDAV | space and content should be writable | |
| create a space, try CRUD file/folder operations with WebDAV on the space with a user that its not shared with | space and content should not be accessible | |
| create a space with a quota, share the space, upload files till the quota is exceeded | upload should work till quota is full, uploads should not work when quota is full | |
| share file/folders from inside a space (see other sharing section) | sharing works and obeys the permissions | |
| create a space, rename the space | new name should be displayed in API calls and web | |
| create a space, change description of the space | new description should be displayed in API calls and web | |
| create a space, set quota, change quota of the space | new quota is obeyed | |
| create a space, set quota, change quota of the space to a value that is lower than the sum of data already stored | new quota is obeyed, new files cannot be uploaded | |
| try the various space operations with invalid data | good error output, server does not crash | |
| try the various space operations without the correct permissions | operations are not possible without sufficient permissions | |
| try the various space operations on personal and virtual spaces | server should not crash, good error responses | |
| try the various space operations sending invalid data (invalid numbers, wrong types, invalid JSON, etc.) | server should not crash, good error responses | |
## Web
Prerequisite:
- [connect ocis to your preferred LDAP server](https://owncloud.dev/ocis/deployment/ocis_ldap/)
- create users and groups in LDAP
- Use your preferred browser (Firefox 94-96, Chrome 92-97, Opera 81-82, Edge 96-97, Safari 14-15) to access the built-in webUI (by default: https://localhost:9200)
| Test Case | Expected Result | Comment |
|--------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------|
| Login with the created user | User logs in. | |
| Create a text file. | Text editor can open, file is saved. | |
| Create a text file with special characters as name | file is created if the name is legal otherwise an error is displayed | |
| Modify a text file. | File can be modified, no problems found. | |
| Rename a file. | File is renamed. | |
| Upload a file. | File is uploaded, no problems found. | |
| Upload multiple files at once. | Files are uploaded, no problems found. | |
| delete all content of a folder at once. | Folder is cleaned, items are visible in the trashbin | |
| Overwrite a file by uploading a new version. | File is uploaded and overwritten, file versions are displayed | |
| Overwrite a file by uploading a new version, restore the original version. | File is restored correctly | |
| upload a huge file | File is uploaded, no problems found. | |
| upload a huge file, cancel the upload, restart the upload | Upload continues at the position where it was cancelled, file is uploaded completely. | |
| Remove a file. | File is removed correctly, it appears in the trashbin. | |
| Restore the deleted file from trashbin | File is restored correctly | |
| Remove multiple files that have the same name but are located in different folders | Files are removed correctly, they appear in the trashbin. | |
| Restore some of the deleted files from trashbin | Files are restored correctly in the correct folders. | |
| Restore some of the deleted files from trashbin, but delete the original containing folder before | Files are restored correctly | |
| Clean files from the trashbin | files are permanently deleted | |
| Create a lot of files, delete a lot of files, empty the trashbin | trashbin is cleaned | |
| Move a file inside a folder. | There are not problems on the process. | |
| Move a file inside a folder that already contains a file with the same name | File is not moved, content in the destination is not overwritten | |
| Create a folder. | Folder is created, no MKCOL problems appear. | |
| Create a folder with special characters as name | Folder is created if the name is legal otherwise an error is displayed | |
| Create a folder with a name of an already existing file/folder | Folder is not created, an error is displayed | |
| Create a folder with a lot of subfolders, use special characters in the name | Folder is created, no MKCOL problems appear. | |
| Delete a folder. | Folder is removed. | |
| Move a folder inside another. | No problems while moving the folder. | |
| open images in mediaviewer | files are displayed correctly. | |
| open videos in mediaviewer | files are displayed correctly. | |
| switch through videos and images in mediaviewer | files are displayed correctly. | |
| Share a file by public link. | Link is created and can be accessed. | |
| Share a folder by public link. | Link is created and can be accessed. | |
| Share a file with another user. | It is shared correctly. | |
| Share a folder with another user. | It is shared correctly. | |
| Share a file with a group. | It is shared correctly. | |
| Share a folder with a group. | It is shared correctly. | |
| Share a folder with userB giving edit permissions. As userB do CRUD operations on items inside the received folder | userB doesn't find any problem while interacting with files. | |
| Use your mobile device to access the UI | All elements reachable | |
| do tests mentioned in the [spaces](#spaces) section using the web-UI | | |
## Desktop Client
Prerequisite:
- [install the desktop client on your preferred OS](https://owncloud.com/desktop-app/)
- start ocis
- connect a new account in the desktop client to ocis e.g. `https://localhost:9200` if you are running the server and the client on the same machine
- accept the self-signed certificate
- you will be redirected to the browser, accept the certificate there also
- login as any user
- allow the "ownCloud desktop app" to access ocis
- select a folder to sync
| Test Case | Expected Result | Comment |
|---------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------|---------|
| Set up two clients with the same user. Change files, add some, delete some, move some, create folders. | Changes sync properly in both clients without errors. | |
| Share a file using contextual menu with userB. | Option to share appears in the contextual menu and file is correctly shared. | |
## Mobile Clients (iOS || Android)
Prerequisite:
- [install the ownCloud app on your mobile](https://owncloud.com/mobile-apps/)
- start oCIS as described in [Basic Remote Setup]({{< ref "../deployment/basic-remote-setup" >}})
- connect a new account in the mobile client to ocis
- accept the self-signed certificate
- you will be redirected to the browser, accept the certificate there also
- login as any user
- allow the mobile app to access ocis
| Test Case | Expected Result | Comment |
|-----------------------------------------------|------------------------------------------|---------|
| Connect to server, see files, download one. | No problems while downloading. | |
| Upload a file using mobile client. | No problems while uploading. | |
| Share a file with userB using mobile client. | File is correctly shared. | |
## other WebDAV clients
Prerequisite:
- start ocis with basic auth
| Test Case | Expected Result | Comment |
|---------------------------------------------------------------|-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| use ocis as webDAV external storage in ownCloud 10 | resource access works | |
| access webDAV with your file-manager | that will not give you a good UX, but ocis should not crash | Urls: https://\<ocis-server\>/remote.php/webdav & https://\<ocis-server\>/remote.php/dav/files/\<username\> |
| access webDAV with the "remote-files" function of LibreOffice | files are accessible and can be written back | |
# Tips for testing
## WebDav
WebDav is accessible under different path
- old: https://\<ocis-server\>/remote.php/webdav
- new: https://\<ocis-server\>/remote.php/dav/files/\<username\>
- spaces: https://\<ocis-server\>/dav/spaces/\<space-id\>
WebDav specifications can be found on http://webdav.org/
here some general WebDav request examples:
variable declaration:
```shell
SERVER_URI=https://localhost:9200
API_PATH=remote.php/webdav
USER=admin
PASSWORD=admin
```
- list content of root folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/" -X PROPFIND`
- list content of sub-folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/f1" -X PROPFIND`
- create a folder: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X MKCOL`
- delete a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X DELETE`
- rename / move a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X MOVE -H "Destination: $SERVER_URI/$API_PATH/renamed"`
- copy a resource: `curl -k -u $USER:$PASSWORD "$SERVER_URI/$API_PATH/folder" -X COPY -H "Destination: $SERVER_URI/$API_PATH/folder-copy"`
## decode HTTPS traffic with wireshark
To decode the HTTPS traffic we need the keys that were used to encrypt the traffic. Those keys are kept secret by the clients, but we can request the clients to save them in a specific file, so that wireshark can use them to decrypt the traffic again.
1. create key file: `touch /tmp/sslkey.log`
2. start wireshark
3. set log filename
- navigate to Edit=>Preferences=>Protocols=>TLS
- in the field `(Pre)-Master-Secret log filename` enter `/tmp/sslkey.log`
4. decode as HTTP
- navigate to Analyze=>Decode As...
- click the + button
- set Field: `TLS Port; Value=9200; Type: Integer, base 10; Default (none); Current HTTP` (adjust the port if you are using another one than 9200)
5. start recording
- use `port 9200` as capture filter to only record ocis packages
- use `http` as display filter to see only decoded traffic
6. run test-software with `SSLKEYLOGFILE=/tmp/sslkey.log` as env. variable e.g.
- curl: `SSLKEYLOGFILE=/tmp/sslkey.log curl -k -u admin:admin https://localhost:9200/ocs/v1.php/cloud/users`
- Browser: `SSLKEYLOGFILE=/tmp/sslkey.log firefox`
- LibreOffice: `SSLKEYLOGFILE=/tmp/sslkey.log libreoffice`
- acceptance tests: `SSLKEYLOGFILE=/tmp/sslkey.log make test-acceptance-api ...`
## format output
- piping **xml** results to `xmllint` gives you nice formats. E.g. `curl -k --user marie:radioactivity "https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares" | xmllint --format -`
- piping **json** results to `jq` gives you nice formats. E.g. `curl -k --user marie:radioactivity "https://localhost:9200/ocs/v1.php/apps/files_sharing/api/v1/shares?format=json" | jq`
## create edge cases
- [Big List of Naughty Strings](https://github.com/minimaxir/big-list-of-naughty-strings)

26
docs/ocis/development/build-docs.md
View File

@@ -1,26 +0,0 @@
---
title: "Documentation"
date: 2020-07-27T08:39:38+00:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: build-docs.md
---
{{< toc >}}
## Build the documentation
{{< hint info >}}
For building the documentation you need to have Make and Go 1.14 or higher installed.
{{< /hint >}}
Just run `make -C docs docs-serve` from within the root level of the oCIS git repository. This will make documentation available on [localhost:1313](http://localhost:1313) and also do a hot reload if you change something in the (non autogenerated) documentation files.
Note that you may get files changed reported like `env_vars.yaml` by just running the make command. You can discard any of these as they come from an internal helper process step necessary. To take care on those changed files, see the [Docs Helpers](https://github.com/owncloud/ocis/tree/master/docs/helpers) readme.
## Add changes to the documentation
Please keep this documentation in sync with the oCIS source code.
Changes on the documentation are automatically applied to [this site]({{< ref "../" >}}) when merged to the master branch.

57
docs/ocis/development/build.md
View File

@@ -1,57 +0,0 @@
---
title: "Build"
date: 2020-02-27T20:35:00+01:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: build.md
---
{{< toc >}}
## Build requirements
see [Development - Getting Started]({{< ref "getting-started/#requirements">}})
## Get the sources
{{< highlight txt >}}
git clone https://github.com/owncloud/ocis.git
cd ocis
{{< / highlight >}}
## Build the oCIS binary
You only need to run following command if you have changed protobuf definitions or the frontend part in one of the extensions. Run the command in the root directory of the repository. Otherwise you can skip this step and proceed to build the oCIS binary.
This will usually modify multiple `embed.go` files because we embed the frontend build output in these `embed.go` files and a timestamp will be updated and also minor differences are expected between different Node.js versions.
{{< highlight txt >}}
make generate
{{< / highlight >}}
The next step is to build the actual oCIS binary. Therefore you need to navigate to the subdirectory `ocis` and start the build process.
{{< highlight txt >}}
cd ocis
make build
{{< / highlight >}}
After the build process finished, you can find the binary within the `bin/` folder (in `ocis/bin` relative to the oCIS repository root folder).
Try to run it: `./bin/ocis h`
## Build a local oCIS docker image
If you are developing and want to run your local changes in a docker or docker-compose setup, you have to build an image locally.
Therefore run following commands in the root of the oCIS repository:
{{< highlight txt >}}
docker build -t owncloud/ocis:dev .
{{< / highlight >}}
Then you can test as usual via
{{< highlight txt >}}
docker run --rm -ti owncloud/ocis:dev
{{< / highlight >}}

56
docs/ocis/development/continuous-integration.md
View File

@@ -1,56 +0,0 @@
---
title: "Continuous Integration"
date: 2020-10-01T20:35:00+01:00
weight: 20
geekdocRepo: https://github.com/owncloud/ocis
geekdocEditPath: edit/master/docs/ocis/development
geekdocFilePath: continuous-integration.md
---
{{< toc >}}
oCIS uses [DRONE](https://www.drone.io/) as CI system. You can find the pipeline logs [here](https://drone.owncloud.com/owncloud/ocis) or in your PR.
## Concepts
The pipeline is defined in [Starlark](https://github.com/bazelbuild/starlark) and transformed to YAML upon pipeline run. This enables us to do a highly dynamic and non repeating pipeline configuration. We enforce Starlark format guidelines with Bazel Buildifier. You can format the .drone.star file by running `make ci-format`.
Upon running the pipeline, your branch gets merged to the master branch. This ensures that we always test your changeset if as it was applied to the master of oCIS. Please note that this does not apply to the pipeline definition (`.drone.star`).
## Things done in CI
- static code analysis
- linting
- running UI tests
- running ownCloud 10 test suite against oCIS
- build and release docker images
- build and release binaries
- build and release documentation
## Flags in commit message and PR title
You may add flags to your commit message or PR title in order to speed up pipeline runs and take load from the CI runners.
- `[CI SKIP]`: no CI is run on the commit or PR
- `[full-ci]`: deactivates the fail early mechanism and runs all available test (as default only smoke tests are run)
### Knowledge base
- My pipeline fails because some CI related files or commands are missing.
Please make sure to rebase your branch onto the latest master of oCIS. It could be that the pipeline definition (`.drone.star`) was changed on the master branch. This is the only file, that will not be auto merged to master upon pipeline run. So things could be out of sync.
- How can I see the YAML drone pipeline definition?
In order to see the Yaml pipeline definition you can use the drone-cli to convert the Starlark file.
```
drone starlark
```
{{< hint info >}}
If you experience a `"build" struct has no .title attribute` you need a newer version of drone-cli.
You currently need to build it yourself from this [source code](https://github.com/drone/drone-cli). If you are not using master as source, please ensure that this [PR](https://github.com/drone/drone-cli/pull/175) is included.
{{< /hint >}}

Some files were not shown because too many files have changed in this diff Show More