From 9def63eb783a21f6b96956b25fcdff279194bc70 Mon Sep 17 00:00:00 2001 From: jkoberg Date: Tue, 21 May 2024 16:11:54 +0200 Subject: [PATCH 1/5] feat(docs): add backup docs Signed-off-by: jkoberg --- docs/ocis/backup.md | 201 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 docs/ocis/backup.md diff --git a/docs/ocis/backup.md b/docs/ocis/backup.md new file mode 100644 index 0000000000..0178d90ea9 --- /dev/null +++ b/docs/ocis/backup.md @@ -0,0 +1,201 @@ +--- +title: "Backup Considerations" +date: 2024-05-07T10:31:00+01:00 +weight: 30 +geekdocRepo: https://github.com/owncloud/ocis +geekdocEditPath: edit/master/docs/ocis +geekdocFilePath: backup.md +--- + +This small guide aims to shed some light on the internal ocis data structure. You can refer to it when you are trying to optimize your backups or if you just curious how ocis stores its data. + +## Ocis data structure + +Ocis stores its data in folder that can be configured via the envvar `OCIS_BASE_DATA_PATH`. + +The default value for the `OCIS_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/etc/dev/ocis` when using the docker container. + +Inside this folder ocis will store all its data. That includes metadata, configuration, queues and stores. The actual bytes of the uploaded files are also stored here by default. If an s3 store is used as blobstore, the blobs need to be backed up seperately. + +### Base Data Path overview + +Listing the contents of the folder will return the following: +```bash + ~/.ocis/:tree -L 1 +. +├── config +├── idm +├── idp +├── nats +├── proxy +├── search +├── storage +├── thumbnails +└── web + +10 directories, 0 files +``` + +### `config` + +Contains basic ocis configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` envvar but for this readme we will assume this envvar is not set) + +```bash + ~/.ocis/config/:tree +. +└── ocis.yaml + +1 directory, 1 file +``` + +* `ocis.yaml`: BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However if not backed up completely, some fields MUST be copied over from the old config: + +| Field Name | Envvar Name | Description | If not backed up | +| --- | --- | --- | --- | +| `idp.ldap.bind_password` | `OCIS_LDAP_BIND_PASSWORD` | Password for the idp | no logins possible | +| `idm.service_user_passwords.idp_password`| `IDM_IDPSVC_PASSWORD` | Same as above | no logins possible | +| `system_user_id` | `OCIS_SYSTEM_USER_ID` | The id of storage-system user | no logins possible | +| `graph.identity.ldap.bind_password` | `GRAPH_LDAP_BIND_PASSWORD` | The password for the idm | no logins possible | +| `idm.service_user_passwords.idm_password` | `IDM_SVC_PASSWORD` | Same as above | no logins possible | +| `idm.service_user_passwords.reva_password`| `IDM_REVASVC_PASSWORD` | The reva password | no logins possible | +| `auth_basic.auth_providers.ldap.bind_password` | `AUTH_BASIC_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `users.drivers.ldap.bind_password` | `USERS_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `groups.drivers.ldap.bind_password` | `GROUPS_LDAP_BIND_PASSWORD` | Same as above | no logins possible | +| `storage_users.mount_id` | `STORAGE_USERS_MOUNT_ID` | The mountid of the storage_users service | sharing data lost | +| `gateway.storage_registry.storage_users_mount_id` | `GATEWAY_STORAGE_USERS_MOUNT_ID` | Same as above | sharing data lost | + +### `idm` + +Note: this folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case. + +Contains the data for the internal ocis identity management. See IDM README. + +```bash + ~/.ocis/idm/:tree +. +├── ldap.crt +├── ldap.key +└── ocis.boltdb + +1 directory, 3 files +``` + +* `ocis.boltdb`: BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, ocis will have no users, therefore also all data is lost. +* `ldap.crt`: BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up. +* `ldap.key`: BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to specify its path. Will be auto-generated if not backed up. + + +### `idp` + +Note: this folder will not appear if you use an external idp. Refer to your idps documentation for backup details in this case. + +Contains the data for the internal ocis identity provider. SEE IDP README. + +```bash + ~/.ocis/idp/:tree +. +├── encryption.key +├── private-key.pem +└── tmp + └── identifier-registration.yaml + +2 directories, 3 files +``` + +* `encryption.key`: BACKUP OPTIONAL. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Will be auto-generated if not backed up. +* `private-key.pem`: BACKUP OPTIONAL. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Will be auto-generated if not backed up. +* `identifier-registration.yml`: BACKUP RECOMMENDED. It holds temporary data of active sessions. Not backing this up will force users to relogin. + +### `nats` + +Note: this folder will not appear if you use an external nats installation + +Contains nats data for streams and stores. SEE NATS README. + +```bash + ~/.ocis/nats/:tree -L 1 +. +└── jetstream + +``` + +* `jetstream`: BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its part. Not backing it up can break history for multiple (non-vital) features such as history or notifications. Ocis functionality is not impacted. + +### `proxy` + +Contains proxy service data. SEE PROXY README. + +```bash + ~/.ocis/proxy/:tree +. +├── server.crt +└── server.key + +1 directory, 2 files +``` + +* `server.crt`: BACKUP OPTIONAL. This is the certificate for the http services. Use `PROXY_TRANSPORT_TLS_CERT` to specify its path. +* `server.key`: BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_TRANSPORT_TLS_KEY` to specify its path. + +### `search` + +Contains the search index. + +```bash + ~/.ocis/search/:tree -L 1 +. +└── bleve + +2 directories, 0 files +``` + +* `bleve`: BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backing it up the search index needs to be recreated. This can take a long time depending on amount of files. + +### `storage` + +Contains ocis meta (and blob) data. + +```bash + ~/.ocis/storage/:tree -L 1 +. +├── metadata +├── ocm +└── users + +4 directories, 0 files +``` + +* `metadata`: BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OCIS_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings. +* `ocm`: BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. +* `users`: BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. Rendering ocis empty, but functionally. + +### `thumbnails` + +Contains thumbnails data. + +```bash + ~/.ocis/thumbnails/:tree -L 1 +. +└── files +``` + +* `files`: OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up thumbnails will be regenerated automatically which leads to some load on thumbnails service. + +### `web` + +Contains web assests such as custom logos. + +```bash + ~/.ocis/web/:tree -L 1 +. +└── assets + +2 directories, 0 files +``` + +* `assests`: BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assests. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. + +### `external services` + +When using an external idp/idm/nats or blobstore its data needs to be backed up separately. Refer to your idp/idm/nats/blobstore documentation for backup details. + From 4b7a8113a66a6255e6dcf2409a6050717ffbb234 Mon Sep 17 00:00:00 2001 From: kobergj Date: Tue, 21 May 2024 16:54:08 +0200 Subject: [PATCH 2/5] feat(docs): backup docs review suggestions Co-authored-by: Martin Signed-off-by: jkoberg --- docs/ocis/backup.md | 46 ++++++++++++++++++++++++--------------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/docs/ocis/backup.md b/docs/ocis/backup.md index 0178d90ea9..93e9c82435 100644 --- a/docs/ocis/backup.md +++ b/docs/ocis/backup.md @@ -7,17 +7,19 @@ geekdocEditPath: edit/master/docs/ocis geekdocFilePath: backup.md --- -This small guide aims to shed some light on the internal ocis data structure. You can refer to it when you are trying to optimize your backups or if you just curious how ocis stores its data. +This small guide aims to shed some light on the internal ocis data structure. You can refer to it when you are trying to optimize your backups or if you are just curious about how ocis stores its data. -## Ocis data structure +Note, as a prerequisite backing up ocis, the instance has to be fully shut down for the time being. -Ocis stores its data in folder that can be configured via the envvar `OCIS_BASE_DATA_PATH`. +## Ocis Data Structure -The default value for the `OCIS_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/etc/dev/ocis` when using the docker container. +Ocis stores its data in a folder that can be configured via the envvar `OCIS_BASE_DATA_PATH`. Without further configuration, services derive from that path when they store data, though individual settings can be configured. -Inside this folder ocis will store all its data. That includes metadata, configuration, queues and stores. The actual bytes of the uploaded files are also stored here by default. If an s3 store is used as blobstore, the blobs need to be backed up seperately. +The default value for the `OCIS_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/var/lib/ocis` when using the docker container. Note: Configuration data is by default stored in `/etc/ocis/` in the container.). -### Base Data Path overview +Inside this folder, ocis will store all its data separated by subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files which are currently uploaded are also stored here by default. If an s3 store is used as blobstore, the blobs need to be backed up separately. Note: See special case for the `config` folder in a docker container. + +### Base Data Path Overview Listing the contents of the folder will return the following: ```bash @@ -36,6 +38,8 @@ Listing the contents of the folder will return the following: 10 directories, 0 files ``` +The following sections describe the content and background of the subdirectories to decide if a backup is required or recommended and its effect when it is not backed up. + ### `config` Contains basic ocis configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` envvar but for this readme we will assume this envvar is not set) @@ -48,7 +52,7 @@ Contains basic ocis configuration created by `ocis init`(Note: The location of t 1 directory, 1 file ``` -* `ocis.yaml`: BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However if not backed up completely, some fields MUST be copied over from the old config: +* `ocis.yaml`: BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: | Field Name | Envvar Name | Description | If not backed up | | --- | --- | --- | --- | @@ -66,9 +70,9 @@ Contains basic ocis configuration created by `ocis init`(Note: The location of t ### `idm` -Note: this folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case. +Note: This folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case. -Contains the data for the internal ocis identity management. See IDM README. +Contains the data for the internal ocis identity management. See the IDM README for more details. ```bash ~/.ocis/idm/:tree @@ -87,9 +91,9 @@ Contains the data for the internal ocis identity management. See IDM README. ### `idp` -Note: this folder will not appear if you use an external idp. Refer to your idps documentation for backup details in this case. +Note: This folder will not appear if you use an external idp. Refer to your idp's documentation for backup details in this case. -Contains the data for the internal ocis identity provider. SEE IDP README. +Contains the data for the internal ocis identity provider. See the IDP README for more details. ```bash ~/.ocis/idp/:tree @@ -108,9 +112,9 @@ Contains the data for the internal ocis identity provider. SEE IDP README. ### `nats` -Note: this folder will not appear if you use an external nats installation +Note: This folder will not appear if you use an external nats installation -Contains nats data for streams and stores. SEE NATS README. +Contains nats data for streams and stores. See the NATS README for more details. ```bash ~/.ocis/nats/:tree -L 1 @@ -119,11 +123,11 @@ Contains nats data for streams and stores. SEE NATS README. ``` -* `jetstream`: BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its part. Not backing it up can break history for multiple (non-vital) features such as history or notifications. Ocis functionality is not impacted. +* `jetstream`: BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The ocis functionality is not impacted if omitted. ### `proxy` -Contains proxy service data. SEE PROXY README. +Contains proxy service data. See the PROXY README for more details. ```bash ~/.ocis/proxy/:tree @@ -149,7 +153,7 @@ Contains the search index. 2 directories, 0 files ``` -* `bleve`: BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backing it up the search index needs to be recreated. This can take a long time depending on amount of files. +* `bleve`: BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files. ### `storage` @@ -167,7 +171,7 @@ Contains ocis meta (and blob) data. * `metadata`: BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OCIS_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings. * `ocm`: BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. -* `users`: BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. Rendering ocis empty, but functionally. +* `users`: BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty ocis instance, which is fully functional accepting new data. Old data is lost. ### `thumbnails` @@ -179,11 +183,11 @@ Contains thumbnails data. └── files ``` -* `files`: OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up thumbnails will be regenerated automatically which leads to some load on thumbnails service. +* `files`: OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service. ### `web` -Contains web assests such as custom logos. +Contains web assets such as custom logos, themes etc. ```bash ~/.ocis/web/:tree -L 1 @@ -193,9 +197,9 @@ Contains web assests such as custom logos. 2 directories, 0 files ``` -* `assests`: BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assests. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. +* `assets`: BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assets. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. ### `external services` -When using an external idp/idm/nats or blobstore its data needs to be backed up separately. Refer to your idp/idm/nats/blobstore documentation for backup details. +When using an external idp/idm/nats or blobstore, its data needs to be backed up separately. Refer to your idp/idm/nats/blobstore documentation for backup details. From d0d319d7ded89282f94a661dec5cae5ffd98041d Mon Sep 17 00:00:00 2001 From: jkoberg Date: Wed, 22 May 2024 09:25:41 +0200 Subject: [PATCH 3/5] feat(docs): fix backup docs Signed-off-by: jkoberg --- docs/ocis/backup.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/ocis/backup.md b/docs/ocis/backup.md index 93e9c82435..7e2786fbfd 100644 --- a/docs/ocis/backup.md +++ b/docs/ocis/backup.md @@ -59,8 +59,6 @@ Contains basic ocis configuration created by `ocis init`(Note: The location of t | `idp.ldap.bind_password` | `OCIS_LDAP_BIND_PASSWORD` | Password for the idp | no logins possible | | `idm.service_user_passwords.idp_password`| `IDM_IDPSVC_PASSWORD` | Same as above | no logins possible | | `system_user_id` | `OCIS_SYSTEM_USER_ID` | The id of storage-system user | no logins possible | -| `graph.identity.ldap.bind_password` | `GRAPH_LDAP_BIND_PASSWORD` | The password for the idm | no logins possible | -| `idm.service_user_passwords.idm_password` | `IDM_SVC_PASSWORD` | Same as above | no logins possible | | `idm.service_user_passwords.reva_password`| `IDM_REVASVC_PASSWORD` | The reva password | no logins possible | | `auth_basic.auth_providers.ldap.bind_password` | `AUTH_BASIC_LDAP_BIND_PASSWORD` | Same as above | no logins possible | | `users.drivers.ldap.bind_password` | `USERS_LDAP_BIND_PASSWORD` | Same as above | no logins possible | @@ -106,9 +104,9 @@ Contains the data for the internal ocis identity provider. See the IDP README fo 2 directories, 3 files ``` -* `encryption.key`: BACKUP OPTIONAL. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Will be auto-generated if not backed up. -* `private-key.pem`: BACKUP OPTIONAL. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Will be auto-generated if not backed up. -* `identifier-registration.yml`: BACKUP RECOMMENDED. It holds temporary data of active sessions. Not backing this up will force users to relogin. +* `encryption.key`: BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Not backing this up will force users to relogin. +* `private-key.pem`: BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin. +* `identifier-registration.yml`: BACKUP OPTIONAL. It holds temporary data of active sessions. Will be recreated if not backed up. ### `nats` From 21e0fd4755cf2732fd60b7cc365b42833cf6bb46 Mon Sep 17 00:00:00 2001 From: kobergj Date: Wed, 22 May 2024 09:50:19 +0200 Subject: [PATCH 4/5] feat(docs): backup review pII Co-authored-by: Martin --- docs/ocis/backup.md | 50 ++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/ocis/backup.md b/docs/ocis/backup.md index 7e2786fbfd..b0f7ca7820 100644 --- a/docs/ocis/backup.md +++ b/docs/ocis/backup.md @@ -42,7 +42,7 @@ The following sections describe the content and background of the subdirectories ### `config` -Contains basic ocis configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` envvar but for this readme we will assume this envvar is not set) +Contains basic ocis configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` envvar but for this readme we will assume this envvar is not set and the default is used.) ```bash ~/.ocis/config/:tree @@ -52,7 +52,8 @@ Contains basic ocis configuration created by `ocis init`(Note: The location of t 1 directory, 1 file ``` -* `ocis.yaml`: BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: +* `ocis.yaml`:\ +BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: | Field Name | Envvar Name | Description | If not backed up | | --- | --- | --- | --- | @@ -82,9 +83,12 @@ Contains the data for the internal ocis identity management. See the IDM README 1 directory, 3 files ``` -* `ocis.boltdb`: BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, ocis will have no users, therefore also all data is lost. -* `ldap.crt`: BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up. -* `ldap.key`: BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to specify its path. Will be auto-generated if not backed up. +* `ocis.boltdb`:\ +BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, ocis will have no users, therefore also all data is lost. +* `ldap.crt`:\ +BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up. +* `ldap.key`:\ +BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to specify its path. Will be auto-generated if not backed up. ### `idp` @@ -104,9 +108,12 @@ Contains the data for the internal ocis identity provider. See the IDP README fo 2 directories, 3 files ``` -* `encryption.key`: BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Not backing this up will force users to relogin. -* `private-key.pem`: BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin. -* `identifier-registration.yml`: BACKUP OPTIONAL. It holds temporary data of active sessions. Will be recreated if not backed up. +* `encryption.key`:\ +BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FILE` to specify its paths. Not backing this up will force users to relogin. +* `private-key.pem`:\ +BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin. +* `identifier-registration.yml`:\ +BACKUP OPTIONAL. It holds temporary data of active sessions. Will be recreated if not backed up. ### `nats` @@ -121,7 +128,8 @@ Contains nats data for streams and stores. See the NATS README for more details. ``` -* `jetstream`: BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The ocis functionality is not impacted if omitted. +* `jetstream`:\ +BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The ocis functionality is not impacted if omitted. ### `proxy` @@ -136,8 +144,10 @@ Contains proxy service data. See the PROXY README for more details. 1 directory, 2 files ``` -* `server.crt`: BACKUP OPTIONAL. This is the certificate for the http services. Use `PROXY_TRANSPORT_TLS_CERT` to specify its path. -* `server.key`: BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_TRANSPORT_TLS_KEY` to specify its path. +* `server.crt`:\ +BACKUP OPTIONAL. This is the certificate for the http services. Use `PROXY_TRANSPORT_TLS_CERT` to specify its path. +* `server.key`:\ +BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_TRANSPORT_TLS_KEY` to specify its path. ### `search` @@ -151,7 +161,8 @@ Contains the search index. 2 directories, 0 files ``` -* `bleve`: BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files. +* `bleve`:\ +BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified via `SEARCH_ENGINE_BLEVE_DATA_PATH`. If not backed up, the search index needs to be recreated. This can take a long time depending on the amount of files. ### `storage` @@ -167,9 +178,12 @@ Contains ocis meta (and blob) data. 4 directories, 0 files ``` -* `metadata`: BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OCIS_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings. -* `ocm`: BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. -* `users`: BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty ocis instance, which is fully functional accepting new data. Old data is lost. +* `metadata`:\ +BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM_OCIS_ROOT`. Not backing it up will remove shares from the system and will also remove custom settings. +* `ocm`:\ +BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. +* `users`:\ +BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty ocis instance, which is fully functional accepting new data. Old data is lost. ### `thumbnails` @@ -181,7 +195,8 @@ Contains thumbnails data. └── files ``` -* `files`: OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service. +* `files`:\ +OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specified via `THUMBNAILS_FILESYSTEMSTORAGE_ROOT`. If not backed up, thumbnails will be regenerated automatically on access which leads to some load on the thumbnails service. ### `web` @@ -195,7 +210,8 @@ Contains web assets such as custom logos, themes etc. 2 directories, 0 files ``` -* `assets`: BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assets. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. +* `assets`:\ +BACKUP RECOMMENDED/OMITABLE. This folder contains custom web assets. Can be specified via `WEB_ASSET_CORE_PATH`. If no custom web assets are used, there is no need for a backup. If those exist but are not backed up, they need to be reuploaded. ### `external services` From 9626bcddee66b281b141c69c2e90f34f3ef5e76f Mon Sep 17 00:00:00 2001 From: kobergj Date: Thu, 23 May 2024 09:05:48 +0200 Subject: [PATCH 5/5] feat(docs): backup review pIII Co-authored-by: Klaas Freitag Signed-off-by: jkoberg --- docs/ocis/backup.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/ocis/backup.md b/docs/ocis/backup.md index b0f7ca7820..093e180b46 100644 --- a/docs/ocis/backup.md +++ b/docs/ocis/backup.md @@ -7,17 +7,17 @@ geekdocEditPath: edit/master/docs/ocis geekdocFilePath: backup.md --- -This small guide aims to shed some light on the internal ocis data structure. You can refer to it when you are trying to optimize your backups or if you are just curious about how ocis stores its data. +This small guide aims to shed some light on the internal Infinite Scale data structure. You can refer to it when you are trying to optimize your backups or if you are just curious about how Infinite Scale stores its data. -Note, as a prerequisite backing up ocis, the instance has to be fully shut down for the time being. +Note, as a prerequisite backing up Infinite Scale, the instance has to be fully shut down for the time being. ## Ocis Data Structure -Ocis stores its data in a folder that can be configured via the envvar `OCIS_BASE_DATA_PATH`. Without further configuration, services derive from that path when they store data, though individual settings can be configured. +Ocis stores its data in a folder that can be configured via the environment variable `OCIS_BASE_DATA_PATH`. Without further configuration, services derive from that path when they store data, though individual settings for certain data types can be configured. The default value for the `OCIS_BASE_DATA_PATH` variable is `$HOME/.ocis` (or `/var/lib/ocis` when using the docker container. Note: Configuration data is by default stored in `/etc/ocis/` in the container.). -Inside this folder, ocis will store all its data separated by subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files which are currently uploaded are also stored here by default. If an s3 store is used as blobstore, the blobs need to be backed up separately. Note: See special case for the `config` folder in a docker container. +Inside this folder, Infinite Scale will store all its data in separate subdirectories. That includes metadata, configurations, queues and stores etc. The actual bytes of files (blobs) are handled by a so called blobstore, which also stores here by default. Depending on the used blobstore, the blobs need to be backed up separately, for example if S3 is used. Note: See special case for the `config` folder in a docker container. ### Base Data Path Overview @@ -42,7 +42,7 @@ The following sections describe the content and background of the subdirectories ### `config` -Contains basic ocis configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` envvar but for this readme we will assume this envvar is not set and the default is used.) +Contains basic Infinite Scale configuration created by `ocis init`(Note: The location of the configuration folder can be specified with the `OCIS_CONFIG_DIR` environment variable, but for this document we will assume this variable is not set and the default is used.) ```bash ~/.ocis/config/:tree @@ -53,7 +53,7 @@ Contains basic ocis configuration created by `ocis init`(Note: The location of t ``` * `ocis.yaml`:\ -BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: +BACKUP RECOMMENDED. Holds Infinite Scale configuration data. The contents can vary depending on your environment variables. In general, most of this file can be recreated again by running `ocis init`. This will recreate secrets and certificates. However, if not backed up completely, some fields MUST be copied over from the old config manually to regain data access after a restore: | Field Name | Envvar Name | Description | If not backed up | | --- | --- | --- | --- | @@ -71,7 +71,7 @@ BACKUP RECOMMENDED. Holds ocis configuration data. The contents can vary dependi Note: This folder will not appear if you use an external idm. Refer to your idms documentation for backup details in this case. -Contains the data for the internal ocis identity management. See the IDM README for more details. +Contains the data for the internal Infinite Scale identity management. See the [IDM README]({{< ref "../services/idm/_index.md" >}}) for more details. ```bash ~/.ocis/idm/:tree @@ -84,7 +84,7 @@ Contains the data for the internal ocis identity management. See the IDM README ``` * `ocis.boltdb`:\ -BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, ocis will have no users, therefore also all data is lost. +BACKUP REQUIRED. This is the boltdb database that stores user data. Use `IDM_DATABASE_PATH` to specify its path. If not backed up, Infinite Scale will have no users, therefore also all data is lost. * `ldap.crt`:\ BACKUP OPTIONAL. This is the certificate for the idm. Use `IDM_LDAPS_CERT` to specify its path. Will be auto-generated if not backed up. * `ldap.key`:\ @@ -95,7 +95,7 @@ BACKUP OPTIONAL. This is the certificate key for the idm. Use `IDM_LDAPS_KEY` to Note: This folder will not appear if you use an external idp. Refer to your idp's documentation for backup details in this case. -Contains the data for the internal ocis identity provider. See the IDP README for more details. +Contains the data for the internal Infinite Scale identity provider. See the [IDP README]({{< ref "../services/idp/_index.md" >}}) for more details. ```bash ~/.ocis/idp/:tree @@ -113,13 +113,13 @@ BACKUP RECOMMENDED. This is the encryption secret. Use `IDP_ENCRYPTION_SECRET_FI * `private-key.pem`:\ BACKUP RECOMMENDED. This is the encryption key. Use `IDP_SIGNING_PRIVATE_KEY_FILES` to specify its paths. Not backing this up will force users to relogin. * `identifier-registration.yml`:\ -BACKUP OPTIONAL. It holds temporary data of active sessions. Will be recreated if not backed up. +BACKUP OPTIONAL. It holds configuration for oidc clients (web, desktop, ios, android). Will be recreated if not backed up. ### `nats` -Note: This folder will not appear if you use an external nats installation +Note: This folder will not appear if you use an external nats installation. In that case, data has to secured in alignment with the external installation. -Contains nats data for streams and stores. See the NATS README for more details. +Contains nats data for streams and stores. See the [NATS README]({{< ref "../services/nats/_index.md" >}}) for more details. ```bash ~/.ocis/nats/:tree -L 1 @@ -129,11 +129,11 @@ Contains nats data for streams and stores. See the NATS README for more details. ``` * `jetstream`:\ -BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The ocis functionality is not impacted if omitted. +BACKUP RECOMMENDED. This folder contains nats data about streams and key-value stores. Use `NATS_NATS_STORE_DIR` to specify its path. Not backing it up can break history for multiple (non-vital) features such as history or notifications. The Infinite Scale functionality is not impacted if omitted. ### `proxy` -Contains proxy service data. See the PROXY README for more details. +Contains proxy service data. See the [PROXY README]({{< ref "../services/proxy/_index.md" >}}) for more details. ```bash ~/.ocis/proxy/:tree @@ -151,7 +151,7 @@ BACKUP OPTIONAL. This is the certificate key for the http services. Use `PROXY_T ### `search` -Contains the search index. +Contains the search index. See the [SEARCH README]({{< ref "../services/search/_index.md" >}}) for more details. ```bash ~/.ocis/search/:tree -L 1 @@ -166,7 +166,7 @@ BACKUP RECOMMENDED/OPTIONAL. This contains the search index. Can be specified vi ### `storage` -Contains ocis meta (and blob) data. +Contains Infinite Scale meta (and blob) data, depending on the blobstore. See the [STORAGE-USERS README]({{< ref "../services/storage-users/_index.md" >}}) for more details. ```bash ~/.ocis/storage/:tree -L 1 @@ -183,11 +183,11 @@ BACKUP REQUIRED. Contains system data. Path can be specified via `STORAGE_SYSTEM * `ocm`:\ BACKUP REQUIRED/OMITABLE. Contains ocm share data. When not using ocm sharing, this folder does not need to be backed up. * `users`:\ -BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty ocis instance, which is fully functional accepting new data. Old data is lost. +BACKUP REQUIRED. Contains user data. Path can be specified via `STORAGE_USERS_OCIS_ROOT`. Not backing it up will remove all spaces and all files. As result, you will have a configured but empty Infinite Scale instance, which is fully functional accepting new data. Old data is lost. ### `thumbnails` -Contains thumbnails data. +Contains thumbnails data. See the [THUMBNAILS README]({{< ref "../services/thumbnails/_index.md" >}}) for more details. ```bash ~/.ocis/thumbnails/:tree -L 1 @@ -200,7 +200,7 @@ OPTIONAL/RECOMMENDED. This folder contains prerendered thumbnails. Can be specif ### `web` -Contains web assets such as custom logos, themes etc. +Contains web assets such as custom logos, themes etc. See the [WEB README]({{< ref "../services/web/_index.md" >}}) for more details. ```bash ~/.ocis/web/:tree -L 1