From 291710deb975e77e7a6c4ff28ebd40db43c3f50a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Thu, 18 Mar 2021 15:59:33 +0000 Subject: [PATCH] incorporate comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Jörn Friedrich Dreyer --- docs/adr/0004-support-hot-migration.md | 24 ++++++++--- docs/ocis/migration.md | 58 +++++++++++++------------- 2 files changed, 46 insertions(+), 36 deletions(-) diff --git a/docs/adr/0004-support-hot-migration.md b/docs/adr/0004-support-hot-migration.md index d7e8b5ceb2..f7a271254c 100644 --- a/docs/adr/0004-support-hot-migration.md +++ b/docs/adr/0004-support-hot-migration.md @@ -13,13 +13,22 @@ Migration is one of the most important topics of the oCIS story. We need to prov ## Decision Drivers - Do not lose file blob or meta data. + - To prevent a sync surge from clients the etag for files should be migrated. + - To prevent internal links from breaking or pointing to wrong files the file id of existing files needs to be migrated. + - To prevent data loss trash and version blobs should be migrated. - Existing shares like public links and federated shares must remain functional after the migration. + - To prevent internal shares the share type, permissions and expiry needs to be migrated. + - To prevent public links from breaking the url token, permissions, expiry and password needs to be migrated. + - *What about federated shares?* + - *What about additional share permissions, eg. comment on office files?* - Legacy clients need to keep working + - To keep existing clients working the `remote.php/webdav` and `dav/files/` webdav endpoints as well as the ocs API need to be available. +- *What about [app passwords/tokens](https://doc.owncloud.com/server/user_manual/personal_settings/security.html#app-passwords-tokens)?* ## Considered Options -- Cold Migration -- Hot Migration +1. Cold Migration: migrate data while systems are not online, so no user interaction happens in between. +2. Hot Migration: one or both systems are online during migration. ## Decision Outcome @@ -39,23 +48,26 @@ Chosen option: "\[option 1\]", because \[justification. e.g., only option, which ### Cold Migration -The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. +The migration happens while the service is offline. File metadata, blobs and share data is exported from ownCloud 10 and imported in oCIS. This can happen user by user, where every user export would contain the file blobs, their metadata, trash, versions, shares and all metadata that belongs to the users storage. To prevent group shares from breaking, users in the same groups must be migrated in batch. Depending on the actual group shares in an instance this may effectively require a complete migration in a single batch. - Good, because oCIS can be tested in a staging system without writing to the production system. -- Good, because file layout can be changed together with the architecture migration. +- Good, because file layout on disk can be changed to support new storage driver capabilities. - Bad, because the export and import might require significant amounts of storage. - Bad, because a rollback to the state before the migration might cause data loss of the changes that happend in between. +- Bad, because the cold migration can mean significant downtime. ### Hot Migration The migration happens in subsequent stages while the service is online. -- Good, because users can switch between the backends on the fly. +- Good, because the admin can migrate users from old to new backend in a controlled way. - Good, because users and admins can learn to trust the new system. +- Good, because there can be preparations even long before the migrations happens in parallel on the oC10 codebase, ie. addition of metadata that is needed while the system operates. +- Good, because the downtime of the system can be fairly small. - Bad, because it is more complex and might drag on for a long time. ## Links -- [Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/1377) because the share manager for ocis should store share information on the storage system. And [storageprovider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)](https://github.com/cs3org/cs3apis/issues/93) finally: [eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/543) \ No newline at end of file +- [Clarify responsibilities of share providers and storage providers · Issue #1377 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/1377) because the share manager for oCIS should store share information on the storage system. And [storageprovider should persist share creator · Issue #93 · cs3org/cs3apis (github.com)](https://github.com/cs3org/cs3apis/issues/93) finally: [eos: store share id in inherited xattr · Issue #543 · cs3org/reva (github.com)](https://github.com/cs3org/reva/issues/543) \ No newline at end of file diff --git a/docs/ocis/migration.md b/docs/ocis/migration.md index 259523b320..b7c0c15e39 100644 --- a/docs/ocis/migration.md +++ b/docs/ocis/migration.md @@ -18,25 +18,29 @@ As an admin I want to migrate certain groups of users before others. As a user, I need a seamless migration and not lose data by any chance. ### Stage-0 -Is the pre-migration stage having a functional ownCloud 10 instance +Is the pre-migration stage having a functional ownCloud 10 instance. ### Stage-1 Introduce OpenID Connect to server and Clients ### Stage-2 -Introduce new web ui, let end users get used to it voluntarily +Install and introduce ownCloud Web and let users test it voluntarily. ### Stage-3 -start oCIS backend and make read only tests on existing data using owncloud storage driver and database backed share manager +Start oCIS backend and make read only tests on existing data using the `owncloud` storage driver which will read (and write) +- blobs from the same datadirectory layout as in ownCloud 10 and +- metadata from the ownCloud 10 databas +The oCIS share manager will read share infomation from the owncloud database as well. +- [ ] *we need a share manager that can read from the oc 10 db as well as from whatever new backend will be used for a pure oCIS setup. Currently, that would be the json file. Or that is migrated after all users have switched to oCIS. ~~~~ jfd* ### Stage-4 -test updating data using oCIS backend +Test writing data with oCIS into the existing ownCloud 10 datafolder using the `owncloud` storage driver. ### Stage-5 -introduce reverse proxy and switch over early adoptors, let admins gain trust in the new backend +Introduce reverse proxy and switch over early adoptors, let admins gain trust in the new backend by comparing metrics of the two systems and having it running in parallel. ### Stage-6 -voluntary transition period and subsequent hard deadline for all users +Voluntary transition period and subsequent hard deadline for all users ### Stage-7 disable oc10 in the proxy, all requests are now handled by oCIS, shut down oc10 web servers and redis (or keep for calendar & contacts only? rip out files from oCIS?) @@ -58,7 +62,11 @@ The fundamental difference between ownCloud 10 and oCIS is that the file metadat ## Data that will be migrated -Currently, oCIS focuses on file sync and share use cases. In ownCloud 10 the files are laid out on disk in the *data directory* using the following layout: +Currently, oCIS focuses on file sync and share use cases. + +### Blob data + +In ownCloud 10 the files are laid out on disk in the *data directory* using the following layout: ``` data ├── einstein @@ -101,6 +109,13 @@ The *data directory* may also contain subfolders for owncloud 10 applications li When an objectstorage is used as the primary storage all file blobs are stored by their file id and a prefix, eg.: `urn:oid:`. +The three types of blobs we need to migrate are stored in +- `files` for file blobs, the current file content, +- `files_trashbin` for trashed files (and their versions) and +- `files_versions` for file blobs of older versions. + +### Filecache table + In both cases the file metadata, including a full replication of the file tree, is stored in the `oc_filecache` table of an ownCloud 10 database. The primary key of a row is the file id. It is used to attach additional metadata like shares, tags, favorites or arbitrary file properties. The `filecache` table itself has more metadata: @@ -127,34 +142,17 @@ The `filecache` table itself has more metadata: > Note: for EOS a hot migration only works seamlessly if file ids in oc10 are already read from eos. otherwise either a mapping from the oc10 filecache file id to the new eos file id has to be created under the assumption that these id sets do not intersect or files and corresponding shares need to be exported and imported offline to generate a new set of ids. While this will preserve public links, user, group and even federated shares, old internal links may still point to different files because they contain the oc10 fileid -### File blob data +### share table - -## Goal - -This ticket is about to decide which data of a typical oC10 installation can and will be migrated to oCIS. - -Examples: -- User files -- Trash -- Versions -- Users, groups and permissions - Public links - Private shares with users and groups -- Federated shares +- Federated shares *incoming shares table ~~~~ jfd* - Guest shares -- Activities -## User Stories +### user data -- As an admin I need a time efficient way of migration. -- As an admin I need to be able to explain which data was migrated and which not, to which consequences. -- As a user I want a seamless migration of my data. -- As a user I do not want to loose work that I put into the system, ie. by setting up shares. -- As a user I want to understand which data can not be migrated. +- Users, groups and permissions (who is admin) -## Acceptance Criteria +### activities -- ADR that describes which data from oC10 will be migrated and which not. -- If data changes it's meaning or used differently (ie. Groups vs Roles) that is explained and the consequences are clear. -- It is explained which data is not migrated. +*dedicated service, not yet implemented, requires decisions about an event system ~~~~ jfd*