From a19dc15c638abbb29c3b33db3d2531fa28db0855 Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Tue, 12 Apr 2022 10:02:16 +0545 Subject: [PATCH 01/20] Bump core commit id for tests --- .drone.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.drone.env b/.drone.env index c3e9ae08dc..8cc1810632 100644 --- a/.drone.env +++ b/.drone.env @@ -1,5 +1,5 @@ # The test runner source for API tests -CORE_COMMITID=574b3c8bc75df34cae8a41ad5ed0524ec08c5519 +CORE_COMMITID=5e9628823c81b39d5792a7fec398da7396bc8c4d CORE_BRANCH=master # The test runner source for UI tests From a05ba053e417a3024c5a1a2599321eed97d2ef94 Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Tue, 12 Apr 2022 10:43:17 +0545 Subject: [PATCH 02/20] Update expected to fail --- tests/acceptance/expected-failures-API-on-OCIS-storage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/acceptance/expected-failures-API-on-OCIS-storage.md b/tests/acceptance/expected-failures-API-on-OCIS-storage.md index 210168733a..57f4063c40 100644 --- a/tests/acceptance/expected-failures-API-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-API-on-OCIS-storage.md @@ -921,8 +921,8 @@ _ocs: api compatibility, return correct status code_ #### [reshared resource is not listed for sharee after accepting share](https://github.com/owncloud/ocis/issues/2214) -- [apiShareReshareToShares2/reShareSubfolder.feature:180](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares2/reShareSubfolder.feature#L180) -- [apiShareReshareToShares2/reShareSubfolder.feature:181](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares2/reShareSubfolder.feature#L181) +- [apiShareReshareToShares2/reShareSubfolder.feature:178](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares2/reShareSubfolder.feature#L178) +- [apiShareReshareToShares2/reShareSubfolder.feature:179](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares2/reShareSubfolder.feature#L179) #### [various sharing settings cannot be set](https://github.com/owncloud/ocis/issues/1328) From ccc77c3771adc4bafa93f8bacf27855c07254f2c Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Tue, 12 Apr 2022 12:13:52 +0545 Subject: [PATCH 03/20] Update commit id --- .drone.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.drone.env b/.drone.env index 8cc1810632..99d6d95153 100644 --- a/.drone.env +++ b/.drone.env @@ -1,5 +1,5 @@ # The test runner source for API tests -CORE_COMMITID=5e9628823c81b39d5792a7fec398da7396bc8c4d +CORE_COMMITID=d21d38aac929918c48d4ee5c44741dc305e875bb CORE_BRANCH=master # The test runner source for UI tests From bf1ee8f9efaac42c0907603cf42df67348dc8c21 Mon Sep 17 00:00:00 2001 From: Viktor Scharf Date: Tue, 12 Apr 2022 08:51:35 +0200 Subject: [PATCH 04/20] [tests-only] apiTest: list of trash files and restore files from the trash (#3462) * apiTest: see trashbin and restore files from trash --- ...ected-failures-localAPI-on-OCIS-storage.md | 2 +- .../apiSpaces/removeSpaceObjects.feature | 155 ++++++---------- .../apiSpaces/restoreSpaceObjects.feature | 67 +++++++ .../features/bootstrap/SpacesContext.php | 174 +++++++++++++++++- 4 files changed, 291 insertions(+), 107 deletions(-) create mode 100644 tests/acceptance/features/apiSpaces/restoreSpaceObjects.feature diff --git a/tests/acceptance/expected-failures-localAPI-on-OCIS-storage.md b/tests/acceptance/expected-failures-localAPI-on-OCIS-storage.md index c70e6b623b..ae26ed68e5 100644 --- a/tests/acceptance/expected-failures-localAPI-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-localAPI-on-OCIS-storage.md @@ -23,6 +23,6 @@ The expected failures in this file are from features in the owncloud/ocis repo. - [apiSpaces/quota.feature:56](https://github.com/owncloud/ocis/blob/master/tests/acceptance/features/apiSpaces/quota.feature#L56) #### [Viewer and editor has the possibility to disable the space](https://github.com/owncloud/ocis/issues/3031) -- [apiSpaces/removeSpaceObjects.feature:121](https://github.com/owncloud/ocis/blob/master/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature#L121) +- [apiSpaces/removeSpaceObjects.feature:74](https://github.com/owncloud/ocis/blob/master/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature#L74) - [apiSpaces/deleteSpaces.feature:73](https://github.com/owncloud/ocis/blob/master/tests/acceptance/features/apiSpaces/deleteSpaces.feature#L73) - [apiSpaces/deleteSpaces.feature:84](https://github.com/owncloud/ocis/blob/master/tests/acceptance/features/apiSpaces/deleteSpaces.feature#L84) diff --git a/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature b/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature index aa15aa89d7..95112c0619 100644 --- a/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature +++ b/tests/acceptance/features/apiSpaces/removeSpaceObjects.feature @@ -3,126 +3,79 @@ Feature: Remove files, folder As a user I want to be able to remove files, folders Users with the editor role can also remove objects - Users with the viewer role cannot remove objects + Users with the viewer role cannot remove objects Note - this feature is run in CI with ACCOUNTS_HASH_DIFFICULTY set to the default for production See https://github.com/owncloud/ocis/issues/1542 and https://github.com/owncloud/ocis/pull/839 Background: - Given user "Alice" has been created with default attributes and without skeleton files - And user "Brian" has been created with default attributes and without skeleton files + Given these users have been created with default attributes and without skeleton files: + | username | + | Alice | + | Brian | And the administrator has given "Alice" the role "Admin" using the settings api + And user "Alice" has created a space "delete objects" with the default quota using the GraphApi + And user "Alice" has created a folder "folderForDeleting/sub1/sub2" in space "delete objects" + And user "Alice" has uploaded a file inside space "delete objects" with content "some content" to "text.txt" -# owner of space (admin permissions) - Scenario: An owner can delete a folder with some subfolders in a Space via the webDav API - Given user "Alice" has created a space "Owner deletes folder" of type "project" with quota "10" - And user "Alice" has created a folder "folderForDeleting/sub1/sub2" in space "Owner deletes folder" - When user "Alice" removes the object "folderForDeleting" from space "Owner deletes folder" - Then the HTTP status code should be "204" - And for user "Alice" the space "Owner deletes folder" should not contain these entries: + Scenario Outline: An user deletes a folder with some subfolders in a Space via the webDav API + Given user "Alice" has shared a space "delete objects" to user "Brian" with role "" + When user "" removes the folder "folderForDeleting" from space "delete objects" + Then the HTTP status code should be "" + And for user "" the space "delete objects" contain these entries: | folderForDeleting | + And as "" folder "folderForDeleting" exist in the trashbin of the space "delete objects" + Examples: + | user | role | code | shouldOrNotBeInSpace | shouldOrNotBeInTrash | + | Alice | manager | 204 | should not | should | + | Brian | manager | 204 | should not | should | + | Brian | editor | 204 | should not | should | + | Brian | viewer | 403 | should | should not | - Scenario: An owner can delete a subfolder in a Space via the webDav API - Given user "Alice" has created a space "Owner deletes subfolder" of type "project" with quota "10" - And user "Alice" has created a subfolder "folder/subFolderForDeleting" in space "Owner deletes subfolder" - When user "Alice" removes the object "folder/subFolderForDeleting" from space "Owner deletes subfolder" - Then the HTTP status code should be "204" - And for user "Alice" the space "Owner deletes subfolder" should contain these entries: - | folder | - And for user "Alice" folder "folder/" of the space "Owner deletes subfolder" should not contain these entries: - | subFolderForDeleting | - - - Scenario: An owner can delete a file in a Space via the webDav API - Given user "Alice" has created a space "Owner deletes file" of type "project" with quota "20" - And user "Alice" has uploaded a file inside space "Owner deletes file" with content "some content" to "text.txt" - When user "Alice" removes the object "text.txt" from space "Owner deletes file" - Then the HTTP status code should be "204" - And for user "Alice" the space "Owner deletes file" should not contain these entries: - | text.txt | - When user "Alice" lists all available spaces via the GraphApi - Then the json responded should contain a space "Owner deletes file" with these key and value pairs: - | key | value | - | name | Owner deletes file | - | quota@@@used | 0 | - -# editor role - - Scenario: An editor can delete a folder with some subfolders in a Space via the webDav API - Given user "Alice" has created a space "Editor deletes folder" of type "project" with quota "10" - And user "Alice" has created a folder "folderForDeleting/sub1/sub2" in space "Editor deletes folder" - And user "Alice" has shared a space "Editor deletes folder" to user "Brian" with role "editor" - When user "Brian" removes the object "folderForDeleting" from space "Editor deletes folder" - Then the HTTP status code should be "204" - And for user "Brian" the space "Editor deletes folder" should not contain these entries: + Scenario Outline: An user deletes a subfolder in a Space via the webDav API + Given user "Alice" has shared a space "delete objects" to user "Brian" with role "" + When user "" removes the folder "folderForDeleting/sub1" from space "delete objects" + Then the HTTP status code should be "" + And for user "" the space "delete objects" should contain these entries: | folderForDeleting | + And for user "" folder "folderForDeleting/" of the space "delete objects" contain these entries: + | sub1 | + And as "" folder "sub1" exist in the trashbin of the space "delete objects" + Examples: + | user | role | code | shouldOrNotBeInSpace | shouldOrNotBeInTrash | + | Alice | manager | 204 | should not | should | + | Brian | manager | 204 | should not | should | + | Brian | editor | 204 | should not | should | + | Brian | viewer | 403 | should | should not | - Scenario: An editor can delete a subfolder in a Space via the webDav API - Given user "Alice" has created a space "Editor deletes subfolder" of type "project" with quota "10" - And user "Alice" has created a subfolder "folder/subFolderForDeleting" in space "Editor deletes subfolder" - And user "Alice" has shared a space "Editor deletes subfolder" to user "Brian" with role "editor" - When user "Brian" removes the object "folder/subFolderForDeleting" from space "Editor deletes subfolder" - Then the HTTP status code should be "204" - And for user "Brian" the space "Editor deletes subfolder" should contain these entries: - | folder | - And for user "Brian" folder "folder/" of the space "Editor deletes subfolder" should not contain these entries: - | subFolderForDeleting | - - - Scenario: An editor can delete a file in a Space via the webDav API - Given user "Alice" has created a space "Editor deletes file" of type "project" with quota "20" - And user "Alice" has uploaded a file inside space "Editor deletes file" with content "some content" to "text.txt" - And user "Alice" has shared a space "Editor deletes file" to user "Brian" with role "editor" - When user "Brian" removes the object "text.txt" from space "Editor deletes file" - Then the HTTP status code should be "204" - And for user "Brian" the space "Editor deletes file" should not contain these entries: + Scenario Outline: An user deletes a file in a Space via the webDav API + Given user "Alice" has shared a space "delete objects" to user "Brian" with role "" + When user "" removes the file "text.txt" from space "delete objects" + Then the HTTP status code should be "" + And for user "" the space "delete objects" contain these entries: | text.txt | - When user "Brian" lists all available spaces via the GraphApi - Then the json responded should contain a space "Editor deletes file" with these key and value pairs: - | key | value | - | name | Editor deletes file | - | quota@@@used | 0 | - -# viewer role - - Scenario: A viewer cannot delete a folder with some subfolders in a Space via the webDav API - Given user "Alice" has created a space "Viewer deletes folder" of type "project" with quota "10" - And user "Alice" has created a folder "folderForDeleting/sub1/sub2" in space "Viewer deletes folder" - And user "Alice" has shared a space "Viewer deletes folder" to user "Brian" with role "viewer" - When user "Brian" removes the object "folderForDeleting" from space "Viewer deletes folder" - Then the HTTP status code should be "403" - And for user "Brian" the space "Viewer deletes folder" should contain these entries: - | folderForDeleting | + And as "" file "text.txt" exist in the trashbin of the space "delete objects" + When user "" lists all available spaces via the GraphApi + Then the json responded should contain a space "delete objects" with these key and value pairs: + | key | value | + | name | delete objects | + | quota@@@used | | + Examples: + | user | role | code | shouldOrNotBeInSpace | shouldOrNotBeInTrash | quotaValue | + | Alice | manager | 204 | should not | should | 0 | + | Brian | manager | 204 | should not | should | 0 | + | Brian | editor | 204 | should not | should | 0 | + | Brian | viewer | 403 | should | should not | 12 | - Scenario: A viewer cannot delete a subfolder in a Space via the webDav API - Given user "Alice" has created a space "Viewer deletes subfolder" of type "project" with quota "10" - And user "Alice" has created a subfolder "folder/subFolderForDeleting" in space "Viewer deletes subfolder" - And user "Alice" has shared a space "Viewer deletes subfolder" to user "Brian" with role "viewer" - When user "Brian" removes the object "folder/subFolderForDeleting" from space "Viewer deletes subfolder" - Then the HTTP status code should be "403" - And for user "Brian" folder "folder/" of the space "Viewer deletes subfolder" should contain these entries: - | subFolderForDeleting | - - - Scenario: A viewer cannot delete a file in a Space via the webDav API - Given user "Alice" has created a space "Viewer deletes file" of type "project" with quota "20" - And user "Alice" has uploaded a file inside space "Viewer deletes file" with content "some content" to "text.txt" - And user "Alice" has shared a space "Viewer deletes file" to user "Brian" with role "viewer" - When user "Brian" removes the object "text.txt" from space "Viewer deletes file" - Then the HTTP status code should be "403" - And for user "Brian" the space "Viewer deletes file" should contain these entries: - | text.txt | - - Scenario: An user is unable to delete a Space via the webDav API Given user "Alice" has created a space "user deletes a space" of type "project" with quota "20" - When user "Alice" removes the object "" from space "user deletes a space" + When user "Alice" removes the folder "" from space "user deletes a space" Then the HTTP status code should be "405" When user "Alice" lists all available spaces via the GraphApi Then the json responded should contain a space "user deletes a space" with these key and value pairs: - | key | value | - | name | user deletes a space | + | key | value | + | name | user deletes a space | diff --git a/tests/acceptance/features/apiSpaces/restoreSpaceObjects.feature b/tests/acceptance/features/apiSpaces/restoreSpaceObjects.feature new file mode 100644 index 0000000000..05f71eec45 --- /dev/null +++ b/tests/acceptance/features/apiSpaces/restoreSpaceObjects.feature @@ -0,0 +1,67 @@ +@api @skipOnOcV10 +Feature: Restore files, folder + As a user with manager and editor role + I want to be able to restore files, folders + Users with the viewer role cannot restore objects + + Note - this feature is run in CI with ACCOUNTS_HASH_DIFFICULTY set to the default for production + See https://github.com/owncloud/ocis/issues/1542 and https://github.com/owncloud/ocis/pull/839 + + Background: + Given these users have been created with default attributes and without skeleton files: + | username | + | Alice | + | Brian | + | Bob | + | Carol | + And the administrator has given "Alice" the role "Admin" using the settings api + And user "Alice" creates a space "restore objects" of type "project" with the default quota using the GraphApi + And user "Alice" has created a folder "newFolder" in space "restore objects" + And user "Alice" has uploaded a file inside space "restore objects" with content "test" to "newFolder/file.txt" + + + Scenario Outline: An user with different role can see deleted objects in trash bin of the space via the webDav API + Given user "Alice" has shared a space "restore objects" to user "Brian" with role "" + And user "Alice" has removed the file "newFolder/file.txt" from space "restore objects" + And user "Alice" has removed the folder "newFolder" from space "restore objects" + When user "" lists all deleted files in the trash bin of the space "restore objects" + Then the HTTP status code should be "207" + And as "" folder "newFolder" should exist in the trashbin of the space "restore objects" + And as "" file "file.txt" should exist in the trashbin of the space "restore objects" + Examples: + | user | role | + | Brian | manager | + | Brian | editor | + | Brian | viewer | + + + Scenario Outline: An user restores a folder with some objects from the trash via the webDav API + Given user "Alice" has shared a space "restore objects" to user "Brian" with role "" + And user "Alice" has removed the folder "newFolder" from space "restore objects" + When user "" restores the folder "newFolder" from the trash of the space "restore objects" to "/newFolder" + Then the HTTP status code should be "" + And for user "" the space "restore objects" contain these entries: + | newFolder | + And as "" folder "newFolder" exist in the trashbin of the space "restore objects" + Examples: + | user | role | code | shouldOrNotBeInSpace | shouldOrNotBeInTrash | + | Alice | manager | 201 | should | should not | + | Brian | manager | 201 | should | should not | + | Brian | editor | 201 | should | should not | + | Brian | viewer | 403 | should not | should | + + + Scenario Outline: An user restores a file from the trash via the webDav API + Given user "Alice" has shared a space "restore objects" to user "Brian" with role "" + And user "Alice" has removed the file "newFolder/file.txt" from space "restore objects" + When user "" restores the file "file.txt" from the trash of the space "restore objects" to "newFolder/file.txt" + Then the HTTP status code should be "" + And for user "" folder "newFolder" of the space "restore objects" contain these files: + | file.txt | + And as "" file "file.txt" exist in the trashbin of the space "restore objects" + Examples: + | user | role | code | shouldOrNotBeInSpace | shouldOrNotBeInTrash | + | Alice | manager | 201 | should | should not | + | Brian | manager | 201 | should | should not | + | Brian | editor | 201 | should | should not | + | Brian | viewer | 403 | should not | should | diff --git a/tests/acceptance/features/bootstrap/SpacesContext.php b/tests/acceptance/features/bootstrap/SpacesContext.php index dd6d6e8311..6556134a74 100644 --- a/tests/acceptance/features/bootstrap/SpacesContext.php +++ b/tests/acceptance/features/bootstrap/SpacesContext.php @@ -49,6 +49,11 @@ class SpacesContext implements Context { */ private OCSContext $ocsContext; + /** + * @var TrashbinContext + */ + private TrashbinContext $trashbinContext; + /** * @var string */ @@ -323,6 +328,7 @@ class SpacesContext implements Context { // Get all the contexts you need in this context $this->featureContext = $environment->getContext('FeatureContext'); $this->ocsContext = $environment->getContext('OCSContext'); + $this->trashbinContext = $environment->getContext('TrashbinContext'); // Run the BeforeScenario function in OCSContext to set it up correctly $this->ocsContext->before($scope); $this->baseUrl = \trim($this->featureContext->getBaseUrl(), "/"); @@ -1432,7 +1438,7 @@ class SpacesContext implements Context { } /** - * @When /^user "([^"]*)" has set the file "([^"]*)" as a (description|space image)\s? in a special section of the "([^"]*)" space$/ + * @Given /^user "([^"]*)" has set the file "([^"]*)" as a (description|space image)\s? in a special section of the "([^"]*)" space$/ * * @param string $user * @param string $file @@ -1516,7 +1522,7 @@ class SpacesContext implements Context { } /** - * @When /^user "([^"]*)" has created a space "([^"]*)" with the default quota using the GraphApi$/ + * @Given /^user "([^"]*)" has created a space "([^"]*)" with the default quota using the GraphApi$/ * * @param string $user * @param string $spaceName @@ -1766,7 +1772,7 @@ class SpacesContext implements Context { } /** - * @When /^user "([^"]*)" removes the object "([^"]*)" from space "([^"]*)"$/ + * @When /^user "([^"]*)" removes the (?:file|folder) "([^"]*)" from space "([^"]*)"$/ * * @param string $user * @param string $object @@ -1818,7 +1824,30 @@ class SpacesContext implements Context { } /** - * @When /^user "([^"]*)" has disabled a space "([^"]*)"$/ + * @Given /^user "([^"]*)" has removed the (?:file|folder) "([^"]*)" from space "([^"]*)"$/ + * + * @param string $user + * @param string $object + * @param string $spaceName + * + * @return void + * @throws GuzzleException + */ + public function sendUserHasRemovedObjectFromSpaceRequest( + string $user, + string $object, + string $spaceName + ): void { + $this->sendRemoveObjectFromSpaceRequest($user, $object, $spaceName); + $expectedHTTPStatus = "204"; + $this->featureContext->theHTTPStatusCodeShouldBe( + $expectedHTTPStatus, + "Expected response status code should be $expectedHTTPStatus" + ); + } + + /** + * @Given /^user "([^"]*)" has disabled a space "([^"]*)"$/ * * @param string $user * @param string $spaceName @@ -1919,7 +1948,7 @@ class SpacesContext implements Context { } /** - * @When /^user "([^"]*)" has restored a disabled space "([^"]*)"$/ + * @Given /^user "([^"]*)" has restored a disabled space "([^"]*)"$/ * * @param string $user * @param string $spaceName @@ -1938,4 +1967,139 @@ class SpacesContext implements Context { "Expected response status code should be $expectedHTTPStatus" ); } + + /** + * @When /^user "([^"]*)" lists all deleted files in the trash bin of the space "([^"]*)"$/ + * + * @param string $user + * @param string $spaceName + * + * @return void + * @throws GuzzleException + */ + public function userListAllDeletedFilesinTrash( + string $user, + string $spaceName + ): void { + $space = $this->getSpaceByName($user, $spaceName); + $fullUrl = $this->baseUrl . "/remote.php/dav/spaces/trash-bin/" . $space["id"]; + $this->featureContext->setResponse( + HttpRequestHelper::sendRequest( + $fullUrl, + "", + 'PROPFIND', + $user, + $this->featureContext->getPasswordForUser($user), + [], + "" + ) + ); + } + + /** + * User get all objects in the trash of project space + * + * method "getTrashbinContentFromResponseXml" borrowed from core repository + * and return array like: + * [1] => Array + * ( + * [href] => /remote.php/dav/spaces/trash-bin/spaceId/objectId/ + * [name] => deleted folder + * [mtime] => 1649147272 + * [original-location] => deleted folder + * ) + * + * @param string $user + * @param string $spaceName + * + * @return array + * @throws GuzzleException + */ + public function getObjectsInTrashbin( + string $user, + string $spaceName + ): array { + $this->userListAllDeletedFilesinTrash($user, $spaceName); + $this->featureContext->theHTTPStatusCodeShouldBe( + 207, + "Expected response status code should be 207" + ); + return $this->trashbinContext->getTrashbinContentFromResponseXml( + $this->featureContext->getResponseXml($this->featureContext->getResponse()) + ); + } + + /** + * @Then /^as "([^"]*)" (?:file|folder|entry) "([^"]*)" should (not|)\s?exist in the trashbin of the space "([^"]*)"$/ + * + * @param string $user + * @param string $object + * @param string $shouldOrNot (not|) + * @param string $spaceName + * + * @return void + * @throws GuzzleException + */ + public function checkExistanceOfObjectsInTrashbin( + string $user, + string $object, + string $shouldOrNot, + string $spaceName + ): void { + $objectsInTrash = $this->getObjectsInTrashbin($user, $spaceName); + + $expectedObject = ""; + foreach ($objectsInTrash as $objectInTrash) { + if ($objectInTrash["name"] === $object) { + $expectedObject = $objectInTrash["name"]; + } + }; + if ($shouldOrNot === "not") { + Assert::assertEmpty($expectedObject, "$object is found in the trash, but should not be there"); + } else Assert::assertNotEmpty($expectedObject, "$object is not found in the trash"); + } + + /** + * @When /^user "([^"]*)" restores the (?:file|folder) "([^"]*)" from the trash of the space "([^"]*)" to "([^"]*)"$/ + * + * @param string $user + * @param string $object + * @param string $spaceName + * @param string $destination + * + * @return void + * @throws GuzzleException + */ + public function userRestoresSpaceObjectsFromTrashRequest( + string $user, + string $object, + string $spaceName, + string $destination + ): void { + $space = $this->getSpaceByName($user, $spaceName); + + // find object in trash + $objectsInTrash = $this->getObjectsInTrashbin($user, $spaceName); + foreach ($objectsInTrash as $objectInTrash) { + if ($objectInTrash["name"] === $object) { + $pathToDeletedObject = $objectInTrash["href"]; + } + }; + + $destination = $this->baseUrl . "/remote.php/dav/spaces/" . $space["id"] . $destination; + $header = ["Destination" => $destination, "Overwrite" => "F"]; + + $fullUrl = $this->baseUrl . $pathToDeletedObject; + $this->featureContext->setResponse( + HttpRequestHelper::sendRequest( + $fullUrl, + "", + 'MOVE', + $user, + $this->featureContext->getPasswordForUser($user), + $header, + "" + ) + ); + } } From 32d739a6cfa3359fb54c6de5c0a7a900da1cdd21 Mon Sep 17 00:00:00 2001 From: Swikriti Tripathi Date: Tue, 12 Apr 2022 12:48:10 +0545 Subject: [PATCH 05/20] Bump Commit id and update expected to fail file --- .drone.env | 2 +- tests/acceptance/expected-failures-API-on-OCIS-storage.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.drone.env b/.drone.env index 99d6d95153..1abaeb34b9 100644 --- a/.drone.env +++ b/.drone.env @@ -1,5 +1,5 @@ # The test runner source for API tests -CORE_COMMITID=d21d38aac929918c48d4ee5c44741dc305e875bb +CORE_COMMITID=8d336099571f9218e09ba5ddf0eb30b19cab7d4f CORE_BRANCH=master # The test runner source for UI tests diff --git a/tests/acceptance/expected-failures-API-on-OCIS-storage.md b/tests/acceptance/expected-failures-API-on-OCIS-storage.md index 57f4063c40..6cac0cf5ec 100644 --- a/tests/acceptance/expected-failures-API-on-OCIS-storage.md +++ b/tests/acceptance/expected-failures-API-on-OCIS-storage.md @@ -1597,8 +1597,8 @@ And other missing implementation of favorites #### [Share inaccessible if folder with same name was deleted and recreated](https://github.com/owncloud/ocis/issues/1787) -- [apiShareReshareToShares1/reShare.feature:269](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares1/reShare.feature#L269) -- [apiShareReshareToShares1/reShare.feature:270](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares1/reShare.feature#L270) +- [apiShareReshareToShares1/reShare.feature:259](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares1/reShare.feature#L259) +- [apiShareReshareToShares1/reShare.feature:260](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiShareReshareToShares1/reShare.feature#L260) #### [Trying to accept a share with invalid ID gives incorrect OCS and HTTP status](https://github.com/owncloud/ocis/issues/2111) From 0e88cb1becc96c65daf2cc091b52e03dbc00e0c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Tue, 12 Apr 2022 11:27:24 +0200 Subject: [PATCH 06/20] Use embeddable ocdav go micro service (#3397) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * allow proxy to route to micro service Signed-off-by: Jörn Friedrich Dreyer * use go micre ocdav service instead of reva frontend Signed-off-by: Jörn Friedrich Dreyer * add missing gateway default config Signed-off-by: Jörn Friedrich Dreyer * update reva branch for testing Signed-off-by: Jörn Friedrich Dreyer * add changelog Signed-off-by: Jörn Friedrich Dreyer * add missing comands Signed-off-by: Jörn Friedrich Dreyer * bump reva version Signed-off-by: jkoberg * tidy Signed-off-by: jkoberg * bump reva again Signed-off-by: jkoberg * a blind mans config change Signed-off-by: jkoberg * add ocdav to must start extensions Signed-off-by: jkoberg * fail when neither backend nor service is set Signed-off-by: jkoberg Co-authored-by: jkoberg --- .drone.star | 2 +- .vscode/launch.json | 2 +- changelog/unreleased/micro-ocdav.md | 5 + changelog/unreleased/update-reva.md | 1 + docs/extensions/idm/setup.md | 2 +- go.mod | 4 +- go.sum | 8 +- ocis/pkg/command/ocdav.go | 28 ++++ ocis/pkg/runtime/service/service.go | 1 + proxy/pkg/config/config.go | 13 +- proxy/pkg/config/defaults/defaultconfig.go | 16 ++- proxy/pkg/proxy/proxy.go | 33 ++++- storage/pkg/command/frontend.go | 8 -- storage/pkg/command/ocdav.go | 134 ++++++++++++++++++ storage/pkg/command/users.go | 2 +- storage/pkg/config/config.go | 86 +++++++---- storage/pkg/config/defaults/defaultconfig.go | 22 ++- .../drone/ocis/proxy.yaml | 12 +- 18 files changed, 310 insertions(+), 69 deletions(-) create mode 100644 changelog/unreleased/micro-ocdav.md create mode 100644 ocis/pkg/command/ocdav.go create mode 100644 storage/pkg/command/ocdav.go diff --git a/.drone.star b/.drone.star index a2e0bc46e8..26c0826d18 100644 --- a/.drone.star +++ b/.drone.star @@ -1696,7 +1696,7 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = "OCIS_STORAGE_READ_ONLY": "false", # General oCIS config # OCIS_RUN_EXTENSIONS specifies to start all extensions except glauth, idp and accounts. These are replaced by external services - "OCIS_RUN_EXTENSIONS": "settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,nats", + "OCIS_RUN_EXTENSIONS": "settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,nats,ocdav", "OCIS_LOG_LEVEL": "error", "OCIS_URL": OCIS_URL, "PROXY_TLS": "true", diff --git a/.vscode/launch.json b/.vscode/launch.json index 405778b585..4332cf2e1a 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -23,7 +23,7 @@ "ACCOUNTS_DEMO_USERS_AND_GROUPS": "true", "IDM_CREATE_DEMO_USERS": "true" // OCIS_RUN_EXTENSIONS allows to start a subset of extensions even in the supervised mode - //"OCIS_RUN_EXTENSIONS": "settings,storage-metadata,glauth,graph,graph-explorer,idp,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,accounts,proxy", + //"OCIS_RUN_EXTENSIONS": "settings,storage-metadata,glauth,graph,graph-explorer,idp,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,accounts,proxy,ocdav", } } ] diff --git a/changelog/unreleased/micro-ocdav.md b/changelog/unreleased/micro-ocdav.md new file mode 100644 index 0000000000..35fa180da8 --- /dev/null +++ b/changelog/unreleased/micro-ocdav.md @@ -0,0 +1,5 @@ +Enhancement: Use embeddable ocdav go micro service + +We now use the reva `pgk/micro/ocdav` package that implements a go micro compatible version of the ocdav service. + +https://github.com/owncloud/ocis/pull/3397 diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index 08809c020a..75da9e4f39 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -4,6 +4,7 @@ Updated reva to version 2.x.x. This update includes: * TODO +https://github.com/owncloud/ocis/pull/3397 https://github.com/owncloud/ocis/pull/3430 https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 diff --git a/docs/extensions/idm/setup.md b/docs/extensions/idm/setup.md index 085df69531..13c13f8b10 100644 --- a/docs/extensions/idm/setup.md +++ b/docs/extensions/idm/setup.md @@ -50,7 +50,7 @@ export STORAGE_LDAP_USERATTRIBUTEFILTER='(&(objectclass=owncloud)({{attr}}={{val export STORAGE_LDAP_USERFINDFILTER='(&(objectclass=owncloud)(|(uid={{query}}*)(cn={{query}}*)(displayname={{query}}*)(mail={{query}}*)(description={{query}}*)))' export STORAGE_LDAP_USERGROUPFILER='(&(objectclass=groupOfNames)(member={{query}}*))' export STORAGE_LDAP_GROUPFILTER='(&(objectclass=groupOfNames)(objectclass=owncloud)(ownclouduuid={{.OpaqueId}}*))' -export OCIS_RUN_EXTENSIONS=settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,idp,nats,idm +export OCIS_RUN_EXTENSIONS=settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,idp,nats,idm,ocdav export OCIS_INSECURE=true bin/ocis server ``` diff --git a/go.mod b/go.mod index 14c0c3a790..a67c814f76 100644 --- a/go.mod +++ b/go.mod @@ -15,14 +15,14 @@ require ( github.com/asim/go-micro/plugins/registry/mdns/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75 - github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75 + github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291 github.com/asim/go-micro/plugins/wrapper/breaker/gobreaker/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/wrapper/monitoring/prometheus/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/wrapper/trace/opencensus/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/blevesearch/bleve/v2 v2.3.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20220328105952-297bef33e13f - github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f + github.com/cs3org/reva/v2 v2.0.0-20220411200520-865ba723f25a github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 diff --git a/go.sum b/go.sum index 857c212273..71a9224729 100644 --- a/go.sum +++ b/go.sum @@ -181,8 +181,8 @@ github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c8 github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:n4K0361IySJ/hmIfaIg8q2NDNAVDL9DUC6Bh7gS4ihg= github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:miPpSGbTA6L2hsHccRNxNSXpIYo6YtlTkoi+fvEHJbw= github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:uQACKU+/RcWKEMdX6/Ewd+TxCPLIsLPt9ym6++B1X68= -github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:ULuhDMrHjkEkj0dxWS+ZM9MD/9ywFPM7kaoEs4OjhSc= -github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:LEsuhy4ZF3UGT9VSOoj/ZP0MZt4brlgwJVG7/LAac+I= +github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291 h1:l1l3pUleHu66DNRmaPwfgyXfVXT7UmyzNXD0uQp9SQE= +github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291/go.mod h1:LEsuhy4ZF3UGT9VSOoj/ZP0MZt4brlgwJVG7/LAac+I= github.com/asim/go-micro/plugins/transport/grpc/v4 v4.0.0-20211019191242-9edc569e68bb h1:rCIGzR4YRddwaWM4pZkbxHIkhFbbuGZOqivnxyKNQz8= github.com/asim/go-micro/plugins/transport/grpc/v4 v4.0.0-20211019191242-9edc569e68bb/go.mod h1:dzSEAsAqQ4mRaYPJ07KcAt6rIjYUJ9w55UXvnyGx4VE= github.com/asim/go-micro/plugins/wrapper/breaker/gobreaker/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:8wLNrBKfdRl6uryUFW6qlBZWNeIbPUYKYkwLG9bI58k= @@ -335,8 +335,8 @@ github.com/crewjam/httperr v0.2.0/go.mod h1:Jlz+Sg/XqBQhyMjdDiC+GNNRzZTD7x39Gu3p github.com/crewjam/saml v0.4.6 h1:XCUFPkQSJLvzyl4cW9OvpWUbRf0gE7VUpU8ZnilbeM4= github.com/crewjam/saml v0.4.6/go.mod h1:ZBOXnNPFzB3CgOkRm7Nd6IVdkG+l/wF+0ZXLqD96t1A= github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= -github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f h1:SNQSbtvhsQ6KvdDJlW536pUMq8RzghQMDCkYYKEFn8c= -github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f/go.mod h1:rE2wCGoiGiUuQ3zj6aD7sZ+BhboLyNsNQ3z3auunWm4= +github.com/cs3org/reva/v2 v2.0.0-20220411200520-865ba723f25a h1:/3filuu6As9ZMznQ3ddShjveqNr0NrjfZynQT3LyTmQ= +github.com/cs3org/reva/v2 v2.0.0-20220411200520-865ba723f25a/go.mod h1:l1dhodFXCp88/Lc0VhzSeaLSQUTn2AdwwkTFtaLCJhk= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= diff --git a/ocis/pkg/command/ocdav.go b/ocis/pkg/command/ocdav.go new file mode 100644 index 0000000000..b04c673ea2 --- /dev/null +++ b/ocis/pkg/command/ocdav.go @@ -0,0 +1,28 @@ +package command + +import ( + "github.com/owncloud/ocis/ocis-pkg/config" + "github.com/owncloud/ocis/ocis/pkg/register" + "github.com/owncloud/ocis/storage/pkg/command" + "github.com/urfave/cli/v2" +) + +// OCDavCommand is the entrypoint for the ocdav command. +func OCDavCommand(cfg *config.Config) *cli.Command { + return &cli.Command{ + Name: "ocdav", + Usage: "start ocdav", + Category: "extensions", + Before: func(ctx *cli.Context) error { + return ParseStorageCommon(ctx, cfg) + }, + Action: func(c *cli.Context) error { + origCmd := command.OCDav(cfg.Storage) + return handleOriginalAction(c, origCmd) + }, + } +} + +func init() { + register.AddCommand(OCDavCommand) +} diff --git a/ocis/pkg/runtime/service/service.go b/ocis/pkg/runtime/service/service.go index 55acba22b9..20fcf21f52 100644 --- a/ocis/pkg/runtime/service/service.go +++ b/ocis/pkg/runtime/service/service.go @@ -108,6 +108,7 @@ func NewService(options ...Option) (*Service, error) { s.ServicesRegistry["web"] = web.NewSutureService s.ServicesRegistry["webdav"] = webdav.NewSutureService s.ServicesRegistry["storage-frontend"] = storage.NewFrontend + s.ServicesRegistry["ocdav"] = storage.NewOCDav s.ServicesRegistry["storage-gateway"] = storage.NewGateway s.ServicesRegistry["storage-userprovider"] = storage.NewUserProvider s.ServicesRegistry["storage-groupprovider"] = storage.NewGroupProvider diff --git a/proxy/pkg/config/config.go b/proxy/pkg/config/config.go index d4934e5581..7beb4d9c4c 100644 --- a/proxy/pkg/config/config.go +++ b/proxy/pkg/config/config.go @@ -43,12 +43,15 @@ type Policy struct { Routes []Route `yaml:"routes"` } -// Route define forwarding routes +// Route defines forwarding routes type Route struct { - Type RouteType `yaml:"type"` - Endpoint string `yaml:"endpoint"` - Backend string `yaml:"backend"` - ApacheVHost bool `yaml:"apache-vhost"` + Type RouteType `yaml:"type"` + Endpoint string `yaml:"endpoint"` + // Backend is a static URL to forward the request to + Backend string `yaml:"backend"` + // Service name to look up in the registry + Service string `yaml:"service"` + ApacheVHost bool `yaml:"apache-vhost"` } // RouteType defines the type of a route diff --git a/proxy/pkg/config/defaults/defaultconfig.go b/proxy/pkg/config/defaults/defaultconfig.go index e2689a9f5b..78e28363aa 100644 --- a/proxy/pkg/config/defaults/defaultconfig.go +++ b/proxy/pkg/config/defaults/defaultconfig.go @@ -96,30 +96,34 @@ func DefaultPolicies() []config.Policy { }, { Endpoint: "/remote.php/", - Backend: "http://localhost:9140", + Service: "ocdav", }, { Endpoint: "/dav/", - Backend: "http://localhost:9140", + Service: "ocdav", }, { Endpoint: "/webdav/", - Backend: "http://localhost:9140", + Service: "ocdav", }, { Endpoint: "/status.php", - Backend: "http://localhost:9140", + Service: "ocdav", }, { Endpoint: "/index.php/", - Backend: "http://localhost:9140", + Service: "ocdav", + }, + { + Endpoint: "/apps/", + Service: "ocdav", }, { Endpoint: "/data", Backend: "http://localhost:9140", }, { - Endpoint: "/app/", + Endpoint: "/app/", // /app or /apps? ocdav only handles /apps Backend: "http://localhost:9140", }, { diff --git a/proxy/pkg/proxy/proxy.go b/proxy/pkg/proxy/proxy.go index 7681defb80..53bb72566b 100644 --- a/proxy/pkg/proxy/proxy.go +++ b/proxy/pkg/proxy/proxy.go @@ -12,10 +12,12 @@ import ( "time" chimiddleware "github.com/go-chi/chi/v5/middleware" + "go-micro.dev/v4/selector" "go.opentelemetry.io/otel/attribute" "github.com/owncloud/ocis/ocis-pkg/log" + "github.com/owncloud/ocis/ocis-pkg/registry" pkgtrace "github.com/owncloud/ocis/ocis-pkg/tracing" "github.com/owncloud/ocis/proxy/pkg/config" "github.com/owncloud/ocis/proxy/pkg/proxy/policy" @@ -86,6 +88,10 @@ func NewMultiHostReverseProxy(opts ...Option) *MultiHostReverseProxy { for _, pol := range options.Config.Policies { for _, route := range pol.Routes { rp.logger.Debug().Str("fwd: ", route.Endpoint) + + if route.Backend == "" && route.Service == "" { + rp.logger.Fatal().Interface("route", route).Msg("neither Backend nor Service is set") + } uri, err2 := url.Parse(route.Backend) if err2 != nil { rp.logger. @@ -95,6 +101,7 @@ func NewMultiHostReverseProxy(opts ...Option) *MultiHostReverseProxy { Msg("malformed url") } + // here the backend is used as a uri rp.AddHost(pol.Name, uri, route) } } @@ -184,9 +191,31 @@ func (p *MultiHostReverseProxy) AddHost(policy string, target *url.URL, rt confi if p.Directors[policy][routeType] == nil { p.Directors[policy][routeType] = make(map[string]func(req *http.Request)) } + + reg := registry.GetRegistry() + sel := selector.NewSelector(selector.Registry(reg)) + p.Directors[policy][routeType][rt.Endpoint] = func(req *http.Request) { - req.URL.Scheme = target.Scheme - req.URL.Host = target.Host + if rt.Service != "" { + // select next node + next, err := sel.Select(rt.Service) + if err != nil { + fmt.Println(fmt.Errorf("could not select %s service from the registry: %v", rt.Service, err)) + return // TODO error? fallback to target.Host & Scheme? + } + node, err := next() + if err != nil { + fmt.Println(fmt.Errorf("could not select next node for service %s: %v", rt.Service, err)) + return // TODO error? fallback to target.Host & Scheme? + } + req.URL.Host = node.Address + req.URL.Scheme = node.Metadata["protocol"] // TODO check property exists? + + } else { + req.URL.Host = target.Host + req.URL.Scheme = target.Scheme + } + // Apache deployments host addresses need to match on req.Host and req.URL.Host // see https://stackoverflow.com/questions/34745654/golang-reverseproxy-with-apache2-sni-hostname-error if rt.ApacheVHost { diff --git a/storage/pkg/command/frontend.go b/storage/pkg/command/frontend.go index 06be385308..e5f4d233b7 100644 --- a/storage/pkg/command/frontend.go +++ b/storage/pkg/command/frontend.go @@ -186,14 +186,6 @@ func frontendConfigFromStruct(c *cli.Context, cfg *config.Config, filesCfg map[s "timeout": 86400, "insecure": true, }, - "ocdav": map[string]interface{}{ - "prefix": cfg.Reva.Frontend.OCDavPrefix, - "files_namespace": cfg.Reva.OCDav.DavFilesNamespace, - "webdav_namespace": cfg.Reva.OCDav.WebdavNamespace, - "timeout": 86400, - "insecure": cfg.Reva.Frontend.OCDavInsecure, - "public_url": cfg.Reva.Frontend.PublicURL, - }, "ocs": map[string]interface{}{ "storage_registry_svc": cfg.Reva.Gateway.Endpoint, "share_prefix": cfg.Reva.Frontend.OCSSharePrefix, diff --git a/storage/pkg/command/ocdav.go b/storage/pkg/command/ocdav.go new file mode 100644 index 0000000000..f3f64bf768 --- /dev/null +++ b/storage/pkg/command/ocdav.go @@ -0,0 +1,134 @@ +package command + +import ( + "context" + "flag" + + "github.com/cs3org/reva/v2/pkg/micro/ocdav" + "github.com/oklog/run" + ociscfg "github.com/owncloud/ocis/ocis-pkg/config" + "github.com/owncloud/ocis/ocis-pkg/sync" + "github.com/owncloud/ocis/storage/pkg/config" + "github.com/owncloud/ocis/storage/pkg/server/debug" + "github.com/owncloud/ocis/storage/pkg/tracing" + "github.com/thejerf/suture/v4" + "github.com/urfave/cli/v2" +) + +// OCDav is the entrypoint for the ocdav command. +// TODO move ocdav cmd to a separate service +func OCDav(cfg *config.Config) *cli.Command { + return &cli.Command{ + Name: "ocdav", + Usage: "start ocdav service", + Before: func(c *cli.Context) error { + if err := loadUserAgent(c, cfg); err != nil { + return err + } + return ParseConfig(c, cfg, "ocdav") + }, + Action: func(c *cli.Context) error { + logger := NewLogger(cfg) + + tracing.Configure(cfg, logger) + + gr := run.Group{} + ctx, cancel := context.WithCancel(context.Background()) + //metrics = metrics.New() + + defer cancel() + + gr.Add(func() error { + s, err := ocdav.Service( + ocdav.Context(ctx), + ocdav.Logger(logger.Logger), + ocdav.Address(cfg.OCDav.Addr), + ocdav.FilesNamespace(cfg.OCDav.FilesNamespace), + ocdav.WebdavNamespace(cfg.OCDav.WebdavNamespace), + ocdav.SharesNamespace(cfg.OCDav.SharesNamespace), + ocdav.Timeout(cfg.OCDav.Timeout), + ocdav.Insecure(cfg.OCDav.Insecure), + ocdav.PublicURL(cfg.OCDav.PublicURL), + ocdav.Prefix(cfg.OCDav.Prefix), + ocdav.GatewaySvc(cfg.OCDav.GatewaySVC), + ocdav.JWTSecret(cfg.OCDav.JWTSecret), + // ocdav.FavoriteManager() // FIXME needs a proper persistence implementation + // ocdav.LockSystem(), // will default to the CS3 lock system + // ocdav.TLSConfig() // tls config for the http server + ) + if err != nil { + return err + } + + return s.Run() + }, func(err error) { + logger.Info().Err(err).Str("server", c.Command.Name).Msg("Shutting down server") + cancel() + }) + + { + server, err := debug.Server( + debug.Name(c.Command.Name+"-debug"), + debug.Addr(cfg.OCDav.DebugAddr), + debug.Logger(logger), + debug.Context(ctx), + debug.Config(cfg), + ) + + if err != nil { + logger.Info(). + Err(err). + Str("server", "debug"). + Msg("Failed to initialize server") + + return err + } + + gr.Add(server.ListenAndServe, func(_ error) { + cancel() + }) + } + + if !cfg.Reva.Frontend.Supervised { + sync.Trap(&gr, cancel) + } + + return gr.Run() + }, + } +} + +// OCDavSutureService allows for the ocdav command to be embedded and supervised by a suture supervisor tree. +type OCDavSutureService struct { + cfg *config.Config +} + +// NewOCDav creates a new ocdav.OCDavSutureService +func NewOCDav(cfg *ociscfg.Config) suture.Service { + cfg.Storage.Commons = cfg.Commons + return OCDavSutureService{ + cfg: cfg.Storage, + } +} + +func (s OCDavSutureService) Serve(ctx context.Context) error { + s.cfg.Reva.Frontend.Context = ctx + f := &flag.FlagSet{} + cmdFlags := OCDav(s.cfg).Flags + for k := range cmdFlags { + if err := cmdFlags[k].Apply(f); err != nil { + return err + } + } + cliCtx := cli.NewContext(nil, f, nil) + if OCDav(s.cfg).Before != nil { + if err := OCDav(s.cfg).Before(cliCtx); err != nil { + return err + } + } + if err := OCDav(s.cfg).Action(cliCtx); err != nil { + return err + } + + return nil +} diff --git a/storage/pkg/command/users.go b/storage/pkg/command/users.go index 2a8a191b04..8e0bd0fbf6 100644 --- a/storage/pkg/command/users.go +++ b/storage/pkg/command/users.go @@ -19,7 +19,7 @@ import ( "github.com/urfave/cli/v2" ) -// Users is the entrypoint for the sharing command. +// Users is the entrypoint for the users command. func Users(cfg *config.Config) *cli.Command { return &cli.Command{ Name: "users", diff --git a/storage/pkg/config/config.go b/storage/pkg/config/config.go index 599d955cca..cfd35175e7 100644 --- a/storage/pkg/config/config.go +++ b/storage/pkg/config/config.go @@ -165,8 +165,6 @@ type FrontendPort struct { DatagatewayPrefix string `yaml:"data_gateway_prefix"` Favorites bool `yaml:"favorites"` ProjectSpaces bool `yaml:"project_spaces"` - OCDavInsecure bool `yaml:"ocdav_insecure"` - OCDavPrefix string `yaml:"ocdav_prefix"` OCSPrefix string `yaml:"ocs_prefix"` OCSSharePrefix string `yaml:"ocs_share_prefix"` OCSHomeNamespace string `yaml:"ocs_home_namespace"` @@ -433,8 +431,26 @@ type LDAPGroupSchema struct { // OCDav defines the available ocdav configuration. type OCDav struct { - WebdavNamespace string `yaml:"webdav_namespace"` - DavFilesNamespace string `yaml:"dav_files_namespace"` + // Addr to listen to with the http server for the ocdav service + Addr string `yaml:"addr"` + Prefix string `yaml:"prefix"` + WebdavNamespace string `yaml:"webdav_namespace"` + FilesNamespace string `yaml:"files_namespace"` + SharesNamespace string `yaml:"shares_namespace"` + // PublicURL used to redirect /s/{token} URLs to + PublicURL string `yaml:"public_url"` + + // Addr to listen to with the debug http server + DebugAddr string `yaml:"debug_addr"` + + // GatewaySVC to forward CS3 requests to TODO use registry + GatewaySVC string `yaml:"gateway_svc"` + // JWTSecret used to verify reva access token + JWTSecret string `yaml:"jwt_secret"` + // Insecure certificates allowed when making requests to the gateway + Insecure bool `yaml:"insecure"` + // Timeout in seconds when making requests to the gateway + Timeout int64 `yaml:"timeout"` } // Archiver defines the available archiver configuration. @@ -455,7 +471,6 @@ type Reva struct { LDAP LDAP `yaml:"ldap"` UserGroupRest UserGroupRest `yaml:"user_group_rest"` UserOwnCloudSQL UserOwnCloudSQL `yaml:"user_owncloud_sql"` - OCDav OCDav `yaml:"ocdav"` Archiver Archiver `yaml:"archiver"` UserStorage StorageConfig `yaml:"user_storage"` MetadataStorage StorageConfig `yaml:"metadata_storage"` @@ -483,7 +498,7 @@ type Reva struct { // Services and Ports will be ignored if this is used Configs map[string]interface{} `yaml:"configs"` // chunking and resumable upload config (TUS) - UploadMaxChunkSize int `yaml:"uppload_max_chunk_size"` + UploadMaxChunkSize int `yaml:"upload_max_chunk_size"` UploadHTTPMethodOverride string `yaml:"upload_http_method_override"` // checksumming capabilities ChecksumSupportedTypes []string `yaml:"checksum_supported_types"` @@ -512,6 +527,7 @@ type Config struct { File string `yaml:"file"` Log *shared.Log `yaml:"log"` Debug Debug `yaml:"debug"` + OCDav OCDav `yaml:"ocdav"` Reva Reva `yaml:"reva"` Tracing Tracing `yaml:"tracing"` Asset Asset `yaml:"asset"` @@ -567,10 +583,6 @@ func structMappings(cfg *Config) []shared.EnvBinding { EnvVars: []string{"OCIS_INSECURE", "STORAGE_FRONTEND_ARCHIVER_INSECURE"}, Destination: &cfg.Reva.Frontend.ArchiverInsecure, }, - { - EnvVars: []string{"OCIS_INSECURE", "STORAGE_FRONTEND_OCDAV_INSECURE"}, - Destination: &cfg.Reva.Frontend.OCDavInsecure, - }, { EnvVars: []string{"OCIS_INSECURE", "STORAGE_OIDC_INSECURE"}, Destination: &cfg.Reva.OIDC.Insecure, @@ -773,18 +785,6 @@ func structMappings(cfg *Config) []shared.EnvBinding { EnvVars: []string{"STORAGE_TRANSFER_SECRET"}, Destination: &cfg.Reva.TransferSecret, }, - { - EnvVars: []string{"STORAGE_CHUNK_FOLDER"}, - Destination: &cfg.Reva.OCDav.WebdavNamespace, - }, - { - EnvVars: []string{"STORAGE_WEBDAV_NAMESPACE"}, - Destination: &cfg.Reva.OCDav.WebdavNamespace, - }, - { - EnvVars: []string{"STORAGE_DAV_FILES_NAMESPACE"}, - Destination: &cfg.Reva.OCDav.DavFilesNamespace, - }, { EnvVars: []string{"STORAGE_ARCHIVER_MAX_NUM_FILES"}, Destination: &cfg.Reva.Archiver.MaxNumFiles, @@ -821,10 +821,6 @@ func structMappings(cfg *Config) []shared.EnvBinding { EnvVars: []string{"STORAGE_FRONTEND_PROJECT_SPACES"}, Destination: &cfg.Reva.Frontend.ProjectSpaces, }, - { - EnvVars: []string{"STORAGE_FRONTEND_OCDAV_PREFIX"}, - Destination: &cfg.Reva.Frontend.OCDavPrefix, - }, { EnvVars: []string{"STORAGE_FRONTEND_OCS_PREFIX"}, Destination: &cfg.Reva.Frontend.OCSPrefix, @@ -1810,5 +1806,43 @@ func structMappings(cfg *Config) []shared.EnvBinding { EnvVars: []string{"STORAGE_PERMISSIONS_ENDPOINT"}, Destination: &cfg.Reva.Permissions.Endpoint, }, + + // ocdav + { + EnvVars: []string{"OCDAV_ADDR"}, + Destination: &cfg.OCDav.Addr, + }, + { + EnvVars: []string{"OCDAV_DEBUG_ADDR"}, + Destination: &cfg.OCDav.DebugAddr, + }, + { + EnvVars: []string{"OCDAV_PREFIX"}, + Destination: &cfg.OCDav.Prefix, + }, + { + EnvVars: []string{"OCDAV_WEBDAV_NAMESPACE"}, + Destination: &cfg.OCDav.WebdavNamespace, + }, + { + EnvVars: []string{"OCDAV_FILES_NAMESPACE"}, + Destination: &cfg.OCDav.FilesNamespace, + }, + { + EnvVars: []string{"OCDAV_SHARES_NAMESPACE"}, + Destination: &cfg.OCDav.SharesNamespace, + }, + { + EnvVars: []string{"OCIS_URL", "OCDAV_PUBLIC_URL"}, + Destination: &cfg.OCDav.PublicURL, + }, + { + EnvVars: []string{"OCIS_INSECURE", "OCDAV_INSECURE"}, + Destination: &cfg.OCDav.Insecure, + }, + { + EnvVars: []string{"OCIS_JWT_SECRET", "OCDAV_JWT_SECRET"}, + Destination: &cfg.OCDav.JWTSecret, + }, } } diff --git a/storage/pkg/config/defaults/defaultconfig.go b/storage/pkg/config/defaults/defaultconfig.go index cdf9741dcf..3e51c8f3a3 100644 --- a/storage/pkg/config/defaults/defaultconfig.go +++ b/storage/pkg/config/defaults/defaultconfig.go @@ -91,10 +91,6 @@ func DefaultConfig() *config.Config { JoinOwnCloudUUID: false, EnableMedialSearch: false, }, - OCDav: config.OCDav{ - WebdavNamespace: defaultStorageNamespace, - DavFilesNamespace: defaultStorageNamespace, - }, Archiver: config.Archiver{ MaxNumFiles: 10000, MaxSize: 1073741824, @@ -235,7 +231,7 @@ func DefaultConfig() *config.Config { Protocol: "", Endpoint: "", DebugAddr: "127.0.0.1:9141", - Services: []string{"datagateway", "ocdav", "ocs", "appprovider"}, + Services: []string{"datagateway", "ocs", "appprovider"}, Config: nil, Context: nil, Supervised: false, @@ -247,8 +243,6 @@ func DefaultConfig() *config.Config { DatagatewayPrefix: "data", Favorites: false, ProjectSpaces: true, - OCDavInsecure: false, // true? - OCDavPrefix: "", OCSPrefix: "ocs", OCSSharePrefix: defaultShareFolder, OCSHomeNamespace: defaultStorageNamespace, @@ -445,6 +439,20 @@ func DefaultConfig() *config.Config { ChecksumPreferredUploadType: "", DefaultUploadProtocol: "tus", }, + // TODO move ocdav config to a separate service + OCDav: config.OCDav{ + Addr: "127.0.0.1:0", // :0 to pick any local free port + DebugAddr: "127.0.0.1:9163", + WebdavNamespace: defaultStorageNamespace, + FilesNamespace: defaultStorageNamespace, + SharesNamespace: defaultShareFolder, + PublicURL: defaultPublicURL, + Prefix: "", + GatewaySVC: defaultGatewayAddr, + Insecure: false, // true? + Timeout: 84300, + JWTSecret: "Pive-Fumkiu4", + }, Tracing: config.Tracing{ Service: "storage", Type: "jaeger", diff --git a/tests/parallelDeployAcceptance/drone/ocis/proxy.yaml b/tests/parallelDeployAcceptance/drone/ocis/proxy.yaml index a472adcf12..50f89d6ac3 100644 --- a/tests/parallelDeployAcceptance/drone/ocis/proxy.yaml +++ b/tests/parallelDeployAcceptance/drone/ocis/proxy.yaml @@ -19,15 +19,17 @@ policies: endpoint: "/remote.php/?preview=1" backend: http://localhost:9115 - endpoint: "/remote.php/" - backend: http://localhost:9140 + service: "ocdav" - endpoint: "/dav/" - backend: http://localhost:9140 + service: "ocdav" - endpoint: "/webdav/" - backend: http://localhost:9140 + service: "ocdav" - endpoint: "/status.php" - backend: http://localhost:9140 + service: "ocdav" - endpoint: "/index.php/" - backend: http://localhost:9140 + service: "ocdav" + - endpoint: "/apps/" + service: "ocdav" - endpoint: "/index.php/login" backend: http://localhost:9100 - endpoint: "/login" From 71ce6251bc0d8d8e07b12dcefc0dbf3b230ea970 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn=20Friedrich=20Dreyer?= Date: Tue, 12 Apr 2022 09:27:51 +0000 Subject: [PATCH 07/20] Automated changelog update [skip ci] --- CHANGELOG.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 253924c86b..75cf8d78de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,8 @@ The following sections list the changes for unreleased. * Enhancement - Add sorting to GraphAPI users and groups: [#3360](https://github.com/owncloud/ocis/issues/3360) * Enhancement - Unify LDAP config settings accross services: [#3476](https://github.com/owncloud/ocis/pull/3476) * Enhancement - Make config dir configurable: [#3440](https://github.com/owncloud/ocis/pull/3440) -* Enhancement - Update reva to v2.x.x: [#3430](https://github.com/owncloud/ocis/pull/3430) +* Enhancement - Use embeddable ocdav go micro service: [#3397](https://github.com/owncloud/ocis/pull/3397) +* Enhancement - Update reva to v2.x.x: [#3397](https://github.com/owncloud/ocis/pull/3397) * Enhancement - Update ownCloud Web to v5.4.0: [#6709](https://github.com/owncloud/web/pull/6709) * Enhancement - Implement audit events for user and groups: [#3467](https://github.com/owncloud/ocis/pull/3467) @@ -115,12 +116,20 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/3440 -* Enhancement - Update reva to v2.x.x: [#3430](https://github.com/owncloud/ocis/pull/3430) +* Enhancement - Use embeddable ocdav go micro service: [#3397](https://github.com/owncloud/ocis/pull/3397) + + We now use the reva `pgk/micro/ocdav` package that implements a go micro compatible version of + the ocdav service. + + https://github.com/owncloud/ocis/pull/3397 + +* Enhancement - Update reva to v2.x.x: [#3397](https://github.com/owncloud/ocis/pull/3397) Updated reva to version 2.x.x. This update includes: * TODO + https://github.com/owncloud/ocis/pull/3397 https://github.com/owncloud/ocis/pull/3430 https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 From adc6c0493aa3f22f8defd2716bb0fdaf7905b9c2 Mon Sep 17 00:00:00 2001 From: jkoberg Date: Tue, 12 Apr 2022 15:02:05 +0200 Subject: [PATCH 08/20] bump reva Signed-off-by: jkoberg --- changelog/unreleased/update-reva.md | 1 + go.mod | 8 +++----- go.sum | 12 ++++++------ 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index 08809c020a..f2a74c5958 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -8,4 +8,5 @@ https://github.com/owncloud/ocis/pull/3430 https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 https://github.com/owncloud/ocis/pull/3497 +https://github.com/owncloud/ocis/pull/3514 diff --git a/go.mod b/go.mod index 14c0c3a790..bf1449b14a 100644 --- a/go.mod +++ b/go.mod @@ -15,14 +15,14 @@ require ( github.com/asim/go-micro/plugins/registry/mdns/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75 - github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75 + github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291 github.com/asim/go-micro/plugins/wrapper/breaker/gobreaker/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/wrapper/monitoring/prometheus/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/asim/go-micro/plugins/wrapper/trace/opencensus/v4 v4.0.0-20220118152736-9e0be6c85d75 github.com/blevesearch/bleve/v2 v2.3.2 github.com/coreos/go-oidc/v3 v3.1.0 - github.com/cs3org/go-cs3apis v0.0.0-20220328105952-297bef33e13f - github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f + github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde + github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665 github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 @@ -272,5 +272,3 @@ require ( // we need to use a fork to make the windows build pass replace github.com/pkg/xattr => github.com/micbar/xattr v0.4.6-0.20220215112335-88e74d648fb7 - -replace github.com/cs3org/go-cs3apis => github.com/kobergj/go-cs3apis v0.0.0-20220406134716-65f04386eb09 // temp fork until cs3api change is merged diff --git a/go.sum b/go.sum index 857c212273..06ea6aee86 100644 --- a/go.sum +++ b/go.sum @@ -181,8 +181,8 @@ github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c8 github.com/asim/go-micro/plugins/registry/nats/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:n4K0361IySJ/hmIfaIg8q2NDNAVDL9DUC6Bh7gS4ihg= github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:miPpSGbTA6L2hsHccRNxNSXpIYo6YtlTkoi+fvEHJbw= github.com/asim/go-micro/plugins/server/grpc/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:uQACKU+/RcWKEMdX6/Ewd+TxCPLIsLPt9ym6++B1X68= -github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:ULuhDMrHjkEkj0dxWS+ZM9MD/9ywFPM7kaoEs4OjhSc= -github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220118152736-9e0be6c85d75/go.mod h1:LEsuhy4ZF3UGT9VSOoj/ZP0MZt4brlgwJVG7/LAac+I= +github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291 h1:l1l3pUleHu66DNRmaPwfgyXfVXT7UmyzNXD0uQp9SQE= +github.com/asim/go-micro/plugins/server/http/v4 v4.0.0-20220317022205-c6d352c83291/go.mod h1:LEsuhy4ZF3UGT9VSOoj/ZP0MZt4brlgwJVG7/LAac+I= github.com/asim/go-micro/plugins/transport/grpc/v4 v4.0.0-20211019191242-9edc569e68bb h1:rCIGzR4YRddwaWM4pZkbxHIkhFbbuGZOqivnxyKNQz8= github.com/asim/go-micro/plugins/transport/grpc/v4 v4.0.0-20211019191242-9edc569e68bb/go.mod h1:dzSEAsAqQ4mRaYPJ07KcAt6rIjYUJ9w55UXvnyGx4VE= github.com/asim/go-micro/plugins/wrapper/breaker/gobreaker/v4 v4.0.0-20220118152736-9e0be6c85d75 h1:8wLNrBKfdRl6uryUFW6qlBZWNeIbPUYKYkwLG9bI58k= @@ -335,8 +335,10 @@ github.com/crewjam/httperr v0.2.0/go.mod h1:Jlz+Sg/XqBQhyMjdDiC+GNNRzZTD7x39Gu3p github.com/crewjam/saml v0.4.6 h1:XCUFPkQSJLvzyl4cW9OvpWUbRf0gE7VUpU8ZnilbeM4= github.com/crewjam/saml v0.4.6/go.mod h1:ZBOXnNPFzB3CgOkRm7Nd6IVdkG+l/wF+0ZXLqD96t1A= github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= -github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f h1:SNQSbtvhsQ6KvdDJlW536pUMq8RzghQMDCkYYKEFn8c= -github.com/cs3org/reva/v2 v2.0.0-20220411161034-134ed5a0801f/go.mod h1:rE2wCGoiGiUuQ3zj6aD7sZ+BhboLyNsNQ3z3auunWm4= +github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde h1:WrD9O8ZaWvsm0eBzpzVBIuczDhqVq50Nmjc7PGHHA9Y= +github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= +github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665 h1:XbFCdhbfdOshRvU5uX5ud90+8A7KGIMp8DVa87Xp+M0= +github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665/go.mod h1:ThLaoO0196KODVwlvCoTSe2EaNS8+TJGv8CXviIVA/w= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= @@ -896,8 +898,6 @@ github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa02 github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4= github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= -github.com/kobergj/go-cs3apis v0.0.0-20220406134716-65f04386eb09 h1:i1caLRatgEscEdtcplmwjxHSVve13rQTuRDxo42FZI8= -github.com/kobergj/go-cs3apis v0.0.0-20220406134716-65f04386eb09/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= github.com/kolo/xmlrpc v0.0.0-20200310150728-e0350524596b/go.mod h1:o03bZfuBwAXHetKXuInt4S7omeXUu62/A845kiycsSQ= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= From 0fab27e2e1b7dd6435d13e79b90c2383ac6673b0 Mon Sep 17 00:00:00 2001 From: jkoberg Date: Tue, 12 Apr 2022 15:49:30 +0200 Subject: [PATCH 09/20] try different wait-for-ocis-server command Signed-off-by: jkoberg --- .drone.star | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.drone.star b/.drone.star index 26c0826d18..67b0760d67 100644 --- a/.drone.star +++ b/.drone.star @@ -1733,9 +1733,9 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = }, { "name": "wait-for-ocis-server", - "image": OC_CI_WAIT_FOR, + "image": OC_CI_ALPINE, "commands": [ - "wait-for -it ocis-server:9200 -t 300", + "curl -k -u admin:admin --retry 10 --retry-all-errors 'https://ocis-server:9200/graph/v1.0/users/ddc2004c-0977-11eb-9d3f-a793888cd0f8'", ], "depends_on": depends_on, }, From bb0a140889aa91b9ca968713bcf880024c7cfd70 Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Tue, 12 Apr 2022 16:37:31 +0200 Subject: [PATCH 10/20] Refresh idm setup docs after LDAP config changes --- docs/extensions/idm/setup.md | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-) diff --git a/docs/extensions/idm/setup.md b/docs/extensions/idm/setup.md index 13c13f8b10..3a5f728dcd 100644 --- a/docs/extensions/idm/setup.md +++ b/docs/extensions/idm/setup.md @@ -19,37 +19,29 @@ the users and groups backend (this setup also disables the glauth and accounts s ``` export GRAPH_IDENTITY_BACKEND=ldap -export GRAPH_LDAP_URI=ldaps://localhost:9235 +export LDAP_URI=ldaps://localhost:9235 +export LDAP_INSECURE="true" +export LDAP_USER_BASE_DN="ou=users,o=libregraph-idm" +export LDAP_USER_SCHEMA_ID="ownclouduuid" +export LDAP_USER_SCHEMA_MAIL="mail" +export LDAP_USER_SCHEMA_USERNAME="uid" +export LDAP_USER_OBJECTCLASS="inetOrgPerson" +export LDAP_GROUP_BASE_DN="ou=groups,o=libregraph-idm" +export LDAP_GROUP_SCHEMA_ID="ownclouduuid" +export LDAP_GROUP_SCHEMA_MAIL="mail" +export LDAP_GROUP_SCHEMA_GROUPNAME="member" +export LDAP_GROUP_OBJECTCLASS="groupOfNames" export GRAPH_LDAP_BIND_DN="uid=libregraph,ou=sysusers,o=libregraph-idm" export GRAPH_LDAP_BIND_PASSWORD=idm -export GRAPH_LDAP_USER_EMAIL_ATTRIBUTE=mail -export GRAPH_LDAP_USER_NAME_ATTRIBUTE=uid -export GRAPH_LDAP_USER_BASE_DN="ou=users,o=libregraph-idm" -export GRAPH_LDAP_GROUP_BASE_DN="ou=groups,o=libregraph-idm" export GRAPH_LDAP_SERVER_WRITE_ENABLED="true" export IDP_INSECURE="true" -export IDP_LDAP_FILTER="(&(objectclass=inetOrgPerson)(objectClass=owncloud))" -export IDP_LDAP_URI=ldaps://localhost:9235 export IDP_LDAP_BIND_DN="uid=idp,ou=sysusers,o=libregraph-idm" export IDP_LDAP_BIND_PASSWORD="idp" -export IDP_LDAP_BASE_DN="ou=users,o=libregraph-idm" export IDP_LDAP_LOGIN_ATTRIBUTE=uid -export IDP_LDAP_UUID_ATTRIBUTE="ownclouduuid" -export IDP_LDAP_UUID_ATTRIBUTE_TYPE=binary export PROXY_ACCOUNT_BACKEND_TYPE=cs3 export OCS_ACCOUNT_BACKEND_TYPE=cs3 -export STORAGE_LDAP_HOSTNAME=localhost -export STORAGE_LDAP_PORT=9235 -export STORAGE_LDAP_INSECURE="true" -export STORAGE_LDAP_BASE_DN="o=libregraph-idm" export STORAGE_LDAP_BIND_DN="uid=reva,ou=sysusers,o=libregraph-idm" export STORAGE_LDAP_BIND_PASSWORD=reva -export STORAGE_LDAP_LOGINFILTER='(&(objectclass=inetOrgPerson)(objectclass=owncloud)(|(uid={{login}})(mail={{login}})))' -export STORAGE_LDAP_USERFILTER='(&(objectclass=inetOrgPerson)(objectclass=owncloud)(|(ownclouduuid={{.OpaqueId}})(uid={{.OpaqueId}})))' -export STORAGE_LDAP_USERATTRIBUTEFILTER='(&(objectclass=owncloud)({{attr}}={{value}}))' -export STORAGE_LDAP_USERFINDFILTER='(&(objectclass=owncloud)(|(uid={{query}}*)(cn={{query}}*)(displayname={{query}}*)(mail={{query}}*)(description={{query}}*)))' -export STORAGE_LDAP_USERGROUPFILER='(&(objectclass=groupOfNames)(member={{query}}*))' -export STORAGE_LDAP_GROUPFILTER='(&(objectclass=groupOfNames)(objectclass=owncloud)(ownclouduuid={{.OpaqueId}}*))' export OCIS_RUN_EXTENSIONS=settings,storage-metadata,graph,graph-explorer,ocs,store,thumbnails,web,webdav,storage-frontend,storage-gateway,storage-userprovider,storage-groupprovider,storage-authbasic,storage-authbearer,storage-authmachine,storage-users,storage-shares,storage-public-link,storage-appprovider,storage-sharing,proxy,idp,nats,idm,ocdav export OCIS_INSECURE=true bin/ocis server From ac8b1295ec5ba131fd79ab65b0bb26f2971c8f16 Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Tue, 12 Apr 2022 15:05:52 +0000 Subject: [PATCH 11/20] Automated changelog update [skip ci] --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 75cf8d78de..94d3e23de0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -134,6 +134,7 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 https://github.com/owncloud/ocis/pull/3497 + https://github.com/owncloud/ocis/pull/3514 * Enhancement - Update ownCloud Web to v5.4.0: [#6709](https://github.com/owncloud/web/pull/6709) From 600c6207a00b412f33ba9bc8cdaaede9cdedea7d Mon Sep 17 00:00:00 2001 From: Michael Barz Date: Tue, 12 Apr 2022 12:21:29 +0200 Subject: [PATCH 12/20] update reva --- changelog/unreleased/update-reva.md | 20 +++++++++++++++++--- go.mod | 2 +- go.sum | 5 +++-- 3 files changed, 21 insertions(+), 6 deletions(-) diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index c06354113e..3c1238b9bb 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -1,13 +1,27 @@ -Enhancement: update reva to v2.x.x +Enhancement: update reva to v2.2.0 -Updated reva to version 2.x.x. This update includes: +Updated reva to version 2.2.0. This update includes: - * TODO +* Bugfix [cs3org/reva#3373](https://github.com/cs3org/reva/pull/3373): Fix the permissions attribute in propfind responses +* Bugfix [cs3org/reva#2721](https://github.com/cs3org/reva/pull/2721): Fix locking and public link scope checker to make the WOPI server work +* Bugfix [cs3org/reva#2668](https://github.com/cs3org/reva/pull/2668): Minor cleanup +* Bugfix [cs3org/reva#2692](https://github.com/cs3org/reva/pull/2692): Ensure that the host in the ocs config endpoint has no protocol +* Bugfix [cs3org/reva#2709](https://github.com/cs3org/reva/pull/2709): Decomposed FS: return precondition failed if already locked +* Change [cs3org/reva#2687](https://github.com/cs3org/reva/pull/2687): Allow link with no or edit permission +* Change [cs3org/reva#2658](https://github.com/cs3org/reva/pull/2658): Small clean up of the ocdav code +* Change [cs3org/reva#2691](https://github.com/cs3org/reva/pull/2691): Decomposed FS: return a reference to the parent +* Enhancement [cs3org/reva#2708](https://github.com/cs3org/reva/pull/2708): Rework LDAP configuration of user and group providers +* Enhancement [cs3org/reva#2665](https://github.com/cs3org/reva/pull/2665): Add embeddable ocdav go micro service +* Enhancement [cs3org/reva#2715](https://github.com/cs3org/reva/pull/2715): Introduced quicklinks +* Enhancement [cs3org/reva#3370](https://github.com/cs3org/reva/pull/3370): Enable all spaces members to list public shares +* Enhancement [cs3org/reva#3370](https://github.com/cs3org/reva/pull/3370): Enable space members to list shares inside the space +* Enhancement [cs3org/reva#2717](https://github.com/cs3org/reva/pull/2717): Add definitions for user and group events https://github.com/owncloud/ocis/pull/3397 https://github.com/owncloud/ocis/pull/3430 https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 https://github.com/owncloud/ocis/pull/3497 +https://github.com/owncloud/ocis/pull/3513 https://github.com/owncloud/ocis/pull/3514 diff --git a/go.mod b/go.mod index bf1449b14a..96bb7680bb 100644 --- a/go.mod +++ b/go.mod @@ -22,7 +22,7 @@ require ( github.com/blevesearch/bleve/v2 v2.3.2 github.com/coreos/go-oidc/v3 v3.1.0 github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde - github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665 + github.com/cs3org/reva/v2 v2.2.0 github.com/disintegration/imaging v1.6.2 github.com/glauth/glauth/v2 v2.0.0-20211021011345-ef3151c28733 github.com/go-chi/chi/v5 v5.0.7 diff --git a/go.sum b/go.sum index 06ea6aee86..5347eda187 100644 --- a/go.sum +++ b/go.sum @@ -335,10 +335,11 @@ github.com/crewjam/httperr v0.2.0/go.mod h1:Jlz+Sg/XqBQhyMjdDiC+GNNRzZTD7x39Gu3p github.com/crewjam/saml v0.4.6 h1:XCUFPkQSJLvzyl4cW9OvpWUbRf0gE7VUpU8ZnilbeM4= github.com/crewjam/saml v0.4.6/go.mod h1:ZBOXnNPFzB3CgOkRm7Nd6IVdkG+l/wF+0ZXLqD96t1A= github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4= +github.com/cs3org/go-cs3apis v0.0.0-20220328105952-297bef33e13f/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde h1:WrD9O8ZaWvsm0eBzpzVBIuczDhqVq50Nmjc7PGHHA9Y= github.com/cs3org/go-cs3apis v0.0.0-20220412090512-93c5918b4bde/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY= -github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665 h1:XbFCdhbfdOshRvU5uX5ud90+8A7KGIMp8DVa87Xp+M0= -github.com/cs3org/reva/v2 v2.0.0-20220412125706-8b210924f665/go.mod h1:ThLaoO0196KODVwlvCoTSe2EaNS8+TJGv8CXviIVA/w= +github.com/cs3org/reva/v2 v2.2.0 h1:OZRaJyz6/mLkR/B77XwYm9fwERRdk6LcU/7lLVP6tFI= +github.com/cs3org/reva/v2 v2.2.0/go.mod h1:l1dhodFXCp88/Lc0VhzSeaLSQUTn2AdwwkTFtaLCJhk= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI= github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= From e674345ef884ee05f0d57638d756ee8cb0af3263 Mon Sep 17 00:00:00 2001 From: jkoberg Date: Tue, 12 Apr 2022 15:02:05 +0200 Subject: [PATCH 13/20] bump reva Signed-off-by: jkoberg --- changelog/unreleased/update-reva.md | 1 - 1 file changed, 1 deletion(-) diff --git a/changelog/unreleased/update-reva.md b/changelog/unreleased/update-reva.md index 3c1238b9bb..9402babc3b 100644 --- a/changelog/unreleased/update-reva.md +++ b/changelog/unreleased/update-reva.md @@ -24,4 +24,3 @@ https://github.com/owncloud/ocis/pull/3482 https://github.com/owncloud/ocis/pull/3497 https://github.com/owncloud/ocis/pull/3513 https://github.com/owncloud/ocis/pull/3514 - From 67abbedb1b08baeaa5689ec0960b5cc55fe94080 Mon Sep 17 00:00:00 2001 From: Willy Kloucek <34452982+wkloucek@users.noreply.github.com> Date: Wed, 13 Apr 2022 06:35:38 +0000 Subject: [PATCH 14/20] Automated changelog update [skip ci] --- CHANGELOG.md | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94d3e23de0..a8d3d1707b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ The following sections list the changes for unreleased. * Enhancement - Unify LDAP config settings accross services: [#3476](https://github.com/owncloud/ocis/pull/3476) * Enhancement - Make config dir configurable: [#3440](https://github.com/owncloud/ocis/pull/3440) * Enhancement - Use embeddable ocdav go micro service: [#3397](https://github.com/owncloud/ocis/pull/3397) -* Enhancement - Update reva to v2.x.x: [#3397](https://github.com/owncloud/ocis/pull/3397) +* Enhancement - Update reva to v2.2.0: [#3397](https://github.com/owncloud/ocis/pull/3397) * Enhancement - Update ownCloud Web to v5.4.0: [#6709](https://github.com/owncloud/web/pull/6709) * Enhancement - Implement audit events for user and groups: [#3467](https://github.com/owncloud/ocis/pull/3467) @@ -123,17 +123,31 @@ The following sections list the changes for unreleased. https://github.com/owncloud/ocis/pull/3397 -* Enhancement - Update reva to v2.x.x: [#3397](https://github.com/owncloud/ocis/pull/3397) +* Enhancement - Update reva to v2.2.0: [#3397](https://github.com/owncloud/ocis/pull/3397) - Updated reva to version 2.x.x. This update includes: + Updated reva to version 2.2.0. This update includes: - * TODO + * Bugfix [cs3org/reva#3373](https://github.com/cs3org/reva/pull/3373): Fix the permissions attribute in propfind responses + * Bugfix [cs3org/reva#2721](https://github.com/cs3org/reva/pull/2721): Fix locking and public link scope checker to make the WOPI server work + * Bugfix [cs3org/reva#2668](https://github.com/cs3org/reva/pull/2668): Minor cleanup + * Bugfix [cs3org/reva#2692](https://github.com/cs3org/reva/pull/2692): Ensure that the host in the ocs config endpoint has no protocol + * Bugfix [cs3org/reva#2709](https://github.com/cs3org/reva/pull/2709): Decomposed FS: return precondition failed if already locked + * Change [cs3org/reva#2687](https://github.com/cs3org/reva/pull/2687): Allow link with no or edit permission + * Change [cs3org/reva#2658](https://github.com/cs3org/reva/pull/2658): Small clean up of the ocdav code + * Change [cs3org/reva#2691](https://github.com/cs3org/reva/pull/2691): Decomposed FS: return a reference to the parent + * Enhancement [cs3org/reva#2708](https://github.com/cs3org/reva/pull/2708): Rework LDAP configuration of user and group providers + * Enhancement [cs3org/reva#2665](https://github.com/cs3org/reva/pull/2665): Add embeddable ocdav go micro service + * Enhancement [cs3org/reva#2715](https://github.com/cs3org/reva/pull/2715): Introduced quicklinks + * Enhancement [cs3org/reva#3370](https://github.com/cs3org/reva/pull/3370): Enable all spaces members to list public shares + * Enhancement [cs3org/reva#3370](https://github.com/cs3org/reva/pull/3370): Enable space members to list shares inside the space + * Enhancement [cs3org/reva#2717](https://github.com/cs3org/reva/pull/2717): Add definitions for user and group events https://github.com/owncloud/ocis/pull/3397 https://github.com/owncloud/ocis/pull/3430 https://github.com/owncloud/ocis/pull/3476 https://github.com/owncloud/ocis/pull/3482 https://github.com/owncloud/ocis/pull/3497 + https://github.com/owncloud/ocis/pull/3513 https://github.com/owncloud/ocis/pull/3514 * Enhancement - Update ownCloud Web to v5.4.0: [#6709](https://github.com/owncloud/web/pull/6709) From 1a3e4ba66d9308e641723bece9b06c0fd6d04a1a Mon Sep 17 00:00:00 2001 From: Willy Kloucek Date: Wed, 13 Apr 2022 09:33:57 +0200 Subject: [PATCH 15/20] ensure svg width does not exceed the page space --- docs/architecture/services-communication.md | 2 +- docs/extensions/storage/_index.md | 4 ++-- docs/extensions/storage/namespaces.md | 2 +- docs/extensions/storage/spaces.md | 2 +- docs/extensions/storage/spacesprovider.md | 10 +++++----- docs/extensions/storage/spacesregistry.md | 3 +-- docs/extensions/storage/terminology.md | 7 +++---- docs/ocis/_index.md | 4 ++-- docs/ocis/adr/0014-microservices-runtime.md | 2 +- docs/ocis/config.md | 2 +- docs/ocis/deployment/monitoring-tracing.md | 2 +- docs/ocis/flow-docs/public-upload-flow.md | 2 +- 12 files changed, 20 insertions(+), 22 deletions(-) diff --git a/docs/architecture/services-communication.md b/docs/architecture/services-communication.md index 7ce661c18c..f3e827426d 100644 --- a/docs/architecture/services-communication.md +++ b/docs/architecture/services-communication.md @@ -7,4 +7,4 @@ geekdocEditPath: edit/master/docs/architecture geekdocFilePath: services-communication.md --- -{{< svg src="/ocis/static/ocis-services-communication.drawio.svg" >}} \ No newline at end of file +{{< figure src="/ocis/static/ocis-services-communication.drawio.svg" >}} diff --git a/docs/extensions/storage/_index.md b/docs/extensions/storage/_index.md index 8ae194026c..1c4e140f15 100644 --- a/docs/extensions/storage/_index.md +++ b/docs/extensions/storage/_index.md @@ -16,7 +16,7 @@ The storage extension wraps [reva](https://github.com/cs3org/reva/) and adds an *Clients* will use the *Spaces Registry* to poll or get notified about changes in all *Spaces* a user has access to. Every *Space* has a dedicated `/dav/spaces/` WebDAV endpoint that is served by a *Spaces Provider* which uses a specific reva storage driver to wrap an underlying *Storage System*. -{{< svg src="extensions/storage/static/overview.drawio.svg" >}} +{{< figure src="/extensions/storage/static/overview.drawio.svg" >}} The dashed lines in the diagram indicate requests that are made to authenticate requests or lookup the storage provider: 1. After authenticating a request, the proxy may either use the CS3 `userprovider` or the accounts service to fetch the user information that will be minted into the `x-access-token`. @@ -31,4 +31,4 @@ The bottom part is lighter because we will deprecate it in favor of using only t In order to reason about the request flow, two aspects in the architecture need to be understood well: 1. What kind of [*namespaces*]({{< ref "./namespaces.md" >}}) are presented at the different WebDAV and CS3 endpoints? 2. What kind of [*resource*]({{< ref "./terminology.md#resources" >}}) [*references*]({{< ref "./terminology.md#references" >}}) are exposed or required: path or id based? -{{< svg src="extensions/storage/static/storage.drawio.svg" >}} +{{< figure src="/extensions/storage/static/storage.drawio.svg" >}} diff --git a/docs/extensions/storage/namespaces.md b/docs/extensions/storage/namespaces.md index 19864a6e7f..227835acd5 100644 --- a/docs/extensions/storage/namespaces.md +++ b/docs/extensions/storage/namespaces.md @@ -10,7 +10,7 @@ geekdocFilePath: namespaces.md A *namespace* is a set of paths with a common prefix. Depending on the endpoint you are talking to you will encounter a different kind of namespace: In ownCloud 10 all paths are considered relative to the users home. The CS3 API uses a global namespace and the *storage providers* use a local namespace with paths relative to the storage providers root. -{{< svg src="extensions/storage/static/namespaces.drawio.svg" >}} +{{< figure src="/extensions/storage/static/namespaces.drawio.svg" >}} The different paths in the namespaces need to be translated while passing [*references*]({{< ref "./terminology.md#references" >}}) from service to service. While the oc10 endpoints all work on paths we internally reference shared resources by id, so the shares don't break when a file is renamed or moved inside a storage [*space*]({{< ref "./spaces" >}}). The following table lists the various namespaces, paths and id based references: diff --git a/docs/extensions/storage/spaces.md b/docs/extensions/storage/spaces.md index 45b445c79c..567e8ddf2e 100644 --- a/docs/extensions/storage/spaces.md +++ b/docs/extensions/storage/spaces.md @@ -186,7 +186,7 @@ The current implementation in oCIS might not yet fully reflect this concept. Fee A storage *space* is a logical concept. It organizes a set of [*resources*]({{< ref "#resources" >}}) in a hierarchical tree. It has a single *owner* (*user* or *group*), a *quota*, *permissions* and is identified by a `storage space id`. -{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} +{{< figure src="/extensions/storage/static/storagespace.drawio.svg" >}} Examples would be every user's personal storage *space*, project storage *spaces* or group storage *spaces*. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A storage [*Spaces Registry*]({{< ref "./spacesregistry.md" >}}) then allows listing the capabilities of storage *spaces*, e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... diff --git a/docs/extensions/storage/spacesprovider.md b/docs/extensions/storage/spacesprovider.md index 5a706ff514..589c9ff29b 100644 --- a/docs/extensions/storage/spacesprovider.md +++ b/docs/extensions/storage/spacesprovider.md @@ -17,18 +17,18 @@ The current implementation in oCIS might not yet fully reflect this concept. Fee A *storage provider* manages [*resources*]({{< ref "#resources" >}}) identified by a [*reference*]({{< ref "#references" >}}) by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "./storagedrivers.md" >}}). -{{< svg src="extensions/storage/static/spacesprovider.drawio.svg" >}} +{{< figure src="/extensions/storage/static/spacesprovider.drawio.svg" >}} ## Frontend The oCIS frontend service starts all services that handle incoming HTTP requests: - *ocdav* for ownCloud flavoured WebDAV -- *ocs* for sharing, user provisioning, capabilities and other OCS API endpoints +- *ocs* for sharing, user provisioning, capabilities and other OCS API endpoints - *datagateway* for up and downloads - TODO: *ocm* -{{< svg src="extensions/storage/static/frontend.drawio.svg" >}} +{{< figure src="/extensions/storage/static/frontend.drawio.svg" >}} ### WebDAV @@ -83,7 +83,7 @@ The API [already returns the storage id](https://doc.owncloud.com/server/develop 3994486 3994485 /Shared/Paris.jpg -``` +``` [Creating shares only takes the **path** as the argument](https://doc.owncloud.com/server/developer_manual/core/apis/ocs-share-api.html#function-arguments) so creating and navigating shares only needs the path. When you update or delete a share it takes the `share id` not the `file id`. {{< /hint >}} @@ -109,4 +109,4 @@ It is used by the reva *gateway* to look up `address` and `port` of the [*storage provider*]({{< ref "#storage-providers" >}}) that should handle a [*reference*]({{< ref "#references" >}}). -{{< svg src="extensions/storage/static/storageregistry.drawio.svg" >}} \ No newline at end of file +{{< figure src="/extensions/storage/static/storageregistry.drawio.svg" >}} diff --git a/docs/extensions/storage/spacesregistry.md b/docs/extensions/storage/spacesregistry.md index d5be48f8ab..4880536e9a 100644 --- a/docs/extensions/storage/spacesregistry.md +++ b/docs/extensions/storage/spacesregistry.md @@ -17,5 +17,4 @@ The current implementation in oCIS might not yet fully reflect this concept. Fee A storage *spaces registry* manages the [*namespace*]({{< ref "./namespaces.md" >}}) for a *user*: it is used by *clients* to look up storage spaces a user has access to, the `/dav/spaces` endpoint to access it via WabDAV, and where the client should mount it in the users personal namespace. -{{< svg src="extensions/storage/static/spacesregistry.drawio.svg" >}} - +{{< figure src="/extensions/storage/static/spacesregistry.drawio.svg" >}} diff --git a/docs/extensions/storage/terminology.md b/docs/extensions/storage/terminology.md index e40e2ad04f..4ff810e673 100644 --- a/docs/extensions/storage/terminology.md +++ b/docs/extensions/storage/terminology.md @@ -64,12 +64,12 @@ Technically, this means that every storage driver needs to have a map of a `uuid ## Technical concepts ### Storage Systems -{{< svg src="extensions/storage/static/storageprovider.drawio.svg" >}} +{{< figure src="/extensions/storage/static/storageprovider.drawio.svg" >}} A *storage provider* manages multiple [*storage spaces*]({{< ref "#storage-space" >}}) by accessing a [*storage system*]({{< ref "#storage-systems" >}}) with a [*storage driver*]({{< ref "#storage-drivers" >}}). -{{< svg src="extensions/storage/static/storageprovider-spaces.drawio.svg" >}} +{{< figure src="/extensions/storage/static/storageprovider-spaces.drawio.svg" >}} ## Storage Space Registries @@ -81,7 +81,7 @@ It is a tree of [*resources*]({{< ref "#resources" >}})*resources* with a single *owner* (*user* or *group*), a *quota* and *permissions*, identified by a `storage space id`. -{{< svg src="extensions/storage/static/storagespace.drawio.svg" >}} +{{< figure src="/extensions/storage/static/storagespace.drawio.svg" >}} Examples would be every user's home storage space, project storage spaces or group storage spaces. While they all serve different purposes and may or may not have workflows like anti virus scanning enabled, we need a way to identify and manage these subtrees in a generic way. By creating a dedicated concept for them this becomes easier and literally makes the codebase cleaner. A [*storage space registry*]({{< ref "#storage-space-registries" >}}) then allows listing the capabilities of [*storage spaces*]({{< ref "#storage-spaces" >}}), e.g. free space, quota, owner, syncable, root etag, upload workflow steps, ... @@ -98,4 +98,3 @@ There might be multiple [*storage drivers*]({{< ref "./storagedrivers.md" >}}) f ### Gateways A *gateway* acts as a facade to the storage related services. It authenticates and forwards API calls that are publicly accessible. - diff --git a/docs/ocis/_index.md b/docs/ocis/_index.md index 766df0526f..73c3bcc15d 100644 --- a/docs/ocis/_index.md +++ b/docs/ocis/_index.md @@ -21,7 +21,7 @@ The below diagram shows the core conceps that are the foundation for the new arc - [*Storage spaces*]({{< ref "../extensions/storage/terminology#storage-spaces" >}}) represent a collection of files and folders. A users personal files are contained in a *storage space*, a group or project drive is a *storage space*, and even incoming shares are treated and implemented as *storage spaces*. Each with properties like owners, permissions, quota and type. - [*Storage providers*]({{< ref "../extensions/storage/terminology#storage-providers" >}}) can hold multiple *storage spaces*. At an oCIS instance, there might be a dedicated *storage provider* responsible for users personal storage spaces. There might be multiple, either to shard the load, provide different levels of redundancy or support custom workflows. Or there might be just one, hosting all types of *storage spaces*. -{{< svg src="ocis/static/idea.drawio.svg" >}} +{{< figure src="/ocis/static/idea.drawio.svg" >}} As an example, Einstein might want to share something with Marie, who has an account at a different identity provider and uses a different storage space registry. The process makes use of [OpenID Connect (OIDC)](https://openid.net/specs/openid-connect-core-1_0.html) for authentication and would look something like this: @@ -63,4 +63,4 @@ We run a huge [test suite](https://github.com/owncloud/core/tree/master/tests), Running `bin/ocis server` will start the below services, all of which can be scaled and deployed on a single node or in a cloud native environment, as needed. -{{< svg src="ocis/static/architecture-overview.drawio.svg" >}} +{{< figure src="/ocis/static/architecture-overview.drawio.svg" >}} diff --git a/docs/ocis/adr/0014-microservices-runtime.md b/docs/ocis/adr/0014-microservices-runtime.md index 820c7f6739..89f68523cf 100644 --- a/docs/ocis/adr/0014-microservices-runtime.md +++ b/docs/ocis/adr/0014-microservices-runtime.md @@ -56,7 +56,7 @@ Number 3: A hybrid solution between framework and in-house. ### Design -{{< svg src="ocis/static/runtime.drawio.svg" >}} +{{< figure src="/ocis/static/runtime.drawio.svg" >}} First of, every ocis service IS a go-micro service, and because go-micro makes use of urfave/cli, a service can be conveniently wrapped inside a subcommand. Writing a supervisor is then a choice. We do use a supervisor to ensure long-running processes and embrace the "let it crash" mentality. The piece we use for this end is called [Suture](https://github.com/thejerf/suture). diff --git a/docs/ocis/config.md b/docs/ocis/config.md index a07e127257..f4ab30cf60 100644 --- a/docs/ocis/config.md +++ b/docs/ocis/config.md @@ -15,7 +15,7 @@ In order to simplify deployments and development the configuration model from oC ## Overview of the approach -{{< svg src="ocis/static/ocis-config-redesign.drawio.svg" >}} +{{< figure src="/ocis/static/ocis-config-redesign.drawio.svg" >}} ## In-depth configuration diff --git a/docs/ocis/deployment/monitoring-tracing.md b/docs/ocis/deployment/monitoring-tracing.md index 26c2c2ee20..139a8e7b13 100644 --- a/docs/ocis/deployment/monitoring-tracing.md +++ b/docs/ocis/deployment/monitoring-tracing.md @@ -17,7 +17,7 @@ This documentation describes how to set up a long running monitoring & tracing i # Overview about the proposed solution -{{< svg src="ocis/static/monitoring_tracing_overview.drawio.svg" >}} +{{< figure src="/ocis/static/monitoring_tracing_overview.drawio.svg" >}} ## Monitoring & tracing clients diff --git a/docs/ocis/flow-docs/public-upload-flow.md b/docs/ocis/flow-docs/public-upload-flow.md index 78adf0c047..ae9a5d025a 100644 --- a/docs/ocis/flow-docs/public-upload-flow.md +++ b/docs/ocis/flow-docs/public-upload-flow.md @@ -12,4 +12,4 @@ geekdocFilePath: public-upload-flow.md The following diagram describes the flow of requests: -{{< svg src="ocis/static/tus-public-upload.svg" >}} +{{< figure src="/ocis/static/tus-public-upload.svg" >}} From 6a21f12ec616039dd48b5fd51fcfbc04f8b3282f Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Wed, 13 Apr 2022 11:31:47 +0200 Subject: [PATCH 16/20] Improve wait-for-ocis check Retry on all HTTP response codes indicating and error. --- .drone.star | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.drone.star b/.drone.star index 67b0760d67..8126cdbb8e 100644 --- a/.drone.star +++ b/.drone.star @@ -1735,7 +1735,7 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = "name": "wait-for-ocis-server", "image": OC_CI_ALPINE, "commands": [ - "curl -k -u admin:admin --retry 10 --retry-all-errors 'https://ocis-server:9200/graph/v1.0/users/ddc2004c-0977-11eb-9d3f-a793888cd0f8'", + "curl -k -u admin:admin --fail --retry-connrefused --retry 10 --retry-all-errors 'https://ocis-server:9200/graph/v1.0/users/ddc2004c-0977-11eb-9d3f-a793888cd0f8'", ], "depends_on": depends_on, }, From 3e0158e4a6a6752283454200343485386bf85141 Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Wed, 13 Apr 2022 12:22:15 +0200 Subject: [PATCH 17/20] Use a different wait for ocis check in the parallel deployment The routing in the parallel deployment is directed to oc10 by default, which might not have and admin user and does not provide the graph API. Switch back to the old wait-for image, which should be good enough in that specific test setup. --- .drone.star | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/.drone.star b/.drone.star index 8126cdbb8e..d5f2a49222 100644 --- a/.drone.star +++ b/.drone.star @@ -1630,6 +1630,14 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = "ACCOUNTS_DEMO_USERS_AND_GROUPS": True, # deprecated, remove after switching to LibreIDM "IDM_CREATE_DEMO_USERS": True, } + wait_for_ocis = { + "name": "wait-for-ocis-server", + "image": OC_CI_ALPINE, + "commands": [ + "curl -k -u admin:admin --fail --retry-connrefused --retry 10 --retry-all-errors 'https://ocis-server:9200/graph/v1.0/users/ddc2004c-0977-11eb-9d3f-a793888cd0f8'", + ], + "depends_on": depends_on, + } else: user = "33:33" environment = { @@ -1710,6 +1718,14 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = "ACCOUNTS_DEMO_USERS_AND_GROUPS": True, # deprecated, remove after switching to LibreIDM "IDM_CREATE_DEMO_USERS": True, } + wait_for_ocis = { + "name": "wait-for-ocis-server", + "image": OC_CI_WAIT_FOR, + "commands": [ + "wait-for -it ocis-server:9200 -t 300", + ], + "depends_on": depends_on, + } # Pass in "default" accounts_hash_difficulty to not set this environment variable. # That will allow OCIS to use whatever its built-in default is. @@ -1731,14 +1747,7 @@ def ocisServer(storage, accounts_hash_difficulty = 4, volumes = [], depends_on = "volumes": volumes, "depends_on": depends_on, }, - { - "name": "wait-for-ocis-server", - "image": OC_CI_ALPINE, - "commands": [ - "curl -k -u admin:admin --fail --retry-connrefused --retry 10 --retry-all-errors 'https://ocis-server:9200/graph/v1.0/users/ddc2004c-0977-11eb-9d3f-a793888cd0f8'", - ], - "depends_on": depends_on, - }, + wait_for_ocis, ] def middlewareService(): From 629f15afcb1cedddc3f95ce65bacce208f5cd013 Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Wed, 13 Apr 2022 11:11:50 +0000 Subject: [PATCH 18/20] Automated changelog update [skip ci] --- CHANGELOG.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a8d3d1707b..8c02ba8897 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ The following sections list the changes for unreleased. -[unreleased]: https://github.com/owncloud/ocis/compare/v1.19.0...master +[unreleased]: https://github.com/owncloud/ocis/compare/v1.19.1...master ## Summary @@ -170,11 +170,28 @@ The following sections list the changes for unreleased. Group member remove https://github.com/owncloud/ocis/pull/3467 +# Changelog for [1.19.1] (2022-03-29) + +The following sections list the changes for 1.19.1. + +[1.19.1]: https://github.com/owncloud/ocis/compare/v1.19.0...v1.19.1 + +## Summary + +* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) + +## Details + +* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) + + URLs for Special items (space image, readme) were broken. + + https://github.com/owncloud/ocis/pull/3419 # Changelog for [1.19.0] (2022-03-29) The following sections list the changes for 1.19.0. -[1.19.0]: https://github.com/owncloud/ocis/compare/v1.19.1...v1.19.0 +[1.19.0]: https://github.com/owncloud/ocis/compare/v1.18.0...v1.19.0 ## Summary @@ -343,23 +360,6 @@ The following sections list the changes for 1.19.0. https://github.com/owncloud/ocis/pull/3291 https://github.com/owncloud/ocis/pull/3375 https://github.com/owncloud/web/releases/tag/v5.3.0 -# Changelog for [1.19.1] (2022-03-29) - -The following sections list the changes for 1.19.1. - -[1.19.1]: https://github.com/owncloud/ocis/compare/v1.18.0...v1.19.1 - -## Summary - -* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) - -## Details - -* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) - - URLs for Special items (space image, readme) were broken. - - https://github.com/owncloud/ocis/pull/3419 # Changelog for [1.18.0] (2022-03-03) The following sections list the changes for 1.18.0. From bca502d2bd09f512d72e507f377a159fc9f8fb41 Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Wed, 13 Apr 2022 15:09:16 +0200 Subject: [PATCH 19/20] Fix idm setup docs --- docs/extensions/idm/setup.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extensions/idm/setup.md b/docs/extensions/idm/setup.md index 3a5f728dcd..6d434dd605 100644 --- a/docs/extensions/idm/setup.md +++ b/docs/extensions/idm/setup.md @@ -29,7 +29,8 @@ export LDAP_USER_OBJECTCLASS="inetOrgPerson" export LDAP_GROUP_BASE_DN="ou=groups,o=libregraph-idm" export LDAP_GROUP_SCHEMA_ID="ownclouduuid" export LDAP_GROUP_SCHEMA_MAIL="mail" -export LDAP_GROUP_SCHEMA_GROUPNAME="member" +export LDAP_GROUP_SCHEMA_GROUPNAME="cn" +export LDAP_GROUP_SCHEMA_MEMBER="member" export LDAP_GROUP_OBJECTCLASS="groupOfNames" export GRAPH_LDAP_BIND_DN="uid=libregraph,ou=sysusers,o=libregraph-idm" export GRAPH_LDAP_BIND_PASSWORD=idm From 30f636abb8390a5546825f41adbe467d9fb7fb5e Mon Sep 17 00:00:00 2001 From: Ralf Haferkamp Date: Wed, 13 Apr 2022 13:34:05 +0000 Subject: [PATCH 20/20] Automated changelog update [skip ci] --- CHANGELOG.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c02ba8897..a8d3d1707b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ The following sections list the changes for unreleased. -[unreleased]: https://github.com/owncloud/ocis/compare/v1.19.1...master +[unreleased]: https://github.com/owncloud/ocis/compare/v1.19.0...master ## Summary @@ -170,28 +170,11 @@ The following sections list the changes for unreleased. Group member remove https://github.com/owncloud/ocis/pull/3467 -# Changelog for [1.19.1] (2022-03-29) - -The following sections list the changes for 1.19.1. - -[1.19.1]: https://github.com/owncloud/ocis/compare/v1.19.0...v1.19.1 - -## Summary - -* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) - -## Details - -* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) - - URLs for Special items (space image, readme) were broken. - - https://github.com/owncloud/ocis/pull/3419 # Changelog for [1.19.0] (2022-03-29) The following sections list the changes for 1.19.0. -[1.19.0]: https://github.com/owncloud/ocis/compare/v1.18.0...v1.19.0 +[1.19.0]: https://github.com/owncloud/ocis/compare/v1.19.1...v1.19.0 ## Summary @@ -360,6 +343,23 @@ The following sections list the changes for 1.19.0. https://github.com/owncloud/ocis/pull/3291 https://github.com/owncloud/ocis/pull/3375 https://github.com/owncloud/web/releases/tag/v5.3.0 +# Changelog for [1.19.1] (2022-03-29) + +The following sections list the changes for 1.19.1. + +[1.19.1]: https://github.com/owncloud/ocis/compare/v1.18.0...v1.19.1 + +## Summary + +* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) + +## Details + +* Bugfix - Return correct special item urls: [#3419](https://github.com/owncloud/ocis/pull/3419) + + URLs for Special items (space image, readme) were broken. + + https://github.com/owncloud/ocis/pull/3419 # Changelog for [1.18.0] (2022-03-03) The following sections list the changes for 1.18.0.