Compare commits

...

15 Commits

Author SHA1 Message Date
Sawjan Gurung
867da079a8 [full-ci][tests-only] test: fix some test flakiness (#2003)
* test: check content after upload

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

test: check content with retry

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* test: check empty body before json decoding

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* test: wait post-processing for webdav requests if applicable

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* test: check token before doing request

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

test: check body before json decoding

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

test: add wait step

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

---------

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2025-12-16 14:43:39 +05:45
Sawjan Gurung
389638878a Merge pull request #2007 from opencloud-eu/ci/fix-translation-pipeline
ci: fix translation pipeline
2025-12-16 14:43:39 +05:45
Saw-jan
949f14a278 test: add test to check mismatch offset during TUS upload
Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2025-12-16 14:43:39 +05:45
Saw-jan
27450c97bb test: proper resource existence check
Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2025-12-16 14:43:39 +05:45
OpenCloud Devops
ac072bee8f 🎉 Release 4.0.1 (#1984)
* 🎉 Release 4.0.1

* 🎉 Release 4.0.1

* 🎉 Release 4.0.1

* 🎉 Release 4.0.1

* 🎉 Release 4.0.1

* 🎉 Release 4.0.1
2025-12-15 14:52:02 +01:00
Viktor Scharf
fa4cd8e279 bump-version-4.0.1 (#2028)
apiAntivirus/antivirus.feature:228 was green in the previous run
2025-12-15 14:39:51 +01:00
Prashant Gurung
7bdbd9c474 skip test related pipelines for ready-release-go PRs (#2011) (#2018)
Signed-off-by: prashant-gurung899 <prasantgrg777@gmail.com>
2025-12-12 17:29:13 +05:45
Florian Schade
1d43bbea17 Merge pull request #2010 from fschade/backport/stable-4.0/pr-2001
[stable-4.0] fix: build time edition channels #2001
2025-12-12 09:25:15 +01:00
Florian Schade
d19968bec7 enhancement: introduce build time edition channels
be careful, the env:OC_EDITION, env:FRONTEND_EDITION, and conf:edition got removed as part of this commit, no deprecation because the flag is build time only!

(cherry picked from commit 40d8aacea4)
2025-12-11 11:52:57 +01:00
Benedikt Kulmann
4962328f0c fix: enforce trailing slash for server url (#2002)
(cherry picked from commit 614d916978)
2025-12-11 10:52:20 +01:00
Viktor Scharf
80cf0ba72c ci: use secret for the chat notifications url (#2006)
Co-authored-by: Jannik Stehle <j.stehle@opencloud.eu>
2025-12-11 10:02:56 +01:00
Michael Barz
26f5cd7493 fix: enhance resource creation with detailed process information (#1978) (#2000)
(cherry picked from commit 64119b3f8a)

Co-authored-by: Stavros Kois <s.kois@outlook.com>
2025-12-11 09:08:19 +01:00
Michael Barz
52cd4abc15 Add release_branch setting to release-helper 2025-12-05 22:27:13 +01:00
Viktor Scharf
4d5851cdff enable release-go plagin in stable-4.0 branch (#1982) 2025-12-05 15:38:58 +01:00
Viktor Scharf
5390322e38 [stable-4.0] replace golang image (#1955) (#1972)
* replace golang image (#1955)

(cherry picked from commit 6ce0cc6b1f)

* fix test

* Update CliContext.php
2025-12-05 10:39:02 +01:00
29 changed files with 484 additions and 70 deletions

View File

@@ -36,8 +36,18 @@ ifndef DATE
DATE := $(shell date -u '+%Y%m%d')
endif
LDFLAGS += -X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn -s -w -X "$(OC_REPO)/pkg/version.String=$(STRING)" -X "$(OC_REPO)/pkg/version.Tag=$(VERSION)" -X "$(OC_REPO)/pkg/version.Date=$(DATE)"
DEBUG_LDFLAGS += -X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn -X "$(OC_REPO)/pkg/version.String=$(STRING)" -X "$(OC_REPO)/pkg/version.Tag=$(VERSION)" -X "$(OC_REPO)/pkg/version.Date=$(DATE)"
LDFLAGS += -X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn -s -w \
-X "$(OC_REPO)/pkg/version.Edition=$(EDITION)" \
-X "$(OC_REPO)/pkg/version.String=$(STRING)" \
-X "$(OC_REPO)/pkg/version.Tag=$(VERSION)" \
-X "$(OC_REPO)/pkg/version.Date=$(DATE)"
DEBUG_LDFLAGS += -X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn \
-X "$(OC_REPO)/pkg/version.Edition=$(EDITION)" \
-X "$(OC_REPO)/pkg/version.String=$(STRING)" \
-X "$(OC_REPO)/pkg/version.Tag=$(VERSION)" \
-X "$(OC_REPO)/pkg/version.Date=$(DATE)"
DOCKER_LDFLAGS += -X "$(OC_REPO)/pkg/config/defaults.BaseDataPathType=path" -X "$(OC_REPO)/pkg/config/defaults.BaseDataPathValue=/var/lib/opencloud"
DOCKER_LDFLAGS += -X "$(OC_REPO)/pkg/config/defaults.BaseConfigPathType=path" -X "$(OC_REPO)/pkg/config/defaults.BaseConfigPathValue=/etc/opencloud"

View File

@@ -18,7 +18,7 @@ OC_CI_ALPINE = "owncloudci/alpine:latest"
OC_CI_BAZEL_BUILDIFIER = "owncloudci/bazel-buildifier:latest"
OC_CI_CLAMAVD = "owncloudci/clamavd"
OC_CI_DRONE_ANSIBLE = "owncloudci/drone-ansible:latest"
OC_CI_GOLANG = "docker.io/golang:1.24"
OC_CI_GOLANG = "registry.heinlein.group/opencloud/golang-ci:1.25"
OC_CI_NODEJS = "owncloudci/nodejs:%s"
OC_CI_PHP = "owncloudci/php:%s"
OC_CI_WAIT_FOR = "owncloudci/wait-for:latest"
@@ -41,7 +41,6 @@ DEFAULT_PHP_VERSION = "8.2"
DEFAULT_NODEJS_VERSION = "20"
CACHE_S3_SERVER = "https://s3.ci.opencloud.eu"
INSTALL_LIBVIPS_COMMAND = "apt-get update; apt-get install libvips42 -y"
dirs = {
"base": "/woodpecker/src/github.com/opencloud-eu/opencloud",
@@ -75,11 +74,11 @@ OC_FED_DOMAIN = "%s:10200" % FED_OC_SERVER_NAME
event = {
"base": {
"event": ["push", "manual"],
"branch": "main",
"branch": "stable-*",
},
"cron": {
"event": "cron",
"branch": "main",
"branch": "stable-*",
},
"pull_request": {
"event": "pull_request",
@@ -480,6 +479,10 @@ def main(ctx):
if ctx.build.event == "cron" and ctx.build.sender == "translation-sync":
return translation_sync(ctx)
is_release_pr = (ctx.build.event == "pull_request" and ctx.build.sender == "openclouders" and "🎉 release" in ctx.build.title.lower())
if is_release_pr:
return [licenseCheck(ctx)]
build_release_helpers = \
readyReleaseGo()
@@ -706,7 +709,7 @@ def restoreGoBinCache():
"name": "extract-go-bin-cache",
"image": OC_UBUNTU,
"commands": [
"tar -xmf %s -C /" % dirs["gobinTarPath"],
"tar -xvmf %s -C /" % dirs["gobinTarPath"],
],
},
]
@@ -1653,6 +1656,7 @@ def dockerRelease(ctx, repo, build_type):
build_args = {
"REVISION": "%s" % ctx.build.commit,
"VERSION": "%s" % (ctx.build.ref.replace("refs/tags/", "") if ctx.build.event == "tag" else "daily"),
"EDITION": "stable" if build_type == "production" else "rolling",
}
# if no additional tag is given, the build-plugin adds latest
@@ -1816,6 +1820,7 @@ def binaryRelease(ctx, arch, depends_on = []):
"image": OC_CI_GOLANG,
"environment": {
"VERSION": (ctx.build.ref.replace("refs/tags/", "") if ctx.build.event == "tag" else "daily"),
"EDITION": "rolling",
"HTTP_PROXY": {
"from_secret": "ci_http_proxy",
},
@@ -1948,6 +1953,7 @@ def readyReleaseGo():
"image": READY_RELEASE_GO,
"settings": {
"git_email": "devops@opencloud.eu",
"release_branch": "stable-4.0",
"forge_type": "github",
"forge_token": {
"from_secret": "github_token",
@@ -2061,7 +2067,9 @@ def notifyMatrix(ctx):
},
"QA_REPO": "https://github.com/opencloud-eu/qa.git",
"QA_REPO_BRANCH": "main",
"CI_WOODPECKER_URL": "https://ci.opencloud.eu/",
"CI_WOODPECKER_URL": {
"from_secret": "oc_ci_url",
},
"CI_REPO_ID": "3",
"CI_WOODPECKER_TOKEN": "no-auth-needed-on-this-repo",
},
@@ -2223,9 +2231,6 @@ def opencloudServer(storage = "decomposed", accounts_hash_difficulty = 4, depend
},
},
"commands": [
"apt-get update",
"apt-get install -y inotify-tools xattr",
INSTALL_LIBVIPS_COMMAND,
"%s init --insecure true" % dirs["opencloudBin"],
"cat $OC_CONFIG_DIR/opencloud.yaml",
"cp tests/config/woodpecker/app-registry.yaml $OC_CONFIG_DIR/app-registry.yaml",
@@ -2269,7 +2274,6 @@ def startOpenCloudService(service = None, name = None, environment = {}):
"detach": True,
"environment": environment,
"commands": [
INSTALL_LIBVIPS_COMMAND,
"%s %s server" % (dirs["opencloudBin"], service),
],
},
@@ -2295,7 +2299,6 @@ def build():
"name": "build",
"image": OC_CI_GOLANG,
"commands": [
"apt-get update; apt-get install libvips-dev -y",
"for i in $(seq 3); do make -C opencloud build ENABLE_VIPS=1 && break || sleep 1; done",
],
"environment": CI_HTTP_PROXY_ENV,
@@ -2344,11 +2347,12 @@ def translation_sync(ctx):
"image": OC_CI_GOLANG,
"commands": [
"make l10n-read",
"mkdir tx && cd tx",
"curl -o- https://raw.githubusercontent.com/transifex/cli/master/install.sh | bash",
". ~/.profile",
"export PATH=$PATH:$(pwd) && cd ..",
"make l10n-push",
"make l10n-pull",
"rm tx",
"rm -rf tx",
"make l10n-clean",
],
"environment": {

View File

@@ -1,5 +1,21 @@
# Changelog
## [4.0.1](https://github.com/opencloud-eu/opencloud/releases/tag/v4.0.1) - 2025-12-15
### ❤️ Thanks to all contributors! ❤️
@ScharfViktor, @fschade, @kulmann, @micbar, @prashant-gurung899
### ✅ Tests
- [stable-4.0] Port #2011 [[#2018](https://github.com/opencloud-eu/opencloud/pull/2018)]
### 🐛 Bug Fixes
- [stable-4.0] fix: build time edition channels #2001 [[#2010](https://github.com/opencloud-eu/opencloud/pull/2010)]
- [stable-4.0] fix: enforce trailing slash for server url [[#2002](https://github.com/opencloud-eu/opencloud/pull/2002)]
- [stable-4.0] fix: enhance resource creation with detailed process information (#1978) [[#2000](https://github.com/opencloud-eu/opencloud/pull/2000)]
## [4.0.0](https://github.com/opencloud-eu/opencloud/releases/tag/v4.0.0) - 2025-12-01
### ❤️ Thanks to all contributors! ❤️

View File

@@ -3,6 +3,7 @@ ARG TARGETOS
ARG TARGETARCH
ARG VERSION
ARG STRING
ARG EDITION
RUN apk add bash make git curl gcc musl-dev libc-dev binutils-gold inotify-tools vips-dev

View File

@@ -105,9 +105,19 @@ func createResource(ctx context.Context, serviceName string) (*resource.Resource
return resource.New(ctx,
// Reads OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME
resource.WithFromEnv(),
// Host and process information
// Host Information
resource.WithHost(),
resource.WithProcess(),
// Process Information
// Resource WithProcessOwner is deliberately omitted because
// inside containers where process might run as an arbitrary
// uid without a username associated this would fail.
resource.WithProcessPID(),
resource.WithProcessCommandArgs(),
resource.WithProcessExecutableName(),
resource.WithProcessExecutablePath(),
resource.WithProcessRuntimeDescription(),
resource.WithProcessRuntimeName(),
resource.WithProcessRuntimeVersion(),
// Service attributes
resource.WithAttributes(
semconv.ServiceName(serviceName),

View File

@@ -0,0 +1,4 @@
package version
// InitEdition exports the private edition initialization func for testing
var InitEdition = initEdition

View File

@@ -1,9 +1,27 @@
package version
import (
"fmt"
"slices"
"strings"
"time"
"github.com/Masterminds/semver"
"github.com/opencloud-eu/reva/v2/pkg/logger"
)
const (
// Dev is used as a placeholder.
Dev = "dev"
// EditionDev indicates the development build channel was used to build the binary.
EditionDev = Dev
// EditionRolling indicates the rolling release build channel was used to build the binary.
EditionRolling = "rolling"
// EditionStable indicates the stable release build channel was used to build the binary.
EditionStable = "stable"
// EditionLTS indicates the lts release build channel was used to build the binary.
EditionLTS = "lts"
)
var (
@@ -16,22 +34,61 @@ var (
// LatestTag is the latest released version plus the dev meta version.
// Will be overwritten by the release pipeline
// Needs a manual change for every tagged release
LatestTag = "4.0.0-rc.3+dev"
LatestTag = "4.0.1+dev"
// Date indicates the build date.
// This has been removed, it looks like you can only replace static strings with recent go versions
//Date = time.Now().Format("20060102")
Date = "dev"
Date = Dev
// Legacy defines the old long 4 number OpenCloud version needed for some clients
Legacy = "0.1.0.0"
// LegacyString defines the old OpenCloud version needed for some clients
LegacyString = "0.1.0"
// Edition describes the build channel (stable, rolling, nightly, daily, dev)
Edition = Dev // default for self-compiled builds
)
func init() { //nolint:gochecknoinits
if err := initEdition(); err != nil {
logger.New().Error().Err(err).Msg("falling back to dev")
}
}
func initEdition() error {
regularEditions := []string{EditionDev, EditionRolling, EditionStable}
versionedEditions := []string{EditionLTS}
if !slices.ContainsFunc(slices.Concat(regularEditions, versionedEditions), func(s string) bool {
isRegularEdition := slices.Contains(regularEditions, Edition)
if isRegularEdition && s == Edition {
return true
}
// handle editions with a version
editionParts := strings.Split(Edition, "-")
if len(editionParts) != 2 { // a versioned edition channel must consist of exactly 2 parts.
return false
}
isVersionedEdition := slices.Contains(versionedEditions, editionParts[0])
if !isVersionedEdition { // not all channels can contain version information
return false
}
_, err := semver.NewVersion(editionParts[1])
return err == nil
}) {
Edition = Dev
return fmt.Errorf(`unknown edition channel "%s"`, Edition)
}
return nil
}
// Compiled returns the compile time of this service.
func Compiled() time.Time {
if Date == "dev" {
if Date == Dev {
return time.Now()
}
t, _ := time.Parse("20060102", Date)

View File

@@ -0,0 +1,65 @@
package version_test
import (
"fmt"
"testing"
"github.com/opencloud-eu/opencloud/pkg/version"
)
func TestChannel(t *testing.T) {
tests := map[string]struct {
got string
valid bool
}{
"no channel, defaults to dev": {
got: "",
valid: false,
},
"dev channel": {
got: version.EditionDev,
valid: true,
},
"rolling channel": {
got: version.EditionRolling,
valid: true,
},
"stable channel": {
got: version.EditionStable,
valid: true,
},
"lts channel without version": {
got: version.EditionLTS,
valid: false,
},
"lts-1.0.0 channel": {
got: fmt.Sprintf("%s-1", version.EditionLTS),
valid: true,
},
"lts-one invalid version": {
got: fmt.Sprintf("%s-one", version.EditionLTS),
valid: false,
},
"known channel with version": {
got: fmt.Sprintf("%s-1", version.EditionStable),
valid: false,
},
"unknown channel": {
got: "foo",
valid: false,
},
}
for name, test := range tests {
t.Run(name, func(t *testing.T) {
version.Edition = test.got
switch err := version.InitEdition(); {
case err != nil && !test.valid && version.Edition != version.Dev: // if a given edition is unknown, the value is always dev
fallthrough
case test.valid != (err == nil):
t.Fatalf("invalid edition: %s", version.Edition)
}
})
}
}

View File

@@ -33,7 +33,7 @@ type Config struct {
EnableFederatedSharingIncoming bool `yaml:"enable_federated_sharing_incoming" env:"OC_ENABLE_OCM;FRONTEND_ENABLE_FEDERATED_SHARING_INCOMING" desc:"Changing this value is NOT supported. Enables support for incoming federated sharing for clients. The backend behaviour is not changed." introductionVersion:"1.0.0"`
EnableFederatedSharingOutgoing bool `yaml:"enable_federated_sharing_outgoing" env:"OC_ENABLE_OCM;FRONTEND_ENABLE_FEDERATED_SHARING_OUTGOING" desc:"Changing this value is NOT supported. Enables support for outgoing federated sharing for clients. The backend behaviour is not changed." introductionVersion:"1.0.0"`
SearchMinLength int `yaml:"search_min_length" env:"FRONTEND_SEARCH_MIN_LENGTH" desc:"Minimum number of characters to enter before a client should start a search for Share receivers. This setting can be used to customize the user experience if e.g too many results are displayed." introductionVersion:"1.0.0"`
Edition string `yaml:"edition" env:"OC_EDITION;FRONTEND_EDITION" desc:"Edition of OpenCloud. Used for branding purposes." introductionVersion:"1.0.0"`
Edition string `desc:"Edition of OpenCloud. Used for branding purposes." introductionVersion:"1.0.0"`
DisableSSE bool `yaml:"disable_sse" env:"OC_DISABLE_SSE;FRONTEND_DISABLE_SSE" desc:"When set to true, clients are informed that the Server-Sent Events endpoint is not accessible." introductionVersion:"1.0.0"`
DisableRadicale bool `yaml:"disable_radicale" env:"FRONTEND_DISABLE_RADICALE" desc:"When set to true, clients are informed that the Radicale (CalDAV/CardDAV) is not accessible." introductionVersion:"4.0.0"`
DefaultLinkPermissions int `yaml:"default_link_permissions" env:"FRONTEND_DEFAULT_LINK_PERMISSIONS" desc:"Defines the default permissions a link is being created with. Possible values are 0 (= internal link, for instance members only) and 1 (= public link with viewer permissions). Defaults to 1." introductionVersion:"1.0.0"`

View File

@@ -5,6 +5,7 @@ import (
"github.com/opencloud-eu/opencloud/pkg/shared"
"github.com/opencloud-eu/opencloud/pkg/structs"
"github.com/opencloud-eu/opencloud/pkg/version"
"github.com/opencloud-eu/opencloud/services/frontend/pkg/config"
)
@@ -87,7 +88,7 @@ func DefaultConfig() *config.Config {
DefaultUploadProtocol: "tus",
DefaultLinkPermissions: 1,
SearchMinLength: 3,
Edition: "",
Edition: version.Edition,
CheckForUpdates: true,
Checksums: config.Checksums{
SupportedTypes: []string{"sha1", "md5", "adler32"},

View File

@@ -346,7 +346,7 @@ func FrontendConfigFromStruct(cfg *config.Config, logger log.Logger) (map[string
},
"version": map[string]interface{}{
"product": "OpenCloud",
"edition": "",
"edition": version.Edition,
"major": version.ParsedLegacy().Major(),
"minor": version.ParsedLegacy().Minor(),
"micro": version.ParsedLegacy().Patch(),

View File

@@ -80,5 +80,5 @@ type Status struct {
Product string
ProductName string
ProductVersion string
Edition string `yaml:"edition" env:"OC_EDITION;OCDAV_EDITION" desc:"Edition of OpenCloud. Used for branding purposes." introductionVersion:"1.0.0"`
Edition string `desc:"Edition of OpenCloud. Used for branding purposes." introductionVersion:"1.0.0"`
}

View File

@@ -92,7 +92,7 @@ func DefaultConfig() *config.Config {
ProductVersion: version.GetString(),
Product: "OpenCloud",
ProductName: "OpenCloud",
Edition: "",
Edition: version.Edition,
},
}
}

View File

@@ -136,6 +136,9 @@ func (p Web) getPayload() (payload []byte, err error) {
p.config.Web.Config.Apps = make([]string, 0)
}
// ensure that the server url has a trailing slash
p.config.Web.Config.Server = strings.TrimRight(p.config.Web.Config.Server, "/") + "/"
return json.Marshal(p.config.Web.Config)
}

View File

@@ -33,6 +33,7 @@ use SimpleXMLElement;
use Sabre\Xml\LibXMLException;
use Sabre\Xml\Reader;
use GuzzleHttp\Pool;
use Symfony\Component\HttpFoundation\Response;
/**
* Helper for HTTP requests
@@ -74,7 +75,6 @@ class HttpRequestHelper {
* than download it all up-front.
* @param int|null $timeout
* @param Client|null $client
* @param string|null $bearerToken
*
* @return ResponseInterface
* @throws GuzzleException
@@ -92,8 +92,42 @@ class HttpRequestHelper {
bool $stream = false,
?int $timeout = 0,
?Client $client = null,
?string $bearerToken = null
): ResponseInterface {
$bearerToken = null;
if (TokenHelper::useBearerToken() && $user && $user !== 'public') {
$bearerToken = TokenHelper::getTokens($user, $password, $url)['access_token'];
// check token is still valid
$parsedUrl = parse_url($url);
$baseUrl = $parsedUrl['scheme'] . '://' . $parsedUrl['host'];
$baseUrl .= isset($parsedUrl['port']) ? ':' . $parsedUrl['port'] : '';
$testUrl = $baseUrl . "/graph/v1.0/use/$user";
if (OcHelper::isTestingOnReva()) {
$url = $baseUrl . "/ocs/v2.php/cloud/users/$user";
}
// check token validity with a GET request
$c = self::createClient(
$user,
$password,
$config,
$cookies,
$stream,
$timeout,
$bearerToken
);
$testReq = self::createRequest($testUrl, $xRequestId, 'GET');
try {
$testRes = $c->send($testReq);
} catch (RequestException $ex) {
$testRes = $ex->getResponse();
if ($testRes && $testRes->getStatusCode() === Response::HTTP_UNAUTHORIZED) {
// token is invalid or expired, get a new one
echo "[INFO] Bearer token expired or invalid, getting a new one...\n";
TokenHelper::clearAllTokens();
$bearerToken = TokenHelper::getTokens($user, $password, $url)['access_token'];
}
}
}
if ($client === null) {
$client = self::createClient(
$user,
@@ -160,6 +194,24 @@ class HttpRequestHelper {
}
HttpLogger::logResponse($response);
// wait for post-processing to finish if applicable
if (WebdavHelper::isDAVRequest($url)
&& \str_starts_with($url, OcHelper::getServerUrl())
&& \in_array($method, ["PUT", "MOVE", "COPY"])
&& \in_array($response->getStatusCode(), [Response::HTTP_CREATED, Response::HTTP_NO_CONTENT])
&& OcConfigHelper::getPostProcessingDelay() === 0
) {
if (\in_array($method, ["MOVE", "COPY"])) {
$url = $headers['Destination'];
}
WebDavHelper::waitForPostProcessingToFinish(
$url,
$user,
$password,
$headers,
);
}
return $response;
}
@@ -203,13 +255,6 @@ class HttpRequestHelper {
} else {
$debugResponses = false;
}
// use basic auth for 'public' user or no user
if ($user === 'public' || $user === null || $user === '') {
$bearerToken = null;
} else {
$useBearerToken = TokenHelper::useBearerToken();
$bearerToken = $useBearerToken ? TokenHelper::getTokens($user, $password, $url)['access_token'] : null;
}
$sendRetryLimit = self::numRetriesOnHttpTooEarly();
$sendCount = 0;
@@ -228,7 +273,6 @@ class HttpRequestHelper {
$stream,
$timeout,
$client,
$bearerToken,
);
if ($response->getStatusCode() >= 400
@@ -256,7 +300,8 @@ class HttpRequestHelper {
// we need to repeat the send request, because we got HTTP_TOO_EARLY or HTTP_CONFLICT
// wait 1 second before sending again, to give the server some time
// to finish whatever post-processing it might be doing.
self::debugResponse($response);
echo "[INFO] Received '" . $response->getStatusCode() .
"' status code, retrying request ($sendCount)...\n";
\sleep(1);
}
} while ($loopAgain);

View File

@@ -30,6 +30,26 @@ use Psr\Http\Message\ResponseInterface;
* A helper class for configuring OpenCloud server
*/
class OcConfigHelper {
public static $postProcessingDelay = 0;
/**
* @return int
*/
public static function getPostProcessingDelay(): int {
return self::$postProcessingDelay;
}
/**
* @param string $postProcessingDelay
*
* @return void
*/
public static function setPostProcessingDelay(string $postProcessingDelay): void {
// extract number from string
$delay = (int) filter_var($postProcessingDelay, FILTER_SANITIZE_NUMBER_INT);
self::$postProcessingDelay = $delay;
}
/**
* @param string $url
* @param string $method

View File

@@ -84,7 +84,9 @@ class TokenHelper {
$tokenData = [
'access_token' => $refreshedToken['access_token'],
'refresh_token' => $refreshedToken['refresh_token'],
'expires_at' => time() + 300 // 5 minutes
// set expiry to 240 (4 minutes) seconds to allow for some buffer
// token actually expires in 300 seconds (5 minutes)
'expires_at' => time() + 240
];
self::$tokenCache[$cacheKey] = $tokenData;
return $tokenData;
@@ -100,7 +102,9 @@ class TokenHelper {
$tokenData = [
'access_token' => $tokens['access_token'],
'refresh_token' => $tokens['refresh_token'],
'expires_at' => time() + 290 // set expiry to 290 seconds to allow for some buffer
// set expiry to 240 (4 minutes) seconds to allow for some buffer
// token actually expires in 300 seconds (5 minutes)
'expires_at' => time() + 240
];
// Save to cache

View File

@@ -923,4 +923,45 @@ class WebDavHelper {
$mtime = new DateTime($xmlPart[0]->__toString());
return $mtime->format('U');
}
/**
* wait until the reqeust doesn't return 425 anymore
*
* @param string $url
* @param ?string $user
* @param ?string $password
* @param ?array $headers
*
* @return void
*/
public static function waitForPostProcessingToFinish(
string $url,
?string $user = null,
?string $password = null,
?array $headers = [],
): void {
$retried = 0;
do {
$response = HttpRequestHelper::sendRequest(
$url,
'check-425-status',
'GET',
$user,
$password,
$headers,
);
$statusCode = $response->getStatusCode();
if ($statusCode !== 425) {
return;
}
$tryAgain = $retried < HttpRequestHelper::numRetriesOnHttpTooEarly();
if ($tryAgain) {
$retried += 1;
echo "[INFO] Waiting for post processing to finish, attempt ($retried)...\n";
// wait 1s and try again
\sleep(1);
}
} while ($tryAgain);
echo "[ERROR] 10 seconds timeout! Post processing did not finish in time.\n";
}
}

View File

@@ -904,7 +904,7 @@ class CliContext implements Context {
$userUuid = $this->featureContext->getAttributeOfCreatedUser($user, 'id');
$storagePath = $this->getUsersStoragePath();
$body = [
"command" => "xattr -p -slz " . escapeshellarg($attribute) . " $storagePath/$userUuid/$file",
"command" => "getfattr -n " . escapeshellarg($attribute) . " --only-values $storagePath/$userUuid/$file",
"raw" => true
];
$this->featureContext->setResponse(CliHelper::runCommand($body));

View File

@@ -2026,8 +2026,12 @@ class FeatureContext extends BehatVariablesContext {
if ($response === null) {
$response = $this->getResponse();
}
$body = (string)$response->getBody();
if (!$body) {
return [];
}
return \json_decode(
(string)$response->getBody(),
$body,
true
);
}

View File

@@ -68,6 +68,7 @@ class OcConfigContext implements Context {
$response->getStatusCode(),
"Failed to set async upload with delayed post processing"
);
OcConfigHelper::setPostProcessingDelay($delayTime);
}
/**
@@ -90,6 +91,9 @@ class OcConfigContext implements Context {
$response->getStatusCode(),
"Failed to set config $configVariable=$configValue"
);
if ($configVariable === "POSTPROCESSING_DELAY") {
OcConfigHelper::setPostProcessingDelay($configValue);
}
}
/**
@@ -184,6 +188,9 @@ class OcConfigContext implements Context {
$envs = [];
foreach ($table->getHash() as $row) {
$envs[$row['config']] = $row['value'];
if ($row['config'] === "POSTPROCESSING_DELAY") {
OcConfigHelper::setPostProcessingDelay($row['value']);
}
}
$response = OcConfigHelper::reConfigureOc($envs);
@@ -200,6 +207,7 @@ class OcConfigContext implements Context {
* @return void
*/
public function rollbackOc(): void {
OcConfigHelper::setPostProcessingDelay('0');
$response = OcConfigHelper::rollbackOc();
Assert::assertEquals(
200,

View File

@@ -607,7 +607,7 @@ trait Provisioning {
Assert::assertEquals(
201,
$response->getStatusCode(),
__METHOD__ . " cannot create user '$userName' using Graph API.\nResponse:" .
__METHOD__ . " cannot create user '$userName'.\nResponse:" .
json_encode($this->getJsonDecodedResponse($response))
);
@@ -1083,7 +1083,7 @@ trait Provisioning {
Assert::assertEquals(
201,
$response->getStatusCode(),
__METHOD__ . " cannot create user '$user' using Graph API.\nResponse:" .
__METHOD__ . " cannot create user '$user'.\nResponse:" .
json_encode($this->getJsonDecodedResponse($response))
);
$userId = $this->getJsonDecodedResponse($response)['id'];

View File

@@ -750,6 +750,9 @@ class SpacesContext implements Context {
} else {
$rawBody = $this->featureContext->getResponse()->getBody()->getContents();
}
if (!$rawBody) {
throw new Exception(__METHOD__ . " - Response body is empty");
}
$drives = json_decode($rawBody, true, 512, JSON_THROW_ON_ERROR);
if (isset($drives["value"])) {
$drives = $drives["value"];

View File

@@ -216,6 +216,44 @@ class TUSContext implements Context {
);
}
/**
* @When user :user sends a chunk to the last created TUS Location with offset :offset and data :data with retry on offset mismatch using the WebDAV API
*
* @param string $user
* @param string $offset
* @param string $data
*
* @return void
*
* @throws GuzzleException
* @throws JsonException
*/
public function userSendsAChunkToTUSLocationWithOffsetAndDataWithRetryOnOffsetMismatch(
string $user,
string $offset,
string $data,
): void {
$resourceLocation = $this->getLastTusResourceLocation();
$retried = 0;
do {
$tryAgain = false;
$response = $this->uploadChunkToTUSLocation($user, $resourceLocation, $offset, $data);
// retry on 409 Conflict (Offset mismatch during TUS upload)
if ($response->getStatusCode() === 409) {
$tryAgain = true;
}
$tryAgain = $tryAgain && $retried < HttpRequestHelper::numRetriesOnHttpTooEarly();
if ($tryAgain) {
$retried += 1;
echo "Offset mismatch during TUS upload, retrying ($retried)...\n";
// wait 1s and try again
\sleep(1);
}
} while ($tryAgain);
$this->featureContext->setResponse($response);
}
/**
* @When user :user sends a chunk to the last created TUS Location with offset :offset and data :data using the WebDAV API
*

View File

@@ -25,6 +25,7 @@ use GuzzleHttp\Exception\GuzzleException;
use PHPUnit\Framework\Assert;
use Psr\Http\Message\ResponseInterface;
use GuzzleHttp\Stream\StreamInterface;
use TestHelpers\OcConfigHelper;
use TestHelpers\OcHelper;
use TestHelpers\UploadHelper;
use TestHelpers\WebDavHelper;
@@ -743,6 +744,7 @@ trait WebDav {
/**
* @When the user waits for :time seconds for postprocessing to finish
* @When the user waits for :time seconds
*
* @param int $time
*
@@ -973,6 +975,61 @@ trait WebDav {
$this->checkDownloadedContentMatches($content, '', $response);
}
/**
* check file content with retry
*
* @param string $user
* @param string $fileName
* @param string $content
*
* @return void
* @throws Exception
*/
public function checkFileContentWithRetry(string $user, string $fileName, string $content): void {
$retried = 0;
do {
$tryAgain = false;
$response = $this->downloadFileAsUserUsingPassword($this->getActualUsername($user), $fileName);
$status = $response->getStatusCode();
$downloadedContent = $response->getBody()->getContents();
if ($status !== 200) {
$tryAgain = true;
$message = "Expected '200' but got '$status'";
} elseif ($downloadedContent !== $content) {
$tryAgain = true;
$message = "Expected content '$content' but got '$downloadedContent'";
}
$tryAgain = $tryAgain && $retried < HttpRequestHelper::numRetriesOnHttpTooEarly();
if ($tryAgain) {
$retried += 1;
echo "[INFO] File content mismatch. $message, checking content again ($retried)...\n";
// break the loop if status is 425 as the request will already be retried
if ($status === HttpRequestHelper::HTTP_TOO_EARLY) {
break;
}
// wait 1s and try again
\sleep(1);
}
} while ($tryAgain);
$this->theHTTPStatusCodeShouldBe(200, '', $response);
$this->checkDownloadedContentMatches($content, '', $response);
}
/**
* @Then as :user the final content of file :fileName should be :content
*
* @param string $user
* @param string $fileName
* @param string $content
*
* @return void
*/
public function asUserFinalContentOfFileShouldBe(string $user, string $fileName, string $content): void {
$this->checkFileContentWithRetry($user, $fileName, $content);
}
/**
* @Then /^the content of the following files for user "([^"]*)" should be "([^"]*)"$/
*
@@ -1264,20 +1321,18 @@ trait WebDav {
$type
);
$statusCode = $response->getStatusCode();
if ($statusCode < 400 || $statusCode > 499) {
try {
$responseXmlObject = HttpRequestHelper::getResponseXml(
$response,
__METHOD__
);
} catch (Exception $e) {
Assert::fail(
"$entry '$path' should not exist. But API returned $statusCode without XML in the body"
);
}
// when checking path with '..' it may return 405 Method Not Allowed
if ($statusCode === 404 || $statusCode === 405) {
return;
}
if ($statusCode === 207) {
$responseXmlObject = HttpRequestHelper::getResponseXml(
$response,
__METHOD__
);
Assert::assertTrue(
$this->isEtagValid($this->getEtagFromResponseXmlObject($responseXmlObject)),
"$entry '$path' should not exist. But API returned $statusCode without an etag in the body"
"$entry '$path' should not exist but found with invalid etag."
);
$isCollection = $responseXmlObject->xpath("//d:prop/d:resourcetype/d:collection");
if (\count($isCollection) === 0) {
@@ -1291,7 +1346,11 @@ trait WebDav {
"$entry '$path' should not exist. But it does."
);
}
return;
}
Assert::fail(
"$entry '$path' should not exist. But API returned $statusCode without XML in the body"
);
}
/**
@@ -2270,6 +2329,11 @@ trait WebDav {
"HTTP status code was not 201 or 204 while trying to upload file '$destination' for user '$user'",
$response
);
// check uploaded content only if post-processing delay is not configured
if (OcConfigHelper::getPostProcessingDelay() === 0) {
$this->checkFileContentWithRetry($user, $destination, $content);
}
return $response->getHeader('oc-fileid');
}

View File

@@ -29,7 +29,7 @@ Feature: create a resources using collaborative posixfs
Scenario: create file
When the administrator creates the file "test.txt" with content "content" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/test.txt" for user "Alice" should be "content"
And as "Alice" the final content of file "test.txt" should be "content"
Scenario: create large file
@@ -41,21 +41,22 @@ Feature: create a resources using collaborative posixfs
Scenario: creates files sequentially in a folder
When the administrator creates 50 files sequentially in the directory "firstFolder" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/firstFolder/file_1.txt" for user "Alice" should be "file 1 content"
And the content of file "/firstFolder/file_50.txt" for user "Alice" should be "file 50 content"
And as "Alice" the final content of file "/firstFolder/file_1.txt" should be "file 1 content"
And as "Alice" the final content of file "/firstFolder/file_50.txt" should be "file 50 content"
Scenario: creates files in parallel in a folder
When the administrator creates 100 files in parallel in the directory "firstFolder" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/firstFolder/parallel_1.txt" for user "Alice" should be "parallel file 1 content"
And the content of file "/firstFolder/parallel_100.txt" for user "Alice" should be "parallel file 100 content"
And as "Alice" the final content of file "/firstFolder/parallel_1.txt" should be "parallel file 1 content"
And as "Alice" the final content of file "/firstFolder/parallel_100.txt" should be "parallel file 100 content"
Scenario: edit file
Given user "Alice" has uploaded file with content "content" to "test.txt"
When the administrator puts the content "new" into the file "test.txt" in the POSIX storage folder of user "Alice"
Then the content of file "/test.txt" for user "Alice" should be "contentnew"
Then the command should be successful
And as "Alice" the final content of file "test.txt" should be "contentnew"
Scenario: read file content
@@ -68,28 +69,28 @@ Feature: create a resources using collaborative posixfs
Given user "Alice" has uploaded file with content "content" to "test.txt"
When the administrator copies the file "test.txt" to the folder "firstFolder" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/firstFolder/test.txt" for user "Alice" should be "content"
And as "Alice" the final content of file "/firstFolder/test.txt" should be "content"
Scenario: rename file
Given user "Alice" has uploaded file with content "content" to "test.txt"
When the administrator renames the file "test.txt" to "new-name.txt" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/new-name.txt" for user "Alice" should be "content"
And as "Alice" the final content of file "/new-name.txt" should be "content"
Scenario: rename a created file
Given the administrator has created the file "test.txt" with content "content" for user "Alice" on the POSIX filesystem
When the administrator renames the file "test.txt" to "test.md" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/test.md" for user "Alice" should be "content"
And as "Alice" the final content of file "/test.md" should be "content"
Scenario: move file to folder
Given user "Alice" has uploaded file with content "content" to "test.txt"
When the administrator moves the file "test.txt" to the folder "firstFolder" for user "Alice" on the POSIX filesystem
Then the command should be successful
And the content of file "/firstFolder/test.txt" for user "Alice" should be "content"
And as "Alice" the final content of file "/firstFolder/test.txt" should be "content"
And as "Alice" file "/test.txt" should not exist
@@ -187,4 +188,4 @@ Feature: create a resources using collaborative posixfs
And the administrator renames the file "test.txt" to "renamed.txt" for user "Alice" on the POSIX filesystem
And the administrator checks the attribute "user.oc.name" of file "renamed.txt" for user "Alice" on the POSIX filesystem
Then the command output should contain "renamed.txt"
And the content of file "/renamed.txt" for user "Alice" should be "content"
And as "Alice" the final content of file "/renamed.txt" should be "content"

View File

@@ -202,7 +202,7 @@ Feature: capabilities
"properties": {
"edition": {
"type": "string",
"enum": ["%edition%"]
"enum": ["dev"]
},
"product": {
"type": "string",
@@ -240,7 +240,7 @@ Feature: capabilities
},
"edition": {
"type": "string",
"enum": ["%edition%"]
"enum": ["dev"]
},
"product": {
"type": "string",

View File

@@ -58,7 +58,7 @@ Feature: default capabilities for normal user
"const": "%versionstring%"
},
"edition": {
"const": "%edition%"
"const": "dev"
},
"productname": {
"const": "%productname%"

View File

@@ -50,8 +50,7 @@ Feature: low level tests for upload of chunks
| Upload-Metadata | filename ZmlsZS50eHQ= |
When user "Alice" sends a chunk to the last created TUS Location with offset "0" and data "123" using the WebDAV API
And user "Alice" sends a chunk to the last created TUS Location with offset "3" and data "4567890" using the WebDAV API
And the user waits for "2" seconds for postprocessing to finish
And user "Alice" sends a chunk to the last created TUS Location with offset "3" and data "0000000" using the WebDAV API
And user "Alice" sends a chunk to the last created TUS Location with offset "3" and data "0000000" with retry on offset mismatch using the WebDAV API
Then the HTTP status code should be "404"
And the content of file "/file.txt" for user "Alice" should be "1234567890"
Examples:
@@ -61,6 +60,22 @@ Feature: low level tests for upload of chunks
| spaces |
Scenario Outline: send last chunk with mismatch offset
Given using <dav-path-version> DAV path
And user "Alice" has created a new TUS resource on the WebDAV API with these headers:
| Upload-Length | 10 |
# ZmlsZS50eHQ= is the base64 encode of file.txt
| Upload-Metadata | filename ZmlsZS50eHQ= |
When user "Alice" sends a chunk to the last created TUS Location with offset "0" and data "123" using the WebDAV API
And user "Alice" sends a chunk to the last created TUS Location with offset "2" and data "34567890" using the WebDAV API
Then the HTTP status code should be "409"
Examples:
| dav-path-version |
| old |
| new |
| spaces |
Scenario Outline: start with uploading not at the beginning of the file
Given using <dav-path-version> DAV path
And user "Alice" has created a new TUS resource on the WebDAV API with these headers: