* add support for specifying a basepath using -p when running the
posixfs scan command, to indicate a directory under which to start
scanning, or a singular file to scan, as opposed to scanning from the
storage root directory as is the default behaviour
* implements https://github.com/opencloud-eu/opencloud/issues/3182
The command no longer times out after 10 minutes when the processing
hasn't been finished yet. Instead it shows progress and logs successful
and failed spaces. The process can also be aborted using ctrl+c.
Example output:
```
$ bin/opencloud search index --all-spaces --insecure --force-rescan
[1/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$eb6a8a0e-2e62-4b40-bce9-360063236676!eb6a8a0e-2e62-4b40-bce9-360063236676 in 5.501190908s
[2/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$34b7c565-da3b-4f53-8559-825e87db4011!34b7c565-da3b-4f53-8559-825e87db4011 in 5.935075697s
[3/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$e08c6026-24f6-4a9e-a805-ed335b34b8da!e08c6026-24f6-4a9e-a805-ed335b34b8da in 6.016350958s
^Caborted, indexing has been stopped
```
Fixes#2592
This speeds up reindexing all spaces which happend sequentially space
after space until now. The level of concurrency can be configured using
the SEARCH_REINDEX_CONCURRENCY env var (3 by default).
The announcement is runtime-managed and injected into config.json; it is not a
static yaml config option. Exclude it from yaml (yaml:"-") so it is not
presented as a configurable knob, keeping the json tag only for the config.json
output.
Replace reva's cache.NewNatsKeyValue (which retries with backoff and can hang
the web service startup) with a direct nats.Connect + jetstream setup like the
graph service, so an unreachable or misconfigured store surfaces as a clear
startup error instead of a hang.
A statically configured announcement is not supported: it cannot be removed via
the runtime API and disappears from the running session once the admin settings
load (they read the store, not config.json). Make the store the single source
of truth, config.json always reflects it and a statically configured
options.announcement is ignored.
Replace the go-micro store with a NATS JetStream key-value bucket, created via
reva's cache.NewNatsKeyValue, so the web service no longer depends on go-micro
stores. The store methods now take a context, threaded through the config.json
render path and the management handlers. Tests use a mockery mock of
jetstream.KeyValue.
Add a logger to the announcement service and log server-side errors before
returning a 5xx, tagged with the request id so they correlate with the
request log line from the logging middleware.
Distinguish the http.MaxBytesReader limit from a malformed body: an
oversized payload now returns 413 Request Entity Too Large instead of a
generic 400.
The permission gates reading the full announcement state (including
disabled ones) as well as writing it, so name it ReadWrite to match the
other management permissions (Accounts.ReadWrite, Settings.ReadWrite, ...)
and use the READWRITE operation.
Make the store authoritative only once it manages an announcement: an
enabled record is exposed, a disabled record clears any static config, and
an empty store falls back to a statically configured
web.config.options.announcement. This matches the review request and stops
a static announcement from being silently dropped.
Always reflect the stored announcement in config.json: expose it while live
and clear it otherwise. A disabled announcement now clears the config output
instead of falling back to a statically configured one, which removes the
precedence ambiguity raised in review.
When walking the tree for creating the search index, ignore "not found"
errors. Search indexing happens asynchronously so the underlying space,
file or folder might just not exist anymore when the indexing starts.
An archive download of a single folder was always named "download.zip" until the
handler resolved the resource name for the Content-Disposition header
(https://github.com/opencloud-eu/reva/pull/661). Add a scenario asserting the
archive of a folder carries a Content-Disposition filename of the folder,
reusing the archiver download step and the header-regex assertion.
Verified red->green on posix and decomposed: red on an opencloud built on a reva
with the change reverted (the archive is "download.zip"), green on the shipped
reva.
* test(coreApiWebdavUploadTUS): assert etag and permissions on the finalizing TUS chunk
The chunked TUS finalize returns OC-ETag, ETag and OC-Perm since
https://github.com/opencloud-eu/reva/pull/718, so a client no longer
needs a follow-up PROPFIND for the new etag and permissions. Add a
scenario asserting these headers on the finalizing chunk, reusing
existing step definitions (the etag is a dynamic quoted hash, hence the
header-regex assertion).
Verified locally red->green: red on stock (headers absent), green on an
opencloud built on a reva carrying that change, on posix and decomposed.
https://github.com/opencloud-eu/opencloud/issues/2409
* test(coreApiWebdavUploadTUS): assert exact OC-Perm and tighten etag regex
---------
Co-authored-by: v.scharf <v.scharf@opencloud.eu>
A file whose name contains a literal "%" 404s on download because the
server-built oc:downloadURL did not percent-encode the path (#2852, fixed in
reva by https://github.com/opencloud-eu/reva/pull/653). Add a scenario mirroring
the web client download flow: read the oc:downloadURL from PROPFIND and GET it
as-is (the signature in the URL authenticates it), asserting 200 and the content.
The new step "user X downloads the P file via its oc:downloadURL" lives in
WebDavPropertiesContext; a regex annotation avoids the ":" in oc:downloadURL
colliding with the turnip :placeholder syntax.
Verified red->green on posix and decomposed: red on an opencloud built on a reva
with the fix reverted (the download 404s), green on the shipped reva.
https://github.com/opencloud-eu/opencloud/issues/2852
dependabot[bot]