Compare commits

...
56 Commits
Author SHA1 Message Date
OpenCloud Devopsandv.scharf d695d283f6 🎉 Release 7.4.0 (#3143)
* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.3.1

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

* 🎉 Release 7.4.0

---------

Co-authored-by: v.scharf <v.scharf@opencloud.eu>
2026-08-03 20:42:51 +02:00
Viktor Scharf 553c85b7ff set introduction version 7.4.0 (#3234) 2026-08-03 20:07:23 +02:00
Viktor Scharf 19e44f9b64 [full-ci] chore: bump web to v7.3.0 (#3223)
* chore: bump web to v7.3.0

* Update WEB_COMMITID in .woodpecker.env
2026-08-03 19:58:05 +02:00
Andre Duffeck c93f50dc37 Merge pull request #3233 from aduffeck/extend-posixfs-consistency-check
Do not check ignored paths
2026-08-03 17:50:35 +02:00
Viktor Scharf bb4cd1ebf5 chore: reva bump -2.48.0 (#3230) 2026-08-03 17:43:51 +02:00
dependabot[bot] e12966a4fd build(deps): bump github.com/open-policy-agent/opa from 1.18.2 to 1.19.0
Bumps [github.com/open-policy-agent/opa](https://github.com/open-policy-agent/opa) from 1.18.2 to 1.19.0.
- [Release notes](https://github.com/open-policy-agent/opa/releases)
- [Changelog](https://github.com/open-policy-agent/opa/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-policy-agent/opa/compare/v1.18.2...v1.19.0)

---
updated-dependencies:
- dependency-name: github.com/open-policy-agent/opa
  dependency-version: 1.19.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 17:00:51 +02:00
André Duffeck b350b355c0 Do not check ignored paths 2026-08-03 16:29:30 +02:00
dependabot[bot] 9df3881128 build(deps): bump google.golang.org/grpc from 1.82.0 to 1.83.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.82.0 to 1.83.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.82.0...v1.83.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.83.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 16:15:46 +02:00
Andre Duffeck 557adfc327 Merge pull request #3220 from aduffeck/extend-posixfs-consistency-check
Extend posixfs consistency check
2026-08-03 14:04:10 +02:00
André Duffeck 97ad525d5b Infer the indexes path from the config instead of calculating it 2026-08-03 13:24:23 +02:00
Ralf Haferkamp 0fd02c5ecf chore(idp): bump dependencies 2026-08-03 13:20:03 +02:00
Viktor Scharf 27a77567dd chore(skills): add skill for bumping reva (#3224)
* chore(skills): add skill for bumping reva

* chore(skills): derive OpenCloud version from open release PR

* chore: exclude .agents from codacy analysis
2026-08-03 13:18:43 +02:00
dependabot[bot] 57a615d6ce build(deps): bump github.com/nats-io/nats-server/v2
Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.14.3 to 2.14.4.
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/RELEASES.md)
- [Commits](https://github.com/nats-io/nats-server/compare/v2.14.3...v2.14.4)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-version: 2.14.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 13:17:14 +02:00
André Duffeck deb3247b99 Move consistency check logic into a checker struct
This makes it possible to get rid of the globals without passing
multiple state vars around.
2026-08-03 12:17:19 +02:00
dependabot[bot] b9adb507f8 build(deps): bump github.com/go-ldap/ldap/v3 from 3.4.13 to 3.4.14
Bumps [github.com/go-ldap/ldap/v3](https://github.com/go-ldap/ldap) from 3.4.13 to 3.4.14.
- [Release notes](https://github.com/go-ldap/ldap/releases)
- [Commits](https://github.com/go-ldap/ldap/compare/v3.4.13...v3.4.14)

---
updated-dependencies:
- dependency-name: github.com/go-ldap/ldap/v3
  dependency-version: 3.4.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 12:16:16 +02:00
André Duffeck c6fba8543f Improve wording 2026-08-03 11:55:14 +02:00
dependabot[bot] 7c2cb14a80 build(deps): bump github.com/prometheus/client_golang
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.23.2 to 1.24.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/v1.24.1/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.23.2...v1.24.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-version: 1.24.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 11:04:15 +02:00
André Duffeck b125f0681f Fix typos 2026-08-03 11:03:45 +02:00
André Duffeck 20d16c5275 Allow for providing multiple paths to check 2026-08-03 10:58:53 +02:00
dependabot[bot] 639e3657cd build(deps): bump github.com/gabriel-vasile/mimetype
Bumps [github.com/gabriel-vasile/mimetype](https://github.com/gabriel-vasile/mimetype) from 1.4.13 to 1.4.15.
- [Release notes](https://github.com/gabriel-vasile/mimetype/releases)
- [Commits](https://github.com/gabriel-vasile/mimetype/compare/v1.4.13...v1.4.15)

---
updated-dependencies:
- dependency-name: github.com/gabriel-vasile/mimetype
  dependency-version: 1.4.15
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 10:44:05 +02:00
André Duffeck 559b10e90a Check if the given path is part of the storage 2026-08-03 09:49:10 +02:00
dependabot[bot] e63f1d542a build(deps): bump golang.org/x/net from 0.56.0 to 0.57.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.56.0 to 0.57.0.
- [Commits](https://github.com/golang/net/compare/v0.56.0...v0.57.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 09:26:21 +02:00
dependabot[bot] 51089765d0 build(deps): bump github.com/beevik/etree from 1.6.0 to 1.7.0
Bumps [github.com/beevik/etree](https://github.com/beevik/etree) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/beevik/etree/releases)
- [Changelog](https://github.com/beevik/etree/blob/main/RELEASE_NOTES.md)
- [Commits](https://github.com/beevik/etree/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/beevik/etree
  dependency-version: 1.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-03 09:25:47 +02:00
André Duffeck 54cced7c56 Streamline logging in the commands 2026-08-03 08:58:19 +02:00
André Duffeck 3d546693c6 Allow for checking specific spaces and files as well 2026-08-03 08:55:49 +02:00
André Duffeck 40ced9997e Also check blobsize and checksums (when --fix-checkums is set) 2026-08-03 08:55:10 +02:00
André Duffeck 3bd188122e Simplify output, remove spinner lib 2026-08-03 08:55:10 +02:00
opencloudeu 9b5849f470 [tx] updated from transifex 2026-07-31 23:16:44 +00:00
Andre Duffeck 942cca8eef Merge pull request #3191 from pbleser-oc/feat/#3182-posixfs-scan-subtree
feat(posixfs): #3182 add basepath option in the "posixfs scan" command
2026-07-31 16:39:48 +02:00
Dominik Schmidt a9f39b85d5 Merge pull request #3140 from opencloud-eu/fix/service-startup-logging
fix(runtime): log service startup errors instead of printing them beside the log
2026-07-31 16:37:19 +02:00
Andre Duffeck 15fc44be94 Merge pull request #3213 from aduffeck/improve-reindex-command
Improve reindex command
2026-07-31 16:34:54 +02:00
André Duffeck 2eb1fe069c Add a --concurrency flag to the index command
It controls the number of spaces that are indexed in parallel. The max
concurrency configured in the server is still considered.
2026-07-31 16:07:18 +02:00
André Duffeck a82b6223aa Remove the SilenceUsage flag again
It will be obsolete once #3140 has been merged
2026-07-31 15:55:36 +02:00
Pascal Bleser 433dea41e8 feat(posixfs): #3182 add basepath option in the "posixfs scan" command
* add support for specifying a basepath using -p when running the
   posixfs scan command, to indicate a directory under which to start
   scanning, or a singular file to scan, as opposed to scanning from the
   storage root directory as is the default behaviour

 * implements https://github.com/opencloud-eu/opencloud/issues/3182
2026-07-31 15:06:50 +02:00
André Duffeck e26a009fc1 Improve "opencloud search index" command
The command no longer times out after 10 minutes when the processing
hasn't been finished yet. Instead it shows progress and logs successful
and failed spaces. The process can also be aborted using ctrl+c.

Example output:

```
$ bin/opencloud search index --all-spaces --insecure --force-rescan
[1/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$eb6a8a0e-2e62-4b40-bce9-360063236676!eb6a8a0e-2e62-4b40-bce9-360063236676 in 5.501190908s
[2/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$34b7c565-da3b-4f53-8559-825e87db4011!34b7c565-da3b-4f53-8559-825e87db4011 in 5.935075697s
[3/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$e08c6026-24f6-4a9e-a805-ed335b34b8da!e08c6026-24f6-4a9e-a805-ed335b34b8da in 6.016350958s
^Caborted, indexing has been stopped
```

Fixes #2592
2026-07-31 14:50:09 +02:00
André Duffeck 49a81c9c1f Cleanup .bingo 2026-07-31 14:50:09 +02:00
Andre Duffeck 7e8179a704 Merge pull request #3207 from aduffeck/concurrent-reindex
Reindex spaces concurrently
2026-07-31 14:49:58 +02:00
André Duffeck d50d8d8900 Rename max concurrency option 2026-07-31 12:04:44 +02:00
André Duffeck 243514ec17 Do not abort reindexing all spaces if one of the spaces fails 2026-07-30 15:34:57 +02:00
André Duffeck be75d8fe95 Reindex spaces concurrently
This speeds up reindexing all spaces which happend sequentially space
after space until now. The level of concurrency can be configured using
the SEARCH_REINDEX_CONCURRENCY env var (3 by default).
2026-07-30 15:25:13 +02:00
Dominik Schmidt bb8cefa33b Merge pull request #3189 from opencloud-eu/feat/announcement-banner
feat: add announcement banner
2026-07-30 10:44:17 +02:00
Viktor Scharf 0a44bbae9e api-test: mark group last-manager removal scenario as flaky (#3194) 2026-07-30 09:50:05 +05:45
opencloudeu fb387a16b6 [tx] updated from transifex 2026-07-29 23:16:50 +00:00
Ralf Haferkamp e358cb6e6e fix(posixfs scan): Setup logger for scan command
Setup a logger for the FS. The nats kv clients for the caches need one
to be able to log NAT connection issues.

Fixes: #3184
2026-07-29 14:29:19 +02:00
Dominik Schmidt 341ebdf6cc refactor: mark the announcement config as output-only (yaml:"-")
The announcement is runtime-managed and injected into config.json; it is not a
static yaml config option. Exclude it from yaml (yaml:"-") so it is not
presented as a configurable knob, keeping the json tag only for the config.json
output.
2026-07-29 12:03:57 +02:00
Dominik Schmidt 6d2d6c38ec fix: fail fast when the announcement store is unreachable
Replace reva's cache.NewNatsKeyValue (which retries with backoff and can hang
the web service startup) with a direct nats.Connect + jetstream setup like the
graph service, so an unreachable or misconfigured store surfaces as a clear
startup error instead of a hang.
2026-07-29 11:53:58 +02:00
Dominik Schmidt d27b7ef40a refactor: drop static config for the announcement, runtime store only
A statically configured announcement is not supported: it cannot be removed via
the runtime API and disappears from the running session once the admin settings
load (they read the store, not config.json). Make the store the single source
of truth, config.json always reflects it and a statically configured
options.announcement is ignored.
2026-07-29 11:52:09 +02:00
Dominik Schmidt 27bdda265f refactor: back the announcement store with plain NATS JetStream
Replace the go-micro store with a NATS JetStream key-value bucket, created via
reva's cache.NewNatsKeyValue, so the web service no longer depends on go-micro
stores. The store methods now take a context, threaded through the config.json
render path and the management handlers. Tests use a mockery mock of
jetstream.KeyValue.
2026-07-29 10:22:43 +02:00
Dominik Schmidt 7d59a83c18 feat: log announcement handler errors with the request id
Add a logger to the announcement service and log server-side errors before
returning a 5xx, tagged with the request id so they correlate with the
request log line from the logging middleware.
2026-07-29 10:08:47 +02:00
Dominik Schmidt 9d472290f2 fix: return 413 when the announcement body exceeds the size cap
Distinguish the http.MaxBytesReader limit from a malformed body: an
oversized payload now returns 413 Request Entity Too Large instead of a
generic 400.
2026-07-29 08:46:46 +02:00
Dominik Schmidt e29d961be3 refactor: rename Announcement.Write to Announcement.ReadWrite
The permission gates reading the full announcement state (including
disabled ones) as well as writing it, so name it ReadWrite to match the
other management permissions (Accounts.ReadWrite, Settings.ReadWrite, ...)
and use the READWRITE operation.
2026-07-29 08:40:29 +02:00
Dominik Schmidt 809d32c4d8 fix: keep a static announcement as fallback when the store is empty
Make the store authoritative only once it manages an announcement: an
enabled record is exposed, a disabled record clears any static config, and
an empty store falls back to a statically configured
web.config.options.announcement. This matches the review request and stops
a static announcement from being silently dropped.
2026-07-29 08:29:58 +02:00
Dominik Schmidt 046f46ce8d fix: make the runtime store authoritative for the announcement
Always reflect the stored announcement in config.json: expose it while live
and clear it otherwise. A disabled announcement now clears the config output
instead of falling back to a statically configured one, which removes the
precedence ambiguity raised in review.
2026-07-29 08:18:56 +02:00
Dominik Schmidt 51dd107767 feat: add announcement banner 2026-07-29 00:39:59 +02:00
Dominik Schmidt fec92d2f22 fix(runtime): keep the usage block for flag errors only 2026-07-15 19:01:03 +02:00
Dominik Schmidt 00dd4f1b7f fix(runtime): log service startup errors instead of printing them beside the log 2026-07-15 18:32:04 +02:00
362 changed files with 40975 additions and 12780 deletions

No files matched your search

+153
View File
@@ -0,0 +1,153 @@
---
name: bumping-reva
description: Use when the user asks to bump, update, or upgrade the OpenCloud reva dependency to a specific version (e.g. "reva bump to 2.48.0", "bump reva to v2.48.0"). Covers editing go.mod, re-vendoring, bumping the OpenCloud LatestTag, the single commit, and opening the PR against main.
---
# Bumping Reva
## Overview
Bumping reva updates the pinned [opencloud-eu/reva](https://github.com/opencloud-eu/reva) dependency (`github.com/opencloud-eu/reva/v2`) to a tagged release, re-vendors the module graph, and bumps the OpenCloud dev version. It touches `go.mod`, `go.sum`, `vendor/**` and `pkg/version/version.go`, lands as one commit, and ships as a PR whose body is the reva changelog for that version.
Template PR: https://github.com/opencloud-eu/opencloud/pull/3127
## Prerequisites
- **The reva release must already be tagged.** A reva release is cut by merging its release PR (title `🎉 Release X.Y.Z`, branch `next-release/main`). The **user merges that PR themselves** — this skill starts *after* the tag `vX.Y.Z` exists. Verify the tag before doing anything (step 1); if it's missing, stop and ask the user to merge the reva release PR first.
- **`gh` (GitHub CLI), authenticated** — tag lookup and PR creation go through `gh api` / `gh pr`. Verify with `gh auth status`; if it fails, ask the user to run `gh auth login` (suggest `! gh auth login` so it runs in-session).
- `gh` needs the system keyring, so run all `gh` commands with the sandbox disabled.
- **Go toolchain + network** — `go get` / `go mod tidy` / `go mod vendor` hit the Go module proxy. Run them with the sandbox disabled (network access).
- `git` and `base64` (decoding the changelog) — standard on macOS/Linux.
## Inputs
- `REVA_VERSION` — the new reva tag, always normalized to a leading `v` (e.g. `v2.48.0`). If the user didn't give it, ask (or take the latest reva release tag).
- `OC_VERSION` — the OpenCloud target for `LatestTag`, **without** the `+dev` suffix (e.g. `7.4.0`). **Do not ask or guess** — derive it from the open OpenCloud release PR (step 3). It can be a new major (e.g. `8.0.0`) when release-please picked up a breaking change.
## What changes
| File | What |
| ------------------------ | ------------------------------------------------------------------- |
| `go.mod` | `github.com/opencloud-eu/reva/v2``REVA_VERSION` (+ indirect deps pulled by `go mod tidy`) |
| `go.sum` | updated by `go get` / `go mod tidy` |
| `vendor/**` | re-vendored by `go mod vendor` (incl. `vendor/modules.txt`) |
| `pkg/version/version.go` | `LatestTag = "<OC_VERSION>+dev"` |
## Procedure
### 1. Verify the reva tag exists
```bash
gh api repos/opencloud-eu/reva/commits/$REVA_VERSION --jq '.sha'
```
- Success (a sha) → the release is tagged, continue.
- `422`/`404` → the tag does not exist yet. The reva release PR (`🎉 Release X.Y.Z`) is probably not merged. **Stop** and tell the user to merge it first. Helpful checks:
```bash
gh api repos/opencloud-eu/reva/releases/latest --jq '.tag_name' # current latest tag
gh pr list --repo opencloud-eu/reva --search '🎉 Release in:title' --state open --json number,title
```
### 2. Bump the dependency and re-vendor
```bash
go get github.com/opencloud-eu/reva/v2@$REVA_VERSION
go mod tidy
go mod vendor
```
(Sandbox disabled — these need network.) Notes:
- Right after a fresh tag the module proxy can lag; if `go get` reports the version as unknown, retry, or use `GOPROXY=direct go get github.com/opencloud-eu/reva/v2@$REVA_VERSION`.
- `go mod tidy` will also bump indirect dependencies that reva pulled in — that is expected (the template PR did the same).
### 3. Bump the OpenCloud dev version
`OC_VERSION` is the version of the **open OpenCloud release PR** — the release-please PR from branch `next-release/main`, titled `🎉 Release X.Y.Z` (e.g. #3143). Derive it, don't ask:
```bash
gh pr list --repo opencloud-eu/opencloud --head next-release/main --state open \
--json title --jq '.[0].title' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1
```
This is the next release target and tracks breaking changes — it may be a new major (e.g. `8.0.0`), not just a minor bump. If no such PR is open, stop and ask the user.
Edit `pkg/version/version.go`:
```go
LatestTag = "<OC_VERSION>+dev" // e.g. "7.4.0+dev"
```
### 4. Fetch the reva changelog for the PR body
```bash
gh api "repos/opencloud-eu/reva/contents/CHANGELOG.md?ref=$REVA_VERSION" --jq '.content' | base64 -d
```
Take **only the section for this version** and trim it exactly like the web bump: start at the first content heading (`### 🐛 Bug Fixes` / `### 📈 Enhancement` / `### 💥 Breaking changes`), drop the `# Changelog` title, the `## [x.y.z] - date` header and the `### ❤️ Thanks to all contributors!` block, and stop before the next `## [...]` version header.
Then **prepend two summary bullets** so the final PR body is:
```
- bump opencloud version to <OC_VERSION>
- reva bump <REVA_VERSION without the leading v>
### 🐛 Bug Fixes
...trimmed reva changelog...
```
Write this to a file for `--body-file` (e.g. in the scratchpad).
### 5. Confirm before committing
The `vendor/` diff is huge — do **not** dump it. Show the user:
```bash
git diff go.mod pkg/version/version.go # the meaningful edits
git diff --stat | tail -1 # vendor churn summary
```
Confirm the reva line in `go.mod` is exactly `github.com/opencloud-eu/reva/v2 REVA_VERSION`, show the target branch (`main`), and the PR body. **Do not commit until the user approves.**
### 6. Commit, push, open PR
- Create a branch (do not commit on `main`), e.g. `reva-bump-2.48.0`.
- Stage everything the bump touched: `git add go.mod go.sum vendor pkg/version/version.go`.
- One commit, conventional-commits format, **empty body**:
```
chore: reva bump -2.48.0
```
- PR base: **`main`** (reva bumps always target main).
- PR title: `[full-ci] chore: reva bump -2.48.0` (the commit message prefixed with `[full-ci] `).
- PR body: the file from step 4.
- Add the label `Type:Maintenance`.
```bash
gh pr create --base main \
--title "[full-ci] chore: reva bump -$REVA_VERSION_NO_V" \
--label "Type:Maintenance" \
--body-file <body-file>
```
(`gh` commands need the sandbox disabled — they require the system keyring.)
## Quick reference
```bash
REVA_VERSION=v2.48.0
gh api repos/opencloud-eu/reva/commits/$REVA_VERSION --jq '.sha' # verify tag exists
OC_VERSION=$(gh pr list --repo opencloud-eu/opencloud --head next-release/main --state open \
--json title --jq '.[0].title' | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1) # from open release PR
go get github.com/opencloud-eu/reva/v2@$REVA_VERSION && go mod tidy && go mod vendor # bump + re-vendor
# edit pkg/version/version.go -> LatestTag = "$OC_VERSION+dev"
gh api "repos/opencloud-eu/reva/contents/CHANGELOG.md?ref=$REVA_VERSION" --jq '.content' | base64 -d # changelog
```
## Common mistakes
- Running the bump before the reva release PR is merged — the tag won't exist and `go get` will fail. Verify the tag first (step 1).
- Forgetting to re-run `go mod vendor` after `go mod tidy`, leaving `vendor/` out of sync with `go.mod`.
- Asking for or hardcoding `OC_VERSION` — always derive it from the open `next-release/main` release PR (step 3); it can even be a new major after a breaking change.
- Reading the changelog from reva `main` instead of the tag (`?ref=$REVA_VERSION`). Always pin to the tag.
- Missing the `[full-ci] ` prefix in the PR title, the `Type:Maintenance` label, or the two summary bullets at the top of the body.
- Dumping the full `vendor/` diff at the confirmation step instead of `go.mod` + `version.go` + a `--stat` summary.
- Committing before the user confirms the diff.
+1
View File
@@ -1,6 +1,7 @@
---
exclude_paths:
- '.github/**'
- '.agents/**'
- 'CHANGELOG.md'
- '**/CHANGELOG.md'
- 'changelog/**'
+1 -1
View File
@@ -1,3 +1,3 @@
# The test runner source for UI tests
WEB_COMMITID=7bc6f96066e0618c301ee34cb7fe219efd47e3c0
WEB_COMMITID=52b3c79f772feae23b7e77058c89c4151087502e
WEB_BRANCH=main
+49
View File
@@ -1,5 +1,54 @@
# Changelog
## [7.4.0](https://github.com/opencloud-eu/opencloud/releases/tag/v7.4.0) - 2026-08-03
### ❤️ Thanks to all contributors! ❤️
@AlexAndBear, @JammingBen, @aduffeck, @dschmidt, @fschade, @michaelstingl, @pbleser-oc, @rhafer, @schweigisito, @v-scharf
### 📈 Enhancement
- Do not check ignored paths [[#3233](https://github.com/opencloud-eu/opencloud/pull/3233)]
- Extend posixfs consistency check [[#3220](https://github.com/opencloud-eu/opencloud/pull/3220)]
- Improve reindex command [[#3213](https://github.com/opencloud-eu/opencloud/pull/3213)]
- Reindex spaces concurrently [[#3207](https://github.com/opencloud-eu/opencloud/pull/3207)]
- feat: add announcement banner [[#3189](https://github.com/opencloud-eu/opencloud/pull/3189)]
- feat: add space viewer with versions role [[#2961](https://github.com/opencloud-eu/opencloud/pull/2961)]
### 🐛 Bug Fixes
- fix(runtime): log service startup errors instead of printing them beside the log [[#3140](https://github.com/opencloud-eu/opencloud/pull/3140)]
- fix(posixfs scan): Setup logger for scan command [[#3185](https://github.com/opencloud-eu/opencloud/pull/3185)]
### ✅ Tests
- api-test: mark group last-manager removal scenario as flaky [[#3194](https://github.com/opencloud-eu/opencloud/pull/3194)]
- rerun flaky tests [[#3183](https://github.com/opencloud-eu/opencloud/pull/3183)]
- api-test: fix removeAccessToDrive.feature:145 [[#3179](https://github.com/opencloud-eu/opencloud/pull/3179)]
- api-test: cover additional unified roles in acceptance tests [[#3169](https://github.com/opencloud-eu/opencloud/pull/3169)]
- test(apiAuthApp): fix flaky token pattern [[#3163](https://github.com/opencloud-eu/opencloud/pull/3163)]
- test(apiArchiver): the single-resource archive is named after the resource [[#3080](https://github.com/opencloud-eu/opencloud/pull/3080)]
- test(coreApiWebdavUploadTUS): assert etag and permissions on the finalizing TUS chunk [[#3078](https://github.com/opencloud-eu/opencloud/pull/3078)]
- test(coreApiWebdavOperations): download a file with a literal "%" via its oc:downloadURL [[#3079](https://github.com/opencloud-eu/opencloud/pull/3079)]
### 📚 Documentation
- maint: clean-up auth-app documentation [[#3155](https://github.com/opencloud-eu/opencloud/pull/3155)]
- ci: sync tests/README.md to docs [[#3164](https://github.com/opencloud-eu/opencloud/pull/3164)]
### 📦️ Dependencies
- [full-ci] chore: bump web to v7.3.0 [[#3223](https://github.com/opencloud-eu/opencloud/pull/3223)]
- build(deps): bump github.com/open-policy-agent/opa from 1.18.2 to 1.19.0 [[#3231](https://github.com/opencloud-eu/opencloud/pull/3231)]
- build(deps): bump google.golang.org/grpc from 1.82.0 to 1.83.0 [[#3232](https://github.com/opencloud-eu/opencloud/pull/3232)]
- chore(idp): bump dependencies [[#3226](https://github.com/opencloud-eu/opencloud/pull/3226)]
- build(deps): bump github.com/nats-io/nats-server/v2 from 2.14.3 to 2.14.4 [[#3221](https://github.com/opencloud-eu/opencloud/pull/3221)]
- build(deps): bump github.com/go-ldap/ldap/v3 from 3.4.13 to 3.4.14 [[#3222](https://github.com/opencloud-eu/opencloud/pull/3222)]
- build(deps): bump github.com/prometheus/client_golang from 1.23.2 to 1.24.1 [[#3218](https://github.com/opencloud-eu/opencloud/pull/3218)]
- build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.13 to 1.4.15 [[#3217](https://github.com/opencloud-eu/opencloud/pull/3217)]
- build(deps): bump golang.org/x/net from 0.56.0 to 0.57.0 [[#3136](https://github.com/opencloud-eu/opencloud/pull/3136)]
- build(deps): bump github.com/beevik/etree from 1.6.0 to 1.7.0 [[#3134](https://github.com/opencloud-eu/opencloud/pull/3134)]
## [7.3.0](https://github.com/opencloud-eu/opencloud/releases/tag/v7.3.0) - 2026-07-14
### 🐛 Bug Fixes
+25 -26
View File
@@ -10,7 +10,7 @@ require (
github.com/MicahParks/keyfunc/v2 v2.1.0
github.com/Nerzal/gocloak/v13 v13.9.0
github.com/bbalet/stopwords v1.0.0
github.com/beevik/etree v1.6.0
github.com/beevik/etree v1.7.0
github.com/blevesearch/bleve/v2 v2.6.0
github.com/cenkalti/backoff v2.2.1+incompatible
github.com/coreos/go-oidc/v3 v3.20.0
@@ -18,12 +18,12 @@ require (
github.com/davidbyttow/govips/v2 v2.18.0
github.com/dhowden/tag v0.0.0-20240417053706-3d75831295e8
github.com/dutchcoders/go-clamd v0.0.0-20170520113014-b970184f4d9e
github.com/gabriel-vasile/mimetype v1.4.13
github.com/gabriel-vasile/mimetype v1.4.15
github.com/ggwhite/go-masker v1.1.0
github.com/go-chi/chi/v5 v5.3.1
github.com/go-chi/render v1.0.3
github.com/go-jose/go-jose/v3 v3.0.5
github.com/go-ldap/ldap/v3 v3.4.13
github.com/go-ldap/ldap/v3 v3.4.14
github.com/go-ldap/ldif v0.0.0-20200320164324-fd88d9b715b3
github.com/go-micro/plugins/v4/client/grpc v1.2.1
github.com/go-micro/plugins/v4/logger/zerolog v1.2.0
@@ -55,26 +55,27 @@ require (
github.com/libregraph/lico v0.67.0
github.com/mna/pigeon v1.3.0
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826
github.com/nats-io/nats-server/v2 v2.14.3
github.com/nats-io/nats-server/v2 v2.14.4
github.com/nats-io/nats.go v1.52.0
github.com/olekukonko/tablewriter v1.1.4
github.com/onsi/ginkgo v1.16.5
github.com/onsi/ginkgo/v2 v2.32.0
github.com/onsi/gomega v1.42.1
github.com/open-policy-agent/opa v1.18.2
github.com/open-policy-agent/opa v1.19.0
github.com/opencloud-eu/icap-client v0.0.0-20250930132611-28a2afe62d89
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20260310090739-853d972b282d
github.com/opencloud-eu/reva/v2 v2.47.0
github.com/opencloud-eu/reva/v2 v2.48.0
github.com/opensearch-project/opensearch-go/v4 v4.6.0
github.com/orcaman/concurrent-map v1.0.0
github.com/pkg/errors v0.9.1
github.com/pkg/xattr v0.4.12
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_golang v1.24.1
github.com/r3labs/sse/v2 v2.10.0
github.com/riandyrn/otelchi v0.12.3
github.com/rogpeppe/go-internal v1.15.0
github.com/rs/cors v1.11.1
github.com/rs/zerolog v1.35.1
github.com/shamaton/msgpack/v2 v2.4.1
github.com/sirupsen/logrus v1.9.4
github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.2
@@ -84,7 +85,6 @@ require (
github.com/test-go/testify v1.1.4
github.com/testcontainers/testcontainers-go v0.43.0
github.com/testcontainers/testcontainers-go/modules/opensearch v0.43.0
github.com/theckman/yacspin v0.13.12
github.com/thejerf/suture/v4 v4.0.6
github.com/tidwall/gjson v1.19.0
github.com/tidwall/sjson v1.2.5
@@ -102,16 +102,16 @@ require (
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.44.0
go.opentelemetry.io/otel/sdk v1.44.0
go.opentelemetry.io/otel/trace v1.44.0
golang.org/x/crypto v0.53.0
golang.org/x/crypto v0.54.0
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f
golang.org/x/image v0.44.0
golang.org/x/net v0.56.0
golang.org/x/net v0.57.0
golang.org/x/oauth2 v0.36.0
golang.org/x/sync v0.22.0
golang.org/x/term v0.45.0
golang.org/x/text v0.40.0
google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa
google.golang.org/grpc v1.82.0
google.golang.org/grpc v1.83.0
google.golang.org/protobuf v1.36.11
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
@@ -135,7 +135,7 @@ require (
github.com/ajg/form v1.5.1 // indirect
github.com/alexedwards/argon2id v1.0.0 // indirect
github.com/amoghe/go-crypt v0.0.0-20220222110647-20eada5f5964 // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/antithesishq/antithesis-sdk-go v0.7.2 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
@@ -203,14 +203,14 @@ require (
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/gdexlab/go-render v1.0.1 // indirect
github.com/go-acme/lego/v4 v4.4.0 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 // indirect
github.com/go-asn1-ber/asn1-ber v1.5.8 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-git/go-git/v5 v5.19.1 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-micro/plugins/v4/events/natsjs v1.2.2 // indirect
github.com/go-micro/plugins/v4/store/nats-js v1.2.1 // indirect
@@ -255,7 +255,7 @@ require (
github.com/json-iterator/go v1.1.12 // indirect
github.com/juliangruber/go-intersect v1.1.0 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/compress v1.19.1 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/kovidgoyal/go-parallel v1.1.1 // indirect
@@ -274,10 +274,10 @@ require (
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/magiconair/properties v1.8.10 // indirect
github.com/mattermost/xml-roundtrip-validator v0.1.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.23 // indirect
github.com/mattn/go-sqlite3 v1.14.42 // indirect
github.com/mattn/go-colorable v0.1.15 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/mattn/go-runewidth v0.0.24 // indirect
github.com/mattn/go-sqlite3 v1.14.49 // indirect
github.com/maxymania/go-system v0.0.0-20170110133659-647cc364bf0b // indirect
github.com/mendsley/gojwk v0.0.0-20141217222730-4d5ec6e58103 // indirect
github.com/miekg/dns v1.1.68 // indirect
@@ -325,8 +325,8 @@ require (
github.com/pquerna/cachecontrol v0.2.0 // indirect
github.com/prometheus/alertmanager v0.33.1 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/prometheus/statsd_exporter v0.22.8 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/rs/xid v1.6.0 // indirect
@@ -342,9 +342,8 @@ require (
github.com/segmentio/ksuid v1.0.4 // indirect
github.com/sercand/kuberesolver/v5 v5.1.1 // indirect
github.com/sergi/go-diff v1.4.0 // indirect
github.com/sethvargo/go-diceware v0.5.0 // indirect
github.com/sethvargo/go-password v0.3.1 // indirect
github.com/shamaton/msgpack/v2 v2.4.1 // indirect
github.com/sethvargo/go-diceware v0.6.0 // indirect
github.com/sethvargo/go-password v0.4.0 // indirect
github.com/shirou/gopsutil/v4 v4.26.5 // indirect
github.com/skeema/knownhosts v1.3.1 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
@@ -363,7 +362,7 @@ require (
github.com/trustelem/zxcvbn v1.0.1 // indirect
github.com/urfave/cli/v2 v2.27.7 // indirect
github.com/valyala/fastjson v1.6.10 // indirect
github.com/vektah/gqlparser/v2 v2.5.34 // indirect
github.com/vektah/gqlparser/v2 v2.5.36 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/wk8/go-ordered-map v1.0.0 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
@@ -390,7 +389,7 @@ require (
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.47.0 // indirect
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260729162451-8efbd57d26e0 // indirect
gopkg.in/cenkalti/backoff.v1 v1.1.0 // indirect
gopkg.in/ini.v1 v1.67.3 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
+52 -54
View File
@@ -115,8 +115,8 @@ github.com/amoghe/go-crypt v0.0.0-20220222110647-20eada5f5964 h1:I9YN9WMo3SUh7p/
github.com/amoghe/go-crypt v0.0.0-20220222110647-20eada5f5964/go.mod h1:eFiR01PwTcpbzXtdMces7zxg6utvFM5puiWHpWB8D/k=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/antithesishq/antithesis-sdk-go v0.7.2 h1:oEEedg1Xgi8drRjqB0f9tfjhLoInE0IYZfZ6zAhQUbY=
github.com/antithesishq/antithesis-sdk-go v0.7.2/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
@@ -132,8 +132,8 @@ github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:W
github.com/aws/aws-sdk-go v1.37.27/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
github.com/bbalet/stopwords v1.0.0 h1:0TnGycCtY0zZi4ltKoOGRFIlZHv0WqpoIGUsObjztfo=
github.com/bbalet/stopwords v1.0.0/go.mod h1:sAWrQoDMfqARGIn4s6dp7OW7ISrshUD8IP2q3KoqPjc=
github.com/beevik/etree v1.6.0 h1:u8Kwy8pp9D9XeITj2Z0XtA5qqZEmtJtuXZRQi+j03eE=
github.com/beevik/etree v1.6.0/go.mod h1:bh4zJxiIr62SOf9pRzN7UUYaEDa9HEKafK25+sLc0Gc=
github.com/beevik/etree v1.7.0 h1:xjBk9O4p4x7D1YajePjfLzdaFC4/uYUENA7P0pv6gXA=
github.com/beevik/etree v1.7.0/go.mod h1:bh4zJxiIr62SOf9pRzN7UUYaEDa9HEKafK25+sLc0Gc=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/beorn7/perks v0.0.0-20160804104726-4c0e84591b9a/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -195,8 +195,6 @@ github.com/bufbuild/protocompile v0.14.1 h1:iA73zAf/fyljNjQKwYzUHD6AD4R8KMasmwa/
github.com/bufbuild/protocompile v0.14.1/go.mod h1:ppVdAIhbr2H8asPk6k4pY7t9zB1OU5DoEw9xY/FUi1c=
github.com/butonic/go-micro/v4 v4.11.1-0.20241115112658-b5d4de5ed9b3 h1:h8Z0hBv5tg/uZMKu8V47+DKWYVQg0lYP8lXDQq7uRpE=
github.com/butonic/go-micro/v4 v4.11.1-0.20241115112658-b5d4de5ed9b3/go.mod h1:eE/tD53n3KbVrzrWxKLxdkGw45Fg1qaNLWjpJMvIUF4=
github.com/bytecodealliance/wasmtime-go/v44 v44.0.0 h1:WRZXnLPIer/TWs5aYPaMlmVcOlzmR6Ur6wjLRIQOhTQ=
github.com/bytecodealliance/wasmtime-go/v44 v44.0.0/go.mod h1:GP93piU+39CoFVCQ5xfHrPOUtL0APlMnkbblJ2d3YY0=
github.com/c-bata/go-prompt v0.2.5/go.mod h1:vFnjEGDIIA/Lib7giyE4E9c50Lvl8j0S+7FVlAwDAVw=
github.com/cenkalti/backoff v2.2.1+incompatible h1:tNowT99t7UNflLxfYYSlKYsBpXdEet03Pg2g16Swow4=
github.com/cenkalti/backoff v2.2.1+incompatible/go.mod h1:90ReRw6GdpyfrHakVjL/QHaoyV4aDUVVkXQJJJ3NXXM=
@@ -281,8 +279,8 @@ github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.1/go.mod h1:ZXNYxsqcloTdSy/rNShjY
github.com/deepmap/oapi-codegen v1.3.11/go.mod h1:suMvK7+rKlx3+tpa8ByptmvoXbAV70wERKTOGH3hLp0=
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f h1:U5y3Y5UE0w7amNe7Z5G/twsBW0KEalRQXZzf8ufSh9I=
github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f/go.mod h1:xH/i4TFMt8koVQZ6WFms69WAsDWr2XsYL3Hkl7jkoLE=
github.com/dgraph-io/badger/v4 v4.9.2 h1:Wb5qw8gElqwV1a8msHTeQKova9b1V10heFKMIiPd80E=
github.com/dgraph-io/badger/v4 v4.9.2/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
github.com/dgraph-io/badger/v4 v4.9.4 h1:bcw+waCpzRZ2nmcSPbnPvDVhiEsn98TKmvnAhK7r7LM=
github.com/dgraph-io/badger/v4 v4.9.4/go.mod h1:nJjaJTUOSsQEBhsq209FmwCvMJzEA3e74RjZw6V2pQI=
github.com/dgraph-io/ristretto v0.2.0 h1:XAfl+7cmoUDWW/2Lx8TGZQjjxIQ2Ley9DSf52dru4WE=
github.com/dgraph-io/ristretto v0.2.0/go.mod h1:8uBHCU/PBV4Ag0CJrP47b9Ofby5dqWNh4FicAdoqFNU=
github.com/dgraph-io/ristretto/v2 v2.2.0 h1:bkY3XzJcXoMuELV8F+vS8kzNgicwQFAaGINAEJdWGOM=
@@ -350,8 +348,8 @@ github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMo
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/gabriel-vasile/mimetype v1.4.15 h1:05iP/CYtZ/w455R/KZM6rZ5ieAdh99UPtd+d3YzLmaI=
github.com/gabriel-vasile/mimetype v1.4.15/go.mod h1:azpTcoLcDZRNgFou5j+APrqQx9HqVPWa6ijYQIIVswQ=
github.com/gdexlab/go-render v1.0.1 h1:rxqB3vo5s4n1kF0ySmoNeSPRYkEsyHgln4jFIQY7v0U=
github.com/gdexlab/go-render v1.0.1/go.mod h1:wRi5nW2qfjiGj4mPukH4UV0IknS1cHD4VgFTmJX5JzM=
github.com/getkin/kin-openapi v0.13.0/go.mod h1:WGRs2ZMM1Q8LR1QBEwUxC6RJEfaBcD0s+pcEVXFuAjw=
@@ -370,8 +368,8 @@ github.com/go-acme/lego/v4 v4.4.0 h1:uHhU5LpOYQOdp3aDU+XY2bajseu8fuExphTL1Ss6/Fc
github.com/go-acme/lego/v4 v4.4.0/go.mod h1:l3+tFUFZb590dWcqhWZegynUthtaHJbG2fevUpoOOE0=
github.com/go-asn1-ber/asn1-ber v1.3.1/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-asn1-ber/asn1-ber v1.4.1/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667 h1:BP4M0CvQ4S3TGls2FvczZtj5Re/2ZzkV9VwqPHH/3Bo=
github.com/go-asn1-ber/asn1-ber v1.5.8-0.20250403174932-29230038a667/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-asn1-ber/asn1-ber v1.5.8 h1:H9AZkK22UOmfX8J84ubyaZxKJZ3FMHVwn8swoMML7iQ=
github.com/go-asn1-ber/asn1-ber v1.5.8/go.mod h1:hEBeB/ic+5LoWskz+yKT7vGhhPYkProFKoKdwZRWMe0=
github.com/go-chi/chi v4.0.2+incompatible/go.mod h1:eB3wogJHnLi3x/kFX2A+IbTBlXxmMeXJVKy9tTv1XzQ=
github.com/go-chi/chi/v5 v5.3.1 h1:3j4HZLGZQ3JpMCrPJF/Jl3mYJfWLKBfNJ6quurUGCf8=
github.com/go-chi/chi/v5 v5.3.1/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
@@ -402,8 +400,8 @@ github.com/go-kit/log v0.2.0/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBj
github.com/go-kit/log v0.2.1 h1:MRVx0/zhvdseW+Gza6N9rVzU/IVzaeE1SFI4raAhmBU=
github.com/go-kit/log v0.2.1/go.mod h1:NwTd00d/i8cPZ3xOwwiv2PO5MOcx78fFErGNcVmBjv0=
github.com/go-ldap/ldap/v3 v3.1.7/go.mod h1:5Zun81jBTabRaI8lzN7E1JjyEl1g6zI6u9pd8luAK4Q=
github.com/go-ldap/ldap/v3 v3.4.13 h1:+x1nG9h+MZN7h/lUi5Q3UZ0fJ1GyDQYbPvbuH38baDQ=
github.com/go-ldap/ldap/v3 v3.4.13/go.mod h1:LxsGZV6vbaK0sIvYfsv47rfh4ca0JXokCoKjZxsszv0=
github.com/go-ldap/ldap/v3 v3.4.14 h1:D6PYdEgsaVzsXyr6w/yDC06Ria4uUhWm+Rb+er8lfAs=
github.com/go-ldap/ldap/v3 v3.4.14/go.mod h1:S4eJUMUNjDkE0ZJtIZdybwyb03sGGLW6gxXT1Hs8VKA=
github.com/go-ldap/ldif v0.0.0-20200320164324-fd88d9b715b3 h1:sfz1YppV05y4sYaW7kXZtrocU/+vimnIWt4cxAYh7+o=
github.com/go-ldap/ldif v0.0.0-20200320164324-fd88d9b715b3/go.mod h1:ZXFhGda43Z2TVbfGZefXyMJzsDHhCh0go3bZUcwTx7o=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
@@ -412,8 +410,8 @@ github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG
github.com/go-logfmt/logfmt v0.5.1 h1:otpy5pqBCBZ1ng9RQ0dPu4PN7ba75Y/aA+UpowDyNVA=
github.com/go-logfmt/logfmt v0.5.1/go.mod h1:WYhtIu8zTZfxdn5+rREduYbwxfcBr/Vr6KEVveWlfTs=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-micro/plugins/v4/client/grpc v1.2.1 h1:7xAwZRCO6mdUtBHsYIQs1/eCTdhCrnjF70GB+AVd6L0=
@@ -715,8 +713,8 @@ github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
@@ -801,23 +799,23 @@ github.com/mattn/go-colorable v0.1.7/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope
github.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY=
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.6/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.23 h1:7ykA0T0jkPpzSvMS5i9uoNn2Xy3R383f9HDx3RybWcw=
github.com/mattn/go-runewidth v0.0.23/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-sqlite3 v1.14.42 h1:MigqEP4ZmHw3aIdIT7T+9TLa90Z6smwcthx+Azv4Cgo=
github.com/mattn/go-sqlite3 v1.14.42/go.mod h1:pjEuOr8IwzLJP2MfGeTb0A35jauH+C2kbHKBr7yXKVQ=
github.com/mattn/go-runewidth v0.0.24 h1:cpokDiIn0MGnhdHwuWnJBITySJ20QyNGnY2kR/ay2DU=
github.com/mattn/go-runewidth v0.0.24/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-sqlite3 v1.14.49 h1:B8jBHC3xhxZgxztrgruTuLucebnULQnx4W7cF7SAE9w=
github.com/mattn/go-sqlite3 v1.14.49/go.mod h1:6JTjA44L93a0QCyJef5YvlPoKXntQPjzWv5gtm9sB6w=
github.com/mattn/go-tty v0.0.0-20180219170247-931426f7535a/go.mod h1:XPvLUNfbS4fJH25nqRHfWLMa1ONC8Amw+mIA639KxkE=
github.com/mattn/go-tty v0.0.3/go.mod h1:ihxohKRERHTVzN+aSVRwACLCeqIoZAWpoICkkvrWyR0=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
@@ -894,8 +892,8 @@ github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRW
github.com/namedotcom/go v0.0.0-20180403034216-08470befbe04/go.mod h1:5sN+Lt1CaY4wsPvgQH/jsuJi4XO2ssZbdsIizr4CVC8=
github.com/nats-io/jwt/v2 v2.8.2 h1:XXRgB60MSTnqsRwejQurVDs/hcv2dkt+86GjI+I/bMc=
github.com/nats-io/jwt/v2 v2.8.2/go.mod h1:Ag/56sq9OblL4JgdYufDd16Egb17Kr/8WwwuO/forVc=
github.com/nats-io/nats-server/v2 v2.14.3 h1:+xjydPt7rkit67G+04TN0mcO2n+8nveZE7tK/PPV53A=
github.com/nats-io/nats-server/v2 v2.14.3/go.mod h1:5IlCtBzfwyzQzPMjmoJ9W2/LKmnJRtNyuOs/OT+NHDY=
github.com/nats-io/nats-server/v2 v2.14.4 h1:efgjZ8cdExAKRuqSg8UPJFprb+l7NlBtSDPhDlw3rO4=
github.com/nats-io/nats-server/v2 v2.14.4/go.mod h1:BltdpOYestjbtQSnVO2zGHdg5SGBZjt+GYTgB9LZq/I=
github.com/nats-io/nats.go v1.52.0 h1:n3avV4VBsCgsdwh71TppsTwtv+QdPs7ntSKM8qJLGsc=
github.com/nats-io/nats.go v1.52.0/go.mod h1:26HypzazeOkyO3/mqd1zZd53STJN0EjCYF9Uy2ZOBno=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
@@ -936,16 +934,16 @@ github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7J
github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
github.com/onsi/gomega v1.42.1 h1:iN1rCUX+44NZ1Dc97MPoeFYbFR0vh8zxoxMFwKdyZ6I=
github.com/onsi/gomega v1.42.1/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg=
github.com/open-policy-agent/opa v1.18.2 h1:VBiLJpioTuk7XTW1JoQi4ILo+FVxD2/8uD8iP9/OcxY=
github.com/open-policy-agent/opa v1.18.2/go.mod h1:9GY+hER4ZEXtxPlMjftVbqJJY9xLtCD3Q0oufRCfAKo=
github.com/open-policy-agent/opa v1.19.0 h1:+j2OCsjMezZEML2T1lI9giJdGJS/PL1XFKgkHPGIhpo=
github.com/open-policy-agent/opa v1.19.0/go.mod h1:pb6Y6klyf7X7X8uXNDflruA9dQC2gMqWROXI5w/kvv0=
github.com/opencloud-eu/go-micro-plugins/v4/store/nats-js-kv v0.0.0-20250512152754-23325793059a h1:Sakl76blJAaM6NxylVkgSzktjo2dS504iDotEFJsh3M=
github.com/opencloud-eu/go-micro-plugins/v4/store/nats-js-kv v0.0.0-20250512152754-23325793059a/go.mod h1:pjcozWijkNPbEtX5SIQaxEW/h8VAVZYTLx+70bmB3LY=
github.com/opencloud-eu/icap-client v0.0.0-20250930132611-28a2afe62d89 h1:W1ms+lP5lUUIzjRGDg93WrQfZJZCaV1ZP3KeyXi8bzY=
github.com/opencloud-eu/icap-client v0.0.0-20250930132611-28a2afe62d89/go.mod h1:vigJkNss1N2QEceCuNw/ullDehncuJNFB6mEnzfq9UI=
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20260310090739-853d972b282d h1:JcqGDiyrcaQwVyV861TUyQgO7uEmsjkhfm7aQd84dOw=
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20260310090739-853d972b282d/go.mod h1:pzatilMEHZFT3qV7C/X3MqOa3NlRQuYhlRhZTL+hN6Q=
github.com/opencloud-eu/reva/v2 v2.47.0 h1:bYul45qS8GmmN9PKplSZ78ZTZ6A+9xp/FfogqFVud18=
github.com/opencloud-eu/reva/v2 v2.47.0/go.mod h1:zdpEKIMDT14w+MGUOWAi+rh+PAZBPUlb7AtIGFWx7Ds=
github.com/opencloud-eu/reva/v2 v2.48.0 h1:G/4Jbv0DWWOfA5u5DtV0CB75pi9Wwtj7JkJQOEBvErs=
github.com/opencloud-eu/reva/v2 v2.48.0/go.mod h1:ZCo/xQM6if+upZa7rJCmdifZ/Y5XHLCrscHytC39yI4=
github.com/opencloud-eu/secure v0.0.0-20260312082735-b6f5cb2244e4 h1:l2oB/RctH+t8r7QBj5p8thfEHCM/jF35aAY3WQ3hADI=
github.com/opencloud-eu/secure v0.0.0-20260312082735-b6f5cb2244e4/go.mod h1:BmF5hyM6tXczk3MpQkFf1hpKSRqCyhqcbiQtiAF7+40=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
@@ -1016,8 +1014,8 @@ github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqr
github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
github.com/prometheus/client_golang v1.12.2/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
github.com/prometheus/client_golang v1.13.0/go.mod h1:vTeo+zgvILHsnnj/39Ou/1fPN5nJFOEMgftOUOmlvYQ=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
github.com/prometheus/client_model v0.0.0-20170216185247-6f3806018612/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
@@ -1037,8 +1035,8 @@ github.com/prometheus/common v0.26.0/go.mod h1:M7rCNAaPfAosfx8veZJCuw84e35h3Cfd9
github.com/prometheus/common v0.32.1/go.mod h1:vu+V0TpY+O6vW9J44gczi3Ap/oXXR10b+M/gUGO4Hls=
github.com/prometheus/common v0.35.0/go.mod h1:phzohg0JFMnBEFGxTDbfu3QyL5GI8gTQJFhYO5B3mfA=
github.com/prometheus/common v0.37.0/go.mod h1:phzohg0JFMnBEFGxTDbfu3QyL5GI8gTQJFhYO5B3mfA=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi/PY=
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
github.com/prometheus/procfs v0.0.0-20170703101242-e645f4e5aaa8/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190117184657-bf6a532e95b1/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
@@ -1049,8 +1047,8 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.7.3/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
github.com/prometheus/procfs v0.8.0/go.mod h1:z7EfXMXOkbkqb9IINtpCn86r/to3BnA0uaxHdg830/4=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
github.com/prometheus/statsd_exporter v0.22.7/go.mod h1:N/TevpjkIh9ccs6nuzY3jQn9dFqnUakOjnEuMPJJJnI=
github.com/prometheus/statsd_exporter v0.22.8 h1:Qo2D9ZzaQG+id9i5NYNGmbf1aa/KxKbB9aKfMS+Yib0=
github.com/prometheus/statsd_exporter v0.22.8/go.mod h1:/DzwbTEaFTE0Ojz5PqcSk6+PFHOPWGxdXVr6yC8eFOM=
@@ -1102,10 +1100,10 @@ github.com/sercand/kuberesolver/v5 v5.1.1 h1:CYH+d67G0sGBj7q5wLK61yzqJJ8gLLC8aep
github.com/sercand/kuberesolver/v5 v5.1.1/go.mod h1:Fs1KbKhVRnB2aDWN12NjKCB+RgYMWZJ294T3BtmVCpQ=
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
github.com/sethvargo/go-diceware v0.5.0 h1:exrQ7GpaBo00GqRVM1N8ChXSsi3oS7tjQiIehsD+yR0=
github.com/sethvargo/go-diceware v0.5.0/go.mod h1:Lg1SyPS7yQO6BBgTN5r4f2MUDkqGfLWsOjHPY0kA8iw=
github.com/sethvargo/go-password v0.3.1 h1:WqrLTjo7X6AcVYfC6R7GtSyuUQR9hGyAj/f1PYQZCJU=
github.com/sethvargo/go-password v0.3.1/go.mod h1:rXofC1zT54N7R8K/h1WDUdkf9BOx5OptoxrMBcrXzvs=
github.com/sethvargo/go-diceware v0.6.0 h1:B3nhMhbBP7KwtTQ7hHRIOmv5FqeD8bJs77RFrV24iWk=
github.com/sethvargo/go-diceware v0.6.0/go.mod h1:lHmdB0xuWaJ06KCraW6bztRT+71Dp+lsXQvborhhsBc=
github.com/sethvargo/go-password v0.4.0 h1:eSidVKQw5C7CmTDAtH3RipBTSjdU1ZRxQaynD2GWLVU=
github.com/sethvargo/go-password v0.4.0/go.mod h1:PO3nYHwUpcHPR0F9woy7a4abZPvzRuqJr0GaeIYTm3k=
github.com/shamaton/msgpack/v2 v2.4.1 h1:JtJ141QoQ3NqgPDsjq2v9VXlaON8SiQOwEaoNLEK/MQ=
github.com/shamaton/msgpack/v2 v2.4.1/go.mod h1:6khjYnkx73f7VQU7wjcFS9DFjs+59naVWJv1TB7qdOI=
github.com/shirou/gopsutil v3.21.11+incompatible h1:+1+c1VGhc88SSonWP6foOcLhvnKlUeu/erjjvaPEYiI=
@@ -1189,10 +1187,10 @@ github.com/testcontainers/testcontainers-go v0.43.0 h1:oEQx5MW2DGd9z3AeEQfB2lPM0
github.com/testcontainers/testcontainers-go v0.43.0/go.mod h1:+VxkT2NQnKOZPKi6praMuMKYHYyOGXr0XSBSlSMCzFo=
github.com/testcontainers/testcontainers-go/modules/opensearch v0.43.0 h1:a1ipjF7d/VxPX1dgVPIk4F+t6YkgMbE2OtBuRQCHJt8=
github.com/testcontainers/testcontainers-go/modules/opensearch v0.43.0/go.mod h1:OWSeUDiGMUy30iMsAltIJIo9uh/CleLv6KyxjYOsgR8=
github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU=
github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0=
github.com/thanhpk/randstr v1.0.6 h1:psAOktJFD4vV9NEVb3qkhRSMvYh4ORRaj1+w/hn4B+o=
github.com/thanhpk/randstr v1.0.6/go.mod h1:M/H2P1eNLZzlDwAzpkkkUvoyNNMbzRGhESZuEQk3r0U=
github.com/theckman/yacspin v0.13.12 h1:CdZ57+n0U6JMuh2xqjnjRq5Haj6v1ner2djtLQRzJr4=
github.com/theckman/yacspin v0.13.12/go.mod h1:Rd2+oG2LmQi5f3zC3yeZAOl245z8QOvrH4OPOJNZxLg=
github.com/thejerf/suture/v4 v4.0.6 h1:QsuCEsCqb03xF9tPAsWAj8QOAJBgQI1c0VqJNaingg8=
github.com/thejerf/suture/v4 v4.0.6/go.mod h1:gu9Y4dXNUWFrByqRt30Rm9/UZ0wzRSt9AJS6xu/ZGxU=
github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
@@ -1229,8 +1227,8 @@ github.com/valyala/fastjson v1.6.10 h1:/yjJg8jaVQdYR3arGxPE2X5z89xrlhS0eGXdv+ADT
github.com/valyala/fastjson v1.6.10/go.mod h1:e6FubmQouUNP73jtMLmcbxS6ydWIpOfhz34TSfO3JaE=
github.com/valyala/fasttemplate v1.0.1/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8=
github.com/valyala/fasttemplate v1.1.0/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8=
github.com/vektah/gqlparser/v2 v2.5.34 h1:MEea5P0qhdcqfBL45ghKE+qr9laidVHTMHjav5h7ckk=
github.com/vektah/gqlparser/v2 v2.5.34/go.mod h1:mFdHLGCio7OGX1fby9ZjTW6FN+qxgmbnBcRIeeScE5s=
github.com/vektah/gqlparser/v2 v2.5.36 h1:CN9mKVHgMkc+XftdOWIhb4HEL8wKSYkFAqhf8booa7s=
github.com/vektah/gqlparser/v2 v2.5.36/go.mod h1:cAJ9qwVgPaUkWv6Gn8vn0mqOE0Ui5Pn56wNy5396XWo=
github.com/vinyldns/go-vinyldns v0.0.0-20200917153823-148a5f6b8f14/go.mod h1:RWc47jtnVuQv6+lY3c768WtXCas/Xi+U5UFc5xULmYg=
github.com/vmihailenco/msgpack/v5 v5.4.1 h1:cQriyiUvjTwOHg8QZaPihLWeRAAVoCpE00IUPn0Bjt8=
github.com/vmihailenco/msgpack/v5 v5.4.1/go.mod h1:GaZTsDaehaPpQVyxrf5mtQlH+pc21PIudVV/E3rRQok=
@@ -1356,8 +1354,8 @@ golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0
golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf4=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1447,8 +1445,8 @@ golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1705,8 +1703,8 @@ google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 h1:XzmzkmB14QhVhgn
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:L43LFes82YgSonw6iTXTxXUX1OlULt4AQtkik4ULL/I=
google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1:Kjn0N0tCrDgiAFW+lGO4JZ3ck44CehvJQMAwj9QF0G8=
google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260729162451-8efbd57d26e0 h1:mJiOtnGp0k/BcSgdu03G2NwnscCfCH+h2QKUBZr18KI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260729162451-8efbd57d26e0/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.19.1/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
@@ -1722,8 +1720,8 @@ google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3Iji
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/grpc v1.82.0 h1:vguDnZUPjE26w09A63VoxZPnvPjB5Riyc0mkXPFmAIU=
google.golang.org/grpc v1.82.0/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
google.golang.org/grpc/examples v0.0.0-20211102180624-670c133e568e h1:m7aQHHqd0q89mRwhwS9Bx2rjyl/hsFAeta+uGrHsQaU=
google.golang.org/grpc/examples v0.0.0-20211102180624-670c133e568e/go.mod h1:gID3PKrg7pWKntu9Ss6zTLJ0ttC0X9IHgREOCZwbCVU=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
+128 -413
View File
@@ -1,17 +1,16 @@
package command
import (
"bufio"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/opencloud-eu/opencloud/opencloud/pkg/register"
"github.com/opencloud-eu/opencloud/pkg/config"
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
"github.com/opencloud-eu/opencloud/pkg/config/parser"
"github.com/opencloud-eu/opencloud/pkg/x/path/filepathx"
storageUsersParser "github.com/opencloud-eu/opencloud/services/storage-users/pkg/config/parser"
"github.com/opencloud-eu/opencloud/services/storage-users/pkg/event"
"github.com/opencloud-eu/opencloud/services/storage-users/pkg/revaconfig"
@@ -19,27 +18,10 @@ import (
"github.com/opencloud-eu/reva/v2/pkg/storage/fs/posix/ignore"
"github.com/opencloud-eu/reva/v2/pkg/storage/fs/posix/options"
"github.com/opencloud-eu/reva/v2/pkg/storage/fs/registry"
"github.com/opencloud-eu/reva/v2/pkg/storage/pkg/decomposedfs/metadata/prefixes"
"github.com/pkg/xattr"
"github.com/rs/zerolog"
"github.com/spf13/cobra"
"github.com/theckman/yacspin"
"github.com/vmihailenco/msgpack/v5"
)
// Define the names of the extended attributes we are working with.
const (
parentIDAttrName = "user.oc.parentid"
idAttrName = "user.oc.id"
nameAttrName = "user.oc.name"
spaceIDAttrName = "user.oc.space.id"
ownerIDAttrName = "user.oc.owner.id"
)
var (
spinner *yacspin.Spinner
restartRequired = false
ignorer *ignore.Ignorer
)
type IDCacher interface {
@@ -94,6 +76,39 @@ func scanCmd(ocCfg *config.Config) *cobra.Command {
os.Exit(1)
}
storageRoot := cfg.Drivers.Posix.Root
root := storageRoot
defaultRoot := true
if v, err := cmd.Flags().GetString("basepath"); err != nil {
fmt.Fprintf(os.Stderr, "Failed to parse command-line parameter '--basepath': %v\n", err)
os.Exit(1)
} else if v != "" {
root = v
if !filepath.IsAbs(v) {
if v, err = filepath.Abs(v); err != nil {
fmt.Fprintf(os.Stderr, "Failed to make the basepath mentioned using '--basepath' absolute: %v\n", err)
os.Exit(1)
} else {
root = v
}
} else {
root = v
}
root = filepath.Clean(root)
defaultRoot = false
}
// ensure that, if a basepath has been indicated, it is under the storage root
if !defaultRoot {
if contained, err := filepathx.IsSameOrContainedBy(storageRoot, root); err != nil {
fmt.Fprintf(os.Stderr, "Failed to determine whether the specified basepath %q is contained by the storage root %q: %v\n", root, storageRoot, err)
os.Exit(1)
} else if !contained {
fmt.Fprintf(os.Stderr, "The specified basepath %q is neither the storage root %q, nor a subdirectory thereof, nor a file underneath it\n", root, storageRoot)
os.Exit(1)
}
}
// We want to initialize the driver but disable scanfs on boot, so we can trigger it manually afterwards
drivers := revaconfig.StorageProviderDrivers(cfg)
drivers["posix"] = revaconfig.Posix(cfg, false, false)
@@ -105,6 +120,11 @@ func scanCmd(ocCfg *config.Config) *cobra.Command {
fmt.Fprintf(os.Stderr, "Failed to create event stream for posix driver: %v\n", err)
os.Exit(1)
}
log := logger("posixfs")
if !defaultRoot {
log = log.With().Str("basepath", root).Logger()
}
f, ok := registry.NewFuncs["posix"]
if !ok {
@@ -112,7 +132,7 @@ func scanCmd(ocCfg *config.Config) *cobra.Command {
os.Exit(1)
}
fs, err := f(drivers["posix"].(map[string]any), fsStream, nil)
fs, err := f(drivers["posix"].(map[string]any), fsStream, &log)
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to initialize filesystem driver '%s': %v\n", cfg.Driver, err)
return err
@@ -124,8 +144,12 @@ func scanCmd(ocCfg *config.Config) *cobra.Command {
os.Exit(1)
}
fmt.Println("Starting posixfs scan...")
err = cacher.WarmupIDCache(cfg.Drivers.Posix.Root, true, false)
if defaultRoot {
fmt.Println("Starting posixfs scan...")
} else {
fmt.Printf("Starting posixfs scan at '%s'...\n", root)
}
err = cacher.WarmupIDCache(root, true, false)
if err != nil {
fmt.Fprintf(os.Stderr, "Scan failed: %v\n", err)
return err
@@ -135,407 +159,98 @@ func scanCmd(ocCfg *config.Config) *cobra.Command {
return nil
},
}
cmd.Flags().StringP("basepath", "p", "", "the root under which to scan files, which may be a directory or a file (when omitted, detaults to using the storage root)")
return cmd
}
// consistencyCmd returns a command to check the consistency of the posixfs storage.
func consistencyCmd(cfg *config.Config) *cobra.Command {
func consistencyCmd(ocCfg *config.Config) *cobra.Command {
consCmd := &cobra.Command{
Use: "consistency",
Short: "check the consistency of the posixfs storage",
Use: "consistency [path ...]",
Short: "Check the consistency of the posixfs storage",
Long: `Check the consistency of the posixfs storage.
You can specify one or more paths to limit the scope of the check.
If no path is provided, the whole storage is checked.
The provided arguments determines the scope of the check:
- a storage root: the whole storage (all personal and project spaces) is checked
- a space root: only that space is checked
- a file or directory: only that single entity is checked (and its children, if it is a directory)`,
Args: cobra.ArbitraryArgs,
PreRunE: func(cmd *cobra.Command, args []string) error {
if err := parser.ParseConfig(ocCfg, true); err != nil {
return configlog.ReturnError(err)
}
// Parse storage users config
ocCfg.StorageUsers.Commons = ocCfg.Commons
return configlog.ReturnFatal(storageUsersParser.ParseConfig(ocCfg.StorageUsers))
},
RunE: func(cmd *cobra.Command, args []string) error {
return checkPosixfsConsistency(cmd, cfg)
cfg := ocCfg.StorageUsers
if len(args) == 0 {
args = []string{cfg.Drivers.Posix.Root}
}
log := logger("posixfs")
recalculateChecksums, _ := cmd.Flags().GetBool("fix-checksums")
drivers := revaconfig.StorageProviderDrivers(cfg)
drivers["posix"] = revaconfig.Posix(cfg, false, false)
opts, err := options.New(drivers["posix"].(map[string]any))
if err != nil {
return err
}
ignorer := ignore.NewIgnorer(opts, &log)
checker := &consistencyChecker{
cfg: cfg,
ignorer: ignorer,
recalculateChecksums: recalculateChecksums,
}
return checker.Check(args)
},
}
consCmd.Flags().StringP("root", "r", "", "Path to the root directory of the posixfs storage")
_ = consCmd.MarkFlagRequired("root")
consCmd.Flags().Bool("fix-checksums", false, "Recalculate and fix the file checksums. This reads every file and can be slow on large storages.")
return consCmd
}
// checkPosixfsConsistency checks the consistency of the posixfs storage.
func checkPosixfsConsistency(cmd *cobra.Command, cfg *config.Config) error {
rootPath, _ := cmd.Flags().GetString("root")
indexesPath := filepath.Join(rootPath, "indexes")
opt, _ := options.New(map[string]interface{}{
"root": rootPath,
})
log := zerolog.Nop()
ignorer = ignore.NewIgnorer(opt, &log)
_, err := os.Stat(indexesPath)
if err != nil {
if os.IsNotExist(err) {
return fmt.Errorf("consistency check failed: '%s' is not a posixfs root", rootPath)
}
return fmt.Errorf("error accessing '%s': %w", indexesPath, err)
}
spinnerCfg := yacspin.Config{
Frequency: 100 * time.Millisecond,
CharSet: yacspin.CharSets[11],
StopCharacter: "✓",
StopColors: []string{"fgGreen"},
StopFailCharacter: "✗",
StopFailColors: []string{"fgRed"},
}
spinner, err = yacspin.New(spinnerCfg)
err = spinner.Start()
if err != nil {
return fmt.Errorf("error creating spinner: %w", err)
}
checkSpaces(filepath.Join(rootPath, "users"))
spinner.Suffix(" Personal spaces check ")
spinner.StopMessage("completed\n")
spinner.Stop()
checkSpaces(filepath.Join(rootPath, "projects"))
spinner.Suffix(" Project spaces check ")
spinner.StopMessage("completed")
spinner.Stop()
if restartRequired {
fmt.Println("\n\n ⚠️ Please restart your openCloud instance to apply changes.")
}
return nil
}
func checkSpaces(basePath string) {
dirEntries, err := os.ReadDir(basePath)
if err != nil {
spinner.Message(fmt.Sprintf("Error reading spaces directory '%s'\n", basePath))
spinner.StopFail()
return
}
for _, entry := range dirEntries {
if entry.IsDir() {
fullPath := filepath.Join(basePath, entry.Name())
checkSpace(fullPath)
}
}
}
func checkSpace(spacePath string) {
spinner.Message("")
spinner.Suffix(fmt.Sprintf(" Checking space '%s'", spacePath))
info, err := os.Stat(spacePath)
if err != nil {
logFailure("Error accessing path '%s': %v", spacePath, err)
return
}
if !info.IsDir() {
logFailure("Error: The provided path '%s' is not a directory\n", spacePath)
return
}
spaceID, err := xattr.Get(spacePath, spaceIDAttrName)
if err != nil || len(spaceID) == 0 {
logFailure("Error: The directory '%s' does not seem to be a space root, it's missing the '%s' attribute\n", spacePath, spaceIDAttrName)
return
}
checkSpaceID(spacePath)
checkNodeIDs(spacePath)
}
func checkSpaceID(spacePath string) {
spinner.Message(" - checking space ID uniqueness")
entries, uniqueIDs, oldestEntry, err := gatherAttributes(spacePath)
if err != nil {
logFailure("Failed to gather attributes: %v", err)
return
}
if len(entries) == 0 {
return
}
if len(uniqueIDs) > 1 {
spinner.Pause()
fmt.Println("\n ⚠ Multiple space IDs found:")
for id := range uniqueIDs {
fmt.Printf(" - %s\n", id)
}
fmt.Printf("\n ⏳ Oldest entry is '%s' (modified on %s).\n",
filepath.Base(oldestEntry.Path), oldestEntry.ModTime.Format(time.RFC1123))
targetID := oldestEntry.ParentID
fmt.Printf(" ✅ Proposed target Parent ID: %s\n", targetID)
fmt.Printf("\n Do you want to unify all parent IDs to '%s'? This will modify %d entries, the directory, and the user index. (y/N): ", targetID, len(entries))
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
input = strings.TrimSpace(strings.ToLower(input))
if input != "y" {
spinner.Unpause()
logFailure("Operation cancelled by user.")
return
}
restartRequired = true
obsoleteIDs := []string{}
for id := range uniqueIDs {
if id != targetID {
obsoleteIDs = append(obsoleteIDs, id)
}
}
fixSpaceID(spacePath, obsoleteIDs, targetID, entries)
spinner.Unpause()
}
}
func walkNodes(dir string, parentID string) int {
fixes := 0
entries, err := os.ReadDir(dir)
if err != nil {
logFailure("Error reading directory '%s': %v", dir, err)
return 0
}
for _, entry := range entries {
fullPath := filepath.Join(dir, entry.Name())
if ignorer.IsIgnored(fullPath) {
continue
}
// Check if the parent ID attribute matches the expected parent ID, if not, fix it.
actualParentID, err := xattr.Get(fullPath, parentIDAttrName)
if err != nil || string(actualParentID) != parentID {
err = xattr.Set(fullPath, parentIDAttrName, []byte(parentID))
if err != nil {
logFailure("Failed to fix parent ID for '%s': %v", fullPath, err)
} else {
spinner.Pause()
fmt.Printf(" + Fixed parent ID for '%s'", fullPath)
spinner.Unpause()
fixes++
restartRequired = true
}
}
// Check that the name attribute matches the actual name of the file/directory, if not, fix it.
nameAttr, err := xattr.Get(fullPath, nameAttrName)
if err != nil || string(nameAttr) != entry.Name() {
err = xattr.Set(fullPath, nameAttrName, []byte(entry.Name()))
if err != nil {
logFailure("Failed to fix name attribute for '%s': %v", fullPath, err)
} else {
spinner.Pause()
fmt.Printf(" + Fixed name attribute for '%s'", fullPath)
spinner.Unpause()
fixes++
restartRequired = true
}
}
if entry.IsDir() {
nodeID, err := xattr.Get(fullPath, idAttrName)
if err != nil || len(nodeID) == 0 {
logFailure("Directory '%s' missing '%s', skipping its children", fullPath, idAttrName)
continue
}
walkNodes(fullPath, string(nodeID))
}
}
return fixes
}
func checkNodeIDs(spacePath string) {
spinner.Message(" - checking nodes")
rootID, err := xattr.Get(spacePath, idAttrName)
if err != nil || len(rootID) == 0 {
logFailure("Space root '%s' missing '%s' attribute", spacePath, idAttrName)
return
}
fixes := walkNodes(spacePath, string(rootID))
if fixes > 0 {
spinner.Pause()
fmt.Printf("\n ✓ Fixed %d incorrect node attributes in %s\n", fixes, filepath.Base(spacePath))
spinner.Unpause()
}
}
func fixSpaceID(spacePath string, obsoleteIDs []string, targetID string, entries []EntryInfo) {
// Set all parentid attributes to the proper space ID
err := setAllParentIDAttributes(entries, targetID)
if err != nil {
logFailure("an error occurred during file attribute update: %v", err)
return
}
// Update space ID itself
fmt.Printf(" Updating directory '%s' with attribute '%s' -> %s\n", filepath.Base(spacePath), idAttrName, targetID)
err = xattr.Set(spacePath, idAttrName, []byte(targetID))
if err != nil {
logFailure("Failed to set attribute on directory '%s': %v", spacePath, err)
return
}
err = xattr.Set(spacePath, spaceIDAttrName, []byte(targetID))
if err != nil {
logFailure("Failed to set attribute on directory '%s': %v", spacePath, err)
return
}
// update the index
err = updateOwnerIndexFile(spacePath, obsoleteIDs)
if err != nil {
logFailure("Could not update the owner index file: %v", err)
}
}
func gatherAttributes(path string) ([]EntryInfo, map[string]struct{}, EntryInfo, error) {
dirEntries, err := os.ReadDir(path)
if err != nil {
return nil, nil, EntryInfo{}, fmt.Errorf("failed to read directory: %w", err)
}
var allEntries []EntryInfo
uniqueIDs := make(map[string]struct{})
var oldestEntry EntryInfo
oldestTime := time.Now().Add(100 * 365 * 24 * time.Hour) // Set to a future date to find the oldest entry
for _, entry := range dirEntries {
fullPath := filepath.Join(path, entry.Name())
if ignorer.IsIgnored(fullPath) {
continue
}
info, err := os.Stat(fullPath)
if err != nil {
fmt.Printf(" - Warning: could not stat %s: %v\n", entry.Name(), err)
continue
}
parentID, err := xattr.Get(fullPath, parentIDAttrName)
if err != nil {
continue // Skip if attribute doesn't exist or can't be read
}
entryInfo := EntryInfo{
Path: fullPath,
ModTime: info.ModTime(),
ParentID: string(parentID),
}
allEntries = append(allEntries, entryInfo)
uniqueIDs[string(parentID)] = struct{}{}
if entryInfo.ModTime.Before(oldestTime) {
oldestTime = entryInfo.ModTime
oldestEntry = entryInfo
}
}
return allEntries, uniqueIDs, oldestEntry, nil
}
func setAllParentIDAttributes(entries []EntryInfo, targetID string) error {
fmt.Printf(" Setting all parent IDs to '%s':\n", targetID)
for _, entry := range entries {
if entry.ParentID == targetID {
fmt.Printf(" - Skipping '%s' (already has target ID).\n", filepath.Base(entry.Path))
continue
}
fmt.Printf(" - Removing all attributes from '%s'. It will be re-assimilated\n", filepath.Base(entry.Path))
filepath.WalkDir(entry.Path, func(path string, d os.DirEntry, err error) error {
if err != nil {
return fmt.Errorf("error walking path '%s': %w", path, err)
}
// Remove all attributes from the file.
if err := removeAttributes(path); err != nil {
fmt.Printf("failed to remove attributes from '%s': %v", path, err)
}
return nil
})
}
return nil
}
// updateOwnerIndexFile handles the logic of reading, modifying, and writing the MessagePack index file.
func updateOwnerIndexFile(basePath string, obsoleteIDs []string) error {
fmt.Printf(" Rewriting index file '%s'\n", basePath)
ownerID, err := xattr.Get(basePath, ownerIDAttrName)
if err != nil {
return fmt.Errorf("could not get owner ID from oldest entry '%s' to find index: %w", basePath, err)
}
indexPath := filepath.Join(basePath, "../../indexes/by-user-id", string(ownerID)+".mpk")
indexPath = filepath.Clean(indexPath)
// Read the MessagePack file
fileData, err := os.ReadFile(indexPath)
if err != nil {
if os.IsNotExist(err) {
return fmt.Errorf("index file does not exist, skipping update")
}
return fmt.Errorf("could not read index file: %w", err)
}
var indexMap map[string]string
if err := msgpack.Unmarshal(fileData, &indexMap); err != nil {
return fmt.Errorf("failed to parse MessagePack index file (is it corrupt?): %w", err)
}
// Remove obsolete IDs from the map
itemsRemoved := 0
for _, id := range obsoleteIDs {
if _, exists := indexMap[id]; exists {
fmt.Printf(" - Removing obsolete ID '%s' from index.\n", id)
delete(indexMap, id)
itemsRemoved++
} else {
fmt.Printf(" - Obsolete ID '%s' not found in index\n", id)
}
}
if itemsRemoved == 0 {
return nil
}
// Write the data back to the file
updatedData, err := msgpack.Marshal(&indexMap)
if err != nil {
return fmt.Errorf("failed to marshal updated index map: %w", err)
}
if err := os.WriteFile(indexPath, updatedData, 0644); err != nil {
return fmt.Errorf("failed to write updated index file: %w", err)
}
fmt.Printf(" ✓ Successfully removed %d item(s) and saved index file.\n", itemsRemoved)
return nil
}
func removeAttributes(path string) error {
attrNames, err := xattr.List(path)
if err != nil {
return fmt.Errorf("failed to list attributes for '%s': %w", path, err)
}
for _, attrName := range attrNames {
if err := xattr.Remove(path, attrName); err != nil {
return fmt.Errorf("failed to remove attribute '%s' from '%s': %w", attrName, path, err)
}
}
return nil
}
func logFailure(message string, args ...any) {
spinner.StopFailMessage(fmt.Sprintf("\n"+message, args...))
spinner.StopFail()
spinner.Start()
fmt.Fprintf(os.Stderr, message+"\n", args...)
}
// findStorageRoot walks up the directory tree starting at path until it finds a
// directory that contains an "indexes" subdirectory which marks the root of a
// posixfs storage. A user directory inside a space might also be named "indexes",
// so to disambiguate we require that the "indexes" directory is an internal
// directory: the storage's own indexes directory is skipped during assimilation
// and therefore never receives a node ID attribute, whereas a regular user
// directory would have one.
func findStorageRoot(path string) (string, error) {
current := path
for {
indexesPath := filepath.Join(current, "indexes")
if info, err := os.Stat(indexesPath); err == nil && info.IsDir() {
if id, err := xattr.Get(indexesPath, prefixes.IDAttr); err != nil || len(id) == 0 {
return current, nil
}
}
parent := filepath.Dir(current)
if parent == current {
return "", fmt.Errorf("'%s' does not appear to be inside a posixfs storage (no 'indexes' directory found)", path)
}
current = parent
}
}
// isSpaceRoot reports whether the given path is a space root, which is
// identified by the presence of the space ID attribute.
func isSpaceRoot(path string) bool {
spaceID, err := xattr.Get(path, prefixes.SpaceIDAttr)
return err == nil && len(spaceID) > 0
}
@@ -0,0 +1,497 @@
// Copyright 2026 OpenCloud GmbH <mail@opencloud.eu>
// SPDX-License-Identifier: Apache-2.0
package command
import (
"bufio"
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/opencloud-eu/opencloud/pkg/x/path/filepathx"
storageUsersConfig "github.com/opencloud-eu/opencloud/services/storage-users/pkg/config"
"github.com/opencloud-eu/reva/v2/pkg/storage/fs/posix/ignore"
"github.com/opencloud-eu/reva/v2/pkg/storage/pkg/decomposedfs/metadata/prefixes"
"github.com/opencloud-eu/reva/v2/pkg/storage/pkg/decomposedfs/node"
"github.com/pkg/xattr"
"github.com/shamaton/msgpack/v2"
)
type consistencyChecker struct {
cfg *storageUsersConfig.Config
ignorer *ignore.Ignorer
recalculateChecksums bool
restartRequired bool
}
// checkPosixfsConsistency checks the consistency of the posixfs storage. The
// given path determines the scope of the check: the whole storage, a single
// space or a single entity within a space.
func (c *consistencyChecker) Check(paths []string) error {
for _, path := range paths {
rootPath, err := findStorageRoot(path)
if err != nil {
return err
}
path = filepath.Clean(path)
if _, err := os.Stat(path); err != nil {
return fmt.Errorf("error accessing '%s': %w", path, err)
}
contained, _ := filepathx.IsSameOrContainedBy(rootPath, path)
switch {
case path == rootPath:
fmt.Println("Checking personal spaces...")
c.checkSpaces(filepath.Join(path, "users"))
fmt.Println("Checking project spaces...")
c.checkSpaces(filepath.Join(path, "projects"))
case isSpaceRoot(path):
fmt.Printf("Checking space '%s'...\n", path)
c.checkSpace(path)
case contained:
if c.ignorer.IsIgnored(path) {
continue
}
fmt.Printf("Checking '%s'...\n", path)
c.checkEntity(path)
default:
return fmt.Errorf("the provided path '%s' is neither a space root nor contained by the storage root '%s'", path, rootPath)
}
}
if c.restartRequired {
fmt.Println("\n\n ⚠️ Please restart your openCloud instance to apply changes.")
}
return nil
}
func (c *consistencyChecker) checkSpaces(basePath string) {
dirEntries, err := os.ReadDir(basePath)
if err != nil {
logFailure("Error reading spaces directory '%s': %v", basePath, err)
return
}
for _, entry := range dirEntries {
if entry.IsDir() {
fullPath := filepath.Join(basePath, entry.Name())
c.checkSpace(fullPath)
}
}
}
func (c *consistencyChecker) checkSpace(spacePath string) {
info, err := os.Stat(spacePath)
if err != nil {
logFailure("Error accessing path '%s': %v", spacePath, err)
return
}
if !info.IsDir() {
logFailure("Error: The provided path '%s' is not a directory\n", spacePath)
return
}
spaceID, err := xattr.Get(spacePath, prefixes.SpaceIDAttr)
if err != nil || len(spaceID) == 0 {
logFailure("Error: The directory '%s' does not seem to be a space root, it's missing the '%s' attribute\n", spacePath, prefixes.SpaceIDAttr)
return
}
c.checkSpaceID(spacePath)
c.checkNodes(spacePath)
}
func (c *consistencyChecker) checkSpaceID(spacePath string) {
entries, uniqueIDs, oldestEntry, err := c.gatherAttributes(spacePath)
if err != nil {
logFailure("Failed to gather attributes: %v", err)
return
}
if len(entries) == 0 {
return
}
if len(uniqueIDs) > 1 {
fmt.Println("\n ⚠ Multiple space IDs found:")
for id := range uniqueIDs {
fmt.Printf(" - %s\n", id)
}
fmt.Printf("\n ⏳ Oldest entry is '%s' (modified on %s).\n",
filepath.Base(oldestEntry.Path), oldestEntry.ModTime.Format(time.RFC1123))
targetID := oldestEntry.ParentID
fmt.Printf(" ✅ Proposed target Parent ID: %s\n", targetID)
fmt.Printf("\n Do you want to unify all parent IDs to '%s'? This will modify %d entries, the directory, and the user index. (y/N): ", targetID, len(entries))
reader := bufio.NewReader(os.Stdin)
input, _ := reader.ReadString('\n')
input = strings.TrimSpace(strings.ToLower(input))
if input != "y" {
logFailure("Operation cancelled by user.")
return
}
c.restartRequired = true
obsoleteIDs := []string{}
for id := range uniqueIDs {
if id != targetID {
obsoleteIDs = append(obsoleteIDs, id)
}
}
c.fixSpaceID(spacePath, obsoleteIDs, targetID, entries)
}
}
func (c *consistencyChecker) walkNodes(dir string, parentID string) int {
fixes := 0
entries, err := os.ReadDir(dir)
if err != nil {
logFailure("Error reading directory '%s': %v", dir, err)
return 0
}
for _, entry := range entries {
fullPath := filepath.Join(dir, entry.Name())
if c.ignorer.IsIgnored(fullPath) {
continue
}
fixes += c.checkNodeAttributes(fullPath, entry.Name(), parentID, entry.IsDir())
if entry.IsDir() {
nodeID, err := xattr.Get(fullPath, prefixes.IDAttr)
if err != nil || len(nodeID) == 0 {
logFailure("Directory '%s' missing '%s', skipping its children", fullPath, prefixes.IDAttr)
continue
}
fixes += c.walkNodes(fullPath, string(nodeID))
}
}
return fixes
}
// checkNodeAttributes checks and fixes the parent ID and name attributes of a
// single node. For files it additionally checks the blobsize and, when
// requested, the checksums. It returns the number of fixes applied.
func (c *consistencyChecker) checkNodeAttributes(path, name, parentID string, isDir bool) int {
fixes := 0
// Check if the parent ID attribute matches the expected parent ID, if not, fix it.
actualParentID, err := xattr.Get(path, prefixes.ParentidAttr)
if err != nil || string(actualParentID) != parentID {
if err := xattr.Set(path, prefixes.ParentidAttr, []byte(parentID)); err != nil {
logFailure("Failed to fix parent ID for '%s': %v", path, err)
} else {
fmt.Printf(" + Fixed parent ID for '%s'\n", path)
fixes++
c.restartRequired = true
}
}
// Check that the name attribute matches the actual name of the file/directory, if not, fix it.
nameAttr, err := xattr.Get(path, prefixes.NameAttr)
if err != nil || string(nameAttr) != name {
if err := xattr.Set(path, prefixes.NameAttr, []byte(name)); err != nil {
logFailure("Failed to fix name attribute for '%s': %v", path, err)
} else {
fmt.Printf(" + Fixed name attribute for '%s'\n", path)
fixes++
c.restartRequired = true
}
}
if !isDir {
fixes += c.checkBlobsize(path)
if c.recalculateChecksums {
fixes += c.fixChecksums(path)
}
}
return fixes
}
// checkEntity checks a single file or directory within a space, including its own
// parent ID, name and (for files) blobsize/checksums. If the entity is a directory
// its children are checked recursively.
func (c *consistencyChecker) checkEntity(path string) {
info, err := os.Stat(path)
if err != nil {
logFailure("Error accessing path '%s': %v", path, err)
return
}
// The expected parent ID is the ID attribute of the containing directory.
parentDir := filepath.Dir(path)
parentID, err := xattr.Get(parentDir, prefixes.IDAttr)
if err != nil || len(parentID) == 0 {
logFailure("Parent directory '%s' is missing the '%s' attribute", parentDir, prefixes.IDAttr)
return
}
fixes := c.checkNodeAttributes(path, info.Name(), string(parentID), info.IsDir())
if info.IsDir() {
nodeID, err := xattr.Get(path, prefixes.IDAttr)
if err != nil || len(nodeID) == 0 {
logFailure("Directory '%s' missing '%s' attribute", path, prefixes.IDAttr)
} else {
fixes += c.walkNodes(path, string(nodeID))
}
}
if fixes > 0 {
fmt.Printf(" ✓ Fixed %d incorrect node attributes for %s\n", fixes, filepath.Base(path))
}
}
// checkBlobsize verifies that the stored blobsize attribute matches the actual
// file size and fixes it if it doesn't. It returns the number of fixes applied.
func (c *consistencyChecker) checkBlobsize(path string) int {
info, err := os.Stat(path)
if err != nil {
logFailure("Error accessing file '%s': %v", path, err)
return 0
}
expectedSize := strconv.FormatInt(info.Size(), 10)
blobsize, err := xattr.Get(path, prefixes.BlobsizeAttr)
if err == nil && string(blobsize) == expectedSize {
return 0
}
if err := xattr.Set(path, prefixes.BlobsizeAttr, []byte(expectedSize)); err != nil {
logFailure("Failed to fix blobsize for '%s': %v", path, err)
return 0
}
fmt.Printf(" + Fixed blobsize for '%s'\n", path)
c.restartRequired = true
return 1
}
// fixChecksums recalculates the sha1, md5 and adler32 checksums of the file and
// updates the stored attributes if they differ. It returns the number of fixes applied.
func (c *consistencyChecker) fixChecksums(path string) int {
sha1h, md5h, adler32h, err := node.CalculateChecksums(context.Background(), path)
if err != nil {
logFailure("Failed to calculate checksums for '%s': %v", path, err)
return 0
}
checksums := map[string][]byte{
prefixes.ChecksumPrefix + "sha1": sha1h.Sum(nil),
prefixes.ChecksumPrefix + "md5": md5h.Sum(nil),
prefixes.ChecksumPrefix + "adler32": adler32h.Sum(nil),
}
fixes := 0
for attrName, sum := range checksums {
current, err := xattr.Get(path, attrName)
if err == nil && bytes.Equal(current, sum) {
continue
}
if err := xattr.Set(path, attrName, sum); err != nil {
logFailure("Failed to fix checksum '%s' for '%s': %v", attrName, path, err)
continue
}
fmt.Printf(" + Fixed checksum '%s' for '%s'\n", attrName, path)
c.restartRequired = true
fixes++
}
return fixes
}
func (c *consistencyChecker) checkNodes(spacePath string) {
rootID, err := xattr.Get(spacePath, prefixes.IDAttr)
if err != nil || len(rootID) == 0 {
logFailure("Space root '%s' missing '%s' attribute", spacePath, prefixes.IDAttr)
return
}
fixes := c.walkNodes(spacePath, string(rootID))
if fixes > 0 {
fmt.Printf(" ✓ Fixed %d incorrect node attributes in %s\n", fixes, filepath.Base(spacePath))
}
}
// fixSpaceID updates the parentid attributes of all entries in a space to a new target ID,
// updates the space's own ID attributes, and removes obsolete IDs from the user index file.
func (c *consistencyChecker) fixSpaceID(spacePath string, obsoleteIDs []string, targetID string, entries []EntryInfo) {
// Set all parentid attributes to the proper space ID
err := setAllParentIDAttributes(entries, targetID)
if err != nil {
logFailure("an error occurred during file attribute update: %v", err)
return
}
// Update space ID itself
fmt.Printf(" Updating directory '%s' with attribute '%s' -> %s\n", filepath.Base(spacePath), prefixes.IDAttr, targetID)
err = xattr.Set(spacePath, prefixes.IDAttr, []byte(targetID))
if err != nil {
logFailure("Failed to set attribute on directory '%s': %v", spacePath, err)
return
}
err = xattr.Set(spacePath, prefixes.SpaceIDAttr, []byte(targetID))
if err != nil {
logFailure("Failed to set attribute on directory '%s': %v", spacePath, err)
return
}
// update the index
err = c.updateOwnerIndexFile(spacePath, obsoleteIDs)
if err != nil {
logFailure("Could not update the owner index file: %v", err)
}
}
func (c *consistencyChecker) gatherAttributes(path string) ([]EntryInfo, map[string]struct{}, EntryInfo, error) {
dirEntries, err := os.ReadDir(path)
if err != nil {
return nil, nil, EntryInfo{}, fmt.Errorf("failed to read directory: %w", err)
}
var allEntries []EntryInfo
uniqueIDs := make(map[string]struct{})
var oldestEntry EntryInfo
oldestTime := time.Now().Add(100 * 365 * 24 * time.Hour) // Set to a future date to find the oldest entry
for _, entry := range dirEntries {
fullPath := filepath.Join(path, entry.Name())
if c.ignorer.IsIgnored(fullPath) {
continue
}
info, err := os.Stat(fullPath)
if err != nil {
fmt.Printf(" - Warning: could not stat %s: %v\n", entry.Name(), err)
continue
}
parentID, err := xattr.Get(fullPath, prefixes.ParentidAttr)
if err != nil {
continue // Skip if attribute doesn't exist or can't be read
}
entryInfo := EntryInfo{
Path: fullPath,
ModTime: info.ModTime(),
ParentID: string(parentID),
}
allEntries = append(allEntries, entryInfo)
uniqueIDs[string(parentID)] = struct{}{}
if entryInfo.ModTime.Before(oldestTime) {
oldestTime = entryInfo.ModTime
oldestEntry = entryInfo
}
}
return allEntries, uniqueIDs, oldestEntry, nil
}
func setAllParentIDAttributes(entries []EntryInfo, targetID string) error {
fmt.Printf(" Setting all parent IDs to '%s':\n", targetID)
for _, entry := range entries {
if entry.ParentID == targetID {
fmt.Printf(" - Skipping '%s' (already has target ID).\n", filepath.Base(entry.Path))
continue
}
fmt.Printf(" - Removing all attributes from '%s'. It will be re-assimilated\n", filepath.Base(entry.Path))
filepath.WalkDir(entry.Path, func(path string, d os.DirEntry, err error) error {
if err != nil {
return fmt.Errorf("error walking path '%s': %w", path, err)
}
// Remove all attributes from the file.
if err := removeAttributes(path); err != nil {
fmt.Printf("failed to remove attributes from '%s': %v", path, err)
}
return nil
})
}
return nil
}
// updateOwnerIndexFile handles the logic of reading, modifying, and writing the MessagePack index file.
func (c *consistencyChecker) updateOwnerIndexFile(basePath string, obsoleteIDs []string) error {
fmt.Printf(" Rewriting index file '%s'\n", basePath)
ownerID, err := xattr.Get(basePath, prefixes.OwnerIDAttr)
if err != nil {
return fmt.Errorf("could not get owner ID from oldest entry '%s' to find index: %w", basePath, err)
}
indexPath := filepath.Join(c.cfg.Drivers.Posix.Root, "indexes", "by-user-id", string(ownerID)+".mpk")
indexPath = filepath.Clean(indexPath)
// Read the MessagePack file
fileData, err := os.ReadFile(indexPath)
if err != nil {
if os.IsNotExist(err) {
return fmt.Errorf("index file does not exist, skipping update")
}
return fmt.Errorf("could not read index file: %w", err)
}
var indexMap map[string]string
if err := msgpack.Unmarshal(fileData, &indexMap); err != nil {
return fmt.Errorf("failed to parse MessagePack index file (is it corrupt?): %w", err)
}
// Remove obsolete IDs from the map
itemsRemoved := 0
for _, id := range obsoleteIDs {
if _, exists := indexMap[id]; exists {
fmt.Printf(" - Removing obsolete ID '%s' from index.\n", id)
delete(indexMap, id)
itemsRemoved++
} else {
fmt.Printf(" - Obsolete ID '%s' not found in index\n", id)
}
}
if itemsRemoved == 0 {
return nil
}
// Write the data back to the file
updatedData, err := msgpack.Marshal(&indexMap)
if err != nil {
return fmt.Errorf("failed to marshal updated index map: %w", err)
}
if err := os.WriteFile(indexPath, updatedData, 0644); err != nil {
return fmt.Errorf("failed to write updated index file: %w", err)
}
fmt.Printf(" ✓ Successfully removed %d item(s) and saved index file.\n", itemsRemoved)
return nil
}
func removeAttributes(path string) error {
attrNames, err := xattr.List(path)
if err != nil {
return fmt.Errorf("failed to list attributes for '%s': %w", path, err)
}
for _, attrName := range attrNames {
if err := xattr.Remove(path, attrName); err != nil {
return fmt.Errorf("failed to remove attribute '%s' from '%s': %w", attrName, path, err)
}
}
return nil
}
+12 -2
View File
@@ -6,11 +6,13 @@ import (
"os/signal"
"syscall"
"github.com/rs/zerolog"
"github.com/spf13/cobra"
"github.com/opencloud-eu/opencloud/opencloud/pkg/register"
"github.com/opencloud-eu/opencloud/pkg/clihelper"
"github.com/opencloud-eu/opencloud/pkg/config"
"github.com/spf13/cobra"
oclog "github.com/opencloud-eu/opencloud/pkg/log"
)
// Execute is the entry point for the opencloud command.
@@ -38,3 +40,11 @@ func Execute() error {
ctx, _ := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT, syscall.SIGHUP)
return app.ExecuteContext(ctx)
}
func logger(name string) zerolog.Logger {
return oclog.NewLogger(
oclog.Name(name),
oclog.Level("info"),
oclog.Pretty(true),
oclog.Color(true)).Logger
}
+2 -12
View File
@@ -9,7 +9,6 @@ import (
"github.com/opencloud-eu/opencloud/pkg/config"
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
"github.com/opencloud-eu/opencloud/pkg/config/parser"
oclog "github.com/opencloud-eu/opencloud/pkg/log"
mregistry "github.com/opencloud-eu/opencloud/pkg/registry"
sharing "github.com/opencloud-eu/opencloud/services/sharing/pkg/config"
sharingparser "github.com/opencloud-eu/opencloud/services/sharing/pkg/config/parser"
@@ -85,7 +84,7 @@ func cleanup(_ *cobra.Command, cfg *config.Config) error {
return configlog.ReturnError(errors.New("cleanup is only implemented for the jsoncs3 share manager"))
}
l := logger()
l := logger("migrate")
zerolog.SetGlobalLevel(zerolog.InfoLevel)
@@ -94,7 +93,7 @@ func cleanup(_ *cobra.Command, cfg *config.Config) error {
if !ok {
return configlog.ReturnError(errors.New("Unknown share manager type '" + driver + "'"))
}
mgr, err := f(rcfg[driver].(map[string]any), l)
mgr, err := f(rcfg[driver].(map[string]any), &l)
if err != nil {
return configlog.ReturnError(err)
}
@@ -167,12 +166,3 @@ func revaShareConfig(cfg *sharing.Config) map[string]any {
},
}
}
func logger() *zerolog.Logger {
log := oclog.NewLogger(
oclog.Name("migrate"),
oclog.Level("info"),
oclog.Pretty(true),
oclog.Color(true)).Logger
return &log
}
+10 -3
View File
@@ -389,9 +389,16 @@ func Start(ctx context.Context, o ...Option) error {
if ev.Restarting {
l = s.Log.Error()
}
l.Str("event", e.String()).Str("service", ev.ServiceName).Str("supervisor", ev.SupervisorName).
Bool("restarting", ev.Restarting).Float64("failures", ev.CurrentFailures).Float64("threshold", ev.FailureThreshold).
Interface("error", ev.Err).Msg("service terminated")
l = l.Str("event", e.String()).Str("service", ev.ServiceName).Str("supervisor", ev.SupervisorName).
Bool("restarting", ev.Restarting).Float64("failures", ev.CurrentFailures).Float64("threshold", ev.FailureThreshold)
// ev.Err is an interface{}: marshaling an error yields {} because
// its fields are unexported, so the message has to go through Err
if err, ok := ev.Err.(error); ok {
l = l.Err(err)
} else {
l = l.Interface("error", ev.Err)
}
l.Msg("service terminated")
case suture.EventBackoff:
s.Log.Warn().Str("event", e.String()).Str("supervisor", ev.SupervisorName).Msg("service backoff")
case suture.EventResume:
+12
View File
@@ -14,5 +14,17 @@ func DefaultApp(app *cobra.Command) *cobra.Command {
// version info
app.Version = fmt.Sprintf("%s (%s <%s>) (%s)", version.String, "OpenCloud GmbH", "support@opencloud.eu", version.Compiled())
// cobra would print the error on top of what main() already prints
app.SilenceErrors = true
// keep the usage block for flag parse errors, drop it once RunE runs.
// Traversing runs the hook even below a subcommand that brings its own,
// e.g. every service below ServiceCommand.
cobra.EnableTraverseRunHooks = true
app.PersistentPreRunE = func(cmd *cobra.Command, _ []string) error {
cmd.SilenceUsage = true
return nil
}
return app
}
+1 -1
View File
@@ -34,7 +34,7 @@ var (
// LatestTag is the latest released version plus the dev meta version.
// Will be overwritten by the release pipeline
// Needs a manual change for every tagged release
LatestTag = "7.3.0+dev"
LatestTag = "7.4.0+dev"
// Date indicates the build date.
// This has been removed, it looks like you can only replace static strings with recent go versions
+26
View File
@@ -1,7 +1,9 @@
package filepathx
import (
"fmt"
"path/filepath"
"strings"
)
// JailJoin joins any number of path elements into a single path,
@@ -10,3 +12,27 @@ import (
func JailJoin(jail string, elem ...string) string {
return filepath.Join(jail, filepath.Join(append([]string{"/"}, elem...)...))
}
// Determines whether the file or directory 'child' is same as or underneath the directory 'parent'.
//
// Note that 'parent' is expected to be a directory.
func IsSameOrContainedBy(parent string, child string) (bool, error) {
absParent, err := filepath.Abs(parent)
if err != nil {
return false, fmt.Errorf("failed to make parent directory absolute: %q: %w", parent, err)
}
absChild, err := filepath.Abs(child)
if err != nil {
return false, fmt.Errorf("failed to make child file/directory absolute: %q: %w", child, err)
}
rel, err := filepath.Rel(absParent, absChild)
if err != nil {
return false, fmt.Errorf("failed to determine the relative path between the parent directory %q and the child file/directory: %q: %w", absParent, absChild, err)
}
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return false, nil
}
return true, nil
}
+23
View File
@@ -1,9 +1,12 @@
package filepathx_test
import (
"fmt"
"strings"
"testing"
"github.com/opencloud-eu/opencloud/pkg/x/path/filepathx"
"github.com/stretchr/testify/require"
)
func TestJailJoin(t *testing.T) {
@@ -61,3 +64,23 @@ func TestJailJoin(t *testing.T) {
})
}
}
func TestIsSameOrContainedBy(t *testing.T) {
for _, tt := range []struct {
parent string
child string
expected bool
}{
{"foo", "foo", true},
{"/foo", "/foo", true},
{"foo", "foo/bar", true},
{"foo", "bar", false},
} {
t.Run(fmt.Sprintf("%s: %s vs %s", t.Name(), strings.ReplaceAll(tt.parent, "/", "."), strings.ReplaceAll(tt.child, "/", ".")), func(t *testing.T) {
require := require.New(t)
b, err := filepathx.IsSameOrContainedBy(tt.parent, tt.child)
require.NoError(err)
require.Equal(tt.expected, b)
})
}
}
@@ -627,13 +627,13 @@ type Bundle struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // @gotags: yaml:"id"
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` // @gotags: yaml:"name"
Type Bundle_Type `protobuf:"varint,3,opt,name=type,proto3,enum=opencloud.messages.settings.v0.Bundle_Type" json:"type,omitempty"` // @gotags: yaml:"type"
Extension string `protobuf:"bytes,4,opt,name=extension,proto3" json:"extension,omitempty"` // @gotags: yaml:"extension"
DisplayName string `protobuf:"bytes,5,opt,name=display_name,json=displayName,proto3" json:"display_name,omitempty"` // @gotags: yaml:"display_name"
Settings []*Setting `protobuf:"bytes,6,rep,name=settings,proto3" json:"settings,omitempty"` // @gotags: yaml:"settings"
Resource *Resource `protobuf:"bytes,7,opt,name=resource,proto3" json:"resource,omitempty"` // @gotags: yaml:"resource"
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty" yaml:"id"` // @gotags: yaml:"id"
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty" yaml:"name"` // @gotags: yaml:"name"
Type Bundle_Type `protobuf:"varint,3,opt,name=type,proto3,enum=opencloud.messages.settings.v0.Bundle_Type" json:"type,omitempty" yaml:"type"` // @gotags: yaml:"type"
Extension string `protobuf:"bytes,4,opt,name=extension,proto3" json:"extension,omitempty" yaml:"extension"` // @gotags: yaml:"extension"
DisplayName string `protobuf:"bytes,5,opt,name=display_name,json=displayName,proto3" json:"display_name,omitempty" yaml:"display_name"` // @gotags: yaml:"display_name"
Settings []*Setting `protobuf:"bytes,6,rep,name=settings,proto3" json:"settings,omitempty" yaml:"settings"` // @gotags: yaml:"settings"
Resource *Resource `protobuf:"bytes,7,opt,name=resource,proto3" json:"resource,omitempty" yaml:"resource"` // @gotags: yaml:"resource"
}
func (x *Bundle) Reset() {
@@ -722,10 +722,10 @@ type Setting struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // @gotags: yaml:"id"
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` // @gotags: yaml:"name"
DisplayName string `protobuf:"bytes,3,opt,name=display_name,json=displayName,proto3" json:"display_name,omitempty"` // @gotags: yaml:"display_name"
Description string `protobuf:"bytes,4,opt,name=description,proto3" json:"description,omitempty"` // @gotags: yaml:"description"
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty" yaml:"id"` // @gotags: yaml:"id"
Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty" yaml:"name"` // @gotags: yaml:"name"
DisplayName string `protobuf:"bytes,3,opt,name=display_name,json=displayName,proto3" json:"display_name,omitempty" yaml:"display_name"` // @gotags: yaml:"display_name"
Description string `protobuf:"bytes,4,opt,name=description,proto3" json:"description,omitempty" yaml:"description"` // @gotags: yaml:"description"
// Types that are assignable to Value:
//
// *Setting_IntValue
@@ -736,7 +736,7 @@ type Setting struct {
// *Setting_PermissionValue
// *Setting_MultiChoiceCollectionValue
Value isSetting_Value `protobuf_oneof:"value"`
Resource *Resource `protobuf:"bytes,11,opt,name=resource,proto3" json:"resource,omitempty"` // @gotags: yaml:"resource"
Resource *Resource `protobuf:"bytes,11,opt,name=resource,proto3" json:"resource,omitempty" yaml:"resource"` // @gotags: yaml:"resource"
}
func (x *Setting) Reset() {
@@ -867,31 +867,31 @@ type isSetting_Value interface {
}
type Setting_IntValue struct {
IntValue *Int `protobuf:"bytes,5,opt,name=int_value,json=intValue,proto3,oneof"` // @gotags: yaml:"int_value"
IntValue *Int `protobuf:"bytes,5,opt,name=int_value,json=intValue,proto3,oneof" yaml:"int_value"` // @gotags: yaml:"int_value"
}
type Setting_StringValue struct {
StringValue *String `protobuf:"bytes,6,opt,name=string_value,json=stringValue,proto3,oneof"` // @gotags: yaml:"string_value"
StringValue *String `protobuf:"bytes,6,opt,name=string_value,json=stringValue,proto3,oneof" yaml:"string_value"` // @gotags: yaml:"string_value"
}
type Setting_BoolValue struct {
BoolValue *Bool `protobuf:"bytes,7,opt,name=bool_value,json=boolValue,proto3,oneof"` // @gotags: yaml:"bool_value"
BoolValue *Bool `protobuf:"bytes,7,opt,name=bool_value,json=boolValue,proto3,oneof" yaml:"bool_value"` // @gotags: yaml:"bool_value"
}
type Setting_SingleChoiceValue struct {
SingleChoiceValue *SingleChoiceList `protobuf:"bytes,8,opt,name=single_choice_value,json=singleChoiceValue,proto3,oneof"` // @gotags: yaml:"single_choice_value"
SingleChoiceValue *SingleChoiceList `protobuf:"bytes,8,opt,name=single_choice_value,json=singleChoiceValue,proto3,oneof" yaml:"single_choice_value"` // @gotags: yaml:"single_choice_value"
}
type Setting_MultiChoiceValue struct {
MultiChoiceValue *MultiChoiceList `protobuf:"bytes,9,opt,name=multi_choice_value,json=multiChoiceValue,proto3,oneof"` // @gotags: yaml:"multi_choice_value"
MultiChoiceValue *MultiChoiceList `protobuf:"bytes,9,opt,name=multi_choice_value,json=multiChoiceValue,proto3,oneof" yaml:"multi_choice_value"` // @gotags: yaml:"multi_choice_value"
}
type Setting_PermissionValue struct {
PermissionValue *Permission `protobuf:"bytes,10,opt,name=permission_value,json=permissionValue,proto3,oneof"` // @gotags: yaml:"permission_value"
PermissionValue *Permission `protobuf:"bytes,10,opt,name=permission_value,json=permissionValue,proto3,oneof" yaml:"permission_value"` // @gotags: yaml:"permission_value"
}
type Setting_MultiChoiceCollectionValue struct {
MultiChoiceCollectionValue *MultiChoiceCollection `protobuf:"bytes,12,opt,name=multi_choice_collection_value,json=multiChoiceCollectionValue,proto3,oneof"` // @gotags: yaml:"multi_choice_collection_value"
MultiChoiceCollectionValue *MultiChoiceCollection `protobuf:"bytes,12,opt,name=multi_choice_collection_value,json=multiChoiceCollectionValue,proto3,oneof" yaml:"multi_choice_collection_value"` // @gotags: yaml:"multi_choice_collection_value"
}
func (*Setting_IntValue) isSetting_Value() {}
@@ -913,11 +913,11 @@ type Int struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Default int64 `protobuf:"varint,1,opt,name=default,proto3" json:"default,omitempty"` // @gotags: yaml:"default"
Min int64 `protobuf:"varint,2,opt,name=min,proto3" json:"min,omitempty"` // @gotags: yaml:"min"
Max int64 `protobuf:"varint,3,opt,name=max,proto3" json:"max,omitempty"` // @gotags: yaml:"max"
Step int64 `protobuf:"varint,4,opt,name=step,proto3" json:"step,omitempty"` // @gotags: yaml:"step"
Placeholder string `protobuf:"bytes,5,opt,name=placeholder,proto3" json:"placeholder,omitempty"` // @gotags: yaml:"placeholder"
Default int64 `protobuf:"varint,1,opt,name=default,proto3" json:"default,omitempty" yaml:"default"` // @gotags: yaml:"default"
Min int64 `protobuf:"varint,2,opt,name=min,proto3" json:"min,omitempty" yaml:"min"` // @gotags: yaml:"min"
Max int64 `protobuf:"varint,3,opt,name=max,proto3" json:"max,omitempty" yaml:"max"` // @gotags: yaml:"max"
Step int64 `protobuf:"varint,4,opt,name=step,proto3" json:"step,omitempty" yaml:"step"` // @gotags: yaml:"step"
Placeholder string `protobuf:"bytes,5,opt,name=placeholder,proto3" json:"placeholder,omitempty" yaml:"placeholder"` // @gotags: yaml:"placeholder"
}
func (x *Int) Reset() {
@@ -992,11 +992,11 @@ type String struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Default string `protobuf:"bytes,1,opt,name=default,proto3" json:"default,omitempty"` // @gotags: yaml:"default"
Required bool `protobuf:"varint,2,opt,name=required,proto3" json:"required,omitempty"` // @gotags: yaml:"required"
MinLength int32 `protobuf:"varint,3,opt,name=min_length,json=minLength,proto3" json:"min_length,omitempty"` // @gotags: yaml:"min_length"
MaxLength int32 `protobuf:"varint,4,opt,name=max_length,json=maxLength,proto3" json:"max_length,omitempty"` // @gotags: yaml:"max_length"
Placeholder string `protobuf:"bytes,5,opt,name=placeholder,proto3" json:"placeholder,omitempty"` // @gotags: yaml:"placeholder"
Default string `protobuf:"bytes,1,opt,name=default,proto3" json:"default,omitempty" yaml:"default"` // @gotags: yaml:"default"
Required bool `protobuf:"varint,2,opt,name=required,proto3" json:"required,omitempty" yaml:"required"` // @gotags: yaml:"required"
MinLength int32 `protobuf:"varint,3,opt,name=min_length,json=minLength,proto3" json:"min_length,omitempty" yaml:"min_length"` // @gotags: yaml:"min_length"
MaxLength int32 `protobuf:"varint,4,opt,name=max_length,json=maxLength,proto3" json:"max_length,omitempty" yaml:"max_length"` // @gotags: yaml:"max_length"
Placeholder string `protobuf:"bytes,5,opt,name=placeholder,proto3" json:"placeholder,omitempty" yaml:"placeholder"` // @gotags: yaml:"placeholder"
}
func (x *String) Reset() {
@@ -1071,8 +1071,8 @@ type Bool struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Default bool `protobuf:"varint,1,opt,name=default,proto3" json:"default,omitempty"` // @gotags: yaml:"default"
Label string `protobuf:"bytes,2,opt,name=label,proto3" json:"label,omitempty"` // @gotags: yaml:"label"
Default bool `protobuf:"varint,1,opt,name=default,proto3" json:"default,omitempty" yaml:"default"` // @gotags: yaml:"default"
Label string `protobuf:"bytes,2,opt,name=label,proto3" json:"label,omitempty" yaml:"label"` // @gotags: yaml:"label"
}
func (x *Bool) Reset() {
@@ -1126,7 +1126,7 @@ type SingleChoiceList struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Options []*ListOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty"` // @gotags: yaml:"options"
Options []*ListOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty" yaml:"options"` // @gotags: yaml:"options"
}
func (x *SingleChoiceList) Reset() {
@@ -1173,7 +1173,7 @@ type MultiChoiceList struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Options []*ListOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty"` // @gotags: yaml:"options"
Options []*ListOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty" yaml:"options"` // @gotags: yaml:"options"
}
func (x *MultiChoiceList) Reset() {
@@ -1220,9 +1220,9 @@ type ListOption struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Value *ListOptionValue `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty"` // @gotags: yaml:"value"
Default bool `protobuf:"varint,2,opt,name=default,proto3" json:"default,omitempty"` // @gotags: yaml:"default"
DisplayValue string `protobuf:"bytes,3,opt,name=display_value,json=displayValue,proto3" json:"display_value,omitempty"` // @gotags: yaml:"display_value"
Value *ListOptionValue `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty" yaml:"value"` // @gotags: yaml:"value"
Default bool `protobuf:"varint,2,opt,name=default,proto3" json:"default,omitempty" yaml:"default"` // @gotags: yaml:"default"
DisplayValue string `protobuf:"bytes,3,opt,name=display_value,json=displayValue,proto3" json:"display_value,omitempty" yaml:"display_value"` // @gotags: yaml:"display_value"
}
func (x *ListOption) Reset() {
@@ -1283,7 +1283,7 @@ type MultiChoiceCollection struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Options []*MultiChoiceCollectionOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty"` // @gotags: yaml:"options"
Options []*MultiChoiceCollectionOption `protobuf:"bytes,1,rep,name=options,proto3" json:"options,omitempty" yaml:"options"` // @gotags: yaml:"options"
}
func (x *MultiChoiceCollection) Reset() {
@@ -1330,10 +1330,10 @@ type MultiChoiceCollectionOption struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Value *MultiChoiceCollectionOptionValue `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty"` // @gotags: yaml:"value"
Key string `protobuf:"bytes,2,opt,name=key,proto3" json:"key,omitempty"` // @gotags: yaml:"key"
Attribute string `protobuf:"bytes,3,opt,name=attribute,proto3" json:"attribute,omitempty"` // @gotags: yaml:"attribute"
DisplayValue string `protobuf:"bytes,4,opt,name=display_value,json=displayValue,proto3" json:"display_value,omitempty"` // @gotags: yaml:"display_value"
Value *MultiChoiceCollectionOptionValue `protobuf:"bytes,1,opt,name=value,proto3" json:"value,omitempty" yaml:"value"` // @gotags: yaml:"value"
Key string `protobuf:"bytes,2,opt,name=key,proto3" json:"key,omitempty" yaml:"key"` // @gotags: yaml:"key"
Attribute string `protobuf:"bytes,3,opt,name=attribute,proto3" json:"attribute,omitempty" yaml:"attribute"` // @gotags: yaml:"attribute"
DisplayValue string `protobuf:"bytes,4,opt,name=display_value,json=displayValue,proto3" json:"display_value,omitempty" yaml:"display_value"` // @gotags: yaml:"display_value"
}
func (x *MultiChoiceCollectionOption) Reset() {
@@ -1474,15 +1474,15 @@ type isMultiChoiceCollectionOptionValue_Option interface {
}
type MultiChoiceCollectionOptionValue_IntValue struct {
IntValue *Int `protobuf:"bytes,1,opt,name=int_value,json=intValue,proto3,oneof"` // @gotags: yaml:"int_value"
IntValue *Int `protobuf:"bytes,1,opt,name=int_value,json=intValue,proto3,oneof" yaml:"int_value"` // @gotags: yaml:"int_value"
}
type MultiChoiceCollectionOptionValue_StringValue struct {
StringValue *String `protobuf:"bytes,2,opt,name=string_value,json=stringValue,proto3,oneof"` // @gotags: yaml:"string_value"
StringValue *String `protobuf:"bytes,2,opt,name=string_value,json=stringValue,proto3,oneof" yaml:"string_value"` // @gotags: yaml:"string_value"
}
type MultiChoiceCollectionOptionValue_BoolValue struct {
BoolValue *Bool `protobuf:"bytes,3,opt,name=bool_value,json=boolValue,proto3,oneof"` // @gotags: yaml:"bool_value"
BoolValue *Bool `protobuf:"bytes,3,opt,name=bool_value,json=boolValue,proto3,oneof" yaml:"bool_value"` // @gotags: yaml:"bool_value"
}
func (*MultiChoiceCollectionOptionValue_IntValue) isMultiChoiceCollectionOptionValue_Option() {}
@@ -1496,8 +1496,8 @@ type Permission struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Operation Permission_Operation `protobuf:"varint,1,opt,name=operation,proto3,enum=opencloud.messages.settings.v0.Permission_Operation" json:"operation,omitempty"` // @gotags: yaml:"operation"
Constraint Permission_Constraint `protobuf:"varint,2,opt,name=constraint,proto3,enum=opencloud.messages.settings.v0.Permission_Constraint" json:"constraint,omitempty"` // @gotags: yaml:"constraint"
Operation Permission_Operation `protobuf:"varint,1,opt,name=operation,proto3,enum=opencloud.messages.settings.v0.Permission_Operation" json:"operation,omitempty" yaml:"operation"` // @gotags: yaml:"operation"
Constraint Permission_Constraint `protobuf:"varint,2,opt,name=constraint,proto3,enum=opencloud.messages.settings.v0.Permission_Constraint" json:"constraint,omitempty" yaml:"constraint"` // @gotags: yaml:"constraint"
}
func (x *Permission) Reset() {
@@ -1552,12 +1552,12 @@ type Value struct {
unknownFields protoimpl.UnknownFields
// id is the id of the Value. It is generated on saving it.
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // @gotags: yaml:"id"
BundleId string `protobuf:"bytes,2,opt,name=bundle_id,json=bundleId,proto3" json:"bundle_id,omitempty"` // @gotags: yaml:"bundle_id"
Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty" yaml:"id"` // @gotags: yaml:"id"
BundleId string `protobuf:"bytes,2,opt,name=bundle_id,json=bundleId,proto3" json:"bundle_id,omitempty" yaml:"bundle_id"` // @gotags: yaml:"bundle_id"
// setting_id is the id of the setting from within its bundle.
SettingId string `protobuf:"bytes,3,opt,name=setting_id,json=settingId,proto3" json:"setting_id,omitempty"` // @gotags: yaml:"setting_id"
AccountUuid string `protobuf:"bytes,4,opt,name=account_uuid,json=accountUuid,proto3" json:"account_uuid,omitempty"` // @gotags: yaml:"account_uuid"
Resource *Resource `protobuf:"bytes,5,opt,name=resource,proto3" json:"resource,omitempty"` // @gotags: yaml:"resource"
SettingId string `protobuf:"bytes,3,opt,name=setting_id,json=settingId,proto3" json:"setting_id,omitempty" yaml:"setting_id"` // @gotags: yaml:"setting_id"
AccountUuid string `protobuf:"bytes,4,opt,name=account_uuid,json=accountUuid,proto3" json:"account_uuid,omitempty" yaml:"account_uuid"` // @gotags: yaml:"account_uuid"
Resource *Resource `protobuf:"bytes,5,opt,name=resource,proto3" json:"resource,omitempty" yaml:"resource"` // @gotags: yaml:"resource"
// Types that are assignable to Value:
//
// *Value_BoolValue
@@ -1682,23 +1682,23 @@ type isValue_Value interface {
}
type Value_BoolValue struct {
BoolValue bool `protobuf:"varint,6,opt,name=bool_value,json=boolValue,proto3,oneof"` // @gotags: yaml:"bool_value"
BoolValue bool `protobuf:"varint,6,opt,name=bool_value,json=boolValue,proto3,oneof" yaml:"bool_value"` // @gotags: yaml:"bool_value"
}
type Value_IntValue struct {
IntValue int64 `protobuf:"varint,7,opt,name=int_value,json=intValue,proto3,oneof"` // @gotags: yaml:"int_value"
IntValue int64 `protobuf:"varint,7,opt,name=int_value,json=intValue,proto3,oneof" yaml:"int_value"` // @gotags: yaml:"int_value"
}
type Value_StringValue struct {
StringValue string `protobuf:"bytes,8,opt,name=string_value,json=stringValue,proto3,oneof"` // @gotags: yaml:"string_value"
StringValue string `protobuf:"bytes,8,opt,name=string_value,json=stringValue,proto3,oneof" yaml:"string_value"` // @gotags: yaml:"string_value"
}
type Value_ListValue struct {
ListValue *ListValue `protobuf:"bytes,9,opt,name=list_value,json=listValue,proto3,oneof"` // @gotags: yaml:"list_value"
ListValue *ListValue `protobuf:"bytes,9,opt,name=list_value,json=listValue,proto3,oneof" yaml:"list_value"` // @gotags: yaml:"list_value"
}
type Value_CollectionValue struct {
CollectionValue *CollectionValue `protobuf:"bytes,10,opt,name=collection_value,json=collectionValue,proto3,oneof"` // @gotags: yaml:"collection_value"
CollectionValue *CollectionValue `protobuf:"bytes,10,opt,name=collection_value,json=collectionValue,proto3,oneof" yaml:"collection_value"` // @gotags: yaml:"collection_value"
}
func (*Value_BoolValue) isValue_Value() {}
@@ -1716,7 +1716,7 @@ type ListValue struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Values []*ListOptionValue `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty"` // @gotags: yaml:"values"
Values []*ListOptionValue `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty" yaml:"values"` // @gotags: yaml:"values"
}
func (x *ListValue) Reset() {
@@ -1836,15 +1836,15 @@ type isListOptionValue_Option interface {
}
type ListOptionValue_StringValue struct {
StringValue string `protobuf:"bytes,1,opt,name=string_value,json=stringValue,proto3,oneof"` // @gotags: yaml:"string_value"
StringValue string `protobuf:"bytes,1,opt,name=string_value,json=stringValue,proto3,oneof" yaml:"string_value"` // @gotags: yaml:"string_value"
}
type ListOptionValue_IntValue struct {
IntValue int64 `protobuf:"varint,2,opt,name=int_value,json=intValue,proto3,oneof"` // @gotags: yaml:"int_value"
IntValue int64 `protobuf:"varint,2,opt,name=int_value,json=intValue,proto3,oneof" yaml:"int_value"` // @gotags: yaml:"int_value"
}
type ListOptionValue_BoolValue struct {
BoolValue bool `protobuf:"varint,3,opt,name=bool_value,json=boolValue,proto3,oneof"` // @gotags: yaml:"bool_value"
BoolValue bool `protobuf:"varint,3,opt,name=bool_value,json=boolValue,proto3,oneof" yaml:"bool_value"` // @gotags: yaml:"bool_value"
}
func (*ListOptionValue_StringValue) isListOptionValue_Option() {}
@@ -1858,7 +1858,7 @@ type CollectionValue struct {
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
Values []*CollectionOption `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty"` // @gotags: yaml:"values"
Values []*CollectionOption `protobuf:"bytes,1,rep,name=values,proto3" json:"values,omitempty" yaml:"values"` // @gotags: yaml:"values"
}
func (x *CollectionValue) Reset() {
@@ -1906,7 +1906,7 @@ type CollectionOption struct {
unknownFields protoimpl.UnknownFields
// required
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty"` // @gotags: yaml:"key"
Key string `protobuf:"bytes,1,opt,name=key,proto3" json:"key,omitempty" yaml:"key"` // @gotags: yaml:"key"
// Types that are assignable to Option:
//
// *CollectionOption_IntValue
@@ -1987,15 +1987,15 @@ type isCollectionOption_Option interface {
}
type CollectionOption_IntValue struct {
IntValue int64 `protobuf:"varint,2,opt,name=int_value,json=intValue,proto3,oneof"` // @gotags: yaml:"int_value"
IntValue int64 `protobuf:"varint,2,opt,name=int_value,json=intValue,proto3,oneof" yaml:"int_value"` // @gotags: yaml:"int_value"
}
type CollectionOption_StringValue struct {
StringValue string `protobuf:"bytes,3,opt,name=string_value,json=stringValue,proto3,oneof"` // @gotags: yaml:"string_value"
StringValue string `protobuf:"bytes,3,opt,name=string_value,json=stringValue,proto3,oneof" yaml:"string_value"` // @gotags: yaml:"string_value"
}
type CollectionOption_BoolValue struct {
BoolValue bool `protobuf:"varint,4,opt,name=bool_value,json=boolValue,proto3,oneof"` // @gotags: yaml:"bool_value"
BoolValue bool `protobuf:"varint,4,opt,name=bool_value,json=boolValue,proto3,oneof" yaml:"bool_value"` // @gotags: yaml:"bool_value"
}
func (*CollectionOption_IntValue) isCollectionOption_Option() {}
@@ -40,7 +40,7 @@ func (_m *SearchProviderService) EXPECT() *SearchProviderService_Expecter {
}
// IndexSpace provides a mock function for the type SearchProviderService
func (_mock *SearchProviderService) IndexSpace(ctx context.Context, in *v0.IndexSpaceRequest, opts ...client.CallOption) (*v0.IndexSpaceResponse, error) {
func (_mock *SearchProviderService) IndexSpace(ctx context.Context, in *v0.IndexSpaceRequest, opts ...client.CallOption) (v0.SearchProvider_IndexSpaceService, error) {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(ctx, in, opts)
@@ -53,16 +53,16 @@ func (_mock *SearchProviderService) IndexSpace(ctx context.Context, in *v0.Index
panic("no return value specified for IndexSpace")
}
var r0 *v0.IndexSpaceResponse
var r0 v0.SearchProvider_IndexSpaceService
var r1 error
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.IndexSpaceRequest, ...client.CallOption) (*v0.IndexSpaceResponse, error)); ok {
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.IndexSpaceRequest, ...client.CallOption) (v0.SearchProvider_IndexSpaceService, error)); ok {
return returnFunc(ctx, in, opts...)
}
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.IndexSpaceRequest, ...client.CallOption) *v0.IndexSpaceResponse); ok {
if returnFunc, ok := ret.Get(0).(func(context.Context, *v0.IndexSpaceRequest, ...client.CallOption) v0.SearchProvider_IndexSpaceService); ok {
r0 = returnFunc(ctx, in, opts...)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*v0.IndexSpaceResponse)
r0 = ret.Get(0).(v0.SearchProvider_IndexSpaceService)
}
}
if returnFunc, ok := ret.Get(1).(func(context.Context, *v0.IndexSpaceRequest, ...client.CallOption) error); ok {
@@ -112,12 +112,12 @@ func (_c *SearchProviderService_IndexSpace_Call) Run(run func(ctx context.Contex
return _c
}
func (_c *SearchProviderService_IndexSpace_Call) Return(indexSpaceResponse *v0.IndexSpaceResponse, err error) *SearchProviderService_IndexSpace_Call {
_c.Call.Return(indexSpaceResponse, err)
func (_c *SearchProviderService_IndexSpace_Call) Return(searchProvider_IndexSpaceService v0.SearchProvider_IndexSpaceService, err error) *SearchProviderService_IndexSpace_Call {
_c.Call.Return(searchProvider_IndexSpaceService, err)
return _c
}
func (_c *SearchProviderService_IndexSpace_Call) RunAndReturn(run func(ctx context.Context, in *v0.IndexSpaceRequest, opts ...client.CallOption) (*v0.IndexSpaceResponse, error)) *SearchProviderService_IndexSpace_Call {
func (_c *SearchProviderService_IndexSpace_Call) RunAndReturn(run func(ctx context.Context, in *v0.IndexSpaceRequest, opts ...client.CallOption) (v0.SearchProvider_IndexSpaceService, error)) *SearchProviderService_IndexSpace_Call {
_c.Call.Return(run)
return _c
}
@@ -12,6 +12,7 @@ import (
_ "google.golang.org/genproto/googleapis/api/annotations"
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
durationpb "google.golang.org/protobuf/types/known/durationpb"
_ "google.golang.org/protobuf/types/known/fieldmaskpb"
reflect "reflect"
sync "sync"
@@ -310,6 +311,7 @@ type IndexSpaceRequest struct {
SpaceId string `protobuf:"bytes,1,opt,name=space_id,json=spaceId,proto3" json:"space_id,omitempty"`
UserId string `protobuf:"bytes,2,opt,name=user_id,json=userId,proto3" json:"user_id,omitempty"`
ForceReindex bool `protobuf:"varint,3,opt,name=force_reindex,json=forceReindex,proto3" json:"force_reindex,omitempty"`
Concurrency int32 `protobuf:"varint,4,opt,name=concurrency,proto3" json:"concurrency,omitempty"`
}
func (x *IndexSpaceRequest) Reset() {
@@ -365,10 +367,28 @@ func (x *IndexSpaceRequest) GetForceReindex() bool {
return false
}
func (x *IndexSpaceRequest) GetConcurrency() int32 {
if x != nil {
return x.Concurrency
}
return 0
}
type IndexSpaceResponse struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields
// The id of the space that has just been indexed.
SpaceId string `protobuf:"bytes,1,opt,name=space_id,json=spaceId,proto3" json:"space_id,omitempty"`
// The duration it took to index this space.
SpaceDuration *durationpb.Duration `protobuf:"bytes,2,opt,name=space_duration,json=spaceDuration,proto3" json:"space_duration,omitempty"`
// The number of spaces that have been indexed so far.
IndexedSpaces int64 `protobuf:"varint,3,opt,name=indexed_spaces,json=indexedSpaces,proto3" json:"indexed_spaces,omitempty"`
// The total number of spaces that are being indexed.
TotalSpaces int64 `protobuf:"varint,4,opt,name=total_spaces,json=totalSpaces,proto3" json:"total_spaces,omitempty"`
// Contains an error message in case indexing this particular space failed.
Error string `protobuf:"bytes,5,opt,name=error,proto3" json:"error,omitempty"`
}
func (x *IndexSpaceResponse) Reset() {
@@ -403,6 +423,41 @@ func (*IndexSpaceResponse) Descriptor() ([]byte, []int) {
return file_opencloud_services_search_v0_search_proto_rawDescGZIP(), []int{5}
}
func (x *IndexSpaceResponse) GetSpaceId() string {
if x != nil {
return x.SpaceId
}
return ""
}
func (x *IndexSpaceResponse) GetSpaceDuration() *durationpb.Duration {
if x != nil {
return x.SpaceDuration
}
return nil
}
func (x *IndexSpaceResponse) GetIndexedSpaces() int64 {
if x != nil {
return x.IndexedSpaces
}
return 0
}
func (x *IndexSpaceResponse) GetTotalSpaces() int64 {
if x != nil {
return x.TotalSpaces
}
return 0
}
func (x *IndexSpaceResponse) GetError() string {
if x != nil {
return x.Error
}
return ""
}
var File_opencloud_services_search_v0_search_proto protoreflect.FileDescriptor
var file_opencloud_services_search_v0_search_proto_rawDesc = []byte{
@@ -422,6 +477,8 @@ var file_opencloud_services_search_v0_search_proto_rawDesc = []byte{
0x69, 0x2f, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x2e, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x1a, 0x20, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74,
0x6f, 0x62, 0x75, 0x66, 0x2f, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x5f, 0x6d, 0x61, 0x73, 0x6b, 0x2e,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72,
0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e,
0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xae, 0x01, 0x0a, 0x0d, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68,
0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x21, 0x0a, 0x09, 0x70, 0x61, 0x67, 0x65, 0x5f,
0x73, 0x69, 0x7a, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x05, 0x42, 0x04, 0xe2, 0x41, 0x01, 0x01,
@@ -465,68 +522,83 @@ var file_opencloud_services_search_v0_search_proto_rawDesc = []byte{
0x74, 0x50, 0x61, 0x67, 0x65, 0x54, 0x6f, 0x6b, 0x65, 0x6e, 0x12, 0x23, 0x0a, 0x0d, 0x74, 0x6f,
0x74, 0x61, 0x6c, 0x5f, 0x6d, 0x61, 0x74, 0x63, 0x68, 0x65, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28,
0x05, 0x52, 0x0c, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x4d, 0x61, 0x74, 0x63, 0x68, 0x65, 0x73, 0x22,
0x6c, 0x0a, 0x11, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65, 0x71,
0x75, 0x65, 0x73, 0x74, 0x12, 0x19, 0x0a, 0x08, 0x73, 0x70, 0x61, 0x63, 0x65, 0x5f, 0x69, 0x64,
0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x73, 0x70, 0x61, 0x63, 0x65, 0x49, 0x64, 0x12,
0x17, 0x0a, 0x07, 0x75, 0x73, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09,
0x52, 0x06, 0x75, 0x73, 0x65, 0x72, 0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x66, 0x6f, 0x72, 0x63,
0x65, 0x5f, 0x72, 0x65, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52,
0x0c, 0x66, 0x6f, 0x72, 0x63, 0x65, 0x52, 0x65, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x22, 0x14, 0x0a,
0x12, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x32, 0xb1, 0x02, 0x0a, 0x0e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x50, 0x72,
0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x85, 0x01, 0x0a, 0x06, 0x53, 0x65, 0x61, 0x72, 0x63,
0x68, 0x12, 0x2b, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65,
0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30,
0x2e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x2c,
0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69,
0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65,
0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x20, 0x82, 0xd3,
0xe4, 0x93, 0x02, 0x1a, 0x3a, 0x01, 0x2a, 0x22, 0x15, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x30,
0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x12, 0x96,
0x01, 0x0a, 0x0a, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x12, 0x2f, 0x2e,
0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63,
0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x64,
0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x30,
0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69,
0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e,
0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65,
0x22, 0x25, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x1f, 0x3a, 0x01, 0x2a, 0x22, 0x1a, 0x2f, 0x61, 0x70,
0x69, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x69, 0x6e, 0x64, 0x65,
0x78, 0x2d, 0x73, 0x70, 0x61, 0x63, 0x65, 0x32, 0xa7, 0x01, 0x0a, 0x0d, 0x49, 0x6e, 0x64, 0x65,
0x78, 0x50, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x95, 0x01, 0x0a, 0x06, 0x53, 0x65,
0x61, 0x72, 0x63, 0x68, 0x12, 0x30, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x94, 0x01, 0x0a, 0x11, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65,
0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x19, 0x0a, 0x08, 0x73, 0x70, 0x61, 0x63, 0x65, 0x5f, 0x69,
0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x73, 0x70, 0x61, 0x63, 0x65, 0x49, 0x64,
0x12, 0x17, 0x0a, 0x07, 0x75, 0x73, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28,
0x09, 0x52, 0x06, 0x75, 0x73, 0x65, 0x72, 0x49, 0x64, 0x12, 0x23, 0x0a, 0x0d, 0x66, 0x6f, 0x72,
0x63, 0x65, 0x5f, 0x72, 0x65, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08,
0x52, 0x0c, 0x66, 0x6f, 0x72, 0x63, 0x65, 0x52, 0x65, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x12, 0x26,
0x0a, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x18, 0x04, 0x20,
0x01, 0x28, 0x05, 0x42, 0x04, 0xe2, 0x41, 0x01, 0x01, 0x52, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75,
0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x22, 0xd1, 0x01, 0x0a, 0x12, 0x49, 0x6e, 0x64, 0x65, 0x78,
0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x19, 0x0a,
0x08, 0x73, 0x70, 0x61, 0x63, 0x65, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
0x07, 0x73, 0x70, 0x61, 0x63, 0x65, 0x49, 0x64, 0x12, 0x40, 0x0a, 0x0e, 0x73, 0x70, 0x61, 0x63,
0x65, 0x5f, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b,
0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62,
0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0d, 0x73, 0x70, 0x61,
0x63, 0x65, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x25, 0x0a, 0x0e, 0x69, 0x6e,
0x64, 0x65, 0x78, 0x65, 0x64, 0x5f, 0x73, 0x70, 0x61, 0x63, 0x65, 0x73, 0x18, 0x03, 0x20, 0x01,
0x28, 0x03, 0x52, 0x0d, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x65, 0x64, 0x53, 0x70, 0x61, 0x63, 0x65,
0x73, 0x12, 0x21, 0x0a, 0x0c, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x5f, 0x73, 0x70, 0x61, 0x63, 0x65,
0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0b, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x53, 0x70,
0x61, 0x63, 0x65, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, 0x05, 0x20,
0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x32, 0xb3, 0x02, 0x0a, 0x0e, 0x53,
0x65, 0x61, 0x72, 0x63, 0x68, 0x50, 0x72, 0x6f, 0x76, 0x69, 0x64, 0x65, 0x72, 0x12, 0x85, 0x01,
0x0a, 0x06, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x12, 0x2b, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63,
0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65,
0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65,
0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x2c, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75,
0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63,
0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x22, 0x20, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x1a, 0x3a, 0x01, 0x2a, 0x22, 0x15,
0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x73,
0x65, 0x61, 0x72, 0x63, 0x68, 0x12, 0x98, 0x01, 0x0a, 0x0a, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53,
0x70, 0x61, 0x63, 0x65, 0x12, 0x2f, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68,
0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x52,
0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x31, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f,
0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72,
0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x49, 0x6e, 0x64, 0x65,
0x78, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x26, 0x82, 0xd3, 0xe4, 0x93, 0x02,
0x20, 0x3a, 0x01, 0x2a, 0x22, 0x1b, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x65,
0x61, 0x72, 0x63, 0x68, 0x2f, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63,
0x68, 0x42, 0xf2, 0x02, 0x5a, 0x4a, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d,
0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70,
0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x67, 0x65, 0x6e,
0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x73,
0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x76, 0x30,
0x92, 0x41, 0xa2, 0x02, 0x12, 0xb7, 0x01, 0x0a, 0x10, 0x4f, 0x70, 0x65, 0x6e, 0x43, 0x6c, 0x6f,
0x75, 0x64, 0x20, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x22, 0x51, 0x0a, 0x0e, 0x4f, 0x70, 0x65,
0x6e, 0x43, 0x6c, 0x6f, 0x75, 0x64, 0x20, 0x47, 0x6d, 0x62, 0x48, 0x12, 0x29, 0x68, 0x74, 0x74,
0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f,
0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65,
0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x1a, 0x14, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x40,
0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x65, 0x75, 0x2a, 0x49, 0x0a, 0x0a,
0x41, 0x70, 0x61, 0x63, 0x68, 0x65, 0x2d, 0x32, 0x2e, 0x30, 0x12, 0x3b, 0x68, 0x74, 0x74, 0x70,
0x73, 0x3a, 0x2f, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f,
0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e,
0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x62, 0x6c, 0x6f, 0x62, 0x2f, 0x6d, 0x61, 0x69, 0x6e, 0x2f,
0x4c, 0x49, 0x43, 0x45, 0x4e, 0x53, 0x45, 0x32, 0x05, 0x31, 0x2e, 0x30, 0x2e, 0x30, 0x2a, 0x02,
0x01, 0x02, 0x32, 0x10, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f,
0x6a, 0x73, 0x6f, 0x6e, 0x3a, 0x10, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f,
0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x72, 0x3e, 0x0a, 0x10, 0x44, 0x65, 0x76, 0x65, 0x6c, 0x6f,
0x70, 0x65, 0x72, 0x20, 0x4d, 0x61, 0x6e, 0x75, 0x61, 0x6c, 0x12, 0x2a, 0x68, 0x74, 0x74, 0x70,
0x73, 0x3a, 0x2f, 0x2f, 0x64, 0x6f, 0x63, 0x73, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f,
0x75, 0x64, 0x2e, 0x65, 0x75, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2f, 0x73,
0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52, 0x65,
0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x30, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75,
0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63,
0x68, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x53, 0x70, 0x61, 0x63, 0x65, 0x52,
0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x25, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x1f, 0x3a,
0x01, 0x2a, 0x22, 0x1a, 0x2f, 0x61, 0x70, 0x69, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x65, 0x61, 0x72,
0x63, 0x68, 0x2f, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x2d, 0x73, 0x70, 0x61, 0x63, 0x65, 0x30, 0x01,
0x32, 0xa7, 0x01, 0x0a, 0x0d, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x50, 0x72, 0x6f, 0x76, 0x69, 0x64,
0x65, 0x72, 0x12, 0x95, 0x01, 0x0a, 0x06, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x12, 0x30, 0x2e,
0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63,
0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x65, 0x61,
0x72, 0x63, 0x68, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a,
0x31, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x73, 0x65, 0x72, 0x76,
0x69, 0x63, 0x65, 0x73, 0x2e, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2e, 0x76, 0x30, 0x2e, 0x53,
0x65, 0x61, 0x72, 0x63, 0x68, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e,
0x73, 0x65, 0x22, 0x26, 0x82, 0xd3, 0xe4, 0x93, 0x02, 0x20, 0x3a, 0x01, 0x2a, 0x22, 0x1b, 0x2f,
0x61, 0x70, 0x69, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x69, 0x6e,
0x64, 0x65, 0x78, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x42, 0xf2, 0x02, 0x5a, 0x4a, 0x67,
0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c,
0x6f, 0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64,
0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x67, 0x65, 0x6e, 0x2f, 0x67, 0x65, 0x6e, 0x2f, 0x6f, 0x70,
0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x2f,
0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x76, 0x30, 0x92, 0x41, 0xa2, 0x02, 0x12, 0xb7, 0x01,
0x0a, 0x10, 0x4f, 0x70, 0x65, 0x6e, 0x43, 0x6c, 0x6f, 0x75, 0x64, 0x20, 0x73, 0x65, 0x61, 0x72,
0x63, 0x68, 0x22, 0x51, 0x0a, 0x0e, 0x4f, 0x70, 0x65, 0x6e, 0x43, 0x6c, 0x6f, 0x75, 0x64, 0x20,
0x47, 0x6d, 0x62, 0x48, 0x12, 0x29, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x67, 0x69,
0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f,
0x75, 0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x1a,
0x14, 0x73, 0x75, 0x70, 0x70, 0x6f, 0x72, 0x74, 0x40, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f,
0x75, 0x64, 0x2e, 0x65, 0x75, 0x2a, 0x49, 0x0a, 0x0a, 0x41, 0x70, 0x61, 0x63, 0x68, 0x65, 0x2d,
0x32, 0x2e, 0x30, 0x12, 0x3b, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x67, 0x69, 0x74,
0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75,
0x64, 0x2d, 0x65, 0x75, 0x2f, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x62,
0x6c, 0x6f, 0x62, 0x2f, 0x6d, 0x61, 0x69, 0x6e, 0x2f, 0x4c, 0x49, 0x43, 0x45, 0x4e, 0x53, 0x45,
0x32, 0x05, 0x31, 0x2e, 0x30, 0x2e, 0x30, 0x2a, 0x02, 0x01, 0x02, 0x32, 0x10, 0x61, 0x70, 0x70,
0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x3a, 0x10, 0x61,
0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x72,
0x3e, 0x0a, 0x10, 0x44, 0x65, 0x76, 0x65, 0x6c, 0x6f, 0x70, 0x65, 0x72, 0x20, 0x4d, 0x61, 0x6e,
0x75, 0x61, 0x6c, 0x12, 0x2a, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x64, 0x6f, 0x63,
0x73, 0x2e, 0x6f, 0x70, 0x65, 0x6e, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2e, 0x65, 0x75, 0x2f, 0x73,
0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x73, 0x2f, 0x73, 0x65, 0x61, 0x72, 0x63, 0x68, 0x2f, 0x62,
0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
}
var (
@@ -551,23 +623,25 @@ var file_opencloud_services_search_v0_search_proto_goTypes = []interface{}{
(*IndexSpaceResponse)(nil), // 5: opencloud.services.search.v0.IndexSpaceResponse
(*v0.Reference)(nil), // 6: opencloud.messages.search.v0.Reference
(*v0.Match)(nil), // 7: opencloud.messages.search.v0.Match
(*durationpb.Duration)(nil), // 8: google.protobuf.Duration
}
var file_opencloud_services_search_v0_search_proto_depIdxs = []int32{
6, // 0: opencloud.services.search.v0.SearchRequest.ref:type_name -> opencloud.messages.search.v0.Reference
7, // 1: opencloud.services.search.v0.SearchResponse.matches:type_name -> opencloud.messages.search.v0.Match
6, // 2: opencloud.services.search.v0.SearchIndexRequest.ref:type_name -> opencloud.messages.search.v0.Reference
7, // 3: opencloud.services.search.v0.SearchIndexResponse.matches:type_name -> opencloud.messages.search.v0.Match
0, // 4: opencloud.services.search.v0.SearchProvider.Search:input_type -> opencloud.services.search.v0.SearchRequest
4, // 5: opencloud.services.search.v0.SearchProvider.IndexSpace:input_type -> opencloud.services.search.v0.IndexSpaceRequest
2, // 6: opencloud.services.search.v0.IndexProvider.Search:input_type -> opencloud.services.search.v0.SearchIndexRequest
1, // 7: opencloud.services.search.v0.SearchProvider.Search:output_type -> opencloud.services.search.v0.SearchResponse
5, // 8: opencloud.services.search.v0.SearchProvider.IndexSpace:output_type -> opencloud.services.search.v0.IndexSpaceResponse
3, // 9: opencloud.services.search.v0.IndexProvider.Search:output_type -> opencloud.services.search.v0.SearchIndexResponse
7, // [7:10] is the sub-list for method output_type
4, // [4:7] is the sub-list for method input_type
4, // [4:4] is the sub-list for extension type_name
4, // [4:4] is the sub-list for extension extendee
0, // [0:4] is the sub-list for field type_name
8, // 4: opencloud.services.search.v0.IndexSpaceResponse.space_duration:type_name -> google.protobuf.Duration
0, // 5: opencloud.services.search.v0.SearchProvider.Search:input_type -> opencloud.services.search.v0.SearchRequest
4, // 6: opencloud.services.search.v0.SearchProvider.IndexSpace:input_type -> opencloud.services.search.v0.IndexSpaceRequest
2, // 7: opencloud.services.search.v0.IndexProvider.Search:input_type -> opencloud.services.search.v0.SearchIndexRequest
1, // 8: opencloud.services.search.v0.SearchProvider.Search:output_type -> opencloud.services.search.v0.SearchResponse
5, // 9: opencloud.services.search.v0.SearchProvider.IndexSpace:output_type -> opencloud.services.search.v0.IndexSpaceResponse
3, // 10: opencloud.services.search.v0.IndexProvider.Search:output_type -> opencloud.services.search.v0.SearchIndexResponse
8, // [8:11] is the sub-list for method output_type
5, // [5:8] is the sub-list for method input_type
5, // [5:5] is the sub-list for extension type_name
5, // [5:5] is the sub-list for extension extendee
0, // [0:5] is the sub-list for field type_name
}
func init() { file_opencloud_services_search_v0_search_proto_init() }
@@ -9,6 +9,7 @@ import (
_ "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/search/v0"
_ "google.golang.org/genproto/googleapis/api/annotations"
proto "google.golang.org/protobuf/proto"
_ "google.golang.org/protobuf/types/known/durationpb"
_ "google.golang.org/protobuf/types/known/fieldmaskpb"
math "math"
)
@@ -45,6 +46,7 @@ func NewSearchProviderEndpoints() []*api.Endpoint {
Name: "SearchProvider.IndexSpace",
Path: []string{"/api/v0/search/index-space"},
Method: []string{"POST"},
Stream: true,
Handler: "rpc",
},
}
@@ -54,7 +56,9 @@ func NewSearchProviderEndpoints() []*api.Endpoint {
type SearchProviderService interface {
Search(ctx context.Context, in *SearchRequest, opts ...client.CallOption) (*SearchResponse, error)
IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...client.CallOption) (*IndexSpaceResponse, error)
// IndexSpace (re)indexes one or all spaces. The response is streamed, sending
// progress information after each space has been indexed.
IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...client.CallOption) (SearchProvider_IndexSpaceService, error)
}
type searchProviderService struct {
@@ -79,27 +83,73 @@ func (c *searchProviderService) Search(ctx context.Context, in *SearchRequest, o
return out, nil
}
func (c *searchProviderService) IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...client.CallOption) (*IndexSpaceResponse, error) {
req := c.c.NewRequest(c.name, "SearchProvider.IndexSpace", in)
out := new(IndexSpaceResponse)
err := c.c.Call(ctx, req, out, opts...)
func (c *searchProviderService) IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...client.CallOption) (SearchProvider_IndexSpaceService, error) {
req := c.c.NewRequest(c.name, "SearchProvider.IndexSpace", &IndexSpaceRequest{})
stream, err := c.c.Stream(ctx, req, opts...)
if err != nil {
return nil, err
}
return out, nil
if err := stream.Send(in); err != nil {
return nil, err
}
return &searchProviderServiceIndexSpace{stream}, nil
}
type SearchProvider_IndexSpaceService interface {
Context() context.Context
SendMsg(interface{}) error
RecvMsg(interface{}) error
CloseSend() error
Close() error
Recv() (*IndexSpaceResponse, error)
}
type searchProviderServiceIndexSpace struct {
stream client.Stream
}
func (x *searchProviderServiceIndexSpace) CloseSend() error {
return x.stream.CloseSend()
}
func (x *searchProviderServiceIndexSpace) Close() error {
return x.stream.Close()
}
func (x *searchProviderServiceIndexSpace) Context() context.Context {
return x.stream.Context()
}
func (x *searchProviderServiceIndexSpace) SendMsg(m interface{}) error {
return x.stream.Send(m)
}
func (x *searchProviderServiceIndexSpace) RecvMsg(m interface{}) error {
return x.stream.Recv(m)
}
func (x *searchProviderServiceIndexSpace) Recv() (*IndexSpaceResponse, error) {
m := new(IndexSpaceResponse)
err := x.stream.Recv(m)
if err != nil {
return nil, err
}
return m, nil
}
// Server API for SearchProvider service
type SearchProviderHandler interface {
Search(context.Context, *SearchRequest, *SearchResponse) error
IndexSpace(context.Context, *IndexSpaceRequest, *IndexSpaceResponse) error
// IndexSpace (re)indexes one or all spaces. The response is streamed, sending
// progress information after each space has been indexed.
IndexSpace(context.Context, *IndexSpaceRequest, SearchProvider_IndexSpaceStream) error
}
func RegisterSearchProviderHandler(s server.Server, hdlr SearchProviderHandler, opts ...server.HandlerOption) error {
type searchProvider interface {
Search(ctx context.Context, in *SearchRequest, out *SearchResponse) error
IndexSpace(ctx context.Context, in *IndexSpaceRequest, out *IndexSpaceResponse) error
IndexSpace(ctx context.Context, stream server.Stream) error
}
type SearchProvider struct {
searchProvider
@@ -115,6 +165,7 @@ func RegisterSearchProviderHandler(s server.Server, hdlr SearchProviderHandler,
Name: "SearchProvider.IndexSpace",
Path: []string{"/api/v0/search/index-space"},
Method: []string{"POST"},
Stream: true,
Handler: "rpc",
}))
return s.Handle(s.NewHandler(&SearchProvider{h}, opts...))
@@ -128,8 +179,44 @@ func (h *searchProviderHandler) Search(ctx context.Context, in *SearchRequest, o
return h.SearchProviderHandler.Search(ctx, in, out)
}
func (h *searchProviderHandler) IndexSpace(ctx context.Context, in *IndexSpaceRequest, out *IndexSpaceResponse) error {
return h.SearchProviderHandler.IndexSpace(ctx, in, out)
func (h *searchProviderHandler) IndexSpace(ctx context.Context, stream server.Stream) error {
m := new(IndexSpaceRequest)
if err := stream.Recv(m); err != nil {
return err
}
return h.SearchProviderHandler.IndexSpace(ctx, m, &searchProviderIndexSpaceStream{stream})
}
type SearchProvider_IndexSpaceStream interface {
Context() context.Context
SendMsg(interface{}) error
RecvMsg(interface{}) error
Close() error
Send(*IndexSpaceResponse) error
}
type searchProviderIndexSpaceStream struct {
stream server.Stream
}
func (x *searchProviderIndexSpaceStream) Close() error {
return x.stream.Close()
}
func (x *searchProviderIndexSpaceStream) Context() context.Context {
return x.stream.Context()
}
func (x *searchProviderIndexSpaceStream) SendMsg(m interface{}) error {
return x.stream.Send(m)
}
func (x *searchProviderIndexSpaceStream) RecvMsg(m interface{}) error {
return x.stream.Recv(m)
}
func (x *searchProviderIndexSpaceStream) Send(m *IndexSpaceResponse) error {
return x.stream.Send(m)
}
// Api Endpoints for IndexProvider service
@@ -1,346 +0,0 @@
// Code generated by protoc-gen-microweb. DO NOT EDIT.
// source: v0.proto
package v0
import (
"bytes"
"encoding/json"
"net/http"
"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
"github.com/golang/protobuf/jsonpb"
merrors "go-micro.dev/v4/errors"
)
type webSearchProviderHandler struct {
r chi.Router
h SearchProviderHandler
}
func (h *webSearchProviderHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
h.r.ServeHTTP(w, r)
}
func (h *webSearchProviderHandler) Search(w http.ResponseWriter, r *http.Request) {
req := &SearchRequest{}
resp := &SearchResponse{}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusPreconditionFailed)
return
}
if err := h.h.Search(
r.Context(),
req,
resp,
); err != nil {
if merr, ok := merrors.As(err); ok && merr.Code == http.StatusNotFound {
http.Error(w, err.Error(), http.StatusNotFound)
} else {
http.Error(w, err.Error(), http.StatusBadRequest)
}
return
}
render.Status(r, http.StatusCreated)
render.JSON(w, r, resp)
}
func (h *webSearchProviderHandler) IndexSpace(w http.ResponseWriter, r *http.Request) {
req := &IndexSpaceRequest{}
resp := &IndexSpaceResponse{}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusPreconditionFailed)
return
}
if err := h.h.IndexSpace(
r.Context(),
req,
resp,
); err != nil {
if merr, ok := merrors.As(err); ok && merr.Code == http.StatusNotFound {
http.Error(w, err.Error(), http.StatusNotFound)
} else {
http.Error(w, err.Error(), http.StatusBadRequest)
}
return
}
render.Status(r, http.StatusCreated)
render.JSON(w, r, resp)
}
func RegisterSearchProviderWeb(r chi.Router, i SearchProviderHandler, middlewares ...func(http.Handler) http.Handler) {
handler := &webSearchProviderHandler{
r: r,
h: i,
}
r.MethodFunc("POST", "/api/v0/search/search", handler.Search)
r.MethodFunc("POST", "/api/v0/search/index-space", handler.IndexSpace)
}
type webIndexProviderHandler struct {
r chi.Router
h IndexProviderHandler
}
func (h *webIndexProviderHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
h.r.ServeHTTP(w, r)
}
func (h *webIndexProviderHandler) Search(w http.ResponseWriter, r *http.Request) {
req := &SearchIndexRequest{}
resp := &SearchIndexResponse{}
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusPreconditionFailed)
return
}
if err := h.h.Search(
r.Context(),
req,
resp,
); err != nil {
if merr, ok := merrors.As(err); ok && merr.Code == http.StatusNotFound {
http.Error(w, err.Error(), http.StatusNotFound)
} else {
http.Error(w, err.Error(), http.StatusBadRequest)
}
return
}
render.Status(r, http.StatusCreated)
render.JSON(w, r, resp)
}
func RegisterIndexProviderWeb(r chi.Router, i IndexProviderHandler, middlewares ...func(http.Handler) http.Handler) {
handler := &webIndexProviderHandler{
r: r,
h: i,
}
r.MethodFunc("POST", "/api/v0/search/index/search", handler.Search)
}
// SearchRequestJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of SearchRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchRequestJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *SearchRequest) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := SearchRequestJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*SearchRequest)(nil)
// SearchRequestJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of SearchRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchRequestJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *SearchRequest) UnmarshalJSON(b []byte) error {
return SearchRequestJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*SearchRequest)(nil)
// SearchResponseJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of SearchResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchResponseJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *SearchResponse) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := SearchResponseJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*SearchResponse)(nil)
// SearchResponseJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of SearchResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchResponseJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *SearchResponse) UnmarshalJSON(b []byte) error {
return SearchResponseJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*SearchResponse)(nil)
// SearchIndexRequestJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of SearchIndexRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchIndexRequestJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *SearchIndexRequest) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := SearchIndexRequestJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*SearchIndexRequest)(nil)
// SearchIndexRequestJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of SearchIndexRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchIndexRequestJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *SearchIndexRequest) UnmarshalJSON(b []byte) error {
return SearchIndexRequestJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*SearchIndexRequest)(nil)
// SearchIndexResponseJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of SearchIndexResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchIndexResponseJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *SearchIndexResponse) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := SearchIndexResponseJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*SearchIndexResponse)(nil)
// SearchIndexResponseJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of SearchIndexResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var SearchIndexResponseJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *SearchIndexResponse) UnmarshalJSON(b []byte) error {
return SearchIndexResponseJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*SearchIndexResponse)(nil)
// IndexSpaceRequestJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of IndexSpaceRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var IndexSpaceRequestJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *IndexSpaceRequest) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := IndexSpaceRequestJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*IndexSpaceRequest)(nil)
// IndexSpaceRequestJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of IndexSpaceRequest. This struct is safe to replace or modify but
// should not be done so concurrently.
var IndexSpaceRequestJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *IndexSpaceRequest) UnmarshalJSON(b []byte) error {
return IndexSpaceRequestJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*IndexSpaceRequest)(nil)
// IndexSpaceResponseJSONMarshaler describes the default jsonpb.Marshaler used by all
// instances of IndexSpaceResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var IndexSpaceResponseJSONMarshaler = new(jsonpb.Marshaler)
// MarshalJSON satisfies the encoding/json Marshaler interface. This method
// uses the more correct jsonpb package to correctly marshal the message.
func (m *IndexSpaceResponse) MarshalJSON() ([]byte, error) {
if m == nil {
return json.Marshal(nil)
}
buf := &bytes.Buffer{}
if err := IndexSpaceResponseJSONMarshaler.Marshal(buf, m); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
var _ json.Marshaler = (*IndexSpaceResponse)(nil)
// IndexSpaceResponseJSONUnmarshaler describes the default jsonpb.Unmarshaler used by all
// instances of IndexSpaceResponse. This struct is safe to replace or modify but
// should not be done so concurrently.
var IndexSpaceResponseJSONUnmarshaler = new(jsonpb.Unmarshaler)
// UnmarshalJSON satisfies the encoding/json Unmarshaler interface. This method
// uses the more correct jsonpb package to correctly unmarshal the message.
func (m *IndexSpaceResponse) UnmarshalJSON(b []byte) error {
return IndexSpaceResponseJSONUnmarshaler.Unmarshal(bytes.NewReader(b), m)
}
var _ json.Unmarshaler = (*IndexSpaceResponse)(nil)
@@ -34,12 +34,22 @@
"paths": {
"/api/v0/search/index-space": {
"post": {
"summary": "IndexSpace (re)indexes one or all spaces. The response is streamed, sending\nprogress information after each space has been indexed.",
"operationId": "SearchProvider_IndexSpace",
"responses": {
"200": {
"description": "A successful response.",
"description": "A successful response.(streaming responses)",
"schema": {
"$ref": "#/definitions/v0IndexSpaceResponse"
"type": "object",
"properties": {
"result": {
"$ref": "#/definitions/v0IndexSpaceResponse"
},
"error": {
"$ref": "#/definitions/rpcStatus"
}
},
"title": "Stream result of v0IndexSpaceResponse"
}
},
"default": {
@@ -332,11 +342,39 @@
},
"forceReindex": {
"type": "boolean"
},
"concurrency": {
"type": "integer",
"format": "int32"
}
}
},
"v0IndexSpaceResponse": {
"type": "object"
"type": "object",
"properties": {
"spaceId": {
"type": "string",
"description": "The id of the space that has just been indexed."
},
"spaceDuration": {
"type": "string",
"description": "The duration it took to index this space."
},
"indexedSpaces": {
"type": "string",
"format": "int64",
"description": "The number of spaces that have been indexed so far."
},
"totalSpaces": {
"type": "string",
"format": "int64",
"description": "The total number of spaces that are being indexed."
},
"error": {
"type": "string",
"description": "Contains an error message in case indexing this particular space failed."
}
}
},
"v0Match": {
"type": "object",
@@ -28,7 +28,9 @@ const (
// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream.
type SearchProviderClient interface {
Search(ctx context.Context, in *SearchRequest, opts ...grpc.CallOption) (*SearchResponse, error)
IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...grpc.CallOption) (*IndexSpaceResponse, error)
// IndexSpace (re)indexes one or all spaces. The response is streamed, sending
// progress information after each space has been indexed.
IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[IndexSpaceResponse], error)
}
type searchProviderClient struct {
@@ -49,22 +51,33 @@ func (c *searchProviderClient) Search(ctx context.Context, in *SearchRequest, op
return out, nil
}
func (c *searchProviderClient) IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...grpc.CallOption) (*IndexSpaceResponse, error) {
func (c *searchProviderClient) IndexSpace(ctx context.Context, in *IndexSpaceRequest, opts ...grpc.CallOption) (grpc.ServerStreamingClient[IndexSpaceResponse], error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(IndexSpaceResponse)
err := c.cc.Invoke(ctx, SearchProvider_IndexSpace_FullMethodName, in, out, cOpts...)
stream, err := c.cc.NewStream(ctx, &SearchProvider_ServiceDesc.Streams[0], SearchProvider_IndexSpace_FullMethodName, cOpts...)
if err != nil {
return nil, err
}
return out, nil
x := &grpc.GenericClientStream[IndexSpaceRequest, IndexSpaceResponse]{ClientStream: stream}
if err := x.ClientStream.SendMsg(in); err != nil {
return nil, err
}
if err := x.ClientStream.CloseSend(); err != nil {
return nil, err
}
return x, nil
}
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type SearchProvider_IndexSpaceClient = grpc.ServerStreamingClient[IndexSpaceResponse]
// SearchProviderServer is the server API for SearchProvider service.
// All implementations must embed UnimplementedSearchProviderServer
// for forward compatibility.
type SearchProviderServer interface {
Search(context.Context, *SearchRequest) (*SearchResponse, error)
IndexSpace(context.Context, *IndexSpaceRequest) (*IndexSpaceResponse, error)
// IndexSpace (re)indexes one or all spaces. The response is streamed, sending
// progress information after each space has been indexed.
IndexSpace(*IndexSpaceRequest, grpc.ServerStreamingServer[IndexSpaceResponse]) error
mustEmbedUnimplementedSearchProviderServer()
}
@@ -78,8 +91,8 @@ type UnimplementedSearchProviderServer struct{}
func (UnimplementedSearchProviderServer) Search(context.Context, *SearchRequest) (*SearchResponse, error) {
return nil, status.Error(codes.Unimplemented, "method Search not implemented")
}
func (UnimplementedSearchProviderServer) IndexSpace(context.Context, *IndexSpaceRequest) (*IndexSpaceResponse, error) {
return nil, status.Error(codes.Unimplemented, "method IndexSpace not implemented")
func (UnimplementedSearchProviderServer) IndexSpace(*IndexSpaceRequest, grpc.ServerStreamingServer[IndexSpaceResponse]) error {
return status.Error(codes.Unimplemented, "method IndexSpace not implemented")
}
func (UnimplementedSearchProviderServer) mustEmbedUnimplementedSearchProviderServer() {}
func (UnimplementedSearchProviderServer) testEmbeddedByValue() {}
@@ -120,24 +133,17 @@ func _SearchProvider_Search_Handler(srv interface{}, ctx context.Context, dec fu
return interceptor(ctx, in, info, handler)
}
func _SearchProvider_IndexSpace_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(IndexSpaceRequest)
if err := dec(in); err != nil {
return nil, err
func _SearchProvider_IndexSpace_Handler(srv interface{}, stream grpc.ServerStream) error {
m := new(IndexSpaceRequest)
if err := stream.RecvMsg(m); err != nil {
return err
}
if interceptor == nil {
return srv.(SearchProviderServer).IndexSpace(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: SearchProvider_IndexSpace_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(SearchProviderServer).IndexSpace(ctx, req.(*IndexSpaceRequest))
}
return interceptor(ctx, in, info, handler)
return srv.(SearchProviderServer).IndexSpace(m, &grpc.GenericServerStream[IndexSpaceRequest, IndexSpaceResponse]{ServerStream: stream})
}
// This type alias is provided for backwards compatibility with existing code that references the prior non-generic stream type by name.
type SearchProvider_IndexSpaceServer = grpc.ServerStreamingServer[IndexSpaceResponse]
// SearchProvider_ServiceDesc is the grpc.ServiceDesc for SearchProvider service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
@@ -149,12 +155,14 @@ var SearchProvider_ServiceDesc = grpc.ServiceDesc{
MethodName: "Search",
Handler: _SearchProvider_Search_Handler,
},
},
Streams: []grpc.StreamDesc{
{
MethodName: "IndexSpace",
Handler: _SearchProvider_IndexSpace_Handler,
StreamName: "IndexSpace",
Handler: _SearchProvider_IndexSpace_Handler,
ServerStreams: true,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "opencloud/services/search/v0/search.proto",
}
+3 -1
View File
@@ -31,7 +31,9 @@ plugins:
opencloud.services.eventhistory.v0;\
opencloud.messages.eventhistory.v0;\
opencloud.services.policies.v0;\
opencloud.messages.policies.v0"
opencloud.messages.policies.v0;\
opencloud.services.search.v0;\
opencloud.messages.search.v0"
- name: openapiv2
path: ../../.bingo/protoc-gen-openapiv2
@@ -9,6 +9,7 @@ import "protoc-gen-openapiv2/options/annotations.proto";
import "google/api/field_behavior.proto";
import "google/api/annotations.proto";
import "google/protobuf/field_mask.proto";
import "google/protobuf/duration.proto";
option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {
info: {
@@ -41,7 +42,9 @@ service SearchProvider {
body: "*"
};
};
rpc IndexSpace(IndexSpaceRequest) returns (IndexSpaceResponse) {
// IndexSpace (re)indexes one or all spaces. The response is streamed, sending
// progress information after each space has been indexed.
rpc IndexSpace(IndexSpaceRequest) returns (stream IndexSpaceResponse) {
option (google.api.http) = {
post: "/api/v0/search/index-space",
body: "*"
@@ -105,7 +108,18 @@ message IndexSpaceRequest {
string space_id = 1;
string user_id = 2;
bool force_reindex = 3;
int32 concurrency = 4 [(google.api.field_behavior) = OPTIONAL];
}
message IndexSpaceResponse {
// The id of the space that has just been indexed.
string space_id = 1;
// The duration it took to index this space.
google.protobuf.Duration space_duration = 2;
// The number of spaces that have been indexed so far.
int64 indexed_spaces = 3;
// The total number of spaces that are being indexed.
int64 total_spaces = 4;
// Contains an error message in case indexing this particular space failed.
string error = 5;
}
+10 -10
View File
@@ -30,19 +30,19 @@
},
"dependencies": {
"@material-ui/core": "^4.12.4",
"@types/node": "^25.7.0",
"@types/react": "^17.0.91",
"@types/node": "^25.9.5",
"@types/react": "^17.0.93",
"@types/react-dom": "^17.0.26",
"@types/react-redux": "^7.1.34",
"@types/redux-logger": "^3.0.13",
"axios": "^1.18.1",
"i18next": "^26.3.0",
"axios": "^1.19.0",
"i18next": "^26.3.6",
"i18next-browser-languagedetector": "^8.2.1",
"i18next-resources-to-backend": "^1.2.1",
"query-string": "^9.3.1",
"i18next-resources-to-backend": "^1.2.3",
"query-string": "^9.4.1",
"react": "^17.0.2",
"react-dom": "^17.0.2",
"react-i18next": "^17.0.8",
"react-i18next": "^17.0.11",
"react-redux": "^8.1.3",
"react-router": "^5.3.4",
"react-router-dom": "5.2.1",
@@ -61,9 +61,9 @@
"css-minimizer-webpack-plugin": "^8.0.0",
"dotenv": "17.4.2",
"dotenv-expand": "^13.0.0",
"gettext-parser": "^9.0.2",
"html-webpack-plugin": "^5.6.7",
"i18next-cli": "^1.65.0",
"gettext-parser": "^9.1.1",
"html-webpack-plugin": "^5.6.8",
"i18next-cli": "^1.67.3",
"i18next-conv": "^17.0.0",
"license-checker-rseidelsohn": "5.0.1",
"mini-css-extract-plugin": "2.9.2",
+764 -744
View File
File diff suppressed because it is too large. Load diff
+7 -4
View File
@@ -20,11 +20,14 @@ supportedArchitectures:
- musl
overrides:
fast-uri: ">=3.1.2"
undici: ">=7.28.0"
postcss: ">=8.5.10"
brace-expansion: ">=5.0.7"
fast-uri: ">=3.1.4"
js-yaml: ">=4.3.0"
postcss: ">=8.5.18"
"serialize-javascript@<7.0.3": ">=7.0.3"
shell-quote: ">=1.8.4"
shell-quote: ">=1.9.0"
svgo: ">=4.0.2"
undici: ">=7.28.0"
"@babel/plugin-transform-modules-systemjs": ">=7.29.4"
"@xmldom/xmldom": "^0.8.13"
@@ -137,6 +137,10 @@ func DefaultPolicies() []config.Policy {
Endpoint: "/branding/logo",
Service: "eu.opencloud.web.web",
},
{
Endpoint: "/announcement",
Service: "eu.opencloud.web.web",
},
{
Endpoint: "/konnect/",
Service: "eu.opencloud.web.idp",
+47 -3
View File
@@ -5,6 +5,10 @@ import (
"crypto/tls"
"errors"
"fmt"
"io"
"os"
"os/signal"
"syscall"
"time"
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
@@ -33,9 +37,14 @@ func Index(cfg *config.Config) *cobra.Command {
forceRescanFlag, _ := cmd.Flags().GetBool("force-rescan")
endpointFlag, _ := cmd.Flags().GetString("endpoint")
insecureFlag, _ := cmd.Flags().GetBool("insecure")
concurrencyFlag, _ := cmd.Flags().GetInt("concurrency")
if spaceFlag == "" && !allSpacesFlag {
return errors.New("either --space or --all-spaces is required")
}
if int(concurrencyFlag) > cfg.ReindexMaxConcurrency {
return fmt.Errorf("concurrency %d exceeds max allowed %d", concurrencyFlag, cfg.ReindexMaxConcurrency)
}
var dialOpts []grpc.DialOption
if cfg.GRPCClientTLS.Mode == "insecure" || insecureFlag {
@@ -54,17 +63,47 @@ func Index(cfg *config.Config) *cobra.Command {
c := searchsvc.NewSearchProviderClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
// Cancel the operation when the user presses Ctrl+C (SIGINT) or the
// process receives SIGTERM. The cancellation propagates over the
// gRPC stream so the server stops indexing.
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
ctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
_, err = c.IndexSpace(ctx, &searchsvc.IndexSpaceRequest{
stream, err := c.IndexSpace(ctx, &searchsvc.IndexSpaceRequest{
SpaceId: spaceFlag,
ForceReindex: forceRescanFlag,
Concurrency: int32(concurrencyFlag),
})
if err != nil {
fmt.Println("failed to index space: " + err.Error())
return err
}
for {
progress, err := stream.Recv()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
// The user aborted (Ctrl+C / SIGTERM). Exit quietly instead
// of dumping a "context canceled" gRPC error.
if errors.Is(ctx.Err(), context.Canceled) {
fmt.Println("aborted, indexing has been stopped")
return nil
}
return err
}
if progress.GetError() != "" {
fmt.Printf("[%d/%d] failed to index space %s: %s\n",
progress.GetIndexedSpaces(), progress.GetTotalSpaces(), progress.GetSpaceId(), progress.GetError())
continue
}
fmt.Printf("[%d/%d] indexed space %s in %s\n",
progress.GetIndexedSpaces(), progress.GetTotalSpaces(), progress.GetSpaceId(), progress.GetSpaceDuration().AsDuration())
}
return nil
},
}
@@ -94,6 +133,11 @@ func Index(cfg *config.Config) *cobra.Command {
false,
"disable TLS for the gRPC connection.",
)
indexCmd.Flags().Int(
"concurrency",
3,
"the number of concurrent indexing operations.",
)
return indexCmd
}
+1
View File
@@ -28,6 +28,7 @@ type Config struct {
Extractor Extractor `yaml:"extractor"`
ContentExtractionSizeLimit uint64 `yaml:"content_extraction_size_limit" env:"SEARCH_CONTENT_EXTRACTION_SIZE_LIMIT" desc:"Maximum file size in bytes that is allowed for content extraction." introductionVersion:"1.0.0"`
BatchSize int `yaml:"batch_size" env:"SEARCH_BATCH_SIZE" desc:"The number of documents to process in a single batch. Defaults to 500." introductionVersion:"1.0.0"`
ReindexMaxConcurrency int `yaml:"reindex_concurrency" env:"SEARCH_REINDEX_MAX_CONCURRENCY" desc:"The maximum number of spaces that are reindexed concurrently when reindexing all spaces." introductionVersion:"7.4.0"`
ServiceAccount ServiceAccount `yaml:"service_account"`
@@ -65,6 +65,7 @@ func DefaultConfig() *config.Config {
},
ContentExtractionSizeLimit: 20 * 1024 * 1024, // Limit content extraction to <20MB files by default
BatchSize: 50,
ReindexMaxConcurrency: 3,
}
}
+79 -7
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"sync"
"time"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
@@ -19,7 +20,9 @@ import (
"github.com/opencloud-eu/reva/v2/pkg/utils"
merrors "go-micro.dev/v4/errors"
"go-micro.dev/v4/metadata"
"golang.org/x/sync/errgroup"
grpcmetadata "google.golang.org/grpc/metadata"
"google.golang.org/protobuf/types/known/durationpb"
"github.com/opencloud-eu/opencloud/pkg/log"
v0 "github.com/opencloud-eu/opencloud/protogen/gen/opencloud/messages/search/v0"
@@ -118,10 +121,27 @@ func (s Service) Search(ctx context.Context, in *searchsvc.SearchRequest, out *s
return nil
}
// IndexSpace (re)indexes all resources of a given space.
func (s Service) IndexSpace(_ context.Context, in *searchsvc.IndexSpaceRequest, _ *searchsvc.IndexSpaceResponse) error {
// IndexSpace (re)indexes all resources of a given space. Progress information is
// streamed back to the caller after every space that has been indexed.
func (s Service) IndexSpace(_ context.Context, in *searchsvc.IndexSpaceRequest, stream searchsvc.SearchProvider_IndexSpaceStream) error {
// Use the stream's context so that indexing stops when the client cancels
// the request or disconnects.
ctx := stream.Context()
if in.GetSpaceId() != "" {
return s.searcher.IndexSpace(&provider.StorageSpaceId{OpaqueId: in.GetSpaceId()}, in.GetForceReindex())
err := s.searcher.IndexSpace(&provider.StorageSpaceId{OpaqueId: in.GetSpaceId()}, in.GetForceReindex())
resp := &searchsvc.IndexSpaceResponse{
SpaceId: in.GetSpaceId(),
IndexedSpaces: 1,
TotalSpaces: 1,
}
if err != nil {
resp.Error = err.Error()
}
if sendErr := stream.Send(resp); sendErr != nil {
return sendErr
}
return err
}
// index all spaces instead
@@ -130,7 +150,7 @@ func (s Service) IndexSpace(_ context.Context, in *searchsvc.IndexSpaceRequest,
return err
}
ctx, err := utils.GetServiceUserContext(s.cfg.ServiceAccount.ServiceAccountID, gwc, s.cfg.ServiceAccount.ServiceAccountSecret)
ctx, err = utils.GetServiceUserContextWithContext(ctx, gwc, s.cfg.ServiceAccount.ServiceAccountID, s.cfg.ServiceAccount.ServiceAccountSecret)
if err != nil {
return err
}
@@ -144,13 +164,65 @@ func (s Service) IndexSpace(_ context.Context, in *searchsvc.IndexSpaceRequest,
return errors.New(resp.GetStatus().GetMessage())
}
for _, space := range resp.GetStorageSpaces() {
if err := s.searcher.IndexSpace(space.GetId(), in.GetForceReindex()); err != nil {
spaces := resp.GetStorageSpaces()
totalSpaces := int64(len(spaces))
// Index all spaces concurrently, limited to a configurable number of spaces
// being reindexed at the same time. The errgroup context is cancelled as
// soon as the client goes away or a stream send fails, so the remaining
// goroutines stop indexing early.
concurrency := max(s.cfg.ReindexMaxConcurrency, 1)
concurrency = min(concurrency, int(in.GetConcurrency()))
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(concurrency)
// Serialize progress updates on the stream, as gRPC streams must not be
// written to from multiple goroutines concurrently.
var (
mu sync.Mutex
indexedCount int64
)
for _, space := range spaces {
// Stop early if the client cancelled the request.
if err := ctx.Err(); err != nil {
return err
}
g.Go(func() error {
s.log.Info().Str("space_id", space.GetId().GetOpaqueId()).Msg("indexing space")
t := time.Now()
indexErr := s.searcher.IndexSpace(space.GetId(), in.GetForceReindex())
if indexErr != nil {
s.log.Error().Err(indexErr).Str("space_id", space.GetId().GetOpaqueId()).Msg("failed to index space")
} else {
s.log.Info().Str("space_id", space.GetId().GetOpaqueId()).Msg("finished indexing space")
}
mu.Lock()
defer mu.Unlock()
// Don't try to send progress on an already cancelled stream.
if err := ctx.Err(); err != nil {
return err
}
indexedCount++
progress := &searchsvc.IndexSpaceResponse{
SpaceId: space.GetId().GetOpaqueId(),
IndexedSpaces: indexedCount,
TotalSpaces: totalSpaces,
SpaceDuration: durationpb.New(time.Since(t)),
}
if indexErr != nil {
progress.Error = indexErr.Error()
}
return stream.Send(progress)
})
}
return nil
return g.Wait()
}
// FromCache pulls a search result from cache
@@ -12,7 +12,7 @@ msgid ""
msgstr ""
"Project-Id-Version: \n"
"Report-Msgid-Bugs-To: EMAIL\n"
"POT-Creation-Date: 2026-07-09 23:15+0000\n"
"POT-Creation-Date: 2026-07-29 23:16+0000\n"
"PO-Revision-Date: 2025-01-27 10:17+0000\n"
"Last-Translator: Andre Nunes, 2026\n"
"Language-Team: Portuguese (https://app.transifex.com/opencloud-eu/teams/204053/pt/)\n"
@@ -80,6 +80,7 @@ func ServiceAccountBundle() *settingsmsg.Bundle {
},
Settings: []*settingsmsg.Setting{
AccountManagementPermission(All),
AnnouncementReadWritePermission(All),
ChangeLogoPermission(All),
CollaborationPublishNotificationPermission(All),
CollaborationManageFontsPermission(All),
@@ -117,6 +118,7 @@ func generateBundleAdminRole() *settingsmsg.Bundle {
},
Settings: []*settingsmsg.Setting{
AccountManagementPermission(All),
AnnouncementReadWritePermission(All),
AutoAcceptSharesPermission(Own),
ChangeLogoPermission(All),
CollaborationPublishNotificationPermission(All),
@@ -29,6 +29,25 @@ func AccountManagementPermission(c settingsmsg.Permission_Constraint) *settingsm
}
}
// AnnouncementReadWritePermission is the permission to read and manage the web announcement banner
func AnnouncementReadWritePermission(c settingsmsg.Permission_Constraint) *settingsmsg.Setting {
return &settingsmsg.Setting{
Id: "52b1994b-1bdb-4c8d-a887-1967dbe8cb11",
Name: "Announcement.ReadWrite",
DisplayName: "Manage announcement",
Description: "This permission permits to read and manage the announcement banner shown to all users.",
Resource: &settingsmsg.Resource{
Type: settingsmsg.Resource_TYPE_SYSTEM,
},
Value: &settingsmsg.Setting_PermissionValue{
PermissionValue: &settingsmsg.Permission{
Operation: settingsmsg.Permission_OPERATION_READWRITE,
Constraint: c,
},
},
}
}
// AutoAcceptSharesPermission is the permission to enable share auto-accept
func AutoAcceptSharesPermission(c settingsmsg.Permission_Constraint) *settingsmsg.Setting {
return &settingsmsg.Setting{
+14
View File
@@ -0,0 +1,14 @@
# maintain v2 separate mocks dir
dir: "{{.InterfaceDir}}/mocks"
structname: "{{.InterfaceName}}"
filename: "{{.InterfaceName | snakecase }}.go"
pkgname: mocks
template: testify
packages:
github.com/nats-io/nats.go/jetstream:
config:
dir: mocks
interfaces:
KeyValue: {}
KeyValueEntry: {}
+1 -1
View File
@@ -1,6 +1,6 @@
SHELL := bash
NAME := web
WEB_ASSETS_VERSION = v7.2.0
WEB_ASSETS_VERSION = v7.3.0
WEB_ASSETS_BRANCH = main
ifneq (, $(shell command -v go 2> /dev/null)) # suppress `command not found warnings` for non go targets in CI
File diff suppressed because it is too large. Load diff
+349
View File
@@ -0,0 +1,349 @@
// Code generated by mockery; DO NOT EDIT.
// github.com/vektra/mockery
// template: testify
package mocks
import (
"time"
"github.com/nats-io/nats.go/jetstream"
mock "github.com/stretchr/testify/mock"
)
// NewKeyValueEntry creates a new instance of KeyValueEntry. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewKeyValueEntry(t interface {
mock.TestingT
Cleanup(func())
}) *KeyValueEntry {
mock := &KeyValueEntry{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
// KeyValueEntry is an autogenerated mock type for the KeyValueEntry type
type KeyValueEntry struct {
mock.Mock
}
type KeyValueEntry_Expecter struct {
mock *mock.Mock
}
func (_m *KeyValueEntry) EXPECT() *KeyValueEntry_Expecter {
return &KeyValueEntry_Expecter{mock: &_m.Mock}
}
// Bucket provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Bucket() string {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Bucket")
}
var r0 string
if returnFunc, ok := ret.Get(0).(func() string); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(string)
}
return r0
}
// KeyValueEntry_Bucket_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Bucket'
type KeyValueEntry_Bucket_Call struct {
*mock.Call
}
// Bucket is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Bucket() *KeyValueEntry_Bucket_Call {
return &KeyValueEntry_Bucket_Call{Call: _e.mock.On("Bucket")}
}
func (_c *KeyValueEntry_Bucket_Call) Run(run func()) *KeyValueEntry_Bucket_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Bucket_Call) Return(s string) *KeyValueEntry_Bucket_Call {
_c.Call.Return(s)
return _c
}
func (_c *KeyValueEntry_Bucket_Call) RunAndReturn(run func() string) *KeyValueEntry_Bucket_Call {
_c.Call.Return(run)
return _c
}
// Created provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Created() time.Time {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Created")
}
var r0 time.Time
if returnFunc, ok := ret.Get(0).(func() time.Time); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(time.Time)
}
return r0
}
// KeyValueEntry_Created_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Created'
type KeyValueEntry_Created_Call struct {
*mock.Call
}
// Created is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Created() *KeyValueEntry_Created_Call {
return &KeyValueEntry_Created_Call{Call: _e.mock.On("Created")}
}
func (_c *KeyValueEntry_Created_Call) Run(run func()) *KeyValueEntry_Created_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Created_Call) Return(time1 time.Time) *KeyValueEntry_Created_Call {
_c.Call.Return(time1)
return _c
}
func (_c *KeyValueEntry_Created_Call) RunAndReturn(run func() time.Time) *KeyValueEntry_Created_Call {
_c.Call.Return(run)
return _c
}
// Delta provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Delta() uint64 {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Delta")
}
var r0 uint64
if returnFunc, ok := ret.Get(0).(func() uint64); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(uint64)
}
return r0
}
// KeyValueEntry_Delta_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Delta'
type KeyValueEntry_Delta_Call struct {
*mock.Call
}
// Delta is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Delta() *KeyValueEntry_Delta_Call {
return &KeyValueEntry_Delta_Call{Call: _e.mock.On("Delta")}
}
func (_c *KeyValueEntry_Delta_Call) Run(run func()) *KeyValueEntry_Delta_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Delta_Call) Return(v uint64) *KeyValueEntry_Delta_Call {
_c.Call.Return(v)
return _c
}
func (_c *KeyValueEntry_Delta_Call) RunAndReturn(run func() uint64) *KeyValueEntry_Delta_Call {
_c.Call.Return(run)
return _c
}
// Key provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Key() string {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Key")
}
var r0 string
if returnFunc, ok := ret.Get(0).(func() string); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(string)
}
return r0
}
// KeyValueEntry_Key_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Key'
type KeyValueEntry_Key_Call struct {
*mock.Call
}
// Key is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Key() *KeyValueEntry_Key_Call {
return &KeyValueEntry_Key_Call{Call: _e.mock.On("Key")}
}
func (_c *KeyValueEntry_Key_Call) Run(run func()) *KeyValueEntry_Key_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Key_Call) Return(s string) *KeyValueEntry_Key_Call {
_c.Call.Return(s)
return _c
}
func (_c *KeyValueEntry_Key_Call) RunAndReturn(run func() string) *KeyValueEntry_Key_Call {
_c.Call.Return(run)
return _c
}
// Operation provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Operation() jetstream.KeyValueOp {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Operation")
}
var r0 jetstream.KeyValueOp
if returnFunc, ok := ret.Get(0).(func() jetstream.KeyValueOp); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(jetstream.KeyValueOp)
}
return r0
}
// KeyValueEntry_Operation_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Operation'
type KeyValueEntry_Operation_Call struct {
*mock.Call
}
// Operation is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Operation() *KeyValueEntry_Operation_Call {
return &KeyValueEntry_Operation_Call{Call: _e.mock.On("Operation")}
}
func (_c *KeyValueEntry_Operation_Call) Run(run func()) *KeyValueEntry_Operation_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Operation_Call) Return(keyValueOp jetstream.KeyValueOp) *KeyValueEntry_Operation_Call {
_c.Call.Return(keyValueOp)
return _c
}
func (_c *KeyValueEntry_Operation_Call) RunAndReturn(run func() jetstream.KeyValueOp) *KeyValueEntry_Operation_Call {
_c.Call.Return(run)
return _c
}
// Revision provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Revision() uint64 {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Revision")
}
var r0 uint64
if returnFunc, ok := ret.Get(0).(func() uint64); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(uint64)
}
return r0
}
// KeyValueEntry_Revision_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Revision'
type KeyValueEntry_Revision_Call struct {
*mock.Call
}
// Revision is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Revision() *KeyValueEntry_Revision_Call {
return &KeyValueEntry_Revision_Call{Call: _e.mock.On("Revision")}
}
func (_c *KeyValueEntry_Revision_Call) Run(run func()) *KeyValueEntry_Revision_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Revision_Call) Return(v uint64) *KeyValueEntry_Revision_Call {
_c.Call.Return(v)
return _c
}
func (_c *KeyValueEntry_Revision_Call) RunAndReturn(run func() uint64) *KeyValueEntry_Revision_Call {
_c.Call.Return(run)
return _c
}
// Value provides a mock function for the type KeyValueEntry
func (_mock *KeyValueEntry) Value() []byte {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Value")
}
var r0 []byte
if returnFunc, ok := ret.Get(0).(func() []byte); ok {
r0 = returnFunc()
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]byte)
}
}
return r0
}
// KeyValueEntry_Value_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Value'
type KeyValueEntry_Value_Call struct {
*mock.Call
}
// Value is a helper method to define mock.On call
func (_e *KeyValueEntry_Expecter) Value() *KeyValueEntry_Value_Call {
return &KeyValueEntry_Value_Call{Call: _e.mock.On("Value")}
}
func (_c *KeyValueEntry_Value_Call) Run(run func()) *KeyValueEntry_Value_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *KeyValueEntry_Value_Call) Return(bytes []byte) *KeyValueEntry_Value_Call {
_c.Call.Return(bytes)
return _c
}
func (_c *KeyValueEntry_Value_Call) RunAndReturn(run func() []byte) *KeyValueEntry_Value_Call {
_c.Call.Return(run)
return _c
}
@@ -0,0 +1,69 @@
// Package announcement persists and serves the web announcement banner.
package announcement
import (
"context"
"encoding/json"
"errors"
"github.com/nats-io/nats.go/jetstream"
)
// _storeKey is the single key under which the announcement is persisted.
const _storeKey = "announcement"
// Announcement is a banner message shown above the top bar to all users.
type Announcement struct {
// Enabled controls whether the announcement is live (injected into config.json).
Enabled bool `json:"enabled"`
BannerText string `json:"bannerText"`
InfoText string `json:"infoText"`
}
// Store persists a single announcement in a NATS JetStream key-value bucket.
type Store struct {
kv jetstream.KeyValue
}
// NewStore returns a new announcement Store backed by the given key-value bucket.
func NewStore(kv jetstream.KeyValue) *Store {
return &Store{kv: kv}
}
// Get returns the currently stored announcement. An unset announcement is returned as the zero value.
func (s *Store) Get(ctx context.Context) (Announcement, error) {
var a Announcement
entry, err := s.kv.Get(ctx, _storeKey)
if err != nil {
if errors.Is(err, jetstream.ErrKeyNotFound) {
return a, nil
}
return a, err
}
if err := json.Unmarshal(entry.Value(), &a); err != nil {
return a, err
}
return a, nil
}
// Set persists the given announcement, overwriting any existing one.
func (s *Store) Set(ctx context.Context, a Announcement) error {
value, err := json.Marshal(a)
if err != nil {
return err
}
_, err = s.kv.Put(ctx, _storeKey, value)
return err
}
// Delete removes the stored announcement. Deleting a missing announcement is a no-op.
func (s *Store) Delete(ctx context.Context) error {
if err := s.kv.Delete(ctx, _storeKey); err != nil && !errors.Is(err, jetstream.ErrKeyNotFound) {
return err
}
return nil
}
@@ -0,0 +1,13 @@
package announcement_test
import (
"testing"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func TestAnnouncement(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Announcement Suite")
}
@@ -0,0 +1,207 @@
package announcement_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
userpb "github.com/cs3org/go-cs3apis/cs3/identity/user/v1beta1"
cs3permissions "github.com/cs3org/go-cs3apis/cs3/permissions/v1beta1"
cs3rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
"github.com/nats-io/nats.go/jetstream"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
revactx "github.com/opencloud-eu/reva/v2/pkg/ctx"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
cs3mocks "github.com/opencloud-eu/reva/v2/tests/cs3mocks/mocks"
"github.com/stretchr/testify/mock"
"google.golang.org/grpc"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/web/mocks"
"github.com/opencloud-eu/opencloud/services/web/pkg/announcement"
)
func newGatewaySelector(allowed bool) pool.Selectable[gateway.GatewayAPIClient] {
code := cs3rpc.Code_CODE_OK
name := "announcement-test-allowed"
if !allowed {
code = cs3rpc.Code_CODE_PERMISSION_DENIED
name = "announcement-test-denied"
}
client := &cs3mocks.GatewayAPIClient{}
client.On("CheckPermission", mock.Anything, mock.Anything).Return(
&cs3permissions.CheckPermissionResponse{Status: &cs3rpc.Status{Code: code}}, nil)
// pool.GetSelector caches by name, so allow/deny must use distinct names
return pool.GetSelector[gateway.GatewayAPIClient](
name,
"eu.opencloud.api.gateway",
func(cc grpc.ClientConnInterface) gateway.GatewayAPIClient { return client },
)
}
func withUser(r *http.Request) *http.Request {
return r.WithContext(revactx.ContextSetUser(r.Context(), &userpb.User{
Id: &userpb.UserId{OpaqueId: "user"},
}))
}
func newService(store *announcement.Store, allowed bool) announcement.Service {
svc, err := announcement.NewService(announcement.ServiceOptions{}.
WithLogger(log.NopLogger()).
WithStore(store).
WithGatewaySelector(newGatewaySelector(allowed)))
Expect(err).ToNot(HaveOccurred())
return svc
}
var _ = Describe("Store", func() {
It("reads the stored announcement", func() {
entry := mocks.NewKeyValueEntry(GinkgoT())
entry.EXPECT().Value().Return([]byte(`{"enabled":true,"bannerText":"hello","infoText":"world"}`))
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Get(mock.Anything, "announcement").Return(entry, nil)
got, err := announcement.NewStore(kv).Get(context.Background())
Expect(err).ToNot(HaveOccurred())
Expect(got.Enabled).To(BeTrue())
Expect(got.BannerText).To(Equal("hello"))
Expect(got.InfoText).To(Equal("world"))
})
It("returns the zero value when unset", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Get(mock.Anything, "announcement").Return(nil, jetstream.ErrKeyNotFound)
got, err := announcement.NewStore(kv).Get(context.Background())
Expect(err).ToNot(HaveOccurred())
Expect(got.BannerText).To(BeEmpty())
})
It("writes the announcement", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Put(mock.Anything, "announcement", mock.Anything).Return(uint64(1), nil)
Expect(announcement.NewStore(kv).Set(context.Background(), announcement.Announcement{BannerText: "hello"})).To(Succeed())
})
It("deletes the announcement", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Delete(mock.Anything, "announcement").Return(nil)
Expect(announcement.NewStore(kv).Delete(context.Background())).To(Succeed())
})
It("treats deleting a missing announcement as a no-op", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Delete(mock.Anything, "announcement").Return(jetstream.ErrKeyNotFound)
Expect(announcement.NewStore(kv).Delete(context.Background())).To(Succeed())
})
})
var _ = Describe("Service", func() {
Describe("NewService", func() {
It("fails when options are missing", func() {
_, err := announcement.NewService(announcement.ServiceOptions{})
Expect(err).To(HaveOccurred())
})
It("succeeds when options are valid", func() {
_, err := announcement.NewService(announcement.ServiceOptions{}.
WithStore(announcement.NewStore(mocks.NewKeyValue(GinkgoT()))).
WithGatewaySelector(newGatewaySelector(true)))
Expect(err).ToNot(HaveOccurred())
})
})
Describe("Get", func() {
It("returns the full stored announcement when permitted", func() {
entry := mocks.NewKeyValueEntry(GinkgoT())
entry.EXPECT().Value().Return([]byte(`{"enabled":true,"bannerText":"hello","infoText":"world"}`))
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Get(mock.Anything, "announcement").Return(entry, nil)
req := withUser(httptest.NewRequest(http.MethodGet, "/announcement", nil))
resp := httptest.NewRecorder()
newService(announcement.NewStore(kv), true).Get(resp, req)
Expect(resp.Code).To(Equal(http.StatusOK))
var got announcement.Announcement
Expect(json.Unmarshal(resp.Body.Bytes(), &got)).To(Succeed())
Expect(got.Enabled).To(BeTrue())
Expect(got.BannerText).To(Equal("hello"))
Expect(got.InfoText).To(Equal("world"))
})
It("is forbidden without permission", func() {
req := withUser(httptest.NewRequest(http.MethodGet, "/announcement", nil))
resp := httptest.NewRecorder()
newService(announcement.NewStore(mocks.NewKeyValue(GinkgoT())), false).Get(resp, req)
Expect(resp.Code).To(Equal(http.StatusForbidden))
})
})
Describe("Set", func() {
It("persists the message when permitted", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Put(mock.Anything, "announcement", mock.Anything).Return(uint64(1), nil)
req := withUser(httptest.NewRequest(http.MethodPut, "/announcement", strings.NewReader(`{"bannerText":"hello"}`)))
resp := httptest.NewRecorder()
newService(announcement.NewStore(kv), true).Set(resp, req)
Expect(resp.Code).To(Equal(http.StatusNoContent))
})
It("is forbidden without permission", func() {
req := withUser(httptest.NewRequest(http.MethodPut, "/announcement", strings.NewReader(`{"bannerText":"hello"}`)))
resp := httptest.NewRecorder()
newService(announcement.NewStore(mocks.NewKeyValue(GinkgoT())), false).Set(resp, req)
Expect(resp.Code).To(Equal(http.StatusForbidden))
})
It("rejects an invalid body", func() {
req := withUser(httptest.NewRequest(http.MethodPut, "/announcement", strings.NewReader(`not json`)))
resp := httptest.NewRecorder()
newService(announcement.NewStore(mocks.NewKeyValue(GinkgoT())), true).Set(resp, req)
Expect(resp.Code).To(Equal(http.StatusBadRequest))
})
It("rejects an oversized body", func() {
body := `{"bannerText":"` + strings.Repeat("a", 300000) + `"}`
req := withUser(httptest.NewRequest(http.MethodPut, "/announcement", strings.NewReader(body)))
resp := httptest.NewRecorder()
newService(announcement.NewStore(mocks.NewKeyValue(GinkgoT())), true).Set(resp, req)
Expect(resp.Code).To(Equal(http.StatusRequestEntityTooLarge))
})
})
Describe("Set with an empty banner text", func() {
It("removes the stored announcement", func() {
kv := mocks.NewKeyValue(GinkgoT())
kv.EXPECT().Delete(mock.Anything, "announcement").Return(nil)
req := withUser(httptest.NewRequest(http.MethodPut, "/announcement", strings.NewReader(`{"enabled":false,"bannerText":"","infoText":""}`)))
resp := httptest.NewRecorder()
newService(announcement.NewStore(kv), true).Set(resp, req)
Expect(resp.Code).To(Equal(http.StatusNoContent))
})
})
})
+192
View File
@@ -0,0 +1,192 @@
package announcement
import (
"encoding/json"
"net/http"
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
permissionsapi "github.com/cs3org/go-cs3apis/cs3/permissions/v1beta1"
rpc "github.com/cs3org/go-cs3apis/cs3/rpc/v1beta1"
"github.com/pkg/errors"
revactx "github.com/opencloud-eu/reva/v2/pkg/ctx"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"github.com/opencloud-eu/opencloud/pkg/log"
)
// _permission is the settings permission required to read and manage the announcement.
const _permission = "Announcement.ReadWrite"
// _maxBodySize caps the announcement request body. The info text is Markdown and ends up in
// the public config.json that every client loads on bootstrap, so it must stay small.
const _maxBodySize = 50 << 10 // 50 KiB
// ServiceOptions defines the options to configure the Service.
type ServiceOptions struct {
logger log.Logger
store *Store
gatewaySelector pool.Selectable[gateway.GatewayAPIClient]
}
// WithLogger sets the logger.
func (o ServiceOptions) WithLogger(l log.Logger) ServiceOptions {
o.logger = l
return o
}
// WithStore sets the announcement store.
func (o ServiceOptions) WithStore(s *Store) ServiceOptions {
o.store = s
return o
}
// WithGatewaySelector sets the gateway selector.
func (o ServiceOptions) WithGatewaySelector(gws pool.Selectable[gateway.GatewayAPIClient]) ServiceOptions {
o.gatewaySelector = gws
return o
}
// validate validates the input parameters.
func (o ServiceOptions) validate() error {
if o.store == nil {
return errors.New("store is required")
}
if o.gatewaySelector == nil {
return errors.New("gatewaySelector is required")
}
return nil
}
// Service exposes the http handlers to manage the announcement.
type Service struct {
logger log.Logger
store *Store
gatewaySelector pool.Selectable[gateway.GatewayAPIClient]
}
// NewService initializes a new Service.
func NewService(options ServiceOptions) (Service, error) {
if err := options.validate(); err != nil {
return Service{}, err
}
return Service{
logger: options.logger,
store: options.store,
gatewaySelector: options.gatewaySelector,
}, nil
}
// logError logs a server-side error together with the request id, so it can be correlated with
// the request log line emitted by the logging middleware.
func (s Service) logError(r *http.Request, err error, msg string) {
s.logger.Error().Err(err).Str(log.RequestIDString, r.Header.Get("X-Request-ID")).Msg(msg)
}
// Get returns the full stored announcement (including disabled ones) for management.
func (s Service) Get(w http.ResponseWriter, r *http.Request) {
gatewayClient, err := s.gatewaySelector.Next()
if err != nil {
s.logError(r, err, "could not select next gateway client")
w.WriteHeader(http.StatusInternalServerError)
return
}
user, ok := revactx.ContextGetUser(r.Context())
if !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
rsp, err := gatewayClient.CheckPermission(r.Context(), &permissionsapi.CheckPermissionRequest{
Permission: _permission,
SubjectRef: &permissionsapi.SubjectReference{
Spec: &permissionsapi.SubjectReference_UserId{
UserId: user.GetId(),
},
},
})
if err != nil {
s.logError(r, err, "could not check permission")
w.WriteHeader(http.StatusInternalServerError)
return
}
if rsp.GetStatus().GetCode() != rpc.Code_CODE_OK {
w.WriteHeader(http.StatusForbidden)
return
}
a, err := s.store.Get(r.Context())
if err != nil {
s.logError(r, err, "could not read announcement from store")
w.WriteHeader(http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(a); err != nil {
s.logError(r, err, "could not encode announcement")
w.WriteHeader(http.StatusInternalServerError)
}
}
// Set persists the announcement provided in the request body.
func (s Service) Set(w http.ResponseWriter, r *http.Request) {
gatewayClient, err := s.gatewaySelector.Next()
if err != nil {
s.logError(r, err, "could not select next gateway client")
w.WriteHeader(http.StatusInternalServerError)
return
}
user, ok := revactx.ContextGetUser(r.Context())
if !ok {
w.WriteHeader(http.StatusUnauthorized)
return
}
rsp, err := gatewayClient.CheckPermission(r.Context(), &permissionsapi.CheckPermissionRequest{
Permission: _permission,
SubjectRef: &permissionsapi.SubjectReference{
Spec: &permissionsapi.SubjectReference_UserId{
UserId: user.GetId(),
},
},
})
if err != nil {
s.logError(r, err, "could not check permission")
w.WriteHeader(http.StatusInternalServerError)
return
}
if rsp.GetStatus().GetCode() != rpc.Code_CODE_OK {
w.WriteHeader(http.StatusForbidden)
return
}
var body Announcement
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, _maxBodySize)).Decode(&body); err != nil {
var maxBytesErr *http.MaxBytesError
if errors.As(err, &maxBytesErr) {
w.WriteHeader(http.StatusRequestEntityTooLarge)
return
}
w.WriteHeader(http.StatusBadRequest)
return
}
// an announcement without a banner text is nothing to show, so remove it entirely
if body.BannerText == "" {
if err := s.store.Delete(r.Context()); err != nil {
s.logError(r, err, "could not delete announcement from store")
w.WriteHeader(http.StatusInternalServerError)
return
}
} else if err := s.store.Set(r.Context(), body); err != nil {
s.logError(r, err, "could not write announcement to store")
w.WriteHeader(http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
+13
View File
@@ -25,9 +25,22 @@ type Config struct {
TokenManager *TokenManager `yaml:"token_manager"`
GatewayAddress string `yaml:"gateway_addr" env:"WEB_GATEWAY_GRPC_ADDR" desc:"The bind address of the GRPC service." introductionVersion:"1.0.0"`
Store Store `yaml:"store"`
Context context.Context `yaml:"-"`
}
// Store configures the NATS JetStream key-value store used to keep runtime managed web settings,
// e.g. the announcement banner.
type Store struct {
Nodes []string `yaml:"nodes" env:"OC_PERSISTENT_STORE_NODES;WEB_STORE_NODES" desc:"A list of nodes to access the NATS JetStream store. See the Environment Variable Types description for more details." introductionVersion:"7.4.0"`
Database string `yaml:"database" env:"WEB_STORE_DATABASE" desc:"The bucket name the store should use." introductionVersion:"7.4.0"`
AuthUsername string `yaml:"username" env:"OC_PERSISTENT_STORE_AUTH_USERNAME;WEB_STORE_AUTH_USERNAME" desc:"The username to authenticate with the store." introductionVersion:"7.4.0"`
AuthPassword string `yaml:"password" env:"OC_PERSISTENT_STORE_AUTH_PASSWORD;WEB_STORE_AUTH_PASSWORD" desc:"The password to authenticate with the store." introductionVersion:"7.4.0"`
EnableTLS bool `yaml:"enable_tls" env:"OC_PERSISTENT_STORE_ENABLE_TLS;WEB_STORE_ENABLE_TLS" desc:"Enable TLS for the connection to the store." introductionVersion:"7.4.0"`
TLSInsecure bool `yaml:"tls_insecure" env:"OC_INSECURE;OC_PERSISTENT_STORE_TLS_INSECURE;WEB_STORE_TLS_INSECURE" desc:"Whether to verify the server TLS certificates." introductionVersion:"7.4.0"`
TLSRootCACertificate string `yaml:"tls_root_ca_certificate" env:"OC_PERSISTENT_STORE_TLS_ROOT_CA_CERTIFICATE;WEB_STORE_TLS_ROOT_CA_CERTIFICATE" desc:"The root CA certificate used to validate the server's TLS certificate. If provided WEB_STORE_TLS_INSECURE will be seen as false." introductionVersion:"7.4.0"`
}
// Asset defines the available asset configuration.
type Asset struct {
CorePath string `yaml:"core_path" env:"WEB_ASSET_CORE_PATH" desc:"Serve OpenCloud Web assets from a path on the filesystem instead of the builtin assets. If not defined, the root directory derives from $OC_BASE_DATA_PATH/web/assets/core" introductionVersion:"1.0.0"`
@@ -85,6 +85,10 @@ func DefaultConfig() *config.Config {
ThemesPath: filepath.Join(defaults.BaseDataPath(), "web/assets/themes"),
},
GatewayAddress: "eu.opencloud.api.gateway",
Store: config.Store{
Nodes: []string{"127.0.0.1:9233"},
Database: "web",
},
Web: config.Web{
ThemeServer: "https://localhost:9200",
ThemePath: "/themes/opencloud/theme.json",
+11
View File
@@ -2,6 +2,7 @@ package config
// Options are the option for the web
type Options struct {
Announcement *Announcement `json:"announcement,omitempty" yaml:"-"`
AccountEditLink *AccountEditLink `json:"accountEditLink,omitempty" yaml:"accountEditLink"`
DisableFeedbackLink bool `json:"disableFeedbackLink,omitempty" yaml:"disableFeedbackLink" env:"WEB_OPTION_DISABLE_FEEDBACK_LINK" desc:"Set this option to 'true' to disable the feedback link in the top bar. Keeping it enabled by setting the value to 'false' or with the absence of the option, allows OpenCloud to get feedback from your user base through a dedicated survey website." introductionVersion:"1.0.0"`
DisableSponsorLink bool `json:"disableSponsorLink,omitempty" yaml:"disableSponsorLink" env:"WEB_OPTION_DISABLE_SPONSOR_LINK" desc:"Set this option to 'true' to disable the sponsor link in the left sidebar. Keeping it enabled by setting the value to 'false' or by leaving the option unset allows OpenCloud to get support from the community through a dedicated sponsorship program on GitHub." introductionVersion:"7.3.0"`
@@ -23,6 +24,16 @@ type Options struct {
OxAppSuite *OxAppSuite `json:"oxAppSuite,omitempty" yaml:"oxAppSuite"`
}
// Announcement is a banner message shown above the top bar to all users. It is managed at runtime
// (via the web service's store and the admin settings UI) and injected into config.json here; a
// value configured statically is ignored.
type Announcement struct {
// BannerText is the short line shown in the banner.
BannerText string `json:"bannerText,omitempty" yaml:"-"`
// InfoText is the (Markdown) detail shown in a dialog when the banner is clicked.
InfoText string `json:"infoText,omitempty" yaml:"-"`
}
// AccountEditLink are the AccountEditLink options
type AccountEditLink struct {
Href string `json:"href,omitempty" yaml:"href" env:"WEB_OPTION_ACCOUNT_EDIT_LINK_HREF" desc:"Set a different target URL for the edit link. Make sure to prepend it with 'http(s)://'." introductionVersion:"1.0.0"`
+33
View File
@@ -1,20 +1,26 @@
package http
import (
"errors"
"fmt"
"path"
"strings"
chimiddleware "github.com/go-chi/chi/v5/middleware"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/opencloud-eu/reva/v2/pkg/rgrpc/todo/pool"
"go-micro.dev/v4"
"github.com/opencloud-eu/opencloud/pkg/cors"
"github.com/opencloud-eu/opencloud/pkg/middleware"
natspkg "github.com/opencloud-eu/opencloud/pkg/nats"
"github.com/opencloud-eu/opencloud/pkg/registry"
"github.com/opencloud-eu/opencloud/pkg/service/http"
"github.com/opencloud-eu/opencloud/pkg/version"
"github.com/opencloud-eu/opencloud/pkg/x/io/fsx"
"github.com/opencloud-eu/opencloud/services/web"
"github.com/opencloud-eu/opencloud/services/web/pkg/announcement"
"github.com/opencloud-eu/opencloud/services/web/pkg/apps"
svc "github.com/opencloud-eu/opencloud/services/web/pkg/service/v0"
)
@@ -76,11 +82,38 @@ func Server(opts ...Option) (http.Service, error) {
fsx.NewBasePathFs(fsx.FromIOFS(web.Assets), "assets/themes"),
)
// NATS JetStream key-value store for runtime managed web settings, e.g. the announcement banner.
// Connect eagerly and fail fast: an unreachable store means the feature would be broken, so a
// clear startup error is preferable to silently degrading.
natsConn, err := nats.Connect(
strings.Join(options.Config.Store.Nodes, ","),
natspkg.Secure(options.Config.Store.EnableTLS, options.Config.Store.TLSInsecure, options.Config.Store.TLSRootCACertificate),
nats.UserInfo(options.Config.Store.AuthUsername, options.Config.Store.AuthPassword),
)
if err != nil {
return http.Service{}, fmt.Errorf("could not connect to nats for the announcement store: %w", err)
}
js, err := jetstream.New(natsConn)
if err != nil {
return http.Service{}, fmt.Errorf("could not create jetstream context for the announcement store: %w", err)
}
kv, err := js.KeyValue(options.Context, options.Config.Store.Database)
if err != nil {
if !errors.Is(err, jetstream.ErrBucketNotFound) {
return http.Service{}, fmt.Errorf("could not open the announcement store bucket %q: %w", options.Config.Store.Database, err)
}
if kv, err = js.CreateKeyValue(options.Context, jetstream.KeyValueConfig{Bucket: options.Config.Store.Database}); err != nil {
return http.Service{}, fmt.Errorf("could not create the announcement store bucket %q: %w", options.Config.Store.Database, err)
}
}
announcementStore := announcement.NewStore(kv)
handle, err := svc.NewService(
svc.Logger(options.Logger),
svc.CoreFS(coreFS.IOFS()),
svc.AppFS(appsFS.IOFS()),
svc.ThemeFS(themeFS),
svc.AnnouncementStore(announcementStore),
svc.AppsHTTPEndpoint(_customAppsEndpoint),
svc.Config(options.Config),
svc.GatewaySelector(gatewaySelector),
+18 -9
View File
@@ -10,6 +10,7 @@ import (
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/x/io/fsx"
"github.com/opencloud-eu/opencloud/services/web/pkg/announcement"
"github.com/opencloud-eu/opencloud/services/web/pkg/config"
)
@@ -18,15 +19,16 @@ type Option func(o *Options)
// Options define the available options for this package.
type Options struct {
Logger log.Logger
Config *config.Config
Middleware []func(http.Handler) http.Handler
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
TraceProvider trace.TracerProvider
AppsHTTPEndpoint string
CoreFS fs.FS
AppFS fs.FS
ThemeFS *fsx.FallbackFS
Logger log.Logger
Config *config.Config
Middleware []func(http.Handler) http.Handler
GatewaySelector pool.Selectable[gateway.GatewayAPIClient]
TraceProvider trace.TracerProvider
AppsHTTPEndpoint string
CoreFS fs.FS
AppFS fs.FS
ThemeFS *fsx.FallbackFS
AnnouncementStore *announcement.Store
}
// newOptions initializes the available default options.
@@ -89,6 +91,13 @@ func ThemeFS(val *fsx.FallbackFS) Option {
}
}
// AnnouncementStore provides a function to set the announcement store option.
func AnnouncementStore(val *announcement.Store) Option {
return func(o *Options) {
o.AnnouncementStore = val
}
}
// AppsHTTPEndpoint provides a function to set the appsHTTPEndpoint option.
func AppsHTTPEndpoint(val string) Option {
return func(o *Options) {
+67 -18
View File
@@ -1,6 +1,7 @@
package svc
import (
"context"
"encoding/json"
"io/fs"
"net/http"
@@ -19,6 +20,7 @@ import (
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/middleware"
"github.com/opencloud-eu/opencloud/pkg/tracing"
"github.com/opencloud-eu/opencloud/services/web/pkg/announcement"
"github.com/opencloud-eu/opencloud/services/web/pkg/assets"
"github.com/opencloud-eu/opencloud/services/web/pkg/config"
"github.com/opencloud-eu/opencloud/services/web/pkg/theme"
@@ -50,10 +52,11 @@ func NewService(opts ...Option) (Service, error) {
)
svc := Web{
logger: options.Logger,
config: options.Config,
mux: m,
gatewaySelector: options.GatewaySelector,
logger: options.Logger,
config: options.Config,
mux: m,
gatewaySelector: options.GatewaySelector,
announcementStore: options.AnnouncementStore,
}
themeService, err := theme.NewService(
@@ -65,6 +68,16 @@ func NewService(opts ...Option) (Service, error) {
return svc, err
}
announcementService, err := announcement.NewService(
announcement.ServiceOptions{}.
WithLogger(options.Logger).
WithStore(options.AnnouncementStore).
WithGatewaySelector(options.GatewaySelector),
)
if err != nil {
return svc, err
}
m.Route(options.Config.HTTP.Root, func(r chi.Router) {
r.Get("/config.json", svc.Config)
r.Route("/branding/logo", func(r chi.Router) {
@@ -75,6 +88,14 @@ func NewService(opts ...Option) (Service, error) {
r.Post("/", themeService.LogoUpload)
r.Delete("/", themeService.LogoReset)
})
r.Route("/announcement", func(r chi.Router) {
r.Use(middleware.ExtractAccountUUID(
account.Logger(options.Logger),
account.JWTSecret(options.Config.TokenManager.JWTSecret),
))
r.Get("/", announcementService.Get)
r.Put("/", announcementService.Set)
})
r.Route("/themes", func(r chi.Router) {
r.Get("/{id}/theme.json", themeService.Get)
r.Mount("/", svc.Static(
@@ -104,10 +125,11 @@ func NewService(opts ...Option) (Service, error) {
// Web defines the handlers for the web service.
type Web struct {
logger log.Logger
config *config.Config
mux *chi.Mux
gatewaySelector pool.Selectable[gateway.GatewayAPIClient]
logger log.Logger
config *config.Config
mux *chi.Mux
gatewaySelector pool.Selectable[gateway.GatewayAPIClient]
announcementStore *announcement.Store
}
// ServeHTTP implements the Service interface.
@@ -115,31 +137,58 @@ func (p Web) ServeHTTP(w http.ResponseWriter, r *http.Request) {
p.mux.ServeHTTP(w, r)
}
func (p Web) getPayload() (payload []byte, err error) {
// render dynamically using config
func (p Web) getPayload(ctx context.Context) (payload []byte, err error) {
// render dynamically using a copy of the config, so per-request values (e.g. the
// announcement) are not written into the shared config concurrently.
webConfig := p.config.Web.Config
// build theme url
if themeServer, err := url.Parse(p.config.Web.ThemeServer); err == nil {
p.config.Web.Config.Theme = themeServer.String() + p.config.Web.ThemePath
webConfig.Theme = themeServer.String() + p.config.Web.ThemePath
} else {
p.config.Web.Config.Theme = p.config.Web.ThemePath
webConfig.Theme = p.config.Web.ThemePath
}
// make apps render as empty array if it is empty
// TODO remove once https://github.com/golang/go/issues/27589 is fixed
if len(p.config.Web.Config.Apps) == 0 {
p.config.Web.Config.Apps = make([]string, 0)
if len(webConfig.Apps) == 0 {
webConfig.Apps = make([]string, 0)
}
// ensure that the server url has a trailing slash
p.config.Web.Config.Server = strings.TrimRight(p.config.Web.Config.Server, "/") + "/"
webConfig.Server = strings.TrimRight(webConfig.Server, "/") + "/"
return json.Marshal(p.config.Web.Config)
// the runtime store is the single source of truth for the announcement banner: expose it
// when live, clear it otherwise. A statically configured value is not supported.
webConfig.Options.Announcement = p.currentAnnouncement(ctx)
return json.Marshal(webConfig)
}
// currentAnnouncement returns the stored announcement for config.json, or nil if unset, disabled
// or unavailable.
func (p Web) currentAnnouncement(ctx context.Context) *config.Announcement {
if p.announcementStore == nil {
return nil
}
a, err := p.announcementStore.Get(ctx)
if err != nil {
p.logger.Error().Err(err).Msg("could not read announcement from store")
return nil
}
// only live (enabled) announcements with a banner line are exposed in the public config.json
if !a.Enabled || a.BannerText == "" {
return nil
}
return &config.Announcement{BannerText: a.BannerText, InfoText: a.InfoText}
}
// Config implements the Service interface.
func (p Web) Config(w http.ResponseWriter, _ *http.Request) {
payload, err := p.getPayload()
func (p Web) Config(w http.ResponseWriter, r *http.Request) {
payload, err := p.getPayload(r.Context())
if err != nil {
http.Error(w, ErrConfigInvalid, http.StatusUnprocessableEntity)
return
@@ -0,0 +1,58 @@
package svc
import (
"context"
"testing"
"github.com/nats-io/nats.go/jetstream"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/services/web/mocks"
"github.com/opencloud-eu/opencloud/services/web/pkg/announcement"
"github.com/opencloud-eu/opencloud/services/web/pkg/config"
)
func TestCurrentAnnouncement(t *testing.T) {
newWeb := func(store *announcement.Store) Web {
return Web{logger: log.NopLogger(), announcementStore: store}
}
// storeReturning builds a store whose backing bucket returns the given JSON for the announcement key.
storeReturning := func(t *testing.T, value string) *announcement.Store {
entry := mocks.NewKeyValueEntry(t)
entry.EXPECT().Value().Return([]byte(value))
kv := mocks.NewKeyValue(t)
kv.EXPECT().Get(mock.Anything, "announcement").Return(entry, nil)
return announcement.NewStore(kv)
}
// emptyStore builds a store whose backing bucket has no announcement.
emptyStore := func(t *testing.T) *announcement.Store {
kv := mocks.NewKeyValue(t)
kv.EXPECT().Get(mock.Anything, "announcement").Return(nil, jetstream.ErrKeyNotFound)
return announcement.NewStore(kv)
}
t.Run("nil when there is no store", func(t *testing.T) {
require.Nil(t, newWeb(nil).currentAnnouncement(context.Background()))
})
t.Run("nil when the store is empty", func(t *testing.T) {
require.Nil(t, newWeb(emptyStore(t)).currentAnnouncement(context.Background()))
})
t.Run("nil when disabled", func(t *testing.T) {
s := storeReturning(t, `{"enabled":false,"bannerText":"hi","infoText":"info"}`)
require.Nil(t, newWeb(s).currentAnnouncement(context.Background()))
})
t.Run("nil when enabled but the banner text is empty", func(t *testing.T) {
s := storeReturning(t, `{"enabled":true,"bannerText":"","infoText":"info"}`)
require.Nil(t, newWeb(s).currentAnnouncement(context.Background()))
})
t.Run("returns banner and info text when enabled with a banner text", func(t *testing.T) {
s := storeReturning(t, `{"enabled":true,"bannerText":"hi","infoText":"info"}`)
require.Equal(t, &config.Announcement{BannerText: "hi", InfoText: "info"}, newWeb(s).currentAnnouncement(context.Background()))
})
}
@@ -1031,19 +1031,7 @@ class SharingNgContext implements Context {
$this->featureContext->shareNgAddToCreatedUserGroupShares($this->getDrivePermissionsList($sharer, $space));
$permissionID = $this->featureContext->shareNgGetLastCreatedUserGroupShareID();
} elseif ($shareType == 'group' && !isset($recipient)) {
// https://github.com/opencloud-eu/opencloud/pull/3179#issuecomment-5103212045
$retried = 0;
do {
$response = $this->getDrivePermissionsList($sharer, $space);
$tryAgain = $response->getStatusCode() === 404
&& $retried < HttpRequestHelper::numRetriesOnHttpTooEarly();
if ($tryAgain) {
$retried += 1;
echo "Drive permissions of space '$space' not available for user '$sharer' yet, retrying ($retried)...\n";
// wait 500ms and try again
\usleep(500 * 1000);
}
} while ($tryAgain);
$response = $this->getDrivePermissionsList($sharer, $space);
$permissionID = $this->featureContext->getJsonDecodedResponse($response)['value'][0]['id'];
} else {
$permissionID = match ($shareType) {
@@ -141,7 +141,7 @@ Feature: Remove access to a drive
Then the HTTP status code should be "403"
And the user "Alice" should have a space called "NewSpace"
@flaky @issue-3193
Scenario: user of a group cannot remove own group from project space if it is the last manager using root endpoint
Given the administrator has assigned the role "Space Admin" to user "Alice" using the Graph API
And group "group1" has been created
+2 -2
View File
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
// Package assert enables defining [test properties] about your program or [workload]. It is part of the [Antithesis Go SDK], which enables Go applications to integrate with the [Antithesis platform].
//
@@ -6,7 +6,7 @@
//
// These functions are no-ops with minimal performance overhead when called outside of the Antithesis environment. However, if the environment variable ANTITHESIS_SDK_LOCAL_OUTPUT is set, these functions will log to the file pointed to by that variable using a structured JSON format defined [here]. This allows you to make use of the Antithesis assertions package in your regular testing, or even in production. In particular, very few assertions frameworks offer a convenient way to define [Sometimes assertions], but they can be quite useful even outside Antithesis.
//
// Each function in this package takes a parameter called message, which is a human readable identifier used to aggregate assertions. Antithesis generates one test property per unique message and this test property will be named "<message>" in the [triage report].
// Each function in this package takes a parameter called message, which is a human readable identifier used to aggregate assertions. Antithesis generates one test property per unique message and this test property will be named "<message>" in the [triage report]. Message must be provided as a string literal.
//
// This test property either passes or fails, which depends upon the evaluation of every assertion that shares its message. Different assertions in different parts of the code should have different message, but the same assertion should always have the same message even if it is moved to a different file.
//
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build !enable_antithesis_sdk
//go:build no_antithesis_sdk
package assert
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package assert
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package assert
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package assert
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package assert
@@ -1,4 +1,4 @@
//go:build !enable_antithesis_sdk
//go:build no_antithesis_sdk
package assert
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package assert
+2 -2
View File
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk
//go:build !no_antithesis_sdk
package internal
@@ -38,7 +38,7 @@ type libHandler interface {
}
const (
errorLogLinePrefix = "[* antithesis-sdk-go *]"
errorLogLinePrefix = "[* antithesis-sdk-go *]"
)
var handler libHandler
+1 -1
View File
@@ -3,7 +3,7 @@ package internal
// --------------------------------------------------------------------------------
// Versions
// --------------------------------------------------------------------------------
const SDK_Version = "0.7.0"
const SDK_Version = "0.7.2"
const Protocol_Version = "1.1.0"
// --------------------------------------------------------------------------------
@@ -1,11 +1,11 @@
//go:build enable_antithesis_sdk && linux && amd64 && cgo
//go:build !no_antithesis_sdk && linux && amd64 && cgo
package internal
import (
"fmt"
"unsafe"
"os"
"unsafe"
)
// --------------------------------------------------------------------------------
@@ -1,4 +1,4 @@
//go:build enable_antithesis_sdk && (!linux || !amd64 || !cgo)
//go:build !no_antithesis_sdk && (!linux || !amd64 || !cgo)
package internal
+24
View File
@@ -1,3 +1,27 @@
Release 1.7.0
=============
**Changes**
**Breaking changes**
* To address a security issue, it was necessary to add a `MaxDepth` option to
`ReadSettings` to limit the depth of XML trees during parsing. A generous
default value of 1024 was chosen to avoid breaking most existing code.
However, if your code is processing XML hierarchies with a depth greater
than 1024, you will need to assign your `Document` a `ReadSettings` that has
a `MaxDepth` set to a higher value.
**Security Fixes**
* Limited the depth of XML trees processed by all `ReadFrom` functions during
parsing.
* Fixed a `CompilePath` index-out-of-range panic that could be caused by a
missing path filter key.
* Sanitized the contents of XML text, comment, ProcInst and Directive tokens
provided by the user.
Release 1.6.0
=============
+31 -17
View File
@@ -13,6 +13,7 @@ import (
"errors"
"io"
"iter"
"maps"
"os"
"slices"
"strings"
@@ -27,6 +28,10 @@ const (
// ErrXML is returned when XML parsing fails due to incorrect formatting.
var ErrXML = errors.New("etree: invalid XML format")
// ErrMaxDepth is returned when the depth of the XML tree being read exceeds
// the maximum depth allowed by ReadSettings.MaxDepth.
var ErrMaxDepth = errors.New("etree: XML tree exceeds maximum depth")
// cdataPrefix is used to detect CDATA text when ReadSettings.PreserveCData is
// true.
var cdataPrefix = []byte("<![CDATA[")
@@ -76,8 +81,18 @@ type ReadSettings struct {
// whether an end element is present. Commonly set to xml.HTMLAutoClose.
// Default: nil.
AutoClose []string
// MaxDepth is the maximum depth of the XML tree to parse. If the depth of
// the XML tree exceeds this value, all ReadFrom* functions return the
// error ErrMaxDepth. If MaxDepth is zero or negative, a depth limit of
// 1024 is used. Default: 0 (i.e., a limit of 1024).
MaxDepth int
}
// defaultMaxDepth is the maximum depth of an XML tree parsed by ReadFrom*
// functions when ReadSettings.MaxDepth is not set to a positive value.
const defaultMaxDepth = 1024
// defaultCharsetReader is used by the xml decoder when the ReadSettings
// CharsetReader value is nil. It behaves as a "pass-through", ignoring
// the requested charset parameter and skipping conversion altogether.
@@ -87,18 +102,9 @@ func defaultCharsetReader(charset string, input io.Reader) (io.Reader, error) {
// dup creates a duplicate of the ReadSettings object.
func (s *ReadSettings) dup() ReadSettings {
var entityCopy map[string]string
if s.Entity != nil {
entityCopy = make(map[string]string)
for k, v := range s.Entity {
entityCopy[k] = v
}
}
return ReadSettings{
CharsetReader: s.CharsetReader,
Permissive: s.Permissive,
Entity: entityCopy,
}
c := *s
c.Entity = maps.Clone(s.Entity)
return c
}
// WriteSettings determine the behavior of the Document's WriteTo* functions.
@@ -913,6 +919,11 @@ func (e *Element) readFrom(ri io.Reader, settings ReadSettings) (n int64, err er
attrCheck := make(map[xml.Name]int)
dec := newDecoder(r, settings)
maxDepth := settings.MaxDepth
if maxDepth <= 0 {
maxDepth = defaultMaxDepth
}
var stack stack[*Element]
stack.push(e)
for {
@@ -942,6 +953,9 @@ func (e *Element) readFrom(ri io.Reader, settings ReadSettings) (n int64, err er
switch t := t.(type) {
case xml.StartElement:
if len(stack.data) > maxDepth {
return r.Bytes(), ErrMaxDepth
}
e := newElement(t.Name.Space, t.Name.Local, top)
if settings.PreserveDuplicateAttrs || len(t.Attr) < 2 {
for _, a := range t.Attr {
@@ -1622,7 +1636,7 @@ func (c *CharData) Index() int {
func (c *CharData) WriteTo(w Writer, s *WriteSettings) {
if c.IsCData() {
w.WriteString(`<![CDATA[`)
w.WriteString(c.Data)
sanitizeCData(w, c.Data)
w.WriteString(`]]>`)
} else {
var m escapeMode
@@ -1704,7 +1718,7 @@ func (c *Comment) Index() int {
// WriteTo serialies the comment to the writer.
func (c *Comment) WriteTo(w Writer, s *WriteSettings) {
w.WriteString("<!--")
w.WriteString(c.Data)
sanitizeComment(w, c.Data)
w.WriteString("-->")
}
@@ -1769,7 +1783,7 @@ func (d *Directive) Index() int {
// WriteTo serializes the XML directive to the writer.
func (d *Directive) WriteTo(w Writer, s *WriteSettings) {
w.WriteString("<!")
w.WriteString(d.Data)
sanitizeDirective(w, d.Data)
w.WriteString(">")
}
@@ -1837,10 +1851,10 @@ func (p *ProcInst) Index() int {
// WriteTo serializes the processing instruction to the writer.
func (p *ProcInst) WriteTo(w Writer, s *WriteSettings) {
w.WriteString("<?")
w.WriteString(p.Target)
sanitizeProcInst(w, p.Target)
if p.Inst != "" {
w.WriteByte(' ')
w.WriteString(p.Inst)
sanitizeProcInst(w, p.Inst)
}
w.WriteString("?>")
}
+114
View File
@@ -384,6 +384,120 @@ func escapeString(w Writer, s string, m escapeMode) {
w.WriteString(s[last:])
}
// sanitizeCData writes the sanitized contents of a CDATA section to the
// writer. XML provides no way to escape the "]]>" sequence within a CDATA
// section, so any occurrence of it is split across two CDATA sections.
func sanitizeCData(w Writer, s string) {
for {
i := strings.Index(s, "]]>")
if i < 0 {
break
}
w.WriteString(s[:i+2])
w.WriteString("]]><![CDATA[")
s = s[i+2:]
}
w.WriteString(s)
}
// sanitizeComment writes the sanitized contents of a comment to the writer.
// An XML comment may not contain the string "--", and it may not end with a
// '-'. Because XML provides no way to escape these sequences, spaces are
// inserted where necessary.
func sanitizeComment(w Writer, s string) {
last, hyphen := 0, false
for i := 0; i < len(s); i++ {
if s[i] != '-' {
hyphen = false
continue
}
if hyphen {
w.WriteString(s[last:i])
w.WriteByte(' ')
last = i
}
hyphen = true
}
w.WriteString(s[last:])
if hyphen {
w.WriteByte(' ')
}
}
// sanitizeProcInst writes the contents of a sanitized processing instruction
// to the writer. XML provides no way to escape the "?>" sequence within a
// processing instruction, so a space is inserted between the two characters.
func sanitizeProcInst(w Writer, s string) {
for {
i := strings.Index(s, "?>")
if i < 0 {
break
}
w.WriteString(s[:i+1])
w.WriteByte(' ')
s = s[i+1:]
}
w.WriteString(s)
}
// sanitizeDirective writes the sanitized contents of an XML directive to the
// writer.
func sanitizeDirective(w Writer, s string) {
// The XML decoder reserves the character following "<!" for comments
// ('-') and CDATA sections ('['), and it treats "<!>" as an unterminated
// directive. Insert a space to avoid conflicts with reserved sequences.
scan := s
if s == "" || s[0] == '-' || s[0] == '[' {
w.WriteByte(' ')
} else {
scan = s[1:]
}
// A directive's contents may legitimately contain '<' and '>' characters,
// so write them without modification when they are balanced.
if isDirectiveBalanced(scan) {
w.WriteString(s)
return
}
// The contents are unbalanced, so escape every character in the string.
escapeString(w, s, escapeNormal)
}
// isDirectiveBalanced returns true if the interpreted portion of an XML
// directive's contents may be enclosed by "<!" and ">" without changing the
// extents of the resulting directive.
func isDirectiveBalanced(s string) bool {
var quote byte
var depth int
for i := 0; i < len(s); i++ {
switch c := s[i]; {
case quote != 0:
if c == quote {
quote = 0
}
case c == '\'' || c == '"':
quote = c
case c == '>':
if depth == 0 {
return false
}
depth--
case c == '<':
if !strings.HasPrefix(s[i+1:], "!--") {
depth++
break
}
j := strings.Index(s[i+4:], "-->")
if j < 0 {
return false
}
i += 4 + j + 2
}
}
return quote == 0 && depth == 0
}
func isInCharacterRange(r rune) bool {
return r == 0x09 ||
r == 0x0A ||
+14 -2
View File
@@ -281,7 +281,11 @@ func (c *compiler) parseSegment(path string) segment {
c.err = ErrPath("path has invalid filter [brackets].")
break
}
seg.filters = append(seg.filters, c.parseFilter(fpath[:len(fpath)-1]))
filter := c.parseFilter(fpath[:len(fpath)-1])
if c.err != ErrPath("") {
break
}
seg.filters = append(seg.filters, filter)
}
return seg
}
@@ -320,7 +324,11 @@ func (c *compiler) parseFilter(path string) filter {
// Filter contains [@attr='val'], [@attr="val"], [fn()='val'],
// [fn()="val"], [tag='val'] or [tag="val"]?
eqindex := strings.IndexByte(path, '=')
if eqindex >= 0 && eqindex+1 < len(path) {
if eqindex == 0 {
c.err = ErrPath("path contains a filter expression with no key.")
return nil
}
if eqindex > 0 && eqindex+1 < len(path) {
quote := path[eqindex+1]
if quote == '\'' || quote == '"' {
rindex := nextIndex(path, quote, eqindex+2)
@@ -334,6 +342,10 @@ func (c *compiler) parseFilter(path string) filter {
switch {
case key[0] == '@':
if len(key) == 1 {
c.err = ErrPath("path contains a filter expression with no key.")
return nil
}
return newFilterAttrVal(key[1:], value)
case strings.HasSuffix(key, "()"):
name := key[:len(key)-2]
+7
View File
@@ -7,6 +7,12 @@ linters:
exclusions:
presets:
- std-error-handling
rules:
# Test fixtures construct CDF binary blobs from known small constants, so
# gosec's integer-overflow checks (G115) add no value there.
- path: internal/cdf/cdf_test\.go
linters:
- gosec
enable:
- gosec # Detects security problems.
# Keep all extras disabled for now to focus on the integer overflow problem.
@@ -31,6 +37,7 @@ linters:
- unused
- usestdlibvars # Detects the possibility to use variables/constants from the Go standard library.
- usetesting # Reports uses of functions with replacement inside the testing package.
- asciicheck # https://daniel.haxx.se/blog/2025/05/16/detecting-malicious-unicode/
settings:
govet:
disable:
+9 -2
View File
@@ -13,8 +13,8 @@
<a href="https://pkg.go.dev/github.com/gabriel-vasile/mimetype">
<img alt="Go Reference" src="https://pkg.go.dev/badge/github.com/gabriel-vasile/mimetype.svg">
</a>
<a href="https://goreportcard.com/report/github.com/gabriel-vasile/mimetype">
<img alt="Go report card" src="https://goreportcard.com/badge/github.com/gabriel-vasile/mimetype">
<a href="https://codecov.io/gh/gabriel-vasile/mimetype">
<img alt="Code coverage" src="https://codecov.io/gh/gabriel-vasile/mimetype/graph/badge.svg">
</a>
<a href="LICENSE">
<img alt="License" src="https://img.shields.io/badge/License-MIT-green.svg">
@@ -103,3 +103,10 @@ shows which file formats are most often misidentified and can help prioritise.
When submitting a PR for detection of a new file format, please make sure to
add a record to the list of testcases in [mimetype_test.go](mimetype_test.go).
For complex files a record can be added in the [testdata](testdata) directory.
Code contributions must respect following rules:
- code must be test covered
- code must be formatted using the `gofmt` tool
- exported names must be documented
**Important**: By submitting a pull request, you agree to allow the project
owner to license your work under the same license as that used by the project.
+1
View File
@@ -0,0 +1 @@
comment: false
+667
View File
@@ -0,0 +1,667 @@
// Package cdf implements parsing of CDF (OLE2) files. It is greatly inspired
// by src/readcdf.c from libmagic. One difference is this implementation is
// permissive of truncated inputs. See readLimit in mimetype.go for the
// reason why truncated inputs need to be handled.
// http://sc.openoffice.org/compdocfileformat.pdf
package cdf
import (
"bytes"
"encoding/binary"
"github.com/gabriel-vasile/mimetype/internal/scan"
)
type CDFType int8
const (
CDFTypeGeneric CDFType = iota
CDFTypeInstaller
CDFTypeDoc
CDFTypePpt
CDFTypeXls
CDFTypeMsg
)
// Detect parses raw as a CDF (OLE2) compound file and returns the document type
// it contains. It returns CDFTypeGeneric for input that is not a CDF file or
// whose type cannot be narrowed down.
func Detect(raw []byte) CDFType {
if len(raw) < 512 {
return CDFTypeGeneric
}
var c cdf
if !parse(raw, &c) {
return CDFTypeGeneric
}
return c.detect()
}
// cdf holds everything we need from a CDF file to do detection.
type cdf struct {
data []byte
secSize int
shortSecSize int
minStdStream uint32
satSecs int32s // list of SAT sector ids; usually a sub-slice of raw input
satEntries int // number of valid SAT entries reachable through satSecs
firstSSAT int32
dirRaw []byte // directory stream bytes (entries are decoded on demand)
sst []byte // short-stream pool (root storage's stream)
sstBuilt bool // whether sst was already loaded (it is loaded lazily)
rootStreamFirst int32 // first sector of the root storage short-stream pool
rootStreamSize uint32 // size of the root storage short-stream pool
rootStorageUUID []byte
}
// parse reads the entire on-disk structure required for type detection. It
// returns true on success and false if the header does not look like a CDF file.
// Truncated or partially malformed bodies are tolerated: sector reads degrade
// to whatever could be collected so detection can still succeed from partial data.
func parse(raw []byte, c *cdf) bool {
if len(raw) < 512 || binary.LittleEndian.Uint64(raw) != cdfMagic {
return false
}
secP2 := binary.LittleEndian.Uint16(raw[30:32])
shortP2 := binary.LittleEndian.Uint16(raw[32:34])
if secP2 > 20 || shortP2 > 20 {
return false
}
c.data = raw
c.secSize = 1 << secP2
c.shortSecSize = 1 << shortP2
c.minStdStream = binary.LittleEndian.Uint32(raw[56:60])
if c.secSize < dirEntrySize {
return false
}
firstDirSec := readSecID(raw[48:52])
c.firstSSAT = readSecID(raw[60:64])
firstMSAT := readSecID(raw[68:72])
nMSAT := binary.LittleEndian.Uint32(raw[72:76])
masterSAT := int32s{b: raw[76 : 76+4*masterSATSize]}
c.buildSAT(masterSAT, firstMSAT, nMSAT)
c.dirRaw = c.readLong(firstDirSec, 0)
c.rootStreamFirst = -1
var d dirEntry
for i, n := 0, c.dirLen(); i < n; i++ {
c.dirAt(i, &d)
if d.typ != dirTypeRootStorage || d.streamFirst < 0 {
continue
}
c.rootStorageUUID = d.storageUUID[:]
// Record where the short-stream pool lives; it is loaded lazily by
// shortStream the first time a short stream is actually read.
c.rootStreamFirst = d.streamFirst
c.rootStreamSize = d.size
break
}
return true
}
func (c *cdf) detect() CDFType {
for _, name := range []string{"\x05SummaryInformation", "\x05DocumentSummaryInformation"} {
if t, ok := c.detectFromSummary(name); ok {
return t
}
}
var d dirEntry
for i, n := 0, c.dirLen(); i < n; i++ {
c.dirAt(i, &d)
if t, ok := lookupSection(d.nameBytes(), d.typ); ok {
return t
}
}
return CDFTypeGeneric
}
// detectFromSummary inspects a (Doc)SummaryInformation stream and tries to
// derive a CDFType from the root-storage CLSID, the property NameOfApplication,
// and finally the names of sibling user streams.
func (c *cdf) detectFromSummary(streamName string) (CDFType, bool) {
if c.rootStorageUUID != nil && bytes.Equal(c.rootStorageUUID, msiCLSID) {
return CDFTypeInstaller, true
}
raw, ok := c.userStream(streamName)
if !ok {
return CDFTypeGeneric, false
}
if app := summaryAppName(raw); len(app) > 0 {
if t, ok := lookupSubstring(app, app2type); ok {
return t, true
}
}
for i, n := 0, c.dirLen(); i < n; i++ {
var d dirEntry
c.dirAt(i, &d)
if d.nameLen == 0 {
continue
}
if t, ok := lookupSubstring(d.nameBytes(), name2type); ok {
return t, true
}
}
return CDFTypeGeneric, true
}
const (
cdfMagic uint64 = 0xE11AB1A1E011CFD0
dirTypeUserStorage = 1
dirTypeUserStream = 2
dirTypeRootStorage = 5
dirEntrySize = 128
masterSATSize = 109 // first 109 SAT secids live in the file header
)
// dirEntry is a single CDF directory record. The UTF-16LE name is pre-decoded
// into an inline ASCII buffer at parse time, avoiding a per-entry heap
// allocation while keeping comparisons trivial. CDF names are at most 32
// UTF-16 code units, so 32 bytes always suffice.
type dirEntry struct {
name [32]byte
nameLen uint8
typ uint8
streamFirst int32
size uint32
storageUUID [16]byte
}
// nameBytes returns the decoded ASCII name without copying.
func (d *dirEntry) nameBytes() []byte { return d.name[:d.nameLen] }
func (c *cdf) ssatAt(i int32) int32 {
for sid := c.firstSSAT; sid >= 0; {
if int(sid) >= c.satLen() {
break // SAT is truncated; stop collecting
}
buf, ok := c.sector(sid)
if !ok {
break
}
lbuf := int32(len(buf) / 4) //nolint:gosec // anything divided by 4 fits int32
if i < lbuf {
return int32(binary.LittleEndian.Uint32(buf[4*i:])) //nolint:gosec // intentional two's-complement reinterpretation of a sector id
}
i -= lbuf
sid = c.satAt(sid)
}
return -1
}
// shortStream returns the root storage short-stream pool, loading it on first
// use. Detection often finishes (e.g. via the root CLSID or a long-stream
// summary) without ever reading a short stream, so building this eagerly would
// be wasted work.
func (c *cdf) shortStream() []byte {
if !c.sstBuilt {
c.sstBuilt = true
if c.rootStreamFirst >= 0 {
c.sst = c.readLong(c.rootStreamFirst, c.rootStreamSize)
}
}
return c.sst
}
// int32s works like a slice of LE int32 and is backed by a slice of bytes.
// int32s could very well be type int32s []byte, but that would mean
// len function can be called on it. We don't want that, we always want to use
// the len method.
type int32s struct {
b []byte
}
func (b int32s) at(i int) int32 {
//nolint:gosec // intentional two's-complement reinterpretation of a sector id
return int32(binary.LittleEndian.Uint32(b.b[4*i:]))
}
func (b int32s) len() int {
return len(b.b) / 4
}
// readSecID reinterprets four little-endian bytes as a signed sector id.
// Every 32-bit pattern is a valid id (values >= 0 are sector numbers,
// negatives are CDF sentinels such as -2 end-of-chain), so the conversion is
// an intentional two's-complement reinterpretation rather than an overflow.
func readSecID(b []byte) int32 {
return int32(binary.LittleEndian.Uint32(b)) //nolint:gosec // intentional two's-complement reinterpretation
}
// satLen is the number of sector ids reachable through the SAT.
func (c *cdf) satLen() int { return c.satEntries }
// satAt returns the i-th sector id from the SAT. Callers must ensure
// i < satLen(). The SAT is not materialized; the entry is fetched directly
// from the input by translating i into (SAT sector index, entry offset).
func (c *cdf) satAt(i int32) int32 {
perSec := c.secSize / 4
secIdx := int(i) / perSec
entryIdx := int(i) % perSec
secID := c.satSecs.at(secIdx)
off := c.secSize*(1+int(secID)) + 4*entryIdx
return readSecID(c.data[off:])
}
// sector returns the bytes of long sector secid. If the file is truncated
// inside the requested sector the result is the available bytes (no padding).
// If the sector starts past EOF or secid is negative, then ok is false.
func (c *cdf) sector(secid int32) (_ []byte, ok bool) {
if secid < 0 {
return nil, false
}
off := int64(c.secSize) * (1 + int64(secid))
if off >= int64(len(c.data)) {
return nil, false
}
// The returned sector might be truncated,
// but we still return it as best effort.
end := min(off+int64(c.secSize), int64(len(c.data)))
// If not even one int32 fits, then fail.
if end-off < 4 {
return nil, false
}
return c.data[off:end], true
}
func (c *cdf) sectorIDs(secid int32) (int32s, bool) {
buf, ok := c.sector(secid)
if !ok {
return int32s{}, ok
}
return int32s{b: buf}, true
}
// buildSAT records the list of SAT sector ids from the master-SAT (header)
// plus any extension blocks chained via firstMSAT. The SAT itself is not
// materialized: satAt computes the requested entry directly from c.data via
// satSecs. In the common case (no extension chain) satSecs is a zero-copy
// sub-slice of the input header.
func (c *cdf) buildSAT(masterSAT int32s, firstMSAT int32, nMSAT uint32) {
// Fast path: no extension chain. masterSAT is already a sub-slice of raw
// input; reuse it directly.
if firstMSAT < 0 || nMSAT == 0 {
c.satSecs = masterSAT
c.satEntries = c.computeSATLen()
return
}
// Slow path: gather sector ids from the header plus the extension chain
// into a fresh buffer. Even here we only allocate space for ids (4 bytes
// each), not the full SAT contents.
maxIDs := len(c.data)/c.secSize + 1
buf := make([]byte, 0, 4*masterSATSize)
for i := 0; i < masterSAT.len(); i++ {
if masterSAT.at(i) < 0 {
break
}
buf = append(buf, masterSAT.b[4*i:4*i+4]...)
}
perSec := c.secSize/4 - 1
mid := firstMSAT
chain:
for j := uint32(0); j < nMSAT && mid >= 0; j++ {
msa, ok := c.sectorIDs(mid)
if !ok {
break
}
for k := 0; k < perSec; k++ {
if k >= msa.len() || msa.at(k) < 0 {
break chain
}
buf = append(buf, msa.b[4*k:4*k+4]...)
if len(buf)/4 > maxIDs {
break chain // cyclic MSAT chain; stop allocating
}
}
if perSec >= msa.len() {
break // no next-MSAT pointer available
}
mid = msa.at(perSec)
}
c.satSecs = int32s{b: buf}
c.satEntries = c.computeSATLen()
}
// computeSATLen walks satSecs and counts how many SAT entries are actually
// reachable in c.data, stopping at the first sentinel id or sector that is not
// fully present in the file.
func (c *cdf) computeSATLen() int {
perSec := c.secSize / 4
total := 0
for i := 0; i < c.satSecs.len(); i++ {
sec := c.satSecs.at(i)
if sec < 0 {
break
}
off := int64(c.secSize) * (1 + int64(sec))
if off >= int64(len(c.data)) {
break
}
avail := int64(len(c.data)) - off
if avail >= int64(c.secSize) {
total += perSec
continue
}
total += int(avail / 4)
break
}
return total
}
// readLong reads a long-sector chain starting at sid. If length > 0 the
// result is truncated to that many bytes. On truncation or any other failure
// it returns whatever sectors were readable.
func (c *cdf) readLong(sid int32, length uint32) []byte {
// Fast path: when the chain is a single physically contiguous run of
// sectors (the common case for the directory and summary streams) the data
// is already laid out sequentially in the input, so return a sub-slice of
// it instead of allocating a buffer and copying every sector.
if sid >= 0 {
maxSec := len(c.data)/c.secSize + 1
n, s := 0, sid
contiguous := true
for s >= 0 {
if int(s) >= c.satLen() {
break // SAT truncated; what remains is still contiguous
}
n++
if n > maxSec {
contiguous = false // cyclic chain; let the slow path guard it
break
}
next := c.satAt(s)
if next >= 0 && int64(next) != int64(s)+1 {
contiguous = false
break
}
s = next
}
if contiguous {
off64 := int64(c.secSize) * (1 + int64(sid))
if off64 >= int64(len(c.data)) {
return nil
}
end64 := min(off64+int64(n)*int64(c.secSize), int64(len(c.data)))
out := c.data[off64:end64]
if length > 0 && int64(length) < int64(len(out)) {
out = out[:length]
}
return out
}
}
// Slow path: gather a fragmented chain into a fresh buffer. Real-world
// writers (MSI builders, edited Office documents) routinely produce
// non-contiguous directory and stream chains, so this fallback is required
// for correct detection on those files.
maxBytes := len(c.data)
out := make([]byte, 0, c.secSize)
for sid >= 0 {
if int(sid) >= c.satLen() {
break // SAT truncated; return what we have
}
buf, ok := c.sector(sid)
if !ok {
break
}
out = append(out, buf...)
if len(out) >= maxBytes {
break // chain longer than the file: cyclic SAT, stop
}
sid = c.satAt(sid)
}
if length > 0 && int64(length) < int64(len(out)) {
out = out[:length]
}
return out
}
// readShort reads a short-sector chain at sid by indexing into the short-stream
// pool. On truncation or if the pool is unavailable it returns whatever was
// readable (possibly nil).
func (c *cdf) readShort(sid int32, length uint32) []byte {
sst := c.shortStream()
if sst == nil {
return nil
}
// TODO: anyway to avoid allocating and copying the bytes?
out := make([]byte, 0, c.shortSecSize)
for sid >= 0 {
off64 := int64(sid) * int64(c.shortSecSize)
if off64+int64(c.shortSecSize) > int64(len(sst)) {
break // short-stream pool truncated or sid out of range
}
off := int(off64)
out = append(out, sst[off:off+c.shortSecSize]...)
if len(out) >= len(sst) {
break // chain longer than the pool: cyclic SSAT, stop
}
sid = c.ssatAt(sid)
}
if length > 0 && int64(length) < int64(len(out)) {
out = out[:length]
}
return out
}
// readChain dispatches to the long or short reader depending on stream size.
func (c *cdf) readChain(sid int32, length uint32) []byte {
if length < c.minStdStream && c.rootStreamFirst >= 0 {
return c.readShort(sid, length)
}
return c.readLong(sid, length)
}
// dirLen returns the number of directory entries in dirRaw.
func (c *cdf) dirLen() int { return len(c.dirRaw) / dirEntrySize }
// dirAt decodes the i-th directory entry into *out. Callers must ensure
// i < dirLen(). The UTF-16LE name is decoded into out.name, ASCII-style,
// stopping at the first NUL.
func (c *cdf) dirAt(i int, out *dirEntry) {
raw := c.dirRaw[i*dirEntrySize:]
nameLen := min(int(binary.LittleEndian.Uint16(raw[64:])), 64)
k := uint8(0)
for j := 0; j < nameLen/2; j++ {
// Names are ASCII; keep the low byte of each little-endian UTF-16
// code unit and stop at the first NUL.
lo, hi := raw[2*j], raw[2*j+1]
if lo == 0 && hi == 0 {
break
}
out.name[k] = lo
k++
}
out.nameLen = k
out.typ = raw[66]
out.streamFirst = readSecID(raw[116:120])
out.size = binary.LittleEndian.Uint32(raw[120:])
copy(out.storageUUID[:], raw[80:96])
}
// userStream finds a user stream by name and returns its bytes.
func (c *cdf) userStream(name string) ([]byte, bool) {
var d dirEntry
for i, n := 0, c.dirLen(); i < n; i++ {
c.dirAt(i, &d)
if d.typ == dirTypeUserStream && string(d.nameBytes()) == name {
buf := c.readChain(d.streamFirst, d.size)
if buf == nil {
return nil, false
}
return buf, true
}
}
return nil, false
}
const (
propIDNameOfApplication = 0x12
typeMask = 0x0fff
typeVector = 0x1000
typeStringASCII = 0x1e
typeStringWide = 0x1f
sectionDeclOffset = 0x1c // section declaration in property-set header
)
// summaryAppName parses a (Doc)SummaryInformation stream and returns the
// value of property NameOfApplication (0x12) as printable ASCII, or nil if
// not present or the stream is malformed. This is the only summary property
// the detection logic ever consults.
func summaryAppName(stream []byte) []byte {
if len(stream) < sectionDeclOffset+20 {
return nil
}
sdOff := binary.LittleEndian.Uint32(stream[sectionDeclOffset+16:])
if uint64(sdOff)+8 > uint64(len(stream)) {
return nil
}
section := stream[sdOff:]
shLen := binary.LittleEndian.Uint32(section[0:])
nProps := binary.LittleEndian.Uint32(section[4:])
if uint64(shLen) > uint64(len(section)) || nProps > 1<<16 || 8+8*nProps > shLen {
return nil
}
for i := uint32(0); i < nProps; i++ {
base := 8 + 8*i
id := binary.LittleEndian.Uint32(section[base:])
if id != propIDNameOfApplication {
continue
}
off := binary.LittleEndian.Uint32(section[base+4:])
if uint64(off)+8 > uint64(shLen) {
return nil
}
typ := binary.LittleEndian.Uint32(section[off:])
if typ&typeVector != 0 {
return nil
}
step := uint32(0)
switch typ & typeMask {
case typeStringASCII:
step = 1
case typeStringWide:
step = 2
default:
return nil
}
slen := binary.LittleEndian.Uint32(section[off+4:])
start := uint64(off) + 8
end := start + uint64(slen)*uint64(step)
if end > uint64(shLen) {
return nil
}
return printableLowBytes(section[start:end], int(step))
}
return nil
}
// printableLowBytes copies the printable low byte of each step-byte unit
// in b, stopping at the first NUL.
func printableLowBytes(b []byte, step int) []byte {
out := make([]byte, 0, len(b)/step)
for i := 0; i+step <= len(b); i += step {
c := b[i]
if c == 0 {
break
}
if c >= 0x20 && c < 0x7f {
out = append(out, c)
}
}
return out
}
// pattern is a case-insensitive substring → CDFType mapping. Entries are
// tested in order; first match wins. needle is stored upper-cased so it can be
// matched case-insensitively by scan.Bytes.Search with scan.IgnoreCase.
type pattern struct {
needle []byte
typ CDFType
}
// app2type maps NameOfApplication values to CDFTypes.
// Mirrors app2mime[] in libmagic. Needles are upper-cased for case-insensitive
// matching via scan.IgnoreCase.
var app2type = []pattern{
{[]byte("WORD"), CDFTypeDoc},
{[]byte("EXCEL"), CDFTypeXls},
{[]byte("POWERPOINT"), CDFTypePpt},
{[]byte("ADVANCED INSTALLER"), CDFTypeInstaller},
{[]byte("INSTALLSHIELD"), CDFTypeInstaller},
{[]byte("MICROSOFT PATCH COMPILER"), CDFTypeInstaller},
{[]byte("NANT"), CDFTypeInstaller},
{[]byte("WINDOWS INSTALLER"), CDFTypeInstaller},
}
// name2type maps directory entry names to CDFTypes.
// Mirrors name2mime[] in libmagic. Needles are upper-cased for case-insensitive
// matching via scan.IgnoreCase.
var name2type = []pattern{
{[]byte("BOOK"), CDFTypeXls},
{[]byte("WORKBOOK"), CDFTypeXls},
{[]byte("WORDDOCUMENT"), CDFTypeDoc},
{[]byte("POWERPOINT"), CDFTypePpt},
{[]byte("DIGITALSIGNATURE"), CDFTypeInstaller},
}
// lookupSubstring returns the CDFType for the first entry in t whose needle
// is a case-insensitive substring of v. Mirrors C's strcasestr semantics
// under the C locale. It allocates nothing: scan.IgnoreCase matches the
// upper-cased needle against input of either case.
func lookupSubstring(v []byte, t []pattern) (CDFType, bool) {
s := scan.Bytes(v)
for _, p := range t {
if i, _ := s.Search(p.needle, scan.IgnoreCase); i != -1 {
return p.typ, true
}
}
return CDFTypeGeneric, false
}
// msiCLSID is the Microsoft Installer root-storage CLSID, in on-disk byte
// order (cdf_directory_t.d_storage_uuid stores two little-endian uint64s).
var msiCLSID = []byte{
0x84, 0x10, 0x0c, 0x00, 0x00, 0x00, 0x00, 0x00,
0xc0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46,
}
// section is a (directory entry name, type) → CDFType mapping.
type section struct {
name string
typ uint8
cdf CDFType
}
// sectionTypes maps distinctive directory entries to CDFTypes — a flattened
// equivalent of sectioninfo[] in libmagic. Used as a fallback when no
// SummaryInformation stream is present. A slice (rather than a map) lets
// lookupSection compare entry names without allocating a string key.
var sectionTypes = []section{
// libmagic uses application/encrypted, but that is not a registered media type.
// For now, we skip identifying that and fall-back on CDFTypeGeneric
// {"EncryptedPackage", dirTypeUserStream, CDFTypeEncrypted},
// {"EncryptedSummary", dirTypeUserStream, CDFTypeEncrypted},
{"Book", dirTypeUserStream, CDFTypeXls},
{"Workbook", dirTypeUserStream, CDFTypeXls},
{"WordDocument", dirTypeUserStream, CDFTypeDoc},
{"PowerPoint Document", dirTypeUserStream, CDFTypePpt},
{"__properties_version1.0", dirTypeUserStream, CDFTypeMsg},
{"__recip_version1.0_#00000000", dirTypeUserStorage, CDFTypeMsg},
}
// lookupSection returns the CDFType for a directory entry whose name and type
// match a sectionTypes entry exactly. The string(name) == comparison is
// optimized by the compiler to avoid allocating.
func lookupSection(name []byte, typ uint8) (CDFType, bool) {
for _, s := range sectionTypes {
if s.typ == typ && string(name) == s.name {
return s.cdf, true
}
}
return CDFTypeGeneric, false
}
+1 -21
View File
@@ -84,19 +84,8 @@ func FromPlain(content []byte) string {
break
}
}
hasHighBit := false
for _, c := range content {
if c >= 0x80 {
hasHighBit = true
break
}
}
if hasHighBit && utf8.Valid(content) {
return "utf-8"
}
// ASCII is a subset of UTF8. Follow W3C recommendation and replace with UTF8.
if ascii(origContent) {
if utf8.Valid(content) {
return "utf-8"
}
@@ -123,15 +112,6 @@ func latin(content []byte) string {
return "iso-8859-1"
}
func ascii(content []byte) bool {
for _, b := range content {
if textChars[b] != T {
return false
}
}
return true
}
// FromXML returns the charset of an XML document. It relies on the XML
// header <?xml version="1.0" encoding="UTF-8"?> and falls back on the plain
// text content.
+3 -3
View File
@@ -12,10 +12,10 @@ import (
type Parser struct {
comma byte
comment byte
s scan.Bytes
s *scan.Bytes
}
func NewParser(comma, comment byte, s scan.Bytes) *Parser {
func NewParser(comma, comment byte, s *scan.Bytes) *Parser {
return &Parser{
comma: comma,
comment: comment,
@@ -55,7 +55,7 @@ func (r *Parser) CountFields(collectIndexes bool) (fields int, fieldPos []int, h
if finished {
return 0, nil, false
}
finished = len(r.s) == 0 && len(line) == 0
finished = len(*r.s) == 0 && len(line) == 0
if len(line) == lengthNL(line) {
line = nil
continue // Skip empty lines.
+5
View File
@@ -10,6 +10,7 @@ const (
QueryGeo = "geo"
QueryHAR = "har"
QueryGLTF = "gltf"
QueryCDX = "cdx"
maxRecursion = 4096
)
@@ -40,6 +41,10 @@ var queries = map[string][]query{
SearchPath: [][]byte{[]byte("asset"), []byte("version")},
SearchVals: [][]byte{[]byte(`"1.0"`), []byte(`"2.0"`)},
}},
QueryCDX: {{
SearchPath: [][]byte{[]byte("bomFormat")},
SearchVals: [][]byte{[]byte(`"CycloneDX"`)},
}},
}
var parserPool = sync.Pool{
+68 -15
View File
@@ -3,6 +3,8 @@ package magic
import (
"bytes"
"encoding/binary"
"github.com/gabriel-vasile/mimetype/internal/mp3"
)
// Flac matches a Free Lossless Audio Codec file.
@@ -51,32 +53,83 @@ func AAC(raw []byte, _ uint32) bool {
return len(raw) > 1 && ((raw[0] == 0xFF && raw[1] == 0xF1) || (raw[0] == 0xFF && raw[1] == 0xF9))
}
// Mp3 matches an mp3 file.
func Mp3(raw []byte, limit uint32) bool {
// MP3 matches a .mp3 file.
func MP3(raw []byte, limit uint32) bool {
if len(raw) < 3 {
return false
}
if bytes.HasPrefix(raw, []byte("ID3")) {
// MP3s with an ID3v2 tag will start with "ID3"
// ID3v1 tags, however appear at the end of the file.
// Any ID3v2 is reported as MP3. Not entirely correct, but the mimesniff
// standard says so. https://mimesniff.spec.whatwg.org/#matching-an-audio-or-video-type-pattern
// Despite the standard only checking for "ID3", we do more validations to
// avoid false positives.
if id3v2(raw) {
return true
}
// Match MP3 files without tags
// If no ID3v2 tag found, then we will look for MP3 frames, but:
// a. Layer III files are a lot more prevalent than Layer I and II.
// b. Layer I frame header has looser constraints than the others: many files
// with regularly repeating 0xFFFF bytes can be misidentified as MP3.
// c. MP3 files are composed of individual frames and those frames can have
// leading garbage bytes: if we want to find all valid MP3s, we have to do a
// linear search. #775, #310
// d. There are file formats that contain MP3s inside: .mo3 and .swa
//
// Given a, b, c and d, this code:
// - initially tries to match by first two bytes in header
// - checks for .mo3 and .swa and disqualifies them
// - does linear search for Layer III
switch binary.BigEndian.Uint16(raw[:2]) & 0xFFFE {
case 0xFFFA:
// MPEG ADTS, layer III, v1
return true
case 0xFFF2:
// MPEG ADTS, layer III, v2
return true
case 0xFFE2:
// MPEG ADTS, layer III, v2.5
case 0xFFFA, 0xFFF2, 0xFFE2, // layer III: v1, v2, v2.5
0xFFFC, 0xFFF4, // layer II: v1, v2
0xFFF5: // layer I: v2
return true
}
// http://lclevy.free.fr/mo3/
if bytes.HasPrefix(raw, []byte("MO3")) {
return false
}
return false
// From PRONOM:
// Macromedia licensed the MP3 technology in 1995 to use in their Shockwave
// product. .swa or Shockwave Audio was originally added as a free plugin
// (Xtras) to SoundEdit 16 to export AIFF files to .swa.
// There is no media type assigned for .swa.
if bytes.HasPrefix(raw, []byte{0x00, 0x00, 0x01, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00}) {
return false
}
_, size := mp3.ExtractFrame(raw)
return size > 0
}
// Based on https://id3.org/Developer%20Information.
func id3v2(raw []byte) bool {
if len(raw) < 10 || !bytes.HasPrefix(raw, []byte("ID3")) {
return false
}
if raw[3] < 2 || raw[3] > 4 { // Version: ID3v2.2 - ID3v2.4.
return false
}
if raw[4] != 0 { // Revision is 0 for all versions.
return false
}
// v2.2 uses 2 bits, v2.3 uses 3 bits and v2.4 uses 4.
// For all versions least significant 4 bits should be 0
if raw[5]&0b1111 != 0 {
return false
}
// Size bytes are synchsafe: most significant bit always 0.
if raw[6]&0x80 != 0 || raw[7]&0x80 != 0 || raw[8]&0x80 != 0 || raw[9]&0x80 != 0 {
return false
}
size := uint32(raw[6])<<21 | uint32(raw[7])<<14 | uint32(raw[8])<<7 | uint32(raw[9])
// Disallow too big frames, let's say 10MB.
return size > 0 && size < 10*1024*1024
}
// Wav matches a Waveform Audio File Format file.
+62 -7
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"debug/macho"
"encoding/binary"
"slices"
)
// Lnk matches Microsoft lnk binary format.
@@ -117,13 +118,7 @@ func Dbf(raw []byte, limit uint32) bool {
0x02, 0x03, 0x04, 0x05, 0x30, 0x31, 0x32, 0x42, 0x62, 0x7B, 0x82,
0x83, 0x87, 0x8A, 0x8B, 0x8E, 0xB3, 0xCB, 0xE5, 0xF5, 0xF4, 0xFB,
}
for _, b := range dbfTypes {
if raw[0] == b {
return true
}
}
return false
return slices.Contains(dbfTypes, raw[0])
}
// ElfObj matches an object file.
@@ -229,3 +224,63 @@ func TzIf(raw []byte, limit uint32) bool {
// Version has to be NUL (0x00), '2' (0x32) or '3' (0x33).
return raw[4] == 0x00 || raw[4] == 0x32 || raw[4] == 0x33
}
// Pyc matches a Python compiled file.
// The signatures are sourced from libmagic v5.47
func Pyc(raw []byte, limit uint32) bool {
if len(raw) < 8 {
return false
}
// python 1.0 through 3.7 signatures, magic/Magdir/python:13:190
pycMagic := []uint32{
0x02099900, 0x03099900, 0x892e0d0a, 0x04170d0a, 0x994e0d0a, 0xfcc40d0a,
0xfdc40d0a, 0x87c60d0a, 0x88c60d0a, 0x2aeb0d0a, 0x2beb0d0a, 0x2ded0d0a,
0x2eed0d0a, 0x3bf20d0a, 0x3cf20d0a, 0x45f20d0a, 0x59f20d0a, 0x63f20d0a,
0x6df20d0a, 0x6ef20d0a, 0x77f20d0a, 0x81f20d0a, 0x8bf20d0a, 0x8cf20d0a,
0x95f20d0a, 0x9ff20d0a, 0xa9f20d0a, 0xb3f20d0a, 0xb4f20d0a, 0xc7f20d0a,
0xd1f20d0a, 0xd2f20d0a, 0xdbf20d0a, 0xe5f20d0a, 0xeff20d0a, 0xf9f20d0a,
0x03f30d0a, 0x04f30d0a, 0x0af30d0a, 0xb80b0d0a, 0xc20b0d0a, 0xcc0b0d0a,
0xd60b0d0a, 0xe00b0d0a, 0xea0b0d0a, 0xf40b0d0a, 0xf50b0d0a, 0xff0b0d0a,
0x090c0d0a, 0x130c0d0a, 0x1d0c0d0a, 0x1f0c0d0a, 0x270c0d0a, 0x3b0c0d0a,
0x450c0d0a, 0x4f0c0d0a, 0x580c0d0a, 0x620c0d0a, 0x6c0c0d0a, 0x760c0d0a,
0x800c0d0a, 0x8a0c0d0a, 0x940c0d0a, 0x9e0c0d0a, 0xb20c0d0a, 0xbc0c0d0a,
0xc60c0d0a, 0xd00c0d0a, 0xda0c0d0a, 0xe40c0d0a, 0xee0c0d0a, 0xf80c0d0a,
0x020d0d0a, 0x0c0d0d0a, 0x160d0d0a, 0x170d0d0a, 0x200d0d0a, 0x210d0d0a,
0x2a0d0d0a, 0x2b0d0d0a, 0x2c0d0d0a, 0x2d0d0d0a, 0x2f0d0d0a, 0x300d0d0a,
0x310d0d0a, 0x320d0d0a, 0x330d0d0a, 0x3e0d0d0a, 0x3f0d0d0a,
}
n := binary.BigEndian.Uint32(raw)
if slices.Contains(pycMagic, n) {
return true
}
if raw[2] == 0x0d && raw[3] == 0x0a {
// Only two bits of flag field are currently used.
if l := binary.LittleEndian.Uint32(raw[4:]); l > 3 {
return false
}
if raw[1] == 0x0d || raw[1] == 0x0e {
return true
}
// PyPy magic numbers, magic/Magdir/python:233
n := binary.LittleEndian.Uint16(raw)
return n == 240 || n == 256 || n == 336 || n == 384 || n == 416
}
return false
}
// Pcap identifies "libpcap" capture files.
// https://www.tcpdump.org/manpages/pcap-savefile.5.html
func Pcap(raw []byte, _ uint32) bool {
if len(raw) < 4 {
return false
}
be := binary.BigEndian.Uint32(raw)
le := binary.LittleEndian.Uint32(raw)
return be == 0xa1b2c3d4 || be == 0xa1b23c4d ||
le == 0xa1b2c3d4 || le == 0xa1b23c4d
}
+41 -8
View File
@@ -3,6 +3,7 @@ package magic
import (
"bytes"
"encoding/binary"
"slices"
)
// Woff matches a Web Open Font Format file.
@@ -29,12 +30,13 @@ func Ttf(raw []byte, limit uint32) bool {
if !bytes.HasPrefix(raw, []byte{0x00, 0x01, 0x00, 0x00}) {
return false
}
// We cannot rely on the first 4 bytes because of false-positives.
// We have to digg deeper into the SFNT tables.
return hasSFNTTable(raw)
}
func hasSFNTTable(raw []byte) bool {
// 49 possible tables as explained below
if len(raw) < 16 || binary.BigEndian.Uint16(raw[4:]) >= 49 {
if len(raw) < 16 {
return false
}
@@ -87,14 +89,45 @@ func hasSFNTTable(raw []byte) bool {
0x6e616d65, // "name"
0x6f706264, // "opbd"
0x4f532f32, // "OS/2"
// The above tables come from the original Apple TTF specification,
// but the later Microsoft specification has additional tables.
// Common tables: https://learn.microsoft.com/en-us/typography/opentype/spec/otvarcommonformats
// Layout tables: https://learn.microsoft.com/en-us/typography/opentype/spec/chapter2
// Even if the Microsoft specification says OpenType, the tables are
// valid for TrueType as well.
0x47535542, // "GSUB"
0x47504f53, // "GPOS"
0x42415345, // "BASE"
0x4a535446, // "JSTF"
0x47444546, // "GDEF"
0x4d415448, // "MATH"
0x43424454, // "CBDT"
0x43424c43, // "CBLC"
0x43464620, // "CFF "
0x43464632, // "CFF2"
0x434f4c52, // "COLR"
0x4350414c, // "CPAL"
0x44534947, // "DSIG"
0x45424454, // "EBDT"
0x45424c43, // "EBLC"
0x48564152, // "HVAR"
0x4c545348, // "LTSH"
0x4d455247, // "MERG"
0x4d564152, // "MVAR"
0x50434c54, // "PCLT"
0x706f7374, // "post"
0x70726570, // "prep"
0x73626978, // "sbix"
0x53544154, // "STAT"
0x53564720, // "SVG "
0x56444d58, // "VDMX"
0x76686561, // "vhea"
0x766d7478, // "vmtx"
0x564f5247, // "VORG"
0x56564152, // "VVAR"
}
ourTable := binary.BigEndian.Uint32(raw[12:16])
for _, t := range possibleTables {
if ourTable == t {
return true
}
}
return false
return slices.Contains(possibleTables, ourTable)
}
// Eot matches an Embedded OpenType font file.
+2 -7
View File
@@ -3,6 +3,7 @@ package magic
import (
"bytes"
"encoding/binary"
"slices"
)
// Shp matches a shape format file.
@@ -39,13 +40,7 @@ func Shp(raw []byte, limit uint32) bool {
31, // MultiPatch
}
for _, st := range shapeTypes {
if st == int(binary.LittleEndian.Uint32(raw[108:112])) {
return true
}
}
return false
return slices.Contains(shapeTypes, int(binary.LittleEndian.Uint32(raw[108:112])))
}
// Shx matches a shape index format file.
+27 -1
View File
@@ -4,6 +4,8 @@ import (
"bytes"
"encoding/binary"
"slices"
"github.com/gabriel-vasile/mimetype/internal/scan"
)
// Png matches a Portable Network Graphics file.
@@ -15,7 +17,31 @@ func Png(raw []byte, _ uint32) bool {
// Apng matches an Animated Portable Network Graphics file.
// https://wiki.mozilla.org/APNG_Specification
func Apng(raw []byte, _ uint32) bool {
return offset(raw, []byte("acTL"), 37)
b := scan.Bytes(raw)
b.Advance(8) // the first 8 bytes matched by regular png
// PNG chunks are composed of:
// 4 bytes: length in big endian
// 4 bytes: chunk type
// length bytes: chunk data
// 4 bytes: CRC
//
// Limit to 32, so we don't waste time on huge inputs.
// acTL chunk must come before any IDAT chunks.
// https://www.w3.org/TR/png-3/#structure
for i := 0; i < 32 && len(b) > 0; i++ {
sz, _ := b.Uint32be()
if bytes.HasPrefix(b, []byte("acTL")) {
return true
}
if bytes.HasPrefix(b, []byte("IDAT")) {
return false
}
if !b.Advance(int(sz + 8)) {
return false
}
}
return false
}
// Jpg matches a Joint Photographic Experts Group file.
+7 -2
View File
@@ -136,6 +136,11 @@ func ftyp(raw []byte, sigs ...[]byte) bool {
return false
}
type shebangSig struct {
sig []byte
flag scan.Flags
}
// A valid shebang starts with the "#!" characters,
// followed by any number of spaces,
// followed by the path to the interpreter,
@@ -146,7 +151,7 @@ func ftyp(raw []byte, sigs ...[]byte) bool {
// #! /usr/bin/env php
//
// /usr/bin/env is the interpreter, php is the first and only argument.
func shebang(b scan.Bytes, matchFlags scan.Flags, sigs ...[]byte) bool {
func shebang(b scan.Bytes, sigs ...shebangSig) bool {
line := b.Line()
if len(line) < 2 || line[0] != '#' || line[1] != '!' {
return false
@@ -154,7 +159,7 @@ func shebang(b scan.Bytes, matchFlags scan.Flags, sigs ...[]byte) bool {
line = line[2:]
line.TrimLWS()
for _, s := range sigs {
if line.Match(s, matchFlags) != -1 {
if line.Match(s.sig, s.flag) != -1 {
return true
}
}
+39 -13
View File
@@ -3,6 +3,8 @@ package magic
import (
"bytes"
"encoding/binary"
"github.com/gabriel-vasile/mimetype/internal/cdf"
)
// Xlsx matches a Microsoft Excel 2007 file.
@@ -47,6 +49,15 @@ func Ole(raw []byte, limit uint32) bool {
// Doc matches a Microsoft Word 97-2003 file.
// See: https://github.com/decalage2/oletools/blob/412ee36ae45e70f42123e835871bac956d958461/oletools/common/clsid.py
func Doc(raw []byte, _ uint32) bool {
fromParsing := cdf.Detect(raw)
if fromParsing == cdf.CDFTypeDoc {
return true
}
if fromParsing != cdf.CDFTypeGeneric {
return false
}
// Fallback for inputs where the CDF directory is past the read limit: match
// the root storage CLSID, which often lies within the first sectors.
clsids := [][]byte{
// Microsoft Word 97-2003 Document (Word.Document.8)
{0x06, 0x09, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46},
@@ -55,19 +66,25 @@ func Doc(raw []byte, _ uint32) bool {
// Microsoft Word Picture (Word.Picture.8)
{0x07, 0x09, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46},
}
for _, clsid := range clsids {
if matchOleClsid(raw, clsid) {
return true
}
}
return false
}
// Ppt matches a Microsoft PowerPoint 97-2003 file or a PowerPoint 95 presentation.
func Ppt(raw []byte, limit uint32) bool {
// Root CLSID test is the safest way to detect identify OLE, however, the format
fromParsing := cdf.Detect(raw)
if fromParsing == cdf.CDFTypePpt {
return true
}
if fromParsing != cdf.CDFTypeGeneric {
return false
}
// Fallback for inputs where the CDF directory is past the read limit.
// Root CLSID test is the safest way to identify the OLE, however, the format
// often places the root CLSID at the end of the file.
if matchOleClsid(raw, []byte{
0x10, 0x8d, 0x81, 0x64, 0x9b, 0x4f, 0xcf, 0x11,
@@ -94,18 +111,21 @@ func Ppt(raw []byte, limit uint32) bool {
}
}
if bytes.HasPrefix(raw[512:], []byte{0xFD, 0xFF, 0xFF, 0xFF}) &&
raw[518] == 0x00 && raw[519] == 0x00 {
return true
}
return lin > 1152 && bytes.Contains(raw[1152:min(4096, lin)],
[]byte("P\x00o\x00w\x00e\x00r\x00P\x00o\x00i\x00n\x00t\x00 D\x00o\x00c\x00u\x00m\x00e\x00n\x00t"))
}
// Xls matches a Microsoft Excel 97-2003 file.
func Xls(raw []byte, limit uint32) bool {
// Root CLSID test is the safest way to detect identify OLE, however, the format
fromParsing := cdf.Detect(raw)
if fromParsing == cdf.CDFTypeXls {
return true
}
if fromParsing != cdf.CDFTypeGeneric {
return false
}
// Fallback for inputs where the CDF directory is past the read limit.
// Root CLSID test is the safest way to identify the OLE, however, the format
// often places the root CLSID at the end of the file.
if matchOleClsid(raw, []byte{
0x10, 0x08, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00,
@@ -148,6 +168,15 @@ func Pub(raw []byte, limit uint32) bool {
// Msg matches a Microsoft Outlook email file.
func Msg(raw []byte, limit uint32) bool {
fromParsing := cdf.Detect(raw)
if fromParsing == cdf.CDFTypeMsg {
return true
}
if fromParsing != cdf.CDFTypeGeneric {
return false
}
// Fallback for inputs where the CDF directory does not carry the streams the
// parser keys on: match the root storage CLSID instead.
return matchOleClsid(raw, []byte{
0x0B, 0x0D, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00,
0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46,
@@ -157,10 +186,7 @@ func Msg(raw []byte, limit uint32) bool {
// Msi matches a Microsoft Windows Installer file.
// http://fileformats.archiveteam.org/wiki/Microsoft_Compound_File
func Msi(raw []byte, limit uint32) bool {
return matchOleClsid(raw, []byte{
0x84, 0x10, 0x0C, 0x00, 0x00, 0x00, 0x00, 0x00,
0xC0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x46,
})
return cdf.Detect(raw) == cdf.CDFTypeInstaller
}
// One matches a Microsoft OneNote file.
+147 -82
View File
@@ -130,6 +130,14 @@ func Xfdf(raw []byte, _ uint32) bool {
return xml(raw, xmlSig{[]byte("<xfdf"), []byte(`xmlns="http://ns.adobe.com/xfdf/"`)})
}
// CDXXML matches a CycloneDX XML BOM file.
// https://cyclonedx.org/docs/1.7/xml/
func CDXXML(raw []byte, _ uint32) bool {
// xmlns is missing the version suffix because there are too many past versions
// and probably future versions to come.
return xml(raw, xmlSig{[]byte("<bom"), []byte(`xmlns="http://cyclonedx.org/schema/bom/`)})
}
// VCard matches a Virtual Contact File.
func VCard(raw []byte, _ uint32) bool {
return ciPrefix(raw, []byte("BEGIN:VCARD\n"), []byte("BEGIN:VCARD\r\n"))
@@ -139,6 +147,14 @@ func VCard(raw []byte, _ uint32) bool {
func ICalendar(raw []byte, _ uint32) bool {
return ciPrefix(raw, []byte("BEGIN:VCALENDAR\n"), []byte("BEGIN:VCALENDAR\r\n"))
}
const (
snone = 0
scws = scan.CompactWS
sfw = scan.FullWord
scwsfw = scan.CompactWS | scan.FullWord
)
func phpPageF(raw []byte, _ uint32) bool {
return ciPrefix(raw,
[]byte("<?PHP"),
@@ -149,66 +165,61 @@ func phpPageF(raw []byte, _ uint32) bool {
}
func phpScriptF(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS,
[]byte("/usr/local/bin/php"),
[]byte("/usr/bin/php"),
[]byte("/usr/bin/env php"),
[]byte("/usr/bin/env -S php"),
shebangSig{[]byte("/usr/local/bin/php"), snone},
shebangSig{[]byte("/usr/bin/php"), snone},
shebangSig{[]byte("/usr/bin/env php"), scws},
shebangSig{[]byte("/usr/bin/env -S php"), scws},
)
}
// Js matches a Javascript file.
func Js(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS,
[]byte("/bin/node"),
[]byte("/usr/bin/node"),
[]byte("/bin/nodejs"),
[]byte("/usr/bin/nodejs"),
[]byte("/usr/bin/env node"),
[]byte("/usr/bin/env -S node"),
[]byte("/usr/bin/env nodejs"),
[]byte("/usr/bin/env -S nodejs"),
shebangSig{[]byte("/bin/node"), snone},
shebangSig{[]byte("/usr/bin/node"), snone},
shebangSig{[]byte("/bin/nodejs"), snone},
shebangSig{[]byte("/usr/bin/nodejs"), snone},
shebangSig{[]byte("/usr/bin/env node"), scws},
shebangSig{[]byte("/usr/bin/env -S node"), scws},
shebangSig{[]byte("/usr/bin/env nodejs"), scws},
shebangSig{[]byte("/usr/bin/env -S nodejs"), scws},
)
}
// Lua matches a Lua programming language file.
func Lua(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS|scan.FullWord,
[]byte("/usr/bin/lua"),
[]byte("/usr/local/bin/lua"),
[]byte("/usr/bin/env lua"),
[]byte("/usr/bin/env -S lua"),
shebangSig{[]byte("/usr/bin/lua"), sfw},
shebangSig{[]byte("/usr/local/bin/lua"), sfw},
shebangSig{[]byte("/usr/bin/env lua"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S lua"), scwsfw},
)
}
// Perl matches a Perl programming language file.
func Perl(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS|scan.FullWord,
[]byte("/usr/bin/perl"),
[]byte("/usr/bin/env perl"),
[]byte("/usr/bin/env -S perl"),
shebangSig{[]byte("/usr/bin/perl"), sfw},
shebangSig{[]byte("/usr/bin/env perl"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S perl"), scwsfw},
)
}
// Python matches a Python programming language file.
func Python(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS,
[]byte("/usr/bin/python"),
[]byte("/usr/local/bin/python"),
[]byte("/usr/bin/env python"),
[]byte("/usr/bin/env -S python"),
[]byte("/usr/bin/python2"),
[]byte("/usr/local/bin/python2"),
[]byte("/usr/bin/env python2"),
[]byte("/usr/bin/env -S python2"),
[]byte("/usr/bin/python3"),
[]byte("/usr/local/bin/python3"),
[]byte("/usr/bin/env python3"),
[]byte("/usr/bin/env -S python3"),
shebangSig{[]byte("/usr/bin/python"), snone},
shebangSig{[]byte("/usr/local/bin/python"), snone},
shebangSig{[]byte("/usr/bin/env python"), scws},
shebangSig{[]byte("/usr/bin/env -S python"), scws},
shebangSig{[]byte("/usr/bin/python2"), snone},
shebangSig{[]byte("/usr/local/bin/python2"), snone},
shebangSig{[]byte("/usr/bin/env python2"), scws},
shebangSig{[]byte("/usr/bin/env -S python2"), scws},
shebangSig{[]byte("/usr/bin/python3"), snone},
shebangSig{[]byte("/usr/local/bin/python3"), snone},
shebangSig{[]byte("/usr/bin/env python3"), scws},
shebangSig{[]byte("/usr/bin/env -S python3"), scws},
)
}
@@ -216,30 +227,28 @@ func Python(raw []byte, _ uint32) bool {
// Ruby matches a Ruby programming language file.
func Ruby(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS,
[]byte("/usr/bin/ruby"),
[]byte("/usr/local/bin/ruby"),
[]byte("/usr/bin/env ruby"),
[]byte("/usr/bin/env -S ruby"),
shebangSig{[]byte("/usr/bin/ruby"), snone},
shebangSig{[]byte("/usr/local/bin/ruby"), snone},
shebangSig{[]byte("/usr/bin/env ruby"), scws},
shebangSig{[]byte("/usr/bin/env -S ruby"), scws},
)
}
// Tcl matches a Tcl programming language file.
func Tcl(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS,
[]byte("/usr/bin/tcl"),
[]byte("/usr/local/bin/tcl"),
[]byte("/usr/bin/env tcl"),
[]byte("/usr/bin/env -S tcl"),
[]byte("/usr/bin/tclsh"),
[]byte("/usr/local/bin/tclsh"),
[]byte("/usr/bin/env tclsh"),
[]byte("/usr/bin/env -S tclsh"),
[]byte("/usr/bin/wish"),
[]byte("/usr/local/bin/wish"),
[]byte("/usr/bin/env wish"),
[]byte("/usr/bin/env -S wish"),
shebangSig{[]byte("/usr/bin/tcl"), snone},
shebangSig{[]byte("/usr/local/bin/tcl"), snone},
shebangSig{[]byte("/usr/bin/env tcl"), scws},
shebangSig{[]byte("/usr/bin/env -S tcl"), scws},
shebangSig{[]byte("/usr/bin/tclsh"), snone},
shebangSig{[]byte("/usr/local/bin/tclsh"), snone},
shebangSig{[]byte("/usr/bin/env tclsh"), scws},
shebangSig{[]byte("/usr/bin/env -S tclsh"), scws},
shebangSig{[]byte("/usr/bin/wish"), snone},
shebangSig{[]byte("/usr/local/bin/wish"), snone},
shebangSig{[]byte("/usr/bin/env wish"), scws},
shebangSig{[]byte("/usr/bin/env -S wish"), scws},
)
}
@@ -251,32 +260,31 @@ func Rtf(raw []byte, _ uint32) bool {
// Shell matches a shell script file.
func Shell(raw []byte, _ uint32) bool {
return shebang(raw,
scan.CompactWS|scan.FullWord,
[]byte("/bin/sh"),
[]byte("/bin/bash"),
[]byte("/usr/local/bin/bash"),
[]byte("/usr/bin/env bash"),
[]byte("/usr/bin/env -S bash"),
[]byte("/bin/csh"),
[]byte("/usr/local/bin/csh"),
[]byte("/usr/bin/env csh"),
[]byte("/usr/bin/env -S csh"),
[]byte("/bin/dash"),
[]byte("/usr/local/bin/dash"),
[]byte("/usr/bin/env dash"),
[]byte("/usr/bin/env -S dash"),
[]byte("/bin/ksh"),
[]byte("/usr/local/bin/ksh"),
[]byte("/usr/bin/env ksh"),
[]byte("/usr/bin/env -S ksh"),
[]byte("/bin/tcsh"),
[]byte("/usr/local/bin/tcsh"),
[]byte("/usr/bin/env tcsh"),
[]byte("/usr/bin/env -S tcsh"),
[]byte("/bin/zsh"),
[]byte("/usr/local/bin/zsh"),
[]byte("/usr/bin/env zsh"),
[]byte("/usr/bin/env -S zsh"),
shebangSig{[]byte("/bin/sh"), sfw},
shebangSig{[]byte("/bin/bash"), sfw},
shebangSig{[]byte("/usr/local/bin/bash"), sfw},
shebangSig{[]byte("/usr/bin/env bash"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S bash"), scwsfw},
shebangSig{[]byte("/bin/csh"), sfw},
shebangSig{[]byte("/usr/local/bin/csh"), sfw},
shebangSig{[]byte("/usr/bin/env csh"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S csh"), scwsfw},
shebangSig{[]byte("/bin/dash"), sfw},
shebangSig{[]byte("/usr/local/bin/dash"), sfw},
shebangSig{[]byte("/usr/bin/env dash"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S dash"), scwsfw},
shebangSig{[]byte("/bin/ksh"), sfw},
shebangSig{[]byte("/usr/local/bin/ksh"), sfw},
shebangSig{[]byte("/usr/bin/env ksh"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S ksh"), scwsfw},
shebangSig{[]byte("/bin/tcsh"), sfw},
shebangSig{[]byte("/usr/local/bin/tcsh"), sfw},
shebangSig{[]byte("/usr/bin/env tcsh"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S tcsh"), scwsfw},
shebangSig{[]byte("/bin/zsh"), sfw},
shebangSig{[]byte("/usr/local/bin/zsh"), sfw},
shebangSig{[]byte("/usr/bin/env zsh"), scwsfw},
shebangSig{[]byte("/usr/bin/env -S zsh"), scwsfw},
)
}
@@ -352,6 +360,12 @@ func GLTF(raw []byte, limit uint32) bool {
return jsonHelper(raw, limit, json.QueryGLTF, json.TokObject)
}
// CDXJSON matches a CycloneDX JSON BOM file.
// https://cyclonedx.org/docs/1.7/json/
func CDXJSON(raw []byte, limit uint32) bool {
return jsonHelper(raw, limit, json.QueryCDX, json.TokObject)
}
// jsonHelper parses raw and tries to match the q query against it. wantToks
// ensures we're not wasting time parsing an input that would not pass anyway,
// ex: the input is a valid JSON array, but we're looking for a JSON object.
@@ -393,8 +407,16 @@ func NdJSON(raw []byte, limit uint32) bool {
var l scan.Bytes
for len(s) != 0 {
l = s.Line()
_, inspected, firstToken, _ := json.Parse(json.QueryNone, l)
if len(l) != inspected {
parsed, inspected, firstToken, _ := json.Parse(json.QueryNone, l)
// Only the last line may be truncated by the read limit; for it, it is
// enough that the parser inspected all of it. Every other line must be a
// complete, valid JSON document, otherwise a single JSON document spread
// over multiple lines would be mistaken for NDJSON. #803
if len(s) == 0 {
if inspected != len(l) {
return false
}
} else if parsed != len(l) {
return false
}
if firstToken == json.TokArray || firstToken == json.TokObject {
@@ -563,6 +585,8 @@ func RFC822(raw []byte, limit uint32) bool {
// Some of the hints are IgnoreCase, some not. I selected based on what libmagic
// does and based on personal observations from sample files.
hints := []rfc822Hint{
// Enron dataset has Message-ID, Message-Id and Message-id.
{[]byte("Message-ID: "), scan.IgnoreCase},
{[]byte("From: "), 0},
{[]byte("To: "), 0},
{[]byte("CC: "), scan.IgnoreCase},
@@ -599,3 +623,44 @@ func lineHasRFC822Hint(b scan.Bytes, hints []rfc822Hint) bool {
}
return false
}
func GEDCOM(raw []byte, limit uint32) bool {
// Skip if empty
if len(raw) == 0 {
return false
}
// GEDCOM header fits within first 4KB
searchLimit := min(len(raw), 4096)
raw = raw[:searchLimit]
b := scan.Bytes(raw)
// Skip BOM if present: UTF-8, UTF-16BE, UTF-16LE
for _, bom := range [][]byte{
{0xEF, 0xBB, 0xBF}, // UTF-8
{0xFE, 0xFF}, // UTF-16BE
{0xFF, 0xFE}, // UTF-16LE
} {
if bytes.HasPrefix(b, bom) {
b.Advance(len(bom))
break // Only one BOM can exist at the start
}
}
b.TrimLWS()
firstLine := b.Line()
if !bytes.Equal(firstLine, []byte("0 HEAD")) {
return false
}
// "1 GEDC" is mandatory in the header
for i := 0; i < 10; i++ {
line := b.Line()
if bytes.Equal(line, []byte("1 GEDC")) {
return true
}
}
return false
}
+17 -4
View File
@@ -17,8 +17,7 @@ func TSV(raw []byte, limit uint32) bool {
func sv(in []byte, comma byte, limit uint32) bool {
s := scan.Bytes(in)
s.DropLastLine(limit)
r := csv.NewParser(comma, '#', s)
r := csv.NewParser(comma, '#', &s)
headerFields, _, hasMore := r.CountFields(false)
if headerFields < 2 || !hasMore {
@@ -30,8 +29,22 @@ func sv(in []byte, comma byte, limit uint32) bool {
if !hasMore && fields == 0 {
break
}
csvLines++
if fields != headerFields {
if fields == headerFields {
csvLines++
} else {
// maybeTruncated signals the input was cut at the read limit,
// meaning the last line may be an incomplete CSV record.
maybeTruncated := limit > 0 && uint64(len(in)) >= uint64(limit)
if maybeTruncated && fields < headerFields {
// Allow the last row to have any number of fields
// if the input is maybeTruncated.
// BUG: if len(input) == limit, then the input is not truncated
// but it is still allowed to have the wrong number of fields
// and it will be reported as valid CSV.
if len(s) == 0 {
break
}
}
return false
}
if csvLines >= 10 {
+2 -4
View File
@@ -49,10 +49,8 @@ func isMatroskaFileTypeMatched(in []byte, flType string) bool {
// The logic of search is: find first instance of \x42\x82 and then
// search for given string after n bytes of above instance.
func isFileTypeNamePresent(in []byte, flType string) bool {
ind, maxInd, lenIn := 0, 4096, len(in)
if lenIn < maxInd { // restricting length to 4096
maxInd = lenIn
}
ind, lenIn := 0, len(in)
maxInd := min(4096, lenIn)
ind = bytes.Index(in[:maxInd], []byte("\x42\x82"))
if ind > 0 && lenIn > ind+2 {
ind += 2
+79 -4
View File
@@ -187,11 +187,11 @@ func msoxml(raw scan.Bytes, searchFor zipEntries, stopAfter int) bool {
return false
}
var zipLocalFileHeader = []byte("PK\003\004")
// next extracts the name of the next zip entry.
func (i *zipIterator) next() []byte {
pk := []byte("PK\003\004")
n := bytes.Index(i.b, pk)
n := bytes.Index(i.b, zipLocalFileHeader)
if n == -1 {
return nil
}
@@ -212,10 +212,85 @@ func (i *zipIterator) next() []byte {
return i.b[:l]
}
// skipZipflingerEntry tries to detect a Zipflinger virtual entry and skips it.
// The detection is based on the following properties:
// - compression method is 0
// - CRC32 is 0
// - compressed size is 0
// - uncompressed size is 0
// - file name is empty
// Returns true if it was found and skipped.
func (i *zipIterator) skipZipflingerEntry() (skipped bool) {
// Make a backup of the data so the inspection does not loses it.
b := i.b
defer func() {
// If no zipflinger was found, restore the original data.
if !skipped {
i.b = b
}
}()
n := bytes.Index(i.b, zipLocalFileHeader)
if n == -1 {
return false
}
if !i.b.Advance(0x08) {
return false
}
// Check compression method
if cm, ok := i.b.Uint16(); !ok || cm != 0 {
return false
}
// Advance up to the CRC32 field
if !i.b.Advance(0x04) {
return false
}
// Check CRC32
if crc32, ok := i.b.Uint32(); !ok || crc32 != 0 {
return false
}
// Check compressed size
if compressedSize, ok := i.b.Uint32(); !ok || compressedSize != 0 {
return false
}
// Check uncompressed size
if uncompressedSize, ok := i.b.Uint32(); !ok || uncompressedSize != 0 {
return false
}
// Check for empty file name
if l, ok := i.b.Uint16(); !ok || l != 0 {
return false
}
// Reached a zipflinger virtual entry: skip extra data
l, ok := i.b.Uint16()
if !ok {
return false
}
if !i.b.Advance(int(l)) {
return false
}
return true
}
// APK matches an Android Package Archive.
// The source of signatures is https://github.com/file/file/blob/1778642b8ba3d947a779a36fcd81f8e807220a19/magic/Magdir/archive#L1820-L1887
func APK(raw []byte, _ uint32) bool {
return zipHas(raw, zipEntries{{
iter := zipIterator{raw}
// If a Zipflinger Virtual Entry is detected, then the data is considered APK
if iter.skipZipflingerEntry() {
return true
}
return zipHas(iter.b, zipEntries{{
name: []byte("AndroidManifest.xml"),
}, {
name: []byte("META-INF/com/android/build/gradle/app-metadata.properties"),
+140
View File
@@ -0,0 +1,140 @@
package mp3
import "bytes"
// minTruncatedSyncMatches is the minimum number of confirmed successive
// header matches required to accept a candidate frame when the buffer ends
// before maxFrameSyncMatches confirmations can be performed.
const minTruncatedSyncMatches = 2
func ExtractFrame(b []byte) (start, size int) {
limit := min(len(b), 2048+headerSize)
for i := 0; i < limit-headerSize; i++ {
j := bytes.IndexByte(b[i:limit-headerSize], 0xFF)
if j < 0 {
break
}
i += j
hdr := header{b[i], b[i+1], b[i+2], b[i+3]}
if !hdr.valid() {
continue
}
frameBytes := hdr.frameBytes()
frameAndPad := frameBytes + hdr.padding()
validHere := frameBytes > 0 && i+frameAndPad <= len(b) && matchFrame(b[i:])
// When the buffer is exactly one frame, matchFrame cannot look ahead for
// a subsequent header to confirm the stream. Trust the validated header.
exact := i == 0 && frameAndPad == len(b)
if validHere || exact {
return i, frameAndPad
}
}
return 0, 0
}
// matchFrame confirms a candidate header by stepping forward and checking that
// subsequent headers are consistent.
func matchFrame(buf []byte) bool {
// maxFrameSyncMatches limits how many valid frames we look at.
const maxFrameSyncMatches = 10
hdr := header{buf[0], buf[1], buf[2], buf[3]}
i := hdr.frameBytes() + hdr.padding()
for nmatch := 0; nmatch < maxFrameSyncMatches; nmatch++ {
if i+headerSize > len(buf) {
return nmatch >= minTruncatedSyncMatches
}
cmp := header{buf[i], buf[i+1], buf[i+2], buf[i+3]}
if !hdr.compatibleWith(cmp) {
return false
}
i += cmp.frameBytes() + cmp.padding()
}
return true
}
const headerSize = 4
type header [headerSize]byte
func (h header) isFreeFormat() bool { return h[2]&0xF0 == 0 }
func (h header) isMPEG1() bool { return h[1]&0x8 != 0 }
func (h header) isMPEG25() bool { return h[1]&0x10 == 0 }
func (h header) rawLayer() byte { return h[1] >> 1 & 3 }
func (h header) rawBitrate() byte { return h[2] >> 4 }
func (h header) rawSampleRate() byte { return h[2] >> 2 & 3 }
func (h header) rawEmphasis() byte { return h[3] & 0b11 }
func (h header) isFrame576() bool { return h[1]&14 == 2 }
func (h header) padding() int {
if h[2]&0x2 != 0 {
return 1
}
return 0
}
// valid reports whether the four bytes form a syntactically valid MP3 header.
func (h header) valid() bool {
return h[0] == 0xff &&
((h[1]&0xF0) == 0xf0 || (h[1]&0xFE) == 0xe2) &&
h.rawLayer() == 1 && // Layer III
h.rawBitrate() != 15 && // Not allowed by spec.
h.rawSampleRate() != 3 &&
h.rawEmphasis() != 2 &&
// The code for extracting frame size for free-format is tedious and
// free-format MP3s are extinct.
!h.isFreeFormat()
}
// compatibleWith reports whether two headers describe frames belonging to the
// same MP3 stream — same MPEG version, layer, sample-rate index.
func (h header) compatibleWith(o header) bool {
return o.valid() &&
(h[1]^o[1])&0xFE == 0 &&
(h[2]^o[2])&0x0C == 0
}
// bitrateKbps returns the bitrate of the frame in kilobits per second.
func (h header) bitrateKbps() int {
// halfrate[mpeg1?][bitrate_idx] holds bitrate/2 in kbps.
halfrate := [2][15]uint8{
{0, 4, 8, 12, 16, 20, 24, 28, 32, 40, 48, 56, 64, 72, 80},
{0, 16, 20, 24, 28, 32, 40, 48, 56, 64, 80, 96, 112, 128, 160},
}
mpeg1 := 0
if h.isMPEG1() {
mpeg1 = 1
}
return 2 * int(halfrate[mpeg1][h.rawBitrate()])
}
// sampleRateHz returns the sampling rate of the frame in Hz.
func (h header) sampleRateHz() int {
base := [3]int{44100, 48000, 32000}[h.rawSampleRate()]
if !h.isMPEG1() {
base >>= 1
}
if h.isMPEG25() {
base >>= 1
}
return base
}
// frameSamples returns the number of audio samples per channel encoded in
// the frame.
func (h header) frameSamples() int {
if h.isFrame576() {
return 576
}
return 1152
}
// frameBytes returns the size of the frame body (header + side info + audio
// data, excluding padding) in bytes.
func (h header) frameBytes() int {
br := h.bitrateKbps()
sr := h.sampleRateHz()
if br == 0 || sr == 0 {
return 0
}
return h.frameSamples() * br * 125 / sr
}
+21 -25
View File
@@ -122,26 +122,6 @@ func (b *Bytes) Line() Bytes {
return line
}
// DropLastLine drops the last incomplete line from b.
//
// mimetype limits itself to ReadLimit bytes when performing a detection.
// This means, for file formats like CSV for NDJSON, the last line of the input
// can be an incomplete line.
// If b length is less than readLimit, it means we received an incomplete file
// and proceed with dropping the last line.
func (b *Bytes) DropLastLine(readLimit uint32) {
if readLimit == 0 || uint64(len(*b)) < uint64(readLimit) {
return
}
for i := len(*b) - 1; i > 0; i-- {
if (*b)[i] == '\n' {
*b = (*b)[:i]
return
}
}
}
func (b *Bytes) Uint16() (uint16, bool) {
if len(*b) < 2 {
return 0, false
@@ -151,6 +131,24 @@ func (b *Bytes) Uint16() (uint16, bool) {
return v, true
}
func (b *Bytes) Uint32() (uint32, bool) {
if len(*b) < 4 {
return 0, false
}
v := binary.LittleEndian.Uint32(*b)
*b = (*b)[4:]
return v, true
}
func (b *Bytes) Uint32be() (uint32, bool) {
if len(*b) < 4 {
return 0, false
}
v := binary.BigEndian.Uint32(*b)
*b = (*b)[4:]
return v, true
}
type Flags int
const (
@@ -205,10 +203,8 @@ func (b Bytes) Match(p []byte, flags Flags) int {
if l == 0 {
return -1
}
// If no flags, or scanning for full word at the end of pattern then
// do a fast HasPrefix check.
// For other flags it's not possible to use HasPrefix.
if flags == 0 || flags&FullWord > 0 {
// Some cases we can handle with a simple bytes.HasPrefix.
if flags == 0 || flags == FullWord {
if bytes.HasPrefix(b, p) {
b = b[len(p):]
p = p[len(p):]
@@ -232,7 +228,7 @@ func (b Bytes) Match(p []byte, flags Flags) int {
return -1
}
b = b[1:]
if !ByteIsWS(p[0]) {
if len(p) > 0 && !ByteIsWS(p[0]) {
b.TrimLWS()
}
} else {
+15 -5
View File
@@ -23,6 +23,14 @@ type MIME struct {
}
// String returns the string representation of the MIME type, e.g., "application/zip".
// String return values can change between releases, for example, when [IANA]
// assigns a new media type. Use [MIME.Is] to avoid breaking changes.
//
// mtype := mimetype.Detect(zipFile)
// if mtype.String() == "application/zip" { /* Plain string comparison is brittle. */ }
// if mtype.Is("application/zip") { /* Will continue to work between releases */ }
//
// [IANA]: https://www.iana.org/assignments/media-types/media-types.xhtml
func (m *MIME) String() string {
return m.mime
}
@@ -38,17 +46,19 @@ func (m *MIME) Extension() string {
// Each MIME type has a non-nil parent, except for the root MIME type.
//
// For example, the application/json and text/html MIME types have text/plain as
// their parent because they are text files who happen to contain JSON or HTML.
// their parent because they are text files that happen to contain JSON or HTML.
// Another example is the ZIP format, which is used as container
// for Microsoft Office files, EPUB files, JAR files, and others.
func (m *MIME) Parent() *MIME {
return m.parent
}
// Is checks whether this MIME type, or any of its aliases, is equal to the
// Is checks whether this MIME type, or any of its [aliases], is equal to the
// expected MIME type. MIME type equality test is done on the "type/subtype"
// section, ignores any optional MIME parameters, ignores any leading and
// trailing whitespace, and is case insensitive.
//
// [aliases]: https://github.com/gabriel-vasile/mimetype/blob/master/supported_mimes.md
func (m *MIME) Is(expectedMIME string) bool {
// Parsing is needed because some detected MIME types contain parameters
// that need to be stripped for the comparison.
@@ -129,7 +139,7 @@ func (m *MIME) flatten() []*MIME {
// hierarchy returns an easy to read list of ancestors for m.
// For example, application/json would return json>txt>root.
func (m *MIME) hierarchy() string {
h := ""
var h strings.Builder
for m := m; m != nil; m = m.Parent() {
e := strings.TrimPrefix(m.Extension(), ".")
if e == "" {
@@ -142,9 +152,9 @@ func (m *MIME) hierarchy() string {
e = "root"
}
}
h += ">" + e
h.WriteString(">" + e)
}
return strings.TrimPrefix(h, ">")
return strings.TrimPrefix(h.String(), ">")
}
// clone creates a new MIME with the provided optional MIME parameters.
+6 -6
View File
@@ -1,6 +1,6 @@
// Package mimetype uses magic number signatures to detect the MIME type of a file.
//
// File formats are stored in a hierarchy with application/octet-stream at its root.
// File formats are stored in a hierarchy with "application/octet-stream" at its root.
// For example, the hierarchy for HTML format is application/octet-stream ->
// text/plain -> text/html.
package mimetype
@@ -12,14 +12,14 @@ import (
"sync/atomic"
)
const defaultLimit uint32 = 3072
const defaultLimit uint32 = 4096
// readLimit is the maximum number of bytes from the input used when detecting.
var readLimit uint32 = defaultLimit
// Detect returns the MIME type found from the provided byte slice.
//
// The result is always a valid MIME type, with application/octet-stream
// The result is always a valid MIME type, with "application/octet-stream"
// returned when identification failed.
func Detect(in []byte) *MIME {
// Using atomic because readLimit can be written at the same time in other goroutine.
@@ -34,7 +34,7 @@ func Detect(in []byte) *MIME {
// DetectReader returns the MIME type of the provided reader.
//
// The result is always a valid MIME type, with application/octet-stream
// The result is always a valid MIME type, with "application/octet-stream"
// returned when identification failed with or without an error.
// Any error returned is related to the reading from the input reader.
//
@@ -72,7 +72,7 @@ func DetectReader(r io.Reader) (*MIME, error) {
// DetectFile returns the MIME type of the provided file.
//
// The result is always a valid MIME type, with application/octet-stream
// The result is always a valid MIME type, with "application/octet-stream"
// returned when identification failed with or without an error.
// Any error returned is related to the opening and reading from the input file.
func DetectFile(path string) (*MIME, error) {
@@ -112,7 +112,7 @@ func SetLimit(limit uint32) {
}
// Extend adds detection for other file formats.
// It is equivalent to calling Extend() on the root MIME type "application/octet-stream".
// It is equivalent to calling [MIME.Extend] on the root MIME type "application/octet-stream".
func Extend(detector func(raw []byte, limit uint32) bool, mime, extension string, aliases ...string) {
root.Extend(detector, mime, extension, aliases...)
}
+10 -5
View File
@@ -1,4 +1,4 @@
## 199 Supported MIME types
## 204 Supported MIME types
This file is automatically generated when running tests. Do not edit manually.
Extension | MIME type <br> Aliases | Hierarchy
@@ -42,7 +42,7 @@ Extension | MIME type <br> Aliases | Hierarchy
**.oga** | **audio/ogg** | oga>ogg>root
**.ogv** | **video/ogg** | ogv>ogg>root
**.png** | **image/png** | png>root
**.png** | **image/vnd.mozilla.apng** | png>png>root
**.apng** | **image/apng** <br> image/vnd.mozilla.apng | apng>png>root
**.jpg** | **image/jpeg** | jpg>root
**.jxl** | **image/jxl** | jxl>root
**.jp2** | **image/jp2** | jp2>root
@@ -67,7 +67,6 @@ Extension | MIME type <br> Aliases | Hierarchy
**.bmp** | **image/bmp** <br> image/x-bmp, image/x-ms-bmp | bmp>root
**.123** | **application/vnd.lotus-1-2-3** | 123>root
**.ico** | **image/x-icon** | ico>root
**.mp3** | **audio/mpeg** <br> audio/x-mpeg, audio/mp3 | mp3>root
**.flac** | **audio/flac** | flac>root
**.midi** | **audio/midi** <br> audio/mid, audio/sp-midi, audio/x-mid, audio/x-midi | midi>root
**.ape** | **audio/ape** | ape>root
@@ -95,7 +94,7 @@ Extension | MIME type <br> Aliases | Hierarchy
**.webm** | **video/webm** <br> audio/webm | webm>root
**.avi** | **video/x-msvideo** <br> video/avi, video/msvideo | avi>root
**.flv** | **video/x-flv** | flv>root
**.mkv** | **video/x-matroska** | mkv>root
**.mkv** | **video/matroska** <br> video/x-matroska | mkv>root
**.asf** | **video/x-ms-asf** <br> video/asf, video/x-ms-wmv | asf>root
**.aac** | **audio/aac** | aac>root
**.voc** | **audio/x-unknown** | voc>root
@@ -116,7 +115,7 @@ Extension | MIME type <br> Aliases | Hierarchy
**.shp** | **application/vnd.shp** | shp>shx>root
**.dbf** | **application/x-dbf** | dbf>root
**.dcm** | **application/dicom** | dcm>root
**.rar** | **application/x-rar-compressed** <br> application/x-rar | rar>root
**.rar** | **application/vnd.rar** <br> application/x-rar-compressed, application/x-rar | rar>root
**.djvu** | **image/vnd.djvu** | djvu>root
**.mobi** | **application/x-mobipocket-ebook** | mobi>root
**.lit** | **application/x-ms-reader** | lit>root
@@ -158,6 +157,9 @@ Extension | MIME type <br> Aliases | Hierarchy
**.hlp** | **application/x-os2-hlp** | hlp>root
**.fm** | **application/vnd.framemaker** | fm>root
**.bufr** | **application/bufr** | bufr>root
**.pyc** | **application/x-bytecode.python** | pyc>root
**.pcap** | **application/vnd.tcpdump.pcap** | pcap>root
**.mp3** | **audio/mpeg** <br> audio/x-mpeg, audio/mp3 | mp3>root
**.txt** | **text/plain** | txt>root
**.svg** | **image/svg+xml** | svg>txt>root
**.html** | **text/html** | html>txt>root
@@ -176,6 +178,7 @@ Extension | MIME type <br> Aliases | Hierarchy
**.xfdf** | **application/vnd.adobe.xfdf** | xfdf>xml>txt>root
**.owl** | **application/owl+xml** | owl>xml>txt>root
**.html** | **application/xhtml+xml** | html>xml>txt>root
**.xml** | **application/vnd.cyclonedx+xml** | xml>xml>txt>root
**.php** | **text/x-php** | php>txt>root
**.js** | **text/javascript** <br> application/x-javascript, application/javascript | js>txt>root
**.lua** | **text/x-lua** | lua>txt>root
@@ -186,6 +189,7 @@ Extension | MIME type <br> Aliases | Hierarchy
**.geojson** | **application/geo+json** | geojson>json>txt>root
**.har** | **application/json** | har>json>txt>root
**.gltf** | **model/gltf+json** | gltf>json>txt>root
**.json** | **application/vnd.cyclonedx+json** | json>json>txt>root
**.ndjson** | **application/x-ndjson** | ndjson>txt>root
**.rtf** | **text/rtf** <br> application/rtf | rtf>txt>root
**.srt** | **application/x-subrip** <br> application/x-srt, text/x-srt | srt>txt>root
@@ -202,3 +206,4 @@ Extension | MIME type <br> Aliases | Hierarchy
**.ppm** | **image/x-portable-pixmap** | ppm>txt>root
**.pam** | **image/x-portable-arbitrarymap** | pam>txt>root
**.eml** | **message/rfc822** | eml>txt>root
**.ged** | **text/vnd.familysearch.gedcom** | ged>txt>root
+26 -15
View File
@@ -19,12 +19,16 @@ var root = newMIME("application/octet-stream", "",
func([]byte, uint32) bool { return true },
xpm, sevenZ, zip, pdf, fdf, ole, ps, psd, p7s, ogg, png, jpg, jxl, jp2, jpx,
jpm, jxs, gif, webp, exe, elf, ar, tar, xar, bz2, fits, tiff, bmp, lotus, ico,
mp3, flac, midi, ape, musePack, amr, wav, aiff, au, mpeg, quickTime, mp4, webM,
flac, midi, ape, musePack, amr, wav, aiff, au, mpeg, quickTime, mp4, webM,
avi, flv, mkv, asf, aac, voc, m3u, rmvb, gzip, class, swf, crx, ttf, woff,
woff2, otf, ttc, eot, wasm, shx, dbf, dcm, rar, djvu, mobi, lit, bpg, cbor,
sqlite3, dwg, nes, lnk, macho, qcp, icns, hdr, mrc, mdb, accdb, zstd, cab,
rpm, xz, lzip, torrent, cpio, tzif, xcf, pat, gbr, glb, cabIS, jxr, parquet,
oneNote, chm, wpd, dxf, grib, zlib, inf, hlp, fm, bufr,
oneNote, chm, wpd, dxf, grib, zlib, inf, hlp, fm, bufr, pyc, pcap,
// MP3 is late because it does a linear search in the input. That means
// containers that embed an MP3, for example: an mp4 file, or a zip without
// compression, would pass as MP3s.
mp3,
// Keep text last because it is the slowest check.
text,
)
@@ -82,16 +86,17 @@ var (
alias("application/x-ogg")
oggAudio = newMIME("audio/ogg", ".oga", magic.OggAudio)
oggVideo = newMIME("video/ogg", ".ogv", magic.OggVideo)
text = newMIME("text/plain", ".txt", magic.Text, svg, html, xml, php, js, lua, perl, python, ruby, json, ndJSON, rtf, srt, tcl, csv, tsv, vCard, iCalendar, warc, vtt, shell, netpbm, netpgm, netppm, netpam, rfc822)
xml = newMIME("text/xml", ".xml", magic.XML, rss, atom, x3d, kml, xliff, collada, gml, gpx, tcx, amf, threemf, xfdf, owl2, xhtml).
text = newMIME("text/plain", ".txt", magic.Text, svg, html, xml, php, js, lua, perl, python, ruby, json, ndJSON, rtf, srt, tcl, csv, tsv, vCard, iCalendar, warc, vtt, shell, netpbm, netpgm, netppm, netpam, rfc822, gedcom)
xml = newMIME("text/xml", ".xml", magic.XML, rss, atom, x3d, kml, xliff, collada, gml, gpx, tcx, amf, threemf, xfdf, owl2, xhtml, cdxxml).
alias("application/xml")
xhtml = newMIME("application/xhtml+xml", ".html", magic.XHTML)
json = newMIME("application/json", ".json", magic.JSON, geoJSON, har, gltf)
json = newMIME("application/json", ".json", magic.JSON, geoJSON, har, gltf, cdxJSON)
har = newMIME("application/json", ".har", magic.HAR)
csv = newMIME("text/csv", ".csv", magic.CSV)
tsv = newMIME("text/tab-separated-values", ".tsv", magic.TSV)
geoJSON = newMIME("application/geo+json", ".geojson", magic.GeoJSON)
ndJSON = newMIME("application/x-ndjson", ".ndjson", magic.NdJSON)
cdxJSON = newMIME("application/vnd.cyclonedx+json", ".json", magic.CDXJSON)
html = newMIME("text/html", ".html", magic.HTML)
php = newMIME("text/x-php", ".php", magic.Php)
rtf = newMIME("text/rtf", ".rtf", magic.Rtf).alias("application/rtf")
@@ -104,6 +109,7 @@ var (
perl = newMIME("text/x-perl", ".pl", magic.Perl)
python = newMIME("text/x-python", ".py", magic.Python).
alias("text/x-script.python", "application/x-python")
pyc = newMIME("application/x-bytecode.python", ".pyc", magic.Pyc)
ruby = newMIME("text/x-ruby", ".rb", magic.Ruby).
alias("application/x-ruby")
shell = newMIME("text/x-shellscript", ".sh", magic.Shell).
@@ -127,13 +133,15 @@ var (
tcx = newMIME("application/vnd.garmin.tcx+xml", ".tcx", magic.Tcx)
amf = newMIME("application/x-amf", ".amf", magic.Amf)
threemf = newMIME("application/vnd.ms-package.3dmanufacturing-3dmodel+xml", ".3mf", magic.Threemf)
cdxxml = newMIME("application/vnd.cyclonedx+xml", ".xml", magic.CDXXML)
png = newMIME("image/png", ".png", magic.Png, apng)
apng = newMIME("image/vnd.mozilla.apng", ".png", magic.Apng)
jpg = newMIME("image/jpeg", ".jpg", magic.Jpg)
jxl = newMIME("image/jxl", ".jxl", magic.Jxl)
jp2 = newMIME("image/jp2", ".jp2", magic.Jp2)
jpx = newMIME("image/jpx", ".jpf", magic.Jpx)
jpm = newMIME("image/jpm", ".jpm", magic.Jpm).
apng = newMIME("image/apng", ".apng", magic.Apng).
alias("image/vnd.mozilla.apng")
jpg = newMIME("image/jpeg", ".jpg", magic.Jpg)
jxl = newMIME("image/jxl", ".jxl", magic.Jxl)
jp2 = newMIME("image/jp2", ".jp2", magic.Jp2)
jpx = newMIME("image/jpx", ".jpf", magic.Jpx)
jpm = newMIME("image/jpm", ".jpm", magic.Jpm).
alias("video/jpm")
jxs = newMIME("image/jxs", ".jxs", magic.Jxs)
xpm = newMIME("image/x-xpixmap", ".xpm", magic.Xpm)
@@ -156,7 +164,7 @@ var (
heifSeq = newMIME("image/heif-sequence", ".heif", magic.HeifSequence)
hdr = newMIME("image/vnd.radiance", ".hdr", magic.Hdr)
avif = newMIME("image/avif", ".avif", magic.AVIF)
mp3 = newMIME("audio/mpeg", ".mp3", magic.Mp3).
mp3 = newMIME("audio/mpeg", ".mp3", magic.MP3).
alias("audio/x-mpeg", "audio/mp3")
flac = newMIME("audio/flac", ".flac", magic.Flac)
midi = newMIME("audio/midi", ".midi", magic.Midi).
@@ -192,7 +200,8 @@ var (
avi = newMIME("video/x-msvideo", ".avi", magic.Avi).
alias("video/avi", "video/msvideo")
flv = newMIME("video/x-flv", ".flv", magic.Flv)
mkv = newMIME("video/x-matroska", ".mkv", magic.Mkv)
mkv = newMIME("video/matroska", ".mkv", magic.Mkv).
alias("video/x-matroska")
asf = newMIME("video/x-ms-asf", ".asf", magic.Asf).
alias("video/asf", "video/x-ms-wmv")
rmvb = newMIME("application/vnd.rn-realmedia-vbr", ".rmvb", magic.Rmvb)
@@ -242,8 +251,8 @@ var (
odc = newMIME("application/vnd.oasis.opendocument.chart", ".odc", magic.Odc).
alias("application/x-vnd.oasis.opendocument.chart")
sxc = newMIME("application/vnd.sun.xml.calc", ".sxc", magic.Sxc)
rar = newMIME("application/x-rar-compressed", ".rar", magic.RAR).
alias("application/x-rar")
rar = newMIME("application/vnd.rar", ".rar", magic.RAR).
alias("application/x-rar-compressed", "application/x-rar")
djvu = newMIME("image/vnd.djvu", ".djvu", magic.DjVu)
mobi = newMIME("application/x-mobipocket-ebook", ".mobi", magic.Mobi)
lit = newMIME("application/x-ms-reader", ".lit", magic.Lit)
@@ -294,4 +303,6 @@ var (
hlp = newMIME("application/x-os2-hlp", ".hlp", magic.Hlp)
fm = newMIME("application/vnd.framemaker", ".fm", magic.FrameMaker)
bufr = newMIME("application/bufr", ".bufr", magic.BUFR)
gedcom = newMIME("text/vnd.familysearch.gedcom", ".ged", magic.GEDCOM)
pcap = newMIME("application/vnd.tcpdump.pcap", ".pcap", magic.Pcap)
)
+23 -10
View File
@@ -19,6 +19,10 @@ import (
// no limit.
var MaxPacketLengthBytes int64 = math.MaxInt32
// MaxNestingDepth specifies the maximum allowed nesting depth when calling ReadPacket, DecodePacket, or
// DecodePacketErr. Set to 0 for no limit.
var MaxNestingDepth int = 1000
type Packet struct {
Identifier
Value interface{}
@@ -218,7 +222,7 @@ func printPacket(out io.Writer, p *Packet, indent int, printBytes bool) {
// ReadPacket reads a single Packet from the reader.
func ReadPacket(reader io.Reader) (*Packet, error) {
p, _, err := readPacket(reader)
p, _, err := readPacket(reader, 0)
if err != nil {
return nil, err
}
@@ -278,7 +282,7 @@ func int64Length(i int64) (numBytes int) {
// DecodePacket decodes the given bytes into a single Packet
// If a decode error is encountered, nil is returned.
func DecodePacket(data []byte) *Packet {
p, _, _ := readPacket(bytes.NewBuffer(data))
p, _, _ := readPacket(bytes.NewBuffer(data), 0)
return p
}
@@ -286,7 +290,7 @@ func DecodePacket(data []byte) *Packet {
// DecodePacketErr decodes the given bytes into a single Packet
// If a decode error is encountered, nil is returned.
func DecodePacketErr(data []byte) (*Packet, error) {
p, _, err := readPacket(bytes.NewBuffer(data))
p, _, err := readPacket(bytes.NewBuffer(data), 0)
if err != nil {
return nil, err
}
@@ -294,12 +298,20 @@ func DecodePacketErr(data []byte) (*Packet, error) {
}
// readPacket reads a single Packet from the reader, returning the number of bytes read.
func readPacket(reader io.Reader) (*Packet, int, error) {
func readPacket(reader io.Reader, depth int) (*Packet, int, error) {
if MaxNestingDepth > 0 && depth >= MaxNestingDepth {
return nil, 0, fmt.Errorf("nesting depth %d exceeds maximum %d", depth, MaxNestingDepth)
}
identifier, length, read, err := readHeader(reader)
if err != nil {
return nil, read, err
}
if length != LengthIndefinite && MaxPacketLengthBytes > 0 && int64(length) > MaxPacketLengthBytes {
return nil, read, fmt.Errorf("length %d greater than maximum %d", length, MaxPacketLengthBytes)
}
p := &Packet{
Identifier: identifier,
}
@@ -326,13 +338,19 @@ func readPacket(reader io.Reader) (*Packet, int, error) {
}
// Read the next packet
child, r, err := readPacket(reader)
child, r, err := readPacket(reader, depth+1)
if err != nil {
return nil, read, unexpectedEOF(err)
}
contentRead += r
read += r
// Enforce the aggregate size limit for constructed packets. Indefinite length declares
// no bound up front, so the content bytes are only known as they are read.
if MaxPacketLengthBytes > 0 && int64(contentRead) > MaxPacketLengthBytes {
return nil, read, fmt.Errorf("length %d greater than maximum %d", contentRead, MaxPacketLengthBytes)
}
// Test is this is the EOC marker for our packet
if isEOCPacket(child) {
if length == LengthIndefinite {
@@ -351,11 +369,6 @@ func readPacket(reader io.Reader) (*Packet, int, error) {
return nil, read, errors.New("indefinite length used with primitive type")
}
// Read definite-length content
if MaxPacketLengthBytes > 0 && int64(length) > MaxPacketLengthBytes {
return nil, read, fmt.Errorf("length %d greater than maximum %d", length, MaxPacketLengthBytes)
}
var content []byte
if length > 0 {
// Read the content and limit it to the parsed length.
+5 -4
View File
@@ -40,7 +40,7 @@ func readLength(reader io.Reader) (length int, read int, err error) {
}
// Accumulate into a 64-bit variable
var length64 int64
var length64 uint64
for i := 0; i < lengthBytes; i++ {
b, err = readByte(reader)
if err != nil {
@@ -53,13 +53,14 @@ func readLength(reader io.Reader) (length int, read int, err error) {
// x.600, 8.1.3.5
length64 <<= 8
length64 |= int64(b)
length64 |= uint64(b)
}
// Cast to a platform-specific integer
length = int(length64)
// Ensure we didn't overflow
if int64(length) != length64 {
// Ensure we didn't overflow or wrap negative. Length octets are unsigned
// (x.600, 8.1.3.5), so a negative result is unrepresentable, not indefinite.
if length < 0 || uint64(length) != length64 {
return 0, read, errors.New("long-form length overflow")
}
+3
View File
@@ -83,6 +83,9 @@ func (l *Conn) Add(addRequest *AddRequest) error {
return err
}
if len(packet.Children) < 2 {
return fmt.Errorf("ldap: malformed response: expected at least 2 children, got %d", len(packet.Children))
}
if packet.Children[1].Tag == ApplicationAddResponse {
err := GetLDAPError(packet)
if err != nil {
+61 -16
View File
@@ -3,13 +3,13 @@ package ldap
import (
"bytes"
"crypto/md5"
"crypto/rand"
"encoding/binary"
"encoding/hex"
enchex "encoding/hex"
"errors"
"fmt"
"io/ioutil"
"math/rand"
"strings"
"unicode/utf16"
@@ -221,12 +221,15 @@ func (l *Conn) DigestMD5Bind(digestMD5BindRequest *DigestMD5BindRequest) (*Diges
}
if len(params) > 0 {
resp := computeResponse(
resp, err := computeResponse(
params,
"ldap/"+strings.ToLower(digestMD5BindRequest.Host),
digestMD5BindRequest.Username,
digestMD5BindRequest.Password,
)
if err != nil {
return nil, fmt.Errorf("compute digest-md5 response: %s", err)
}
packet = ber.Encode(ber.ClassUniversal, ber.TypeConstructed, ber.TagSequence, nil, "LDAP Request")
packet.AppendChild(ber.NewInteger(ber.ClassUniversal, ber.TypePrimitive, ber.TagInteger, l.nextMessageID(), "MessageID"))
@@ -291,12 +294,22 @@ func parseParams(str string) (map[string]string, error) {
m := make(map[string]string)
var key, value string
var state int
var escaped bool
for i := 0; i <= len(str); i++ {
switch state {
case 0: // reading key
if i == len(str) {
return nil, fmt.Errorf("syntax error on %d", i)
}
// The digest-challenge is an RFC 2068 #rule (RFC 2831 section 2.1.1),
// which permits optional linear whitespace around the comma directive
// separators. Directive names are tokens that never contain
// whitespace, so skip it here; otherwise a directive following
// "..., name" is keyed with a leading space and the lookups in
// computeResponse (realm, nonce, authzid) miss it.
if str[i] == ' ' || str[i] == '\t' {
continue
}
if str[i] != '=' {
key += string(str[i])
continue
@@ -307,6 +320,14 @@ func parseParams(str string) (map[string]string, error) {
m[key] = value
break
}
// Linear whitespace outside a quoted string is not part of the
// value: an unquoted value is a token and a quoted value's content
// is read in the quoted state below. Skipping it lets a challenge
// using the whitespace the #rule allows (e.g. `nonce="n" , qop=auth`)
// parse the same as the unspaced form.
if str[i] == ' ' || str[i] == '\t' {
continue
}
switch str[i] {
case ',':
m[key] = value
@@ -325,20 +346,34 @@ func parseParams(str string) (map[string]string, error) {
if i == len(str) {
return nil, fmt.Errorf("syntax error on %d", i)
}
if str[i] != '"' {
switch {
case escaped:
// RFC 2831 section 7.1 quoted-pair: a backslash escapes the
// following character, so the next byte is taken literally
// (this is how a server sends a literal " or \ in a realm or
// nonce).
value += string(str[i])
} else {
escaped = false
case str[i] == '\\':
escaped = true
case str[i] == '"':
state = 1
default:
value += string(str[i])
}
}
}
return m, nil
}
func computeResponse(params map[string]string, uri, username, password string) string {
func computeResponse(params map[string]string, uri, username, password string) (string, error) {
nc := "00000001"
qop := "auth"
cnonce := enchex.EncodeToString(randomBytes(16))
rb, err := randomBytes(16)
if err != nil {
return "", err
}
cnonce := enchex.EncodeToString(rb)
x := username + ":" + params["realm"] + ":" + password
y := md5Hash([]byte(x))
@@ -361,14 +396,24 @@ func computeResponse(params map[string]string, uri, username, password string) s
resp := enchex.EncodeToString(md5Hash([]byte(kd)))
return fmt.Sprintf(
`username="%s",realm="%s",nonce="%s",cnonce="%s",nc=00000001,qop=%s,digest-uri="%s",response=%s`,
username,
params["realm"],
params["nonce"],
quotedStringEscape(username),
quotedStringEscape(params["realm"]),
quotedStringEscape(params["nonce"]),
cnonce,
qop,
uri,
quotedStringEscape(uri),
resp,
)
), nil
}
// quotedStringEscape escapes the two characters that may not appear unescaped
// inside a DIGEST-MD5 quoted string per RFC 2831 section 7.1: the backslash
// and the double quote. The backslash is replaced first so the quotes escaped
// afterwards are not doubled.
func quotedStringEscape(s string) string {
s = strings.ReplaceAll(s, `\`, `\\`)
s = strings.ReplaceAll(s, `"`, `\"`)
return s
}
func md5Hash(b []byte) []byte {
@@ -377,12 +422,12 @@ func md5Hash(b []byte) []byte {
return hasher.Sum(nil)
}
func randomBytes(len int) []byte {
b := make([]byte, len)
for i := 0; i < len; i++ {
b[i] = byte(rand.Intn(256))
func randomBytes(length int) ([]byte, error) {
b := make([]byte, length)
if _, err := rand.Read(b); err != nil {
return nil, err
}
return b
return b, nil
}
var externalBindRequest = requestFunc(func(envelope *ber.Packet) error {
+3
View File
@@ -46,6 +46,9 @@ func (l *Conn) Compare(dn, attribute, value string) (bool, error) {
return false, err
}
if len(packet.Children) < 2 {
return false, fmt.Errorf("ldap: malformed response: expected at least 2 children, got %d", len(packet.Children))
}
if packet.Children[1].Tag == ApplicationCompareResponse {
err := GetLDAPError(packet)
+55 -11
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"net"
"net/url"
"strings"
"sync"
"sync/atomic"
"time"
@@ -112,7 +113,11 @@ type Conn struct {
outstandingRequests uint
messageMutex sync.Mutex
err error
// errMutex guards err only. It is a leaf lock: processMessages and reader
// record errors while another goroutine may hold messageMutex, so err must
// not share messageMutex or those writers could deadlock.
errMutex sync.Mutex
err error
}
var _ Client = &Conn{}
@@ -160,10 +165,18 @@ type DialContext struct {
func (dc *DialContext) dial(u *url.URL) (net.Conn, error) {
if u.Scheme == "ldapi" {
if u.Path == "" || u.Path == "/" {
u.Path = "/var/run/slapd/ldapi"
// RFC 4516 (and draft-chu-ldap-ldapi) put the socket path in the
// host component, percent-encoded; the path is an optional DN.
// parseLDAPURL has already decoded the host. Accept the older
// ldapi:///path form too so existing callers keep working.
path := u.Host
if path == "" {
path = u.Path
}
return dc.dialer.Dial("unix", u.Path)
if path == "" || path == "/" {
path = "/var/run/slapd/ldapi"
}
return dc.dialer.Dial("unix", path)
}
host, port, err := net.SplitHostPort(u.Host)
@@ -222,12 +235,33 @@ func DialTLS(network, addr string, config *tls.Config) (*Conn, error) {
return conn, nil
}
// parseLDAPURL parses an LDAP URL. It defers to net/url for the common
// ldap/ldaps/cldap schemes, but handles ldapi specially: the spec puts the
// unix socket path in the host, percent-encoded with %2F, and net/url rejects
// that as invalid. Pull the host out manually and decode it.
func parseLDAPURL(addr string) (*url.URL, error) {
const ldapi = "ldapi://"
if !strings.HasPrefix(addr, ldapi) {
return url.Parse(addr)
}
rest := addr[len(ldapi):]
host, path := rest, ""
if i := strings.IndexByte(rest, '/'); i >= 0 {
host, path = rest[:i], rest[i:]
}
decodedHost, err := url.PathUnescape(host)
if err != nil {
return nil, fmt.Errorf("ldapi: invalid host %q: %w", host, err)
}
return &url.URL{Scheme: "ldapi", Host: decodedHost, Path: path}, nil
}
// DialURL connects to the given ldap URL.
// The following schemas are supported: ldap://, ldaps://, ldapi://,
// and cldap:// (RFC1798, deprecated but used by Active Directory).
// On success a new Conn for the connection is returned.
func DialURL(addr string, opts ...DialOpt) (*Conn, error) {
u, err := url.Parse(addr)
u, err := parseLDAPURL(addr)
if err != nil {
return nil, NewError(ErrorNetwork, err)
}
@@ -338,11 +372,21 @@ func (l *Conn) nextMessageID() int64 {
// GetLastError returns the last recorded error from goroutines like processMessages and reader.
// Only the last recorded error will be returned.
func (l *Conn) GetLastError() error {
l.messageMutex.Lock()
defer l.messageMutex.Unlock()
l.errMutex.Lock()
defer l.errMutex.Unlock()
return l.err
}
// setError records the connection's last error. The background goroutines that
// call it (processMessages, reader, the per-request timeout helper and the
// SearchAsync worker) run concurrently with callers of GetLastError, so the
// write must take the mutex the getter reads under.
func (l *Conn) setError(err error) {
l.errMutex.Lock()
defer l.errMutex.Unlock()
l.err = err
}
// StartTLS sends the command to start a TLS session and then creates a new TLS Client
func (l *Conn) StartTLS(config *tls.Config) error {
if l.isTLS {
@@ -491,7 +535,7 @@ func (l *Conn) sendProcessMessage(message *messagePacket) bool {
func (l *Conn) processMessages() {
defer func() {
if err := recover(); err != nil {
l.err = fmt.Errorf("ldap: recovered panic in processMessages: %v", err)
l.setError(fmt.Errorf("ldap: recovered panic in processMessages: %v", err))
}
for messageID, msgCtx := range l.messageContexts {
// If we are closing due to an error, inform anyone who
@@ -541,7 +585,7 @@ func (l *Conn) processMessages() {
timer := time.NewTimer(time.Duration(requestTimeout))
defer func() {
if err := recover(); err != nil {
l.err = fmt.Errorf("ldap: recovered panic in RequestTimeout: %v", err)
l.setError(fmt.Errorf("ldap: recovered panic in RequestTimeout: %v", err))
}
timer.Stop()
@@ -563,7 +607,7 @@ func (l *Conn) processMessages() {
if msgCtx, ok := l.messageContexts[message.MessageID]; ok {
msgCtx.sendResponse(&PacketResponse{message.Packet, nil}, time.Duration(l.getTimeout()))
} else {
l.err = fmt.Errorf("ldap: received unexpected message %d, %v", message.MessageID, l.IsClosing())
l.setError(fmt.Errorf("ldap: received unexpected message %d, %v", message.MessageID, l.IsClosing()))
l.Debug.PrintPacket(message.Packet)
}
case MessageTimeout:
@@ -590,7 +634,7 @@ func (l *Conn) reader() {
cleanstop := false
defer func() {
if err := recover(); err != nil {
l.err = fmt.Errorf("ldap: recovered panic in reader: %v", err)
l.setError(fmt.Errorf("ldap: recovered panic in reader: %v", err))
}
if !cleanstop {
l.Close()
+84 -27
View File
@@ -565,12 +565,20 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
case 1:
// just type, no criticality or value
packet.Children[0].Description = "Control Type (" + ControlTypeMap[ControlType] + ")"
ControlType = packet.Children[0].Value.(string)
ct, ok := packet.Children[0].Value.(string)
if !ok {
return nil, fmt.Errorf("control type is not a string: %T", packet.Children[0].Value)
}
ControlType = ct
case 2:
packet.Children[0].Description = "Control Type (" + ControlTypeMap[ControlType] + ")"
if packet.Children[0].Value != nil {
ControlType = packet.Children[0].Value.(string)
ct, ok := packet.Children[0].Value.(string)
if !ok {
return nil, fmt.Errorf("control type is not a string: %T", packet.Children[0].Value)
}
ControlType = ct
} else if packet.Children[0].Data != nil {
ControlType = packet.Children[0].Data.String()
} else {
@@ -579,9 +587,9 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
// Children[1] could be criticality or value (both are optional)
// duck-type on whether this is a boolean
if _, ok := packet.Children[1].Value.(bool); ok {
if crit, ok := packet.Children[1].Value.(bool); ok {
packet.Children[1].Description = "Criticality"
Criticality = packet.Children[1].Value.(bool)
Criticality = crit
} else {
packet.Children[1].Description = "Control Value"
value = packet.Children[1]
@@ -589,10 +597,18 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
case 3:
packet.Children[0].Description = "Control Type (" + ControlTypeMap[ControlType] + ")"
ControlType = packet.Children[0].Value.(string)
ct, ok := packet.Children[0].Value.(string)
if !ok {
return nil, fmt.Errorf("control type is not a string: %T", packet.Children[0].Value)
}
ControlType = ct
packet.Children[1].Description = "Criticality"
Criticality = packet.Children[1].Value.(bool)
crit, ok := packet.Children[1].Value.(bool)
if !ok {
return nil, fmt.Errorf("criticality is not a bool: %T", packet.Children[1].Value)
}
Criticality = crit
packet.Children[2].Description = "Control Value"
value = packet.Children[2]
@@ -606,6 +622,9 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
case ControlTypeManageDsaIT:
return NewControlManageDsaIT(Criticality), nil
case ControlTypePaging:
if value == nil {
return nil, fmt.Errorf("paging control value is missing")
}
value.Description += " (Paging)"
c := new(ControlPaging)
if value.Value != nil {
@@ -617,11 +636,21 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
value.Value = nil
value.AppendChild(valueChildren)
}
if len(value.Children) == 0 {
return nil, fmt.Errorf("paging control value is empty")
}
value = value.Children[0]
value.Description = "Search Control Value"
if len(value.Children) < 2 {
return nil, fmt.Errorf("paging control value has %d children, expected 2", len(value.Children))
}
value.Children[0].Description = "Paging Size"
value.Children[1].Description = "Cookie"
c.PagingSize = uint32(value.Children[0].Value.(int64))
pagingSize, ok := value.Children[0].Value.(int64)
if !ok {
return nil, fmt.Errorf("paging size is not an integer: %T", value.Children[0].Value)
}
c.PagingSize = uint32(pagingSize)
c.Cookie = value.Children[1].Data.Bytes()
value.Children[1].Value = c.Cookie
return c, nil
@@ -729,7 +758,16 @@ func DecodeControl(packet *ber.Packet) (Control, error) {
c.ControlType = ControlType
c.Criticality = Criticality
if value != nil {
c.ControlValue = value.Value.(string)
// A non-conforming or malicious server can send a non-string
// (or nil) value here; the previous unchecked cast panicked
// the calling goroutine, see #561. Fall back to the raw bytes
// when the value isn't a string so we surface an error
// instead of crashing.
if s, ok := value.Value.(string); ok {
c.ControlValue = s
} else if value.Data != nil {
c.ControlValue = value.Data.String()
}
}
return c, nil
}
@@ -921,28 +959,44 @@ func (c *ControlServerSideSorting) GetControlType() string {
}
func NewControlServerSideSorting(value *ber.Packet) (*ControlServerSideSorting, error) {
sortKeys := []*SortKey{}
val, err := ber.DecodePacketErr(value.Data.Bytes())
if err != nil {
return nil, fmt.Errorf("decode packet err: %s", err)
}
val := value.Children[1].Children
if len(val) != 1 {
if len(val.Children) == 0 {
return nil, fmt.Errorf("no sequence value in packet")
}
sequences := val[0].Children
var sortKeys []*SortKey
for i, sequence := range sequences {
sortKey := new(SortKey)
if len(sequence.Children) < 2 {
return nil, fmt.Errorf("attributeType or matchingRule is missing from sequence %d", i)
for i, sequence := range val.Children {
if len(sequence.Children) < 1 || len(sequence.Children) > 3 {
return nil, fmt.Errorf("attributeType is missing from sequence %d", i)
}
sortKey.AttributeType = sequence.Children[0].Value.(string)
sortKey.MatchingRule = sequence.Children[1].Value.(string)
sortKey := new(SortKey)
if len(sequence.Children) == 3 {
sortKey.Reverse = sequence.Children[2].Value.(bool)
for _, child := range sequence.Children {
switch {
case child.ClassType == ber.ClassUniversal && child.Tag == ber.TagOctetString:
// A constructed-form OCTET STRING matches this case but leaves
// Value nil; guard the assertion so a malformed attributeType is
// rejected below rather than panicking.
if attrType, ok := child.Value.(string); ok {
sortKey.AttributeType = attrType
}
case child.ClassType == ber.ClassContext && child.Tag == 0:
sortKey.MatchingRule = child.Data.String()
case child.ClassType == ber.ClassContext && child.Tag == 1:
b := child.Data.Bytes()
sortKey.Reverse = len(b) > 0 && b[0] != 0
}
}
if sortKey.AttributeType == "" {
return nil, fmt.Errorf("attributeType is missing from sequence %d", i)
}
sortKeys = append(sortKeys, sortKey)
@@ -959,7 +1013,6 @@ func (c *ControlServerSideSorting) Encode() *ber.Packet {
packet := ber.Encode(ber.ClassUniversal, ber.TypeConstructed, ber.TagSequence, nil, "Control")
control := ber.NewString(ber.ClassUniversal, ber.TypePrimitive, ber.TagOctetString, c.GetControlType(), "Control Type")
value := ber.Encode(ber.ClassUniversal, ber.TypePrimitive, ber.TagOctetString, nil, "Control Value")
seqs := ber.Encode(ber.ClassUniversal, ber.TypeConstructed, ber.TagSequence, nil, "SortKeyList")
for _, f := range c.SortKeys {
@@ -968,9 +1021,11 @@ func (c *ControlServerSideSorting) Encode() *ber.Packet {
seq.AppendChild(
ber.NewString(ber.ClassUniversal, ber.TypePrimitive, ber.TagOctetString, f.AttributeType, "attributeType"),
)
seq.AppendChild(
ber.NewString(ber.ClassContext, ber.TypePrimitive, 0, f.MatchingRule, "orderingRule"),
)
if f.MatchingRule != "" {
seq.AppendChild(
ber.NewString(ber.ClassContext, ber.TypePrimitive, 0, f.MatchingRule, "orderingRule"),
)
}
if f.Reverse {
seq.AppendChild(
ber.NewBoolean(ber.ClassContext, ber.TypePrimitive, 1, f.Reverse, "reverseOrder"),
@@ -980,7 +1035,7 @@ func (c *ControlServerSideSorting) Encode() *ber.Packet {
seqs.AppendChild(seq)
}
value.AppendChild(seqs)
value := ber.Encode(ber.ClassUniversal, ber.TypePrimitive, ber.TagOctetString, string(seqs.Bytes()), "Control Value")
packet.AppendChild(control)
packet.AppendChild(value)
@@ -1060,6 +1115,8 @@ func NewControlServerSideSortingResult(pkt *ber.Packet) (*ControlServerSideSorti
return nil, err
}
control.Result = ControlServerSideSortingCode(codeInt)
return control, nil
}
+3
View File
@@ -52,6 +52,9 @@ func (l *Conn) Del(delRequest *DelRequest) error {
return err
}
if len(packet.Children) < 2 {
return fmt.Errorf("ldap: malformed response: expected at least 2 children, got %d", len(packet.Children))
}
if packet.Children[1].Tag == ApplicationDelResponse {
err := GetLDAPError(packet)
if err != nil {
Loaded 100 of 362 files, more files were not shown because too many files have changed in this diff. Show more