From 8053275166500c5b3b00b6257d42f8b019453907 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Thu, 23 Apr 2026 11:05:38 +0200 Subject: [PATCH 01/13] docs(adr): add 0005 unified search index mapping MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Propose a single central Go-struct + overrides map as the source of truth for the search index layout across bleve and OpenSearch — the same definition drives the per-backend index mapping, the write-time adapter, the hit-decoding path, and the KQL compiler's case-folding rules, so the two backends cannot drift silently again. Also records the end-to-end case-handling principle for facet data (indexed as case-preserving keywords so aggregation buckets return correct display values), the sibling-field pattern for geopoint on Location, and the rationale for replacing the two backends' implicit defaults with an explicit, backend-agnostic contract. PR #2659 is a proof-of-concept implementation the proposal emerged from; scope and APIs there will be revisited once this ADR lands. --- docs/adr/0005-unified-search-index-mapping.md | 255 ++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 docs/adr/0005-unified-search-index-mapping.md diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md new file mode 100644 index 0000000000..7147221fe6 --- /dev/null +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -0,0 +1,255 @@ +--- +title: "5. Unified Search Index Mapping" +--- + +* Status: proposed +* Deciders: [] +* Date: 2026-04-23 + +Reference: https://github.com/opencloud-eu/opencloud/pull/2659, a +proof-of-concept that demonstrates the proposal end-to-end. It is +not presumed accepted by this ADR; scope and APIs there will be +revisited once the decision here lands. + +## Context and Problem Statement + +The search service supports two backends, bleve (embedded) and +OpenSearch (external). Each backend currently carries its own, +independently maintained description of the index layout: + +- The bleve backend hand-builds a document mapping that explicitly + declares only Name, Tags, Favorites and Content. Everything else, + including the entire facet block (audio, image, photo, location), + is left to bleve's dynamic mapping. +- The OpenSearch backend ships a static JSON template that covers a + similar but not identical subset, plus a few OpenSearch-specific + primitives (path_hierarchy analyzer, wildcard MimeType). It does + not list the facet sub-fields either; they are produced by + OpenSearch's dynamic templating at first write. +- The graph DriveItem assembly path keeps its own private copy of a + reflection-based walker to turn CS3 ArbitraryMetadata back into + typed libregraph facets, parallel to the search service's + reflection helpers but maintained separately. +- The bleve KQL compiler keeps a hand-maintained set of field names + whose query values need to be pre-lowercased, with a comment that + literally says "Keep in sync with index.go". + +The current implementation has four concrete problems: + +1. **The two backends do not behave the same.** Both rely on their + own implicit defaults for fields that are not explicitly + declared. The inferred shapes differ: bleve produces keyword- + analyzed text, OpenSearch produces a `text + keyword` multi-field + with auto-detected dates. Nobody has written down which behavior + is the intended one. +2. **The indexed facet metadata is effectively unreachable.** The + audio, image, photo and location sub-fields go into the index and + take up space, but no caller of the search service can actually + use them. The KQL compiler does not support the dot syntax needed + to refer to `audio.artist` on the OpenSearch side, and until + #2633 the bleve-side compiler unconditionally lowercased query + values even against case-preserving fields. The hit and REPORT + paths do not expose facet values back to consumers either. +3. **Drift risk.** The OpenSearch JSON template is a subset of what + actually gets indexed. Even where it overlaps with the bleve + mapping it diverges on analyzer choices. Because point 2 means + nothing queries the diverging fields, the divergence has been + invisible, but it would surface the moment the first working + cross-backend facet query landed. +4. **Per-facet cost.** Adding a new facet (motionPhoto, etc.) + requires coordinated edits across the proto message, both backend + mappings, the bleve hit converters, the OpenSearch convert + closures, the search service's metadata persistence, the graph + DriveItem assembly, and the KQL compiler's lowercasing set. Most + of those edits are boilerplate following a copy-paste pattern. + Adding a genuinely new index capability (geopoint, wildcard, + ...) means wiring it in at every one of those sites, and there + is no single place to hook a type-specific adapter. + +### A note on backwards compatibility + +The unreachability above (problem 2) has a useful corollary for +this ADR: **changing the indexed shape of the facet fields cannot +break any existing client of the search service**, because no +client successfully reads them today. The behavior changes +discussed below are therefore additive in a literal sense; nothing +that works today stops working as a result. + +## Decision Drivers + +* **Predictable OpenCloud API behavior independent of backend.** + Consumers of the search service should be able to rely on the + documented behavior of the API, not on which backend happens to + be configured. Today the same query can give different results + depending on whether bleve or OpenSearch is wired in (bleve's + dynamic default is `keyword`, exact match; OpenSearch's dynamic + default is `text + keyword`, also matches sub-tokens of a + string). That is backend-implementation leakage, and trying to + keep the two implicit defaults synchronized has not worked. +* Single source of truth for the indexed schema, so the two backends + cannot drift silently again. +* Reduce the per-facet cost so future facets (motionPhoto and + whatever comes next) can be added with minimal boilerplate. +* Establish a single place to hook index-type-specific behavior, so + a new capability needs to be implemented at most once per backend + and then becomes available for any field uniformly. +* A one-time reindex is an acceptable upgrade path. Both bleve and + OpenSearch store their mapping alongside the data; existing + indexes keep serving queries against their stored shape without + any automatic reshaping. Benefiting from the new behavior is done + by creating a fresh index and re-ingesting, which is the normal + reindex flow, rather than by inventing migration tooling. + +## Considered Options + +### Option 1: Do nothing, keep relying on implicit backend defaults + +Accept that bleve and OpenSearch each fall back to their own +dynamic-mapping defaults for whatever is not explicitly declared, +and treat the observable search behavior of OpenCloud as "whatever +the configured backend happens to do". Adding a facet stays a +copy-paste coordination across half a dozen sites; the existing +divergence between bleve (keyword) and OpenSearch (`text + keyword` +multi-field plus auto-date detection) stays silently in place +until a working query actually reaches the diverging field and +returns different answers on the two backends. + +Low upfront work, but it makes the OpenCloud API behavior a +function of the backend rather than a contract, and it keeps the +per-facet boilerplate cost for every new field. + +### Option 2: Generate one backend's mapping from the other + +Treat one backend as canonical (likely bleve, because Go types) and +derive the other. Partial answer; it still does not help the reader +path or the graph walker, and still leaves per-facet boilerplate in +non-mapping code. + +### Option 3: A struct-driven mapping (chosen) + +Let the Go struct that represents an indexed document, together +with a small overrides map, be the single source of truth. A +reflection-based helper walks the struct via json tags and emits +each backend's index mapping. The same definition drives the +write-time path, the hit-decoding path, and the query compiler's +case-folding rules. Any future field follows one declaration in +one place and falls through the whole pipeline consistently. + +## Decision Outcome + +Adopt Option 3. The Go struct that represents an indexed document, +together with a small overrides map, becomes the single source of +truth for the search index. The bleve and OpenSearch index +mappings, the write-time conversion, the hit-decoding path, and +the query compiler's case-folding rules are all derived from that +same definition. Drift between backends is prevented by +construction, because there is no second place to edit. + +The overrides surface stays small. Each entry declares one of a +handful of things per field: a semantic type for fields whose +intent cannot be inferred from the Go type (for example a path- +analyzed field, a fulltext field, a wildcard field, a geopoint +field), an analyzer name where one is needed, or a backend- +specific inclusion knob. Any field that needs something beyond the +inferred defaults gets one line in the overrides map and that one +line flows through every derived piece. Overrides are validated at +startup so a typo fails loudly instead of silently disabling a +setting. + +A practical consequence of having one place to hook things: when a +new capability is needed (a geopoint representation, a sibling +field for a different aggregation behavior, a different analyzer +for a class of fields, ...) it can be implemented once per backend +in the central pipeline. After that, turning the capability on for +a specific field is a single override entry, and both backends +adopt it the same way. This ADR does not decide which capabilities +to add, only that they will land in this uniform shape rather than +through coordinated per-site edits. + +### Facet values are indexed as case-preserving keywords + +All facet sub-fields, meaning any leaf inside `audio`, `photo`, +`image`, `location`, are indexed as plain keywords on both +backends. No tokenization, no case folding. The raw value the +extractor saw, or the CS3 ArbitraryMetadata string, is what lands +in the index. + +This is the single intended semantic for facets across bleve and +OpenSearch, and it is driven by what aggregations need. +Aggregation buckets ("group all files by `audio.artist`", "list +distinct `photo.cameraMake`") return bucket keys drawn from the +indexed terms. If the indexing analyzer lowercases (OpenSearch's +default `text + keyword` multi-field against the text leg, or a +`lowercaseKeyword`-style analyzer), the buckets come back lower- +cased: a distinct-artists query would answer `motörhead` and +`queen` instead of the original display casings, and two tag +writers using `Motörhead` versus `MOTÖRHEAD` would collapse into a +single bucket labelled `motörhead`. For a metadata display use +case (thumbnails, facet filters in the UI, distinct lists) that +behavior is not what we want. + +Keyword-indexed values preserve case and produce correctly- +labelled buckets on both backends. If, in the future, case- +insensitive search on a facet property is also wanted alongside +the case-preserving bucket view, that is a strict superset and +can be added on top. It is much easier to bolt onto one +consistent backend-agnostic state than to retrofit it onto the +current divergent, implicit per-backend behavior. + +The query side lines up with this: the KQL parser preserves the +user's casing, and the per-backend compiler decides per field +whether to fold that value to lowercase before handing it to the +backend, based on the same overrides map the index mapping uses. +Fields whose override selects a lowercasing analyzer (Name, Tags, +Favorites, Content) get their query value folded so user input +matches the tokens the index produced; everything else, including +all facet sub-fields, compares case-sensitive on both sides. The +case-sensitivity alignment of the compiler with the index mapping +was started in #2633; this ADR completes it by deriving both +sides from the same source. + +### Behavior changes, applied to newly-created indexes + +Existing indexes keep their stored mapping; the new shape applies +only to indexes created after the proposal lands. + +- OpenSearch `Tags` and `Favorites` change from dynamic `keyword` + to `text + lowercaseKeyword`, matching the bleve mapping. The + analyzer is registered in the new index settings. +- OpenSearch facet sub-strings (`audio.*`, `photo.*`, `image.*`) + change from the dynamic `text + keyword` multi-field to + `keyword`-only, implementing the principle above. As noted in + the context section, the tokenized leg is not actually reachable + from user queries today, so no working path regresses. + +### Known trade-off + +The write-time pipeline produces the document as a generic map via +a json round-trip. The OpenSearch write path already does the +equivalent today via the same json-based conversion helper, so +that path is unchanged. The bleve write path, which previously +handed the struct directly to bleve's reflective indexer, now goes +through the same map-producing step and pays roughly the same +cost. On hot paths (initial indexing of a large space) this is +measurable but not significant; if it ever matters, a direct +reflection walker can replace the json round-trip without changing +any call site. + +### Follow-ups out of scope for this ADR + +- **WebDAV REPORT facet exposure.** The current webdav search + endpoint renders none of the facet fields back to the client. + This is a missing feature, not a regression of the proposal; + its natural resolution is to let the graph-search endpoint + (not yet in tree) take over once graph search lands. +- **Graph search hit conversion.** Once graph search is wired up, + it needs to translate proto hits back into libregraph DriveItems. + The same facet-copy helper used internally on the search service + side is reusable. +- **reva's PROPFIND facet listing** uses its own hand-maintained + per-facet key lists. reva deliberately does not depend on the + libregraph Go types, so unifying those key sets is a reva-side + decision tracked separately. +- **Write-path performance.** The json round-trip in the bleve + write path is an optional optimisation target with no call-site + impact when it lands. From 2223b8ed9d23567c1f5026f13541fec76ca5a6ef Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Tue, 12 May 2026 09:24:36 +0200 Subject: [PATCH 02/13] docs(adr): include location.* in the OpenSearch facet sub-strings list The bullet enumerating which OpenSearch facet sub-fields change from text+keyword to keyword-only listed audio, photo and image but omitted location, which is in the same dynamic-template-handled facet block per the context section. Add it for consistency. --- docs/adr/0005-unified-search-index-mapping.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index 7147221fe6..93cd4305a4 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -216,8 +216,8 @@ only to indexes created after the proposal lands. - OpenSearch `Tags` and `Favorites` change from dynamic `keyword` to `text + lowercaseKeyword`, matching the bleve mapping. The analyzer is registered in the new index settings. -- OpenSearch facet sub-strings (`audio.*`, `photo.*`, `image.*`) - change from the dynamic `text + keyword` multi-field to +- OpenSearch facet sub-strings (`audio.*`, `photo.*`, `image.*`, + `location.*`) change from the dynamic `text + keyword` multi-field to `keyword`-only, implementing the principle above. As noted in the context section, the tokenized leg is not actually reachable from user queries today, so no working path regresses. From 03cfbacafacdf10e761785fb77afcbebd61b7113 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Wed, 1 Jul 2026 19:49:05 +0200 Subject: [PATCH 03/13] docs(adr): add concrete mtime/name backend-drift examples --- docs/adr/0005-unified-search-index-mapping.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index 93cd4305a4..3e3024f01f 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -41,7 +41,16 @@ The current implementation has four concrete problems: declared. The inferred shapes differ: bleve produces keyword- analyzed text, OpenSearch produces a `text + keyword` multi-field with auto-detected dates. Nobody has written down which behavior - is the intended one. + is the intended one. Two concrete instances surfaced while building + #2659: + - **mtime** is stored as an RFC3339 string. OpenSearch's dynamic + mapping auto-detects it as `date`; bleve leaves it `keyword`. So + `mtime:>...` is a chronological range on OpenSearch but a + lexicographic string compare on bleve. + - **name/tags**: bleve indexes a single lowercase token (exact or + wildcard match only); OpenSearch word-tokenizes, so a bare + `name:report` matches "My Report.txt" on OpenSearch but not on + bleve. 2. **The indexed facet metadata is effectively unreachable.** The audio, image, photo and location sub-fields go into the index and take up space, but no caller of the search service can actually From 874d8fdac2b2b8895c974182bdfff70d594c745a Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 13:54:07 +0200 Subject: [PATCH 04/13] test(search): fail the parity suite when the committed matrix is stale The suite already regenerates README.md on every run; now a run whose content differs from the committed file also fails, so CI catches a stale matrix. The fresh content is on disk after the run, committing it is the fix. --- services/search/pkg/parity/matrix_test.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/services/search/pkg/parity/matrix_test.go b/services/search/pkg/parity/matrix_test.go index 5630ac0cfb..48f5df66ef 100644 --- a/services/search/pkg/parity/matrix_test.go +++ b/services/search/pkg/parity/matrix_test.go @@ -226,8 +226,15 @@ func writeMatrix(report types.Report) { matrixNames(row.answered["bleve"]), matrixNames(row.answered["opensearch"]), matrixVerdict(row)) } + previous, _ := os.ReadFile(matrixFile) if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) + return + } + if string(previous) != out.String() { + // the committed matrix must match what the suite answers; the fresh + // content is already on disk, committing it is the fix + Fail(matrixFile + " was out of date, the suite regenerated it: commit the change") } } From b1533a31355c378a28a1b0eaa4f80817dccf32cd Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 13:58:46 +0200 Subject: [PATCH 05/13] test(search): write the matrix only with UPDATE_MATRIX A plain run compares and fails on a stale README without touching the working tree; UPDATE_MATRIX=true regenerates it. --- services/search/pkg/parity/matrix_test.go | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/services/search/pkg/parity/matrix_test.go b/services/search/pkg/parity/matrix_test.go index 48f5df66ef..c10fed5e56 100644 --- a/services/search/pkg/parity/matrix_test.go +++ b/services/search/pkg/parity/matrix_test.go @@ -226,15 +226,17 @@ func writeMatrix(report types.Report) { matrixNames(row.answered["bleve"]), matrixNames(row.answered["opensearch"]), matrixVerdict(row)) } - previous, _ := os.ReadFile(matrixFile) - if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { - fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) + if os.Getenv("UPDATE_MATRIX") != "" { + if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { + fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) + } return } + + // the committed matrix must match what the suite answers + previous, _ := os.ReadFile(matrixFile) if string(previous) != out.String() { - // the committed matrix must match what the suite answers; the fresh - // content is already on disk, committing it is the fix - Fail(matrixFile + " was out of date, the suite regenerated it: commit the change") + Fail(matrixFile + " is out of date: regenerate it with UPDATE_MATRIX=true and commit the change") } } From feb13d426bbb5f65c4de155168de477046ed0616 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 13:59:33 +0200 Subject: [PATCH 06/13] test(search): name the update flag in the generated header --- services/search/pkg/parity/README.md | 2 +- services/search/pkg/parity/matrix_test.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/services/search/pkg/parity/README.md b/services/search/pkg/parity/README.md index 67c02ab784..fb86424d73 100644 --- a/services/search/pkg/parity/README.md +++ b/services/search/pkg/parity/README.md @@ -1,6 +1,6 @@ # Engine parity -Written by the parity suite (`go test ./services/search/pkg/parity/`), do not edit. +Written by the parity suite (`UPDATE_MATRIX=true go test ./services/search/pkg/parity/`), do not edit. Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as expected, `❌ known` when an engine's divergence is documented in the case (`engineOverrides`), `❌` when it is not. `✅ stale` when every engine diff --git a/services/search/pkg/parity/matrix_test.go b/services/search/pkg/parity/matrix_test.go index c10fed5e56..6347ae6571 100644 --- a/services/search/pkg/parity/matrix_test.go +++ b/services/search/pkg/parity/matrix_test.go @@ -171,7 +171,7 @@ func writeMatrix(report types.Report) { out := &strings.Builder{} out.WriteString("# Engine parity\n\n") - out.WriteString("Written by the parity suite (`go test ./services/search/pkg/parity/`), do not edit.\n") + out.WriteString("Written by the parity suite (`UPDATE_MATRIX=true go test ./services/search/pkg/parity/`), do not edit.\n") out.WriteString("Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as\n") out.WriteString("expected, `❌ known` when an engine's divergence is documented in the case\n") out.WriteString("(`engineOverrides`), `❌` when it is not. `✅ stale` when every engine\n") From fe1e7632fa4410fc5ca20bd9a606fe94e0c67abd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 31 Aug 2026 14:44:13 +0000 Subject: [PATCH 07/13] build(deps): bump go.opentelemetry.io/otel from 1.45.0 to 1.46.0 Bumps [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) from 1.45.0 to 1.46.0. - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.45.0...v1.46.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/otel dependency-version: 1.46.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- go.mod | 6 +- go.sum | 12 +-- vendor/go.opentelemetry.io/otel/.golangci.yml | 3 +- vendor/go.opentelemetry.io/otel/CHANGELOG.md | 32 +++++++- vendor/go.opentelemetry.io/otel/Makefile | 30 ++++++-- vendor/go.opentelemetry.io/otel/README.md | 7 ++ vendor/go.opentelemetry.io/otel/VERSIONING.md | 6 ++ .../otel/attribute/hash.go | 74 ++++++++++++++++--- .../otel/attribute/internal/xxhash/xxhash.go | 5 ++ .../go.opentelemetry.io/otel/attribute/set.go | 3 + vendor/go.opentelemetry.io/otel/renovate.json | 7 ++ vendor/go.opentelemetry.io/otel/version.go | 2 +- vendor/go.opentelemetry.io/otel/versions.yaml | 8 +- vendor/modules.txt | 6 +- 14 files changed, 164 insertions(+), 37 deletions(-) diff --git a/go.mod b/go.mod index ec4d735f3b..49a35abf39 100644 --- a/go.mod +++ b/go.mod @@ -97,11 +97,11 @@ require ( go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 go.opentelemetry.io/contrib/zpages v0.70.0 - go.opentelemetry.io/otel v1.45.0 + go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0 go.opentelemetry.io/otel/sdk v1.45.0 - go.opentelemetry.io/otel/trace v1.45.0 + go.opentelemetry.io/otel/trace v1.46.0 golang.org/x/crypto v0.55.0 golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f golang.org/x/image v0.45.0 @@ -378,7 +378,7 @@ require ( go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 // indirect - go.opentelemetry.io/otel/metric v1.45.0 // indirect + go.opentelemetry.io/otel/metric v1.46.0 // indirect go.opentelemetry.io/proto/otlp v1.11.0 // indirect go.uber.org/multierr v1.11.0 // indirect go.uber.org/zap v1.27.0 // indirect diff --git a/go.sum b/go.sum index b6fdb20f0c..7b014067d6 100644 --- a/go.sum +++ b/go.sum @@ -1299,22 +1299,22 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAy go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04= go.opentelemetry.io/contrib/zpages v0.70.0 h1:uBcclHekIrCRLO4KZfXiebDcWB52QQXaHGgvhUJ1K5I= go.opentelemetry.io/contrib/zpages v0.70.0/go.mod h1:Nz9A1+68HVoJrk+cg92zJbcbHFAa+Jbl1uINwAlcK94= -go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU= -go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ= +go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc= +go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE= go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 h1:QRefszxJmfPdjXUUm3j6iDzY03mTPXMjqErFqQ67vUg= go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0/go.mod h1:Tiz03lTBVBrm7eWZBOidzEaYaJa8tjwGUGv6d8mlTyk= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 h1:fG5MCxGz8+2VtrN/WgqSpJFctVz24gpxj8CxkKmc8Ww= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0/go.mod h1:BmAYTn+3ysbRe+IU2msxmf5Rx3g6DHvex+tWI3LdhYI= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0 h1:lsA/S1bxgdbyFGkTj+3meEdJ6ADVU7QoFstV6MXgE68= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0/go.mod h1:L7u+MirGoB1bjeLH66+xDykF4RC8C3RN7lIFpBiewUo= -go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M= -go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s= +go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= +go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o= go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw= go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA= go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o= go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA= -go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag= -go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc= +go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= +go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk= go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= diff --git a/vendor/go.opentelemetry.io/otel/.golangci.yml b/vendor/go.opentelemetry.io/otel/.golangci.yml index 8a7f1ec5d9..a0ba70805e 100644 --- a/vendor/go.opentelemetry.io/otel/.golangci.yml +++ b/vendor/go.opentelemetry.io/otel/.golangci.yml @@ -143,8 +143,7 @@ linters: - name: constant-logical-expr - name: context-as-argument arguments: - - allow-types-before: '*testing.T' - disabled: true + - allow-types-before: '*testing.T,*testing.B' - name: context-keys-type - name: deep-exit - name: defer diff --git a/vendor/go.opentelemetry.io/otel/CHANGELOG.md b/vendor/go.opentelemetry.io/otel/CHANGELOG.md index 2db588ea27..4d473a1e85 100644 --- a/vendor/go.opentelemetry.io/otel/CHANGELOG.md +++ b/vendor/go.opentelemetry.io/otel/CHANGELOG.md @@ -11,6 +11,34 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm +## [1.46.0/0.68.0/0.22.0/0.0.19] - 2026-08-25 + +This release is the last to support [Go 1.25]. +The next release will require at least [Go 1.26]. + +### Added + +- Support testing of [Go 1.27]. (#8811) +- Support `http/json` protocol in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#8273, #8775, #8831) +- Add `Hasher` struct and methods in `go.opentelemetry.io/otel/attribute` to compute authoritative `Distinct` hashes incrementally for attribute filtering and deduplication. (#8598) + +### Changed + +- Lazily evaluate filtered and dropped attributes on measurement hot paths in `go.opentelemetry.io/otel/sdk/metric` to avoid unnecessary attribute set allocations. (#8598) +- Add `ErrExporterShutdown` to `go.opentelemetry.io/otel/sdk/log` and return it from the `go.opentelemetry.io/otel/exporters/stdout/stdoutlog`, `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`, and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp` exporters when `Export` is called after `Shutdown`. (#8773) +- Clarify in `go.opentelemetry.io/otel/log` that calling `Logger.Enabled` is optional and that cached results can become stale. (#8764) + +### Fixed + +- Export dropped attribute counts in OTLP log records from `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8829) +- Name span events created from OpenTracing logs after the `event` log field, falling back to `log`, instead of always using an empty name in `go.opentelemetry.io/otel/bridge/opentracing`. (#8648) +- Count exception attributes omitted due to the attribute count limit as dropped in `go.opentelemetry.io/otel/sdk/log`. (#8796) +- Prevent log record and instrumentation scope attributes with empty keys from reaching processors and exporters in `go.opentelemetry.io/otel/sdk/log`. (#8797) +- Fix a data race when span attributes are read concurrently in `go.opentelemetry.io/otel/sdk/trace`. (#8706) +- Prevent a panic in `(*Set).Filter` when called on a nil receiver in `go.opentelemetry.io/otel/attribute`. (#8792) +- The simple span and log processors record `otel.sdk.processor.{span,log}.processed` when the record is submitted to the exporter instead of after the export completes, and no longer set `error.type` from the export outcome, in `go.opentelemetry.io/otel/sdk/trace` and `go.opentelemetry.io/otel/sdk/log`. (#8705) +- Prevent `Resource.MarshalLog` from panicking on nil resources in `go.opentelemetry.io/otel/sdk/resource`. (#8758) + ## [1.45.0/0.67.0/0.21.0/0.0.18] - 2026-08-03 ### Added @@ -3782,7 +3810,8 @@ It contains api and sdk for trace and meter. - CircleCI build CI manifest files. - CODEOWNERS file to track owners of this project. -[Unreleased]: https://github.com/open-telemetry/opentelemetry-go/compare/v1.45.0...HEAD +[Unreleased]: https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...HEAD +[1.46.0/0.68.0/0.22.0/0.0.19]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.46.0 [1.45.0/0.67.0/0.21.0/0.0.18]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.45.0 [1.44.0/0.66.0/0.20.0/0.0.17]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.44.0 [1.43.0/0.65.0/0.19.0]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.43.0 @@ -3887,6 +3916,7 @@ It contains api and sdk for trace and meter. +[Go 1.27]: https://go.dev/doc/go1.27 [Go 1.26]: https://go.dev/doc/go1.26 [Go 1.25]: https://go.dev/doc/go1.25 [Go 1.24]: https://go.dev/doc/go1.24 diff --git a/vendor/go.opentelemetry.io/otel/Makefile b/vendor/go.opentelemetry.io/otel/Makefile index d4711257df..c674ef8a17 100644 --- a/vendor/go.opentelemetry.io/otel/Makefile +++ b/vendor/go.opentelemetry.io/otel/Makefile @@ -69,8 +69,13 @@ $(GORELEASE): PACKAGE=golang.org/x/exp/cmd/gorelease GOVULNCHECK = $(TOOLS)/govulncheck $(TOOLS)/govulncheck: PACKAGE=golang.org/x/vuln/cmd/govulncheck +AFFECTEDMODS = $(TOOLS)/affectedmods +AFFECTEDMODS_FILES := $(sort $(shell find $(TOOLS_MOD_DIR)/affectedmods -type f)) +$(TOOLS)/affectedmods: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/affectedmods +$(TOOLS)/affectedmods: $(AFFECTEDMODS_FILES) + .PHONY: tools -tools: $(CROSSLINK) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(VERIFYREADMES) $(MULTIMOD) $(SEMCONVKIT) $(GOTMPL) $(GORELEASE) +tools: $(CROSSLINK) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(VERIFYREADMES) $(MULTIMOD) $(SEMCONVKIT) $(GOTMPL) $(GORELEASE) $(AFFECTEDMODS) # Virtualized python tools via docker @@ -125,9 +130,11 @@ go-work: $(CROSSLINK) # Build -.PHONY: build +.PHONY: build cross-build build: $(OTEL_GO_MOD_DIRS:%=build/%) $(OTEL_GO_MOD_DIRS:%=build-tests/%) +# Cross-platform builds cannot execute the binaries produced by build-tests (go test), so use a compile-only target. +cross-build: $(OTEL_GO_MOD_DIRS:%=build/%) build/%: DIR=$* build/%: @echo "$(GO) build $(DIR)/..." \ @@ -163,6 +170,9 @@ test/%: | grep -v third_party \ | xargs $(GO) test -timeout $(TIMEOUT)s $(ARGS) +.PHONY: test-tools +test-tools: test/$(TOOLS_MOD_DIR) + COVERAGE_MODE = atomic COVERAGE_PROFILE = coverage.out .PHONY: test-coverage @@ -187,15 +197,23 @@ benchmark/%: # sdk/metric is split into two shards to work around CodSpeed limitations. # See https://github.com/CodSpeedHQ/codspeed-go/issues/56 -BENCHMARK_SHARDS := $(filter-out ./sdk/metric,$(OTEL_GO_MOD_DIRS)) ./sdk/metric/root ./sdk/metric/internal -benchmark/./sdk/metric/root: +BENCHMARK_SHARDS := $(filter-out ./sdk/metric,$(OTEL_GO_MOD_DIRS)) ./sdk/metric/. ./sdk/metric/internal +benchmark/./sdk/metric/.: cd ./sdk/metric && $(GO) test -run='^$$' -bench=. $(ARGS) . ./exemplar/... -benchmark/./sdk/metric/internal: - cd ./sdk/metric && $(GO) test -run='^$$' -bench=. $(ARGS) ./internal/... print-sharded-benchmarks: @echo $(BENCHMARK_SHARDS) | jq -cR 'split(" ")' +# Print the JSON list of benchmark shards whose code changed since +# BASE_REF (default: main). Filters print-sharded-benchmarks +# down to shards whose underlying module contains a changed *.go file. +# Non-Go changes (docs, workflows, Makefile, go.mod/go.sum, tooling) emit []. +# Override the diff base via BASE_REF, or pass ARGS=-all to emit the full list. +BASE_REF ?= main +.PHONY: print-affected-benchmarks +print-affected-benchmarks: + @$(MAKE) -s print-sharded-benchmarks | $(GO) -C $(TOOLS_MOD_DIR) run ./affectedmods $(if $(strip $(BASE_REF)),-base=$(BASE_REF),) $(ARGS) + .PHONY: golangci-lint golangci-lint-fix golangci-lint-fix: ARGS=--fix golangci-lint-fix: golangci-lint diff --git a/vendor/go.opentelemetry.io/otel/README.md b/vendor/go.opentelemetry.io/otel/README.md index ce97e9a251..9291d1470d 100644 --- a/vendor/go.opentelemetry.io/otel/README.md +++ b/vendor/go.opentelemetry.io/otel/README.md @@ -53,18 +53,25 @@ Currently, this project supports the following environments. | OS | Go Version | Architecture | |----------|------------|--------------| +| Ubuntu | 1.27 | amd64 | | Ubuntu | 1.26 | amd64 | | Ubuntu | 1.25 | amd64 | +| Ubuntu | 1.27 | 386 | | Ubuntu | 1.26 | 386 | | Ubuntu | 1.25 | 386 | +| Ubuntu | 1.27 | arm64 | | Ubuntu | 1.26 | arm64 | | Ubuntu | 1.25 | arm64 | +| macOS | 1.27 | amd64 | | macOS | 1.26 | amd64 | | macOS | 1.25 | amd64 | +| macOS | 1.27 | arm64 | | macOS | 1.26 | arm64 | | macOS | 1.25 | arm64 | +| Windows | 1.27 | amd64 | | Windows | 1.26 | amd64 | | Windows | 1.25 | amd64 | +| Windows | 1.27 | 386 | | Windows | 1.26 | 386 | | Windows | 1.25 | 386 | diff --git a/vendor/go.opentelemetry.io/otel/VERSIONING.md b/vendor/go.opentelemetry.io/otel/VERSIONING.md index b27c9e84f5..56eaa7d220 100644 --- a/vendor/go.opentelemetry.io/otel/VERSIONING.md +++ b/vendor/go.opentelemetry.io/otel/VERSIONING.md @@ -12,6 +12,12 @@ is designed so the following goals can be achieved. * [Semantic import versioning](https://github.com/golang/go/wiki/Modules#semantic-import-versioning) will be used. + * Stable module compatibility is understood in terms of the [Go 1 + compatibility guidelines](https://go.dev/doc/go1compat). Code that + compiled against an older version of a package should continue to compile + against newer versions of that package, subject to the exceptions in the + Go 1 compatibility guidelines and any additional exceptions documented + below. * Versions will comply with [semver 2.0](https://semver.org/spec/v2.0.0.html) with the following exceptions. * New methods may be added to exported API interfaces. All exported diff --git a/vendor/go.opentelemetry.io/otel/attribute/hash.go b/vendor/go.opentelemetry.io/otel/attribute/hash.go index f651eb13d9..54c26e67a0 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/hash.go +++ b/vendor/go.opentelemetry.io/otel/attribute/hash.go @@ -33,21 +33,73 @@ const ( emptyID uint64 = 7305809155345288421 // "__empty_" (little endian) ) -// hashKVs returns a new xxHash64 hash of kvs. -func hashKVs(kvs []KeyValue) uint64 { - h := xxhash.New() - for _, kv := range kvs { - h = hashKV(h, kv) - } - sum := h.Sum64() - // Remap 0 to a non-zero value for non-empty input because hash == 0 is a reserved sentinel (treated as empty/invalid). - const remappedZeroHash uint64 = 1 - if sum == 0 && len(kvs) > 0 { - return remappedZeroHash +// Hasher computes a Distinct value from KeyValue attributes supplied with +// Write. +// +// A Hasher must be obtained from [NewHasher]. The zero value is not usable and +// its methods will panic. +type Hasher struct { + h xxhash.Hash +} + +// NewHasher returns a new Hasher. +func NewHasher() *Hasher { + return &Hasher{h: xxhash.New()} +} + +// Reset resets h to its initial state so it can be reused. +func (h *Hasher) Reset() { + h.h.Reset() +} + +// Write adds kv to the hash. +// +// Write requires attributes to be supplied in ascending key order with no +// duplicate keys. To produce the same Distinct as Set.Equivalent, write +// attributes in ascending key order, with no more than one value for each key. +// If the source contains duplicate keys, retain the last value for each key +// before calling Write. +func (h *Hasher) Write(kv KeyValue) { + // hashKV mutates the digest h.h refers to in place and returns the same + // Hash value it was passed. Discarding the result keeps the digest pointer + // from flowing back into h, which would force the digest to be heap + // allocated for every Hasher. Keeping Write this small also keeps it within + // the inlining budget, which matters because hashKVs calls it per attribute. + _ = hashKV(h.h, kv) +} + +// Distinct returns the identifier for the attributes written to h. When Write +// is called as described above, it returns the same value as [Set.Equivalent]. +func (h *Hasher) Distinct() Distinct { + // No count of written attributes is needed to detect the empty case. The + // sum of a digest with nothing written to it is emptyHash, which is + // non-zero (0xef46db3751d8e999), so it passes through remapZeroHash + // unchanged and matches emptySet.Equivalent. + return Distinct{hash: remapZeroHash(h.h.Sum64())} +} + +// remapZeroHash remaps a 0 sum to a non-zero value, because hash == 0 is a +// reserved sentinel (treated as empty/invalid). +func remapZeroHash(sum uint64) uint64 { + if sum == 0 { + return 1 } return sum } +// hashKVs returns a new xxHash64 hash of kvs. +// +// This routes through [Hasher] so that Set hashing and Hasher cannot disagree: +// there is exactly one implementation of how attributes are mixed and how the +// final sum is framed. +func hashKVs(kvs []KeyValue) uint64 { + h := NewHasher() + for _, kv := range kvs { + h.Write(kv) + } + return h.Distinct().hash +} + // hashKV returns the xxHash64 hash of kv with h as the base. func hashKV(h xxhash.Hash, kv KeyValue) xxhash.Hash { h = h.String(string(kv.Key)) diff --git a/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go b/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go index c851179cca..4fcdd2dbea 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go +++ b/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go @@ -62,3 +62,8 @@ func (h Hash) String(val string) Hash { func (h Hash) Sum64() uint64 { return h.d.Sum64() } + +// Reset resets the hash to its initial state. +func (h Hash) Reset() { + h.d.Reset() +} diff --git a/vendor/go.opentelemetry.io/otel/attribute/set.go b/vendor/go.opentelemetry.io/otel/attribute/set.go index 87d6f96200..56a70a9f6d 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/set.go +++ b/vendor/go.opentelemetry.io/otel/attribute/set.go @@ -313,6 +313,9 @@ func filteredToFront(slice []KeyValue, keep Filter) int { // Filter returns a filtered copy of this Set. See the documentation for // NewSetWithSortableFiltered for more details. func (l *Set) Filter(re Filter) (Set, []KeyValue) { + if l == nil { + return emptySet, nil + } if re == nil { return *l, nil } diff --git a/vendor/go.opentelemetry.io/otel/renovate.json b/vendor/go.opentelemetry.io/otel/renovate.json index fa5acf2d3b..5008ab2bf1 100644 --- a/vendor/go.opentelemetry.io/otel/renovate.json +++ b/vendor/go.opentelemetry.io/otel/renovate.json @@ -15,6 +15,13 @@ "matchDepTypes": ["indirect"], "enabled": true }, + { + "description": "Disable Go module major updates to v2+", + "matchManagers": ["gomod"], + "matchUpdateTypes": ["major"], + "matchNewValue": "/^v?([2-9]|[1-9][0-9]+)\\./", + "enabled": false + }, { "matchPackageNames": ["go.opentelemetry.io/build-tools/**"], "groupName": "build-tools" diff --git a/vendor/go.opentelemetry.io/otel/version.go b/vendor/go.opentelemetry.io/otel/version.go index ea79030969..f599bf285d 100644 --- a/vendor/go.opentelemetry.io/otel/version.go +++ b/vendor/go.opentelemetry.io/otel/version.go @@ -5,5 +5,5 @@ package otel // Version is the current release version of OpenTelemetry in use. func Version() string { - return "1.45.0" + return "1.46.0" } diff --git a/vendor/go.opentelemetry.io/otel/versions.yaml b/vendor/go.opentelemetry.io/otel/versions.yaml index 3b1ad7301a..f329d171d9 100644 --- a/vendor/go.opentelemetry.io/otel/versions.yaml +++ b/vendor/go.opentelemetry.io/otel/versions.yaml @@ -3,7 +3,7 @@ module-sets: stable-v1: - version: v1.45.0 + version: v1.46.0 modules: - go.opentelemetry.io/otel - go.opentelemetry.io/otel/bridge/opencensus @@ -22,12 +22,12 @@ module-sets: - go.opentelemetry.io/otel/sdk/metric - go.opentelemetry.io/otel/trace experimental-metrics: - version: v0.67.0 + version: v0.68.0 modules: - go.opentelemetry.io/otel/exporters/prometheus - go.opentelemetry.io/otel/metric/x experimental-logs: - version: v0.21.0 + version: v0.22.0 modules: - go.opentelemetry.io/otel/log - go.opentelemetry.io/otel/log/logtest @@ -37,7 +37,7 @@ module-sets: - go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp - go.opentelemetry.io/otel/exporters/stdout/stdoutlog experimental-schema: - version: v0.0.18 + version: v0.0.19 modules: - go.opentelemetry.io/otel/schema excluded-modules: diff --git a/vendor/modules.txt b/vendor/modules.txt index 8079ad739c..2eff9ed031 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -2323,7 +2323,7 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/internal/semconv ## explicit; go 1.25.0 go.opentelemetry.io/contrib/zpages go.opentelemetry.io/contrib/zpages/internal -# go.opentelemetry.io/otel v1.45.0 +# go.opentelemetry.io/otel v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel go.opentelemetry.io/otel/attribute @@ -2366,7 +2366,7 @@ go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/counter go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/observ go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/x -# go.opentelemetry.io/otel/metric v1.45.0 +# go.opentelemetry.io/otel/metric v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel/metric go.opentelemetry.io/otel/metric/embedded @@ -2382,7 +2382,7 @@ go.opentelemetry.io/otel/sdk/trace go.opentelemetry.io/otel/sdk/trace/internal/env go.opentelemetry.io/otel/sdk/trace/internal/observ go.opentelemetry.io/otel/sdk/trace/tracetest -# go.opentelemetry.io/otel/trace v1.45.0 +# go.opentelemetry.io/otel/trace v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel/trace go.opentelemetry.io/otel/trace/embedded From 65ea519099ee724c2118e02c14ef9a4f3514cd27 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 21:16:43 +0200 Subject: [PATCH 08/13] docs(adr): update to the implemented state --- docs/adr/0005-unified-search-index-mapping.md | 66 +++++-------------- 1 file changed, 16 insertions(+), 50 deletions(-) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index 3e3024f01f..449f52ae0d 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -2,14 +2,11 @@ title: "5. Unified Search Index Mapping" --- -* Status: proposed -* Deciders: [] -* Date: 2026-04-23 +* Status: accepted +* Deciders: @aduffeck, @butonic, @dschmidt, @fschade +* Date: 2026-04-23, accepted and updated to the implemented state 2026-08-31 -Reference: https://github.com/opencloud-eu/opencloud/pull/2659, a -proof-of-concept that demonstrates the proposal end-to-end. It is -not presumed accepted by this ADR; scope and APIs there will be -revisited once the decision here lands. +Reference: implemented by https://github.com/opencloud-eu/opencloud/pull/3345 (reflection-based mapping, search siblings, shared query lowering) and https://github.com/opencloud-eu/opencloud/pull/3197 (schema versioning and startup checks). https://github.com/opencloud-eu/opencloud/pull/2659 was the original proof-of-concept. ## Context and Problem Statement @@ -157,9 +154,9 @@ construction, because there is no second place to edit. The overrides surface stays small. Each entry declares one of a handful of things per field: a semantic type for fields whose intent cannot be inferred from the Go type (for example a path- -analyzed field, a fulltext field, a wildcard field, a geopoint -field), an analyzer name where one is needed, or a backend- -specific inclusion knob. Any field that needs something beyond the +analyzed field, a fulltext field, a geopoint field), or search- +behavior flags (case-insensitivity, word breaking, inclusion in +the catch-all field). Any field that needs something beyond the inferred defaults gets one line in the overrides map and that one line flows through every derived piece. Overrides are validated at startup so a typo fails loudly instead of silently disabling a @@ -177,11 +174,7 @@ through coordinated per-site edits. ### Facet values are indexed as case-preserving keywords -All facet sub-fields, meaning any leaf inside `audio`, `photo`, -`image`, `location`, are indexed as plain keywords on both -backends. No tokenization, no case folding. The raw value the -extractor saw, or the CS3 ArbitraryMetadata string, is what lands -in the index. +All facet sub-fields, meaning any leaf inside `audio`, `photo`, `image`, `location` and the facets that followed (`video`, `motionPhoto`, `livePhoto`), keep a case-preserving keyword as their stored base field on both backends. The raw value the extractor saw, or the CS3 ArbitraryMetadata string, is what lands in the index, and it is what returning, sorting and aggregations read. This is the single intended semantic for facets across bleve and OpenSearch, and it is driven by what aggregations need. @@ -197,39 +190,13 @@ single bucket labelled `motörhead`. For a metadata display use case (thumbnails, facet filters in the UI, distinct lists) that behavior is not what we want. -Keyword-indexed values preserve case and produce correctly- -labelled buckets on both backends. If, in the future, case- -insensitive search on a facet property is also wanted alongside -the case-preserving bucket view, that is a strict superset and -can be added on top. It is much easier to bolt onto one -consistent backend-agnostic state than to retrofit it onto the -current divergent, implicit per-backend behavior. +Searching is layered on top as exactly the strict superset the proposal reserved for later, and it shipped with the implementation: every keyword field additionally gets search-only sibling fields derived from the same definition, a `_lowercase` keyword sibling (doc values disabled; serves wildcards and `=` whole-value matches) and a `_words` text sibling (`words` analyzer: dots to spaces, unicode tokenization, lowercasing, no stemming; serves token and phrase matches). Case-insensitive, word-broken search is the default for every keyword field including facets; fields opt out per override where that is wrong: opaque ids (`ID`, `RootID`, `ParentID`, `Favorites`, `livePhoto.contentId`), the POSIX `Path`, the normalized `MimeType`, and `Content`, which is a fulltext field of its own. Aggregation buckets keep their display casing because they read the base field, never the siblings. -The query side lines up with this: the KQL parser preserves the -user's casing, and the per-backend compiler decides per field -whether to fold that value to lowercase before handing it to the -backend, based on the same overrides map the index mapping uses. -Fields whose override selects a lowercasing analyzer (Name, Tags, -Favorites, Content) get their query value folded so user input -matches the tokens the index produced; everything else, including -all facet sub-fields, compares case-sensitive on both sides. The -case-sensitivity alignment of the compiler with the index mapping -was started in #2633; this ADR completes it by deriving both -sides from the same source. +The query side derives from the same source: the shared lowering pass resolves field names case-insensitively, folds values and routes each match to the right sibling (wildcards to `_lowercase`, tokens and phrases to `_words`, `=` as a whole-value term on `_lowercase`), and both backend compilers consume that one decision. The engine parity suite pins the resulting behavior against bleve and OpenSearch, so a divergence fails CI instead of surfacing in production. The case-sensitivity alignment started in #2633 is completed by deriving both sides from the same source. -### Behavior changes, applied to newly-created indexes +### Schema versioning and upgrades -Existing indexes keep their stored mapping; the new shape applies -only to indexes created after the proposal lands. - -- OpenSearch `Tags` and `Favorites` change from dynamic `keyword` - to `text + lowercaseKeyword`, matching the bleve mapping. The - analyzer is registered in the new index settings. -- OpenSearch facet sub-strings (`audio.*`, `photo.*`, `image.*`, - `location.*`) change from the dynamic `text + keyword` multi-field to - `keyword`-only, implementing the principle above. As noted in - the context section, the tokenized leg is not actually reachable - from user queries today, so no working path regresses. +Index names carry a schema version derived from the single `search.SchemaVersion` constant (`opencloud-resource-v4`, `bleve-v4`). On startup the service classifies the stored mapping against the code: additive changes (new fields, unchanged analyzers) are reconciled in place without a version bump, breaking changes make the service refuse to start and name the reindex steps. The upgrade path is a plain reindex (`opencloud search index --all-spaces`) into the new versioned index; older indexes stay untouched and can be deleted afterwards (services/search/MIGRATION.md). Golden mapping tests on both backends pin the rendered mappings and reuse the same classifier to tell a contributor whether a change needs only a golden regeneration or a version bump. ### Known trade-off @@ -250,11 +217,10 @@ any call site. endpoint renders none of the facet fields back to the client. This is a missing feature, not a regression of the proposal; its natural resolution is to let the graph-search endpoint - (not yet in tree) take over once graph search lands. -- **Graph search hit conversion.** Once graph search is wired up, - it needs to translate proto hits back into libregraph DriveItems. - The same facet-copy helper used internally on the search service - side is reusable. + (proposed in #3211) take over once graph search lands. +- **Graph search hit conversion.** Graph search (#3211) translates + proto hits back into libregraph DriveItems with the same + facet-copy helper the search service uses internally. - **reva's PROPFIND facet listing** uses its own hand-maintained per-facet key lists. reva deliberately does not depend on the libregraph Go types, so unifying those key sets is a reva-side From c6ddc13055672655db8df6688cb52a634d200ade Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 21:20:54 +0200 Subject: [PATCH 09/13] docs(adr): date the context section --- docs/adr/0005-unified-search-index-mapping.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index 449f52ae0d..4921312119 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -10,6 +10,8 @@ Reference: implemented by https://github.com/opencloud-eu/opencloud/pull/3345 (r ## Context and Problem Statement +This section describes the state at decision time (April 2026); the implementation has since resolved the problems listed here. + The search service supports two backends, bleve (embedded) and OpenSearch (external). Each backend currently carries its own, independently maintained description of the index layout: From b149316349698631548235d300645fdddc27710d Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Mon, 31 Aug 2026 21:22:22 +0200 Subject: [PATCH 10/13] docs(adr): keep the facet reachability gap out of the problem list --- docs/adr/0005-unified-search-index-mapping.md | 32 ++++++++----------- 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index 4921312119..b0822a9bad 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -33,7 +33,7 @@ independently maintained description of the index layout: whose query values need to be pre-lowercased, with a comment that literally says "Keep in sync with index.go". -The current implementation has four concrete problems: +The current implementation has three concrete problems: 1. **The two backends do not behave the same.** Both rely on their own implicit defaults for fields that are not explicitly @@ -50,21 +50,15 @@ The current implementation has four concrete problems: wildcard match only); OpenSearch word-tokenizes, so a bare `name:report` matches "My Report.txt" on OpenSearch but not on bleve. -2. **The indexed facet metadata is effectively unreachable.** The - audio, image, photo and location sub-fields go into the index and - take up space, but no caller of the search service can actually - use them. The KQL compiler does not support the dot syntax needed - to refer to `audio.artist` on the OpenSearch side, and until - #2633 the bleve-side compiler unconditionally lowercased query - values even against case-preserving fields. The hit and REPORT - paths do not expose facet values back to consumers either. -3. **Drift risk.** The OpenSearch JSON template is a subset of what +2. **Drift risk.** The OpenSearch JSON template is a subset of what actually gets indexed. Even where it overlaps with the bleve - mapping it diverges on analyzer choices. Because point 2 means - nothing queries the diverging fields, the divergence has been - invisible, but it would surface the moment the first working - cross-backend facet query landed. -4. **Per-facet cost.** Adding a new facet (motionPhoto, etc.) + mapping it diverges on analyzer choices. Because the facet + fields were not reachable from user queries at the time (no dot + syntax in the KQL compilers, no facet exposure on the hit and + REPORT paths), the divergence has been invisible, but it would + surface the moment the first working cross-backend facet query + landed. +3. **Per-facet cost.** Adding a new facet (motionPhoto, etc.) requires coordinated edits across the proto message, both backend mappings, the bleve hit converters, the OpenSearch convert closures, the search service's metadata persistence, the graph @@ -76,10 +70,10 @@ The current implementation has four concrete problems: ### A note on backwards compatibility -The unreachability above (problem 2) has a useful corollary for -this ADR: **changing the indexed shape of the facet fields cannot -break any existing client of the search service**, because no -client successfully reads them today. The behavior changes +That the facet fields were unreachable at decision time has a +useful corollary for this ADR: **changing the indexed shape of the +facet fields cannot break any existing client of the search +service**, because no client successfully read them. The behavior changes discussed below are therefore additive in a literal sense; nothing that works today stops working as a result. From 7bf64e329cc7b22affb16d641976509d9d0ef540 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Tue, 1 Sep 2026 07:26:32 +0200 Subject: [PATCH 11/13] docs(adr): apply review wording suggestion --- docs/adr/0005-unified-search-index-mapping.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md index b0822a9bad..a98d52b301 100644 --- a/docs/adr/0005-unified-search-index-mapping.md +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -73,7 +73,7 @@ The current implementation has three concrete problems: That the facet fields were unreachable at decision time has a useful corollary for this ADR: **changing the indexed shape of the facet fields cannot break any existing client of the search -service**, because no client successfully read them. The behavior changes +service**, because no client could successfully read them. The behavior changes discussed below are therefore additive in a literal sense; nothing that works today stops working as a result. From ef41429838cc7b160f1baa1ea41b836e0eee6f08 Mon Sep 17 00:00:00 2001 From: Dominik Schmidt Date: Tue, 1 Sep 2026 07:35:15 +0200 Subject: [PATCH 12/13] test(search): rename the update flag to UPDATE_SEARCH_PARITY_MATRIX --- services/search/pkg/parity/README.md | 2 +- services/search/pkg/parity/matrix_test.go | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/services/search/pkg/parity/README.md b/services/search/pkg/parity/README.md index fb86424d73..05845cbc92 100644 --- a/services/search/pkg/parity/README.md +++ b/services/search/pkg/parity/README.md @@ -1,6 +1,6 @@ # Engine parity -Written by the parity suite (`UPDATE_MATRIX=true go test ./services/search/pkg/parity/`), do not edit. +Written by the parity suite (`UPDATE_SEARCH_PARITY_MATRIX=true go test ./services/search/pkg/parity/`), do not edit. Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as expected, `❌ known` when an engine's divergence is documented in the case (`engineOverrides`), `❌` when it is not. `✅ stale` when every engine diff --git a/services/search/pkg/parity/matrix_test.go b/services/search/pkg/parity/matrix_test.go index 6347ae6571..a1d51394d7 100644 --- a/services/search/pkg/parity/matrix_test.go +++ b/services/search/pkg/parity/matrix_test.go @@ -171,7 +171,7 @@ func writeMatrix(report types.Report) { out := &strings.Builder{} out.WriteString("# Engine parity\n\n") - out.WriteString("Written by the parity suite (`UPDATE_MATRIX=true go test ./services/search/pkg/parity/`), do not edit.\n") + out.WriteString("Written by the parity suite (`UPDATE_SEARCH_PARITY_MATRIX=true go test ./services/search/pkg/parity/`), do not edit.\n") out.WriteString("Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as\n") out.WriteString("expected, `❌ known` when an engine's divergence is documented in the case\n") out.WriteString("(`engineOverrides`), `❌` when it is not. `✅ stale` when every engine\n") @@ -226,7 +226,7 @@ func writeMatrix(report types.Report) { matrixNames(row.answered["bleve"]), matrixNames(row.answered["opensearch"]), matrixVerdict(row)) } - if os.Getenv("UPDATE_MATRIX") != "" { + if os.Getenv("UPDATE_SEARCH_PARITY_MATRIX") != "" { if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) } @@ -236,7 +236,7 @@ func writeMatrix(report types.Report) { // the committed matrix must match what the suite answers previous, _ := os.ReadFile(matrixFile) if string(previous) != out.String() { - Fail(matrixFile + " is out of date: regenerate it with UPDATE_MATRIX=true and commit the change") + Fail(matrixFile + " is out of date: regenerate it with UPDATE_SEARCH_PARITY_MATRIX=true and commit the change") } } From 9e8403818c6d8ee8b419b83e0f5b4a351bed7dbd Mon Sep 17 00:00:00 2001 From: Florian Schade Date: Tue, 1 Sep 2026 06:47:27 +0200 Subject: [PATCH 13/13] test(search): cover chinese search in the parity suite --- services/search/pkg/parity/README.md | 44 +++++++++++++++++ services/search/pkg/parity/parity_test.go | 1 + services/search/pkg/parity/query_cjk_test.go | 51 ++++++++++++++++++++ 3 files changed, 96 insertions(+) create mode 100644 services/search/pkg/parity/query_cjk_test.go diff --git a/services/search/pkg/parity/README.md b/services/search/pkg/parity/README.md index 67c02ab784..ed7dc08df2 100644 --- a/services/search/pkg/parity/README.md +++ b/services/search/pkg/parity/README.md @@ -149,6 +149,50 @@ Fixtures: | CONTENT-09 | `Content:"alan@example.org"` | links.txt | links.txt | links.txt | ✅ | | CONTENT-10 | `Content:opencloud` | links.txt | links.txt | links.txt | ✅ | +### cjk + +Fixtures: + +- `报告.txt`, Content = "这是一个关于年度销售的中文文档" +- `说明书.pdf`, MimeType = application/pdf, Content = "产品说明与安装步骤" +- `手册.txt`, Content = "学生手册" +- `图片`, folder +- `english.txt`, Content = "annual sales report" + +| Case | Query | expected | bleve | OpenSearch | same? | +|---|---|---|---|---|---| +| CJK-01 | `Content:中文` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-02 | `Content:中文文档` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-03 | `Content:"中文文档"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-04 | `Content:销售` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-05 | `Content:说明` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-06 | `name:报告` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-07 | `name:"*报告*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-08 | `name:"报告.txt"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-09 | `报告` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-10 | `说明书` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-11 | `name:图片` | 图片 | 图片 | 图片 | ✅ | +| CJK-12 | `name:"*图片*"` | 图片 | 图片 | 图片 | ✅ | +| CJK-13 | `图片` | 图片 | 图片 | 图片 | ✅ | +| CJK-14 | `Content:学生` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-15 | `Content:手册` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-16 | `Content:生手` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-17 | `name:"报*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-18 | `name:"*告*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-19 | `name:"*册*"` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-20 | `name:"图?"` | 图片 | 图片 | 图片 | ✅ | +| CJK-21 | `name:"报?.txt"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-22 | `name:"*报?*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-23 | `Content:中文*` | no match | no match | no match | ✅ | +| CJK-24 | `Content:*销售*` | no match | no match | no match | ✅ | +| CJK-25 | `Content:*文档` | no match | no match | no match | ✅ | +| CJK-26 | `Content:文*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-27 | `Content:*文*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-28 | `Content:*文` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-29 | `Content:销*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-30 | `Content:说*` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-31 | `Content:*明` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | + ### favorites Fixtures: diff --git a/services/search/pkg/parity/parity_test.go b/services/search/pkg/parity/parity_test.go index 3bacdd762b..41695d18b0 100644 --- a/services/search/pkg/parity/parity_test.go +++ b/services/search/pkg/parity/parity_test.go @@ -97,6 +97,7 @@ func queryGroups() []queryGroup { tagsGroup(), titleGroup(), contentGroup(), + cjkGroup(), favoritesGroup(), mediatypeGroup(), pathGroup(), diff --git a/services/search/pkg/parity/query_cjk_test.go b/services/search/pkg/parity/query_cjk_test.go new file mode 100644 index 0000000000..f855c06b28 --- /dev/null +++ b/services/search/pkg/parity/query_cjk_test.go @@ -0,0 +1,51 @@ +package parity + +import ( + "github.com/opencloud-eu/opencloud/services/search/pkg/search" +) + +func cjkGroup() queryGroup { + return queryGroup{ + name: "cjk", + fixtures: []search.Resource{ + fixtureDoc("报告.txt", withContent("这是一个关于年度销售的中文文档")), + fixtureDoc("说明书.pdf", withContent("产品说明与安装步骤"), withMime("application/pdf")), + fixtureDoc("手册.txt", withContent("学生手册")), + fixtureFolder("图片"), + fixtureDoc("english.txt", withContent("annual sales report")), + }, + cases: []queryCase{ + {id: 1, query: `Content:中文`, want: []string{"报告.txt"}}, + {id: 2, query: `Content:中文文档`, want: []string{"报告.txt"}}, + {id: 3, query: `Content:"中文文档"`, want: []string{"报告.txt"}}, + {id: 4, query: `Content:销售`, want: []string{"报告.txt"}}, + {id: 5, query: `Content:说明`, want: []string{"说明书.pdf"}}, + {id: 6, query: `name:报告`, want: []string{"报告.txt"}}, + {id: 7, query: `name:"*报告*"`, want: []string{"报告.txt"}}, + {id: 8, query: `name:"报告.txt"`, want: []string{"报告.txt"}}, + {id: 9, query: `报告`, want: []string{"报告.txt"}}, + {id: 10, query: `说明书`, want: []string{"说明书.pdf"}}, + {id: 11, query: `name:图片`, want: []string{"图片"}}, + {id: 12, query: `name:"*图片*"`, want: []string{"图片"}}, + {id: 13, query: `图片`, want: []string{"图片"}}, + {id: 14, query: `Content:学生`, want: []string{"手册.txt"}}, + {id: 15, query: `Content:手册`, want: []string{"手册.txt"}}, + {id: 16, query: `Content:生手`, want: []string{"手册.txt"}}, + {id: 17, query: `name:"报*"`, want: []string{"报告.txt"}}, + {id: 18, query: `name:"*告*"`, want: []string{"报告.txt"}}, + {id: 19, query: `name:"*册*"`, want: []string{"手册.txt"}}, + {id: 20, query: `name:"图?"`, want: []string{"图片"}}, + {id: 21, query: `name:"报?.txt"`, want: []string{"报告.txt"}}, + {id: 22, query: `name:"*报?*"`, want: []string{"报告.txt"}}, + {id: 23, query: `Content:中文*`}, + {id: 24, query: `Content:*销售*`}, + {id: 25, query: `Content:*文档`}, + {id: 26, query: `Content:文*`, want: []string{"报告.txt"}}, + {id: 27, query: `Content:*文*`, want: []string{"报告.txt"}}, + {id: 28, query: `Content:*文`, want: []string{"报告.txt"}}, + {id: 29, query: `Content:销*`, want: []string{"报告.txt"}}, + {id: 30, query: `Content:说*`, want: []string{"说明书.pdf"}}, + {id: 31, query: `Content:*明`, want: []string{"说明书.pdf"}}, + }, + } +}