diff --git a/docs/adr/0005-unified-search-index-mapping.md b/docs/adr/0005-unified-search-index-mapping.md new file mode 100644 index 0000000000..a98d52b301 --- /dev/null +++ b/docs/adr/0005-unified-search-index-mapping.md @@ -0,0 +1,226 @@ +--- +title: "5. Unified Search Index Mapping" +--- + +* Status: accepted +* Deciders: @aduffeck, @butonic, @dschmidt, @fschade +* Date: 2026-04-23, accepted and updated to the implemented state 2026-08-31 + +Reference: implemented by https://github.com/opencloud-eu/opencloud/pull/3345 (reflection-based mapping, search siblings, shared query lowering) and https://github.com/opencloud-eu/opencloud/pull/3197 (schema versioning and startup checks). https://github.com/opencloud-eu/opencloud/pull/2659 was the original proof-of-concept. + +## Context and Problem Statement + +This section describes the state at decision time (April 2026); the implementation has since resolved the problems listed here. + +The search service supports two backends, bleve (embedded) and +OpenSearch (external). Each backend currently carries its own, +independently maintained description of the index layout: + +- The bleve backend hand-builds a document mapping that explicitly + declares only Name, Tags, Favorites and Content. Everything else, + including the entire facet block (audio, image, photo, location), + is left to bleve's dynamic mapping. +- The OpenSearch backend ships a static JSON template that covers a + similar but not identical subset, plus a few OpenSearch-specific + primitives (path_hierarchy analyzer, wildcard MimeType). It does + not list the facet sub-fields either; they are produced by + OpenSearch's dynamic templating at first write. +- The graph DriveItem assembly path keeps its own private copy of a + reflection-based walker to turn CS3 ArbitraryMetadata back into + typed libregraph facets, parallel to the search service's + reflection helpers but maintained separately. +- The bleve KQL compiler keeps a hand-maintained set of field names + whose query values need to be pre-lowercased, with a comment that + literally says "Keep in sync with index.go". + +The current implementation has three concrete problems: + +1. **The two backends do not behave the same.** Both rely on their + own implicit defaults for fields that are not explicitly + declared. The inferred shapes differ: bleve produces keyword- + analyzed text, OpenSearch produces a `text + keyword` multi-field + with auto-detected dates. Nobody has written down which behavior + is the intended one. Two concrete instances surfaced while building + #2659: + - **mtime** is stored as an RFC3339 string. OpenSearch's dynamic + mapping auto-detects it as `date`; bleve leaves it `keyword`. So + `mtime:>...` is a chronological range on OpenSearch but a + lexicographic string compare on bleve. + - **name/tags**: bleve indexes a single lowercase token (exact or + wildcard match only); OpenSearch word-tokenizes, so a bare + `name:report` matches "My Report.txt" on OpenSearch but not on + bleve. +2. **Drift risk.** The OpenSearch JSON template is a subset of what + actually gets indexed. Even where it overlaps with the bleve + mapping it diverges on analyzer choices. Because the facet + fields were not reachable from user queries at the time (no dot + syntax in the KQL compilers, no facet exposure on the hit and + REPORT paths), the divergence has been invisible, but it would + surface the moment the first working cross-backend facet query + landed. +3. **Per-facet cost.** Adding a new facet (motionPhoto, etc.) + requires coordinated edits across the proto message, both backend + mappings, the bleve hit converters, the OpenSearch convert + closures, the search service's metadata persistence, the graph + DriveItem assembly, and the KQL compiler's lowercasing set. Most + of those edits are boilerplate following a copy-paste pattern. + Adding a genuinely new index capability (geopoint, wildcard, + ...) means wiring it in at every one of those sites, and there + is no single place to hook a type-specific adapter. + +### A note on backwards compatibility + +That the facet fields were unreachable at decision time has a +useful corollary for this ADR: **changing the indexed shape of the +facet fields cannot break any existing client of the search +service**, because no client could successfully read them. The behavior changes +discussed below are therefore additive in a literal sense; nothing +that works today stops working as a result. + +## Decision Drivers + +* **Predictable OpenCloud API behavior independent of backend.** + Consumers of the search service should be able to rely on the + documented behavior of the API, not on which backend happens to + be configured. Today the same query can give different results + depending on whether bleve or OpenSearch is wired in (bleve's + dynamic default is `keyword`, exact match; OpenSearch's dynamic + default is `text + keyword`, also matches sub-tokens of a + string). That is backend-implementation leakage, and trying to + keep the two implicit defaults synchronized has not worked. +* Single source of truth for the indexed schema, so the two backends + cannot drift silently again. +* Reduce the per-facet cost so future facets (motionPhoto and + whatever comes next) can be added with minimal boilerplate. +* Establish a single place to hook index-type-specific behavior, so + a new capability needs to be implemented at most once per backend + and then becomes available for any field uniformly. +* A one-time reindex is an acceptable upgrade path. Both bleve and + OpenSearch store their mapping alongside the data; existing + indexes keep serving queries against their stored shape without + any automatic reshaping. Benefiting from the new behavior is done + by creating a fresh index and re-ingesting, which is the normal + reindex flow, rather than by inventing migration tooling. + +## Considered Options + +### Option 1: Do nothing, keep relying on implicit backend defaults + +Accept that bleve and OpenSearch each fall back to their own +dynamic-mapping defaults for whatever is not explicitly declared, +and treat the observable search behavior of OpenCloud as "whatever +the configured backend happens to do". Adding a facet stays a +copy-paste coordination across half a dozen sites; the existing +divergence between bleve (keyword) and OpenSearch (`text + keyword` +multi-field plus auto-date detection) stays silently in place +until a working query actually reaches the diverging field and +returns different answers on the two backends. + +Low upfront work, but it makes the OpenCloud API behavior a +function of the backend rather than a contract, and it keeps the +per-facet boilerplate cost for every new field. + +### Option 2: Generate one backend's mapping from the other + +Treat one backend as canonical (likely bleve, because Go types) and +derive the other. Partial answer; it still does not help the reader +path or the graph walker, and still leaves per-facet boilerplate in +non-mapping code. + +### Option 3: A struct-driven mapping (chosen) + +Let the Go struct that represents an indexed document, together +with a small overrides map, be the single source of truth. A +reflection-based helper walks the struct via json tags and emits +each backend's index mapping. The same definition drives the +write-time path, the hit-decoding path, and the query compiler's +case-folding rules. Any future field follows one declaration in +one place and falls through the whole pipeline consistently. + +## Decision Outcome + +Adopt Option 3. The Go struct that represents an indexed document, +together with a small overrides map, becomes the single source of +truth for the search index. The bleve and OpenSearch index +mappings, the write-time conversion, the hit-decoding path, and +the query compiler's case-folding rules are all derived from that +same definition. Drift between backends is prevented by +construction, because there is no second place to edit. + +The overrides surface stays small. Each entry declares one of a +handful of things per field: a semantic type for fields whose +intent cannot be inferred from the Go type (for example a path- +analyzed field, a fulltext field, a geopoint field), or search- +behavior flags (case-insensitivity, word breaking, inclusion in +the catch-all field). Any field that needs something beyond the +inferred defaults gets one line in the overrides map and that one +line flows through every derived piece. Overrides are validated at +startup so a typo fails loudly instead of silently disabling a +setting. + +A practical consequence of having one place to hook things: when a +new capability is needed (a geopoint representation, a sibling +field for a different aggregation behavior, a different analyzer +for a class of fields, ...) it can be implemented once per backend +in the central pipeline. After that, turning the capability on for +a specific field is a single override entry, and both backends +adopt it the same way. This ADR does not decide which capabilities +to add, only that they will land in this uniform shape rather than +through coordinated per-site edits. + +### Facet values are indexed as case-preserving keywords + +All facet sub-fields, meaning any leaf inside `audio`, `photo`, `image`, `location` and the facets that followed (`video`, `motionPhoto`, `livePhoto`), keep a case-preserving keyword as their stored base field on both backends. The raw value the extractor saw, or the CS3 ArbitraryMetadata string, is what lands in the index, and it is what returning, sorting and aggregations read. + +This is the single intended semantic for facets across bleve and +OpenSearch, and it is driven by what aggregations need. +Aggregation buckets ("group all files by `audio.artist`", "list +distinct `photo.cameraMake`") return bucket keys drawn from the +indexed terms. If the indexing analyzer lowercases (OpenSearch's +default `text + keyword` multi-field against the text leg, or a +`lowercaseKeyword`-style analyzer), the buckets come back lower- +cased: a distinct-artists query would answer `motörhead` and +`queen` instead of the original display casings, and two tag +writers using `Motörhead` versus `MOTÖRHEAD` would collapse into a +single bucket labelled `motörhead`. For a metadata display use +case (thumbnails, facet filters in the UI, distinct lists) that +behavior is not what we want. + +Searching is layered on top as exactly the strict superset the proposal reserved for later, and it shipped with the implementation: every keyword field additionally gets search-only sibling fields derived from the same definition, a `_lowercase` keyword sibling (doc values disabled; serves wildcards and `=` whole-value matches) and a `_words` text sibling (`words` analyzer: dots to spaces, unicode tokenization, lowercasing, no stemming; serves token and phrase matches). Case-insensitive, word-broken search is the default for every keyword field including facets; fields opt out per override where that is wrong: opaque ids (`ID`, `RootID`, `ParentID`, `Favorites`, `livePhoto.contentId`), the POSIX `Path`, the normalized `MimeType`, and `Content`, which is a fulltext field of its own. Aggregation buckets keep their display casing because they read the base field, never the siblings. + +The query side derives from the same source: the shared lowering pass resolves field names case-insensitively, folds values and routes each match to the right sibling (wildcards to `_lowercase`, tokens and phrases to `_words`, `=` as a whole-value term on `_lowercase`), and both backend compilers consume that one decision. The engine parity suite pins the resulting behavior against bleve and OpenSearch, so a divergence fails CI instead of surfacing in production. The case-sensitivity alignment started in #2633 is completed by deriving both sides from the same source. + +### Schema versioning and upgrades + +Index names carry a schema version derived from the single `search.SchemaVersion` constant (`opencloud-resource-v4`, `bleve-v4`). On startup the service classifies the stored mapping against the code: additive changes (new fields, unchanged analyzers) are reconciled in place without a version bump, breaking changes make the service refuse to start and name the reindex steps. The upgrade path is a plain reindex (`opencloud search index --all-spaces`) into the new versioned index; older indexes stay untouched and can be deleted afterwards (services/search/MIGRATION.md). Golden mapping tests on both backends pin the rendered mappings and reuse the same classifier to tell a contributor whether a change needs only a golden regeneration or a version bump. + +### Known trade-off + +The write-time pipeline produces the document as a generic map via +a json round-trip. The OpenSearch write path already does the +equivalent today via the same json-based conversion helper, so +that path is unchanged. The bleve write path, which previously +handed the struct directly to bleve's reflective indexer, now goes +through the same map-producing step and pays roughly the same +cost. On hot paths (initial indexing of a large space) this is +measurable but not significant; if it ever matters, a direct +reflection walker can replace the json round-trip without changing +any call site. + +### Follow-ups out of scope for this ADR + +- **WebDAV REPORT facet exposure.** The current webdav search + endpoint renders none of the facet fields back to the client. + This is a missing feature, not a regression of the proposal; + its natural resolution is to let the graph-search endpoint + (proposed in #3211) take over once graph search lands. +- **Graph search hit conversion.** Graph search (#3211) translates + proto hits back into libregraph DriveItems with the same + facet-copy helper the search service uses internally. +- **reva's PROPFIND facet listing** uses its own hand-maintained + per-facet key lists. reva deliberately does not depend on the + libregraph Go types, so unifying those key sets is a reva-side + decision tracked separately. +- **Write-path performance.** The json round-trip in the bleve + write path is an optional optimisation target with no call-site + impact when it lands. diff --git a/go.mod b/go.mod index 99d8e7f44f..b609e9ef20 100644 --- a/go.mod +++ b/go.mod @@ -97,11 +97,11 @@ require ( go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 go.opentelemetry.io/contrib/zpages v0.70.0 - go.opentelemetry.io/otel v1.45.0 + go.opentelemetry.io/otel v1.46.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0 go.opentelemetry.io/otel/sdk v1.45.0 - go.opentelemetry.io/otel/trace v1.45.0 + go.opentelemetry.io/otel/trace v1.46.0 golang.org/x/crypto v0.55.0 golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f golang.org/x/image v0.45.0 @@ -378,7 +378,7 @@ require ( go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 // indirect - go.opentelemetry.io/otel/metric v1.45.0 // indirect + go.opentelemetry.io/otel/metric v1.46.0 // indirect go.opentelemetry.io/proto/otlp v1.11.0 // indirect go.uber.org/multierr v1.11.0 // indirect go.uber.org/zap v1.27.0 // indirect diff --git a/go.sum b/go.sum index c4b2576064..6ffc6d9bba 100644 --- a/go.sum +++ b/go.sum @@ -1299,22 +1299,22 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAy go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04= go.opentelemetry.io/contrib/zpages v0.70.0 h1:uBcclHekIrCRLO4KZfXiebDcWB52QQXaHGgvhUJ1K5I= go.opentelemetry.io/contrib/zpages v0.70.0/go.mod h1:Nz9A1+68HVoJrk+cg92zJbcbHFAa+Jbl1uINwAlcK94= -go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU= -go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ= +go.opentelemetry.io/otel v1.46.0 h1:FHt5/CDyVxi/8IM1CH7VE/rRgq3kLHa2mSTVMO8AWyc= +go.opentelemetry.io/otel v1.46.0/go.mod h1:Gj3SEScelsNC45tp4nSxRYlS+f5iez7W8XPMCt905kE= go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 h1:QRefszxJmfPdjXUUm3j6iDzY03mTPXMjqErFqQ67vUg= go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0/go.mod h1:Tiz03lTBVBrm7eWZBOidzEaYaJa8tjwGUGv6d8mlTyk= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 h1:fG5MCxGz8+2VtrN/WgqSpJFctVz24gpxj8CxkKmc8Ww= go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0/go.mod h1:BmAYTn+3ysbRe+IU2msxmf5Rx3g6DHvex+tWI3LdhYI= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0 h1:lsA/S1bxgdbyFGkTj+3meEdJ6ADVU7QoFstV6MXgE68= go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.45.0/go.mod h1:L7u+MirGoB1bjeLH66+xDykF4RC8C3RN7lIFpBiewUo= -go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M= -go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s= +go.opentelemetry.io/otel/metric v1.46.0 h1:yBnkXvgV7AXFILZc5K6IZe/CBFF3OS7BJ8ov6/lj0K8= +go.opentelemetry.io/otel/metric v1.46.0/go.mod h1:iPmdWqifKUdzziPkvvzIJXITl56fQx2mGM/DHLB3/2o= go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw= go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA= go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o= go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA= -go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag= -go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc= +go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= +go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk= go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E= go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= diff --git a/services/search/pkg/parity/README.md b/services/search/pkg/parity/README.md index 67c02ab784..e829530f25 100644 --- a/services/search/pkg/parity/README.md +++ b/services/search/pkg/parity/README.md @@ -1,6 +1,6 @@ # Engine parity -Written by the parity suite (`go test ./services/search/pkg/parity/`), do not edit. +Written by the parity suite (`UPDATE_SEARCH_PARITY_MATRIX=true go test ./services/search/pkg/parity/`), do not edit. Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as expected, `❌ known` when an engine's divergence is documented in the case (`engineOverrides`), `❌` when it is not. `✅ stale` when every engine @@ -149,6 +149,50 @@ Fixtures: | CONTENT-09 | `Content:"alan@example.org"` | links.txt | links.txt | links.txt | ✅ | | CONTENT-10 | `Content:opencloud` | links.txt | links.txt | links.txt | ✅ | +### cjk + +Fixtures: + +- `报告.txt`, Content = "这是一个关于年度销售的中文文档" +- `说明书.pdf`, MimeType = application/pdf, Content = "产品说明与安装步骤" +- `手册.txt`, Content = "学生手册" +- `图片`, folder +- `english.txt`, Content = "annual sales report" + +| Case | Query | expected | bleve | OpenSearch | same? | +|---|---|---|---|---|---| +| CJK-01 | `Content:中文` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-02 | `Content:中文文档` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-03 | `Content:"中文文档"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-04 | `Content:销售` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-05 | `Content:说明` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-06 | `name:报告` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-07 | `name:"*报告*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-08 | `name:"报告.txt"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-09 | `报告` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-10 | `说明书` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-11 | `name:图片` | 图片 | 图片 | 图片 | ✅ | +| CJK-12 | `name:"*图片*"` | 图片 | 图片 | 图片 | ✅ | +| CJK-13 | `图片` | 图片 | 图片 | 图片 | ✅ | +| CJK-14 | `Content:学生` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-15 | `Content:手册` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-16 | `Content:生手` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-17 | `name:"报*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-18 | `name:"*告*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-19 | `name:"*册*"` | 手册.txt | 手册.txt | 手册.txt | ✅ | +| CJK-20 | `name:"图?"` | 图片 | 图片 | 图片 | ✅ | +| CJK-21 | `name:"报?.txt"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-22 | `name:"*报?*"` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-23 | `Content:中文*` | no match | no match | no match | ✅ | +| CJK-24 | `Content:*销售*` | no match | no match | no match | ✅ | +| CJK-25 | `Content:*文档` | no match | no match | no match | ✅ | +| CJK-26 | `Content:文*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-27 | `Content:*文*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-28 | `Content:*文` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-29 | `Content:销*` | 报告.txt | 报告.txt | 报告.txt | ✅ | +| CJK-30 | `Content:说*` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | +| CJK-31 | `Content:*明` | 说明书.pdf | 说明书.pdf | 说明书.pdf | ✅ | + ### favorites Fixtures: diff --git a/services/search/pkg/parity/matrix_test.go b/services/search/pkg/parity/matrix_test.go index 5630ac0cfb..a1d51394d7 100644 --- a/services/search/pkg/parity/matrix_test.go +++ b/services/search/pkg/parity/matrix_test.go @@ -171,7 +171,7 @@ func writeMatrix(report types.Report) { out := &strings.Builder{} out.WriteString("# Engine parity\n\n") - out.WriteString("Written by the parity suite (`go test ./services/search/pkg/parity/`), do not edit.\n") + out.WriteString("Written by the parity suite (`UPDATE_SEARCH_PARITY_MATRIX=true go test ./services/search/pkg/parity/`), do not edit.\n") out.WriteString("Every case runs against bleve and OpenSearch. `same?` is ✅ when both answer as\n") out.WriteString("expected, `❌ known` when an engine's divergence is documented in the case\n") out.WriteString("(`engineOverrides`), `❌` when it is not. `✅ stale` when every engine\n") @@ -226,8 +226,17 @@ func writeMatrix(report types.Report) { matrixNames(row.answered["bleve"]), matrixNames(row.answered["opensearch"]), matrixVerdict(row)) } - if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { - fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) + if os.Getenv("UPDATE_SEARCH_PARITY_MATRIX") != "" { + if err := os.WriteFile(matrixFile, []byte(out.String()), 0o644); err != nil { + fmt.Fprintf(os.Stderr, "failed to write %s: %v\n", matrixFile, err) + } + return + } + + // the committed matrix must match what the suite answers + previous, _ := os.ReadFile(matrixFile) + if string(previous) != out.String() { + Fail(matrixFile + " is out of date: regenerate it with UPDATE_SEARCH_PARITY_MATRIX=true and commit the change") } } diff --git a/services/search/pkg/parity/parity_test.go b/services/search/pkg/parity/parity_test.go index 3bacdd762b..41695d18b0 100644 --- a/services/search/pkg/parity/parity_test.go +++ b/services/search/pkg/parity/parity_test.go @@ -97,6 +97,7 @@ func queryGroups() []queryGroup { tagsGroup(), titleGroup(), contentGroup(), + cjkGroup(), favoritesGroup(), mediatypeGroup(), pathGroup(), diff --git a/services/search/pkg/parity/query_cjk_test.go b/services/search/pkg/parity/query_cjk_test.go new file mode 100644 index 0000000000..f855c06b28 --- /dev/null +++ b/services/search/pkg/parity/query_cjk_test.go @@ -0,0 +1,51 @@ +package parity + +import ( + "github.com/opencloud-eu/opencloud/services/search/pkg/search" +) + +func cjkGroup() queryGroup { + return queryGroup{ + name: "cjk", + fixtures: []search.Resource{ + fixtureDoc("报告.txt", withContent("这是一个关于年度销售的中文文档")), + fixtureDoc("说明书.pdf", withContent("产品说明与安装步骤"), withMime("application/pdf")), + fixtureDoc("手册.txt", withContent("学生手册")), + fixtureFolder("图片"), + fixtureDoc("english.txt", withContent("annual sales report")), + }, + cases: []queryCase{ + {id: 1, query: `Content:中文`, want: []string{"报告.txt"}}, + {id: 2, query: `Content:中文文档`, want: []string{"报告.txt"}}, + {id: 3, query: `Content:"中文文档"`, want: []string{"报告.txt"}}, + {id: 4, query: `Content:销售`, want: []string{"报告.txt"}}, + {id: 5, query: `Content:说明`, want: []string{"说明书.pdf"}}, + {id: 6, query: `name:报告`, want: []string{"报告.txt"}}, + {id: 7, query: `name:"*报告*"`, want: []string{"报告.txt"}}, + {id: 8, query: `name:"报告.txt"`, want: []string{"报告.txt"}}, + {id: 9, query: `报告`, want: []string{"报告.txt"}}, + {id: 10, query: `说明书`, want: []string{"说明书.pdf"}}, + {id: 11, query: `name:图片`, want: []string{"图片"}}, + {id: 12, query: `name:"*图片*"`, want: []string{"图片"}}, + {id: 13, query: `图片`, want: []string{"图片"}}, + {id: 14, query: `Content:学生`, want: []string{"手册.txt"}}, + {id: 15, query: `Content:手册`, want: []string{"手册.txt"}}, + {id: 16, query: `Content:生手`, want: []string{"手册.txt"}}, + {id: 17, query: `name:"报*"`, want: []string{"报告.txt"}}, + {id: 18, query: `name:"*告*"`, want: []string{"报告.txt"}}, + {id: 19, query: `name:"*册*"`, want: []string{"手册.txt"}}, + {id: 20, query: `name:"图?"`, want: []string{"图片"}}, + {id: 21, query: `name:"报?.txt"`, want: []string{"报告.txt"}}, + {id: 22, query: `name:"*报?*"`, want: []string{"报告.txt"}}, + {id: 23, query: `Content:中文*`}, + {id: 24, query: `Content:*销售*`}, + {id: 25, query: `Content:*文档`}, + {id: 26, query: `Content:文*`, want: []string{"报告.txt"}}, + {id: 27, query: `Content:*文*`, want: []string{"报告.txt"}}, + {id: 28, query: `Content:*文`, want: []string{"报告.txt"}}, + {id: 29, query: `Content:销*`, want: []string{"报告.txt"}}, + {id: 30, query: `Content:说*`, want: []string{"说明书.pdf"}}, + {id: 31, query: `Content:*明`, want: []string{"说明书.pdf"}}, + }, + } +} diff --git a/vendor/go.opentelemetry.io/otel/.golangci.yml b/vendor/go.opentelemetry.io/otel/.golangci.yml index 8a7f1ec5d9..a0ba70805e 100644 --- a/vendor/go.opentelemetry.io/otel/.golangci.yml +++ b/vendor/go.opentelemetry.io/otel/.golangci.yml @@ -143,8 +143,7 @@ linters: - name: constant-logical-expr - name: context-as-argument arguments: - - allow-types-before: '*testing.T' - disabled: true + - allow-types-before: '*testing.T,*testing.B' - name: context-keys-type - name: deep-exit - name: defer diff --git a/vendor/go.opentelemetry.io/otel/CHANGELOG.md b/vendor/go.opentelemetry.io/otel/CHANGELOG.md index 2db588ea27..4d473a1e85 100644 --- a/vendor/go.opentelemetry.io/otel/CHANGELOG.md +++ b/vendor/go.opentelemetry.io/otel/CHANGELOG.md @@ -11,6 +11,34 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm +## [1.46.0/0.68.0/0.22.0/0.0.19] - 2026-08-25 + +This release is the last to support [Go 1.25]. +The next release will require at least [Go 1.26]. + +### Added + +- Support testing of [Go 1.27]. (#8811) +- Support `http/json` protocol in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#8273, #8775, #8831) +- Add `Hasher` struct and methods in `go.opentelemetry.io/otel/attribute` to compute authoritative `Distinct` hashes incrementally for attribute filtering and deduplication. (#8598) + +### Changed + +- Lazily evaluate filtered and dropped attributes on measurement hot paths in `go.opentelemetry.io/otel/sdk/metric` to avoid unnecessary attribute set allocations. (#8598) +- Add `ErrExporterShutdown` to `go.opentelemetry.io/otel/sdk/log` and return it from the `go.opentelemetry.io/otel/exporters/stdout/stdoutlog`, `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc`, and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp` exporters when `Export` is called after `Shutdown`. (#8773) +- Clarify in `go.opentelemetry.io/otel/log` that calling `Logger.Enabled` is optional and that cached results can become stale. (#8764) + +### Fixed + +- Export dropped attribute counts in OTLP log records from `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` and `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp`. (#8829) +- Name span events created from OpenTracing logs after the `event` log field, falling back to `log`, instead of always using an empty name in `go.opentelemetry.io/otel/bridge/opentracing`. (#8648) +- Count exception attributes omitted due to the attribute count limit as dropped in `go.opentelemetry.io/otel/sdk/log`. (#8796) +- Prevent log record and instrumentation scope attributes with empty keys from reaching processors and exporters in `go.opentelemetry.io/otel/sdk/log`. (#8797) +- Fix a data race when span attributes are read concurrently in `go.opentelemetry.io/otel/sdk/trace`. (#8706) +- Prevent a panic in `(*Set).Filter` when called on a nil receiver in `go.opentelemetry.io/otel/attribute`. (#8792) +- The simple span and log processors record `otel.sdk.processor.{span,log}.processed` when the record is submitted to the exporter instead of after the export completes, and no longer set `error.type` from the export outcome, in `go.opentelemetry.io/otel/sdk/trace` and `go.opentelemetry.io/otel/sdk/log`. (#8705) +- Prevent `Resource.MarshalLog` from panicking on nil resources in `go.opentelemetry.io/otel/sdk/resource`. (#8758) + ## [1.45.0/0.67.0/0.21.0/0.0.18] - 2026-08-03 ### Added @@ -3782,7 +3810,8 @@ It contains api and sdk for trace and meter. - CircleCI build CI manifest files. - CODEOWNERS file to track owners of this project. -[Unreleased]: https://github.com/open-telemetry/opentelemetry-go/compare/v1.45.0...HEAD +[Unreleased]: https://github.com/open-telemetry/opentelemetry-go/compare/v1.46.0...HEAD +[1.46.0/0.68.0/0.22.0/0.0.19]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.46.0 [1.45.0/0.67.0/0.21.0/0.0.18]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.45.0 [1.44.0/0.66.0/0.20.0/0.0.17]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.44.0 [1.43.0/0.65.0/0.19.0]: https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.43.0 @@ -3887,6 +3916,7 @@ It contains api and sdk for trace and meter. +[Go 1.27]: https://go.dev/doc/go1.27 [Go 1.26]: https://go.dev/doc/go1.26 [Go 1.25]: https://go.dev/doc/go1.25 [Go 1.24]: https://go.dev/doc/go1.24 diff --git a/vendor/go.opentelemetry.io/otel/Makefile b/vendor/go.opentelemetry.io/otel/Makefile index d4711257df..c674ef8a17 100644 --- a/vendor/go.opentelemetry.io/otel/Makefile +++ b/vendor/go.opentelemetry.io/otel/Makefile @@ -69,8 +69,13 @@ $(GORELEASE): PACKAGE=golang.org/x/exp/cmd/gorelease GOVULNCHECK = $(TOOLS)/govulncheck $(TOOLS)/govulncheck: PACKAGE=golang.org/x/vuln/cmd/govulncheck +AFFECTEDMODS = $(TOOLS)/affectedmods +AFFECTEDMODS_FILES := $(sort $(shell find $(TOOLS_MOD_DIR)/affectedmods -type f)) +$(TOOLS)/affectedmods: PACKAGE=go.opentelemetry.io/otel/$(TOOLS_MOD_DIR)/affectedmods +$(TOOLS)/affectedmods: $(AFFECTEDMODS_FILES) + .PHONY: tools -tools: $(CROSSLINK) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(VERIFYREADMES) $(MULTIMOD) $(SEMCONVKIT) $(GOTMPL) $(GORELEASE) +tools: $(CROSSLINK) $(GOLANGCI_LINT) $(MISSPELL) $(GOCOVMERGE) $(STRINGER) $(VERIFYREADMES) $(MULTIMOD) $(SEMCONVKIT) $(GOTMPL) $(GORELEASE) $(AFFECTEDMODS) # Virtualized python tools via docker @@ -125,9 +130,11 @@ go-work: $(CROSSLINK) # Build -.PHONY: build +.PHONY: build cross-build build: $(OTEL_GO_MOD_DIRS:%=build/%) $(OTEL_GO_MOD_DIRS:%=build-tests/%) +# Cross-platform builds cannot execute the binaries produced by build-tests (go test), so use a compile-only target. +cross-build: $(OTEL_GO_MOD_DIRS:%=build/%) build/%: DIR=$* build/%: @echo "$(GO) build $(DIR)/..." \ @@ -163,6 +170,9 @@ test/%: | grep -v third_party \ | xargs $(GO) test -timeout $(TIMEOUT)s $(ARGS) +.PHONY: test-tools +test-tools: test/$(TOOLS_MOD_DIR) + COVERAGE_MODE = atomic COVERAGE_PROFILE = coverage.out .PHONY: test-coverage @@ -187,15 +197,23 @@ benchmark/%: # sdk/metric is split into two shards to work around CodSpeed limitations. # See https://github.com/CodSpeedHQ/codspeed-go/issues/56 -BENCHMARK_SHARDS := $(filter-out ./sdk/metric,$(OTEL_GO_MOD_DIRS)) ./sdk/metric/root ./sdk/metric/internal -benchmark/./sdk/metric/root: +BENCHMARK_SHARDS := $(filter-out ./sdk/metric,$(OTEL_GO_MOD_DIRS)) ./sdk/metric/. ./sdk/metric/internal +benchmark/./sdk/metric/.: cd ./sdk/metric && $(GO) test -run='^$$' -bench=. $(ARGS) . ./exemplar/... -benchmark/./sdk/metric/internal: - cd ./sdk/metric && $(GO) test -run='^$$' -bench=. $(ARGS) ./internal/... print-sharded-benchmarks: @echo $(BENCHMARK_SHARDS) | jq -cR 'split(" ")' +# Print the JSON list of benchmark shards whose code changed since +# BASE_REF (default: main). Filters print-sharded-benchmarks +# down to shards whose underlying module contains a changed *.go file. +# Non-Go changes (docs, workflows, Makefile, go.mod/go.sum, tooling) emit []. +# Override the diff base via BASE_REF, or pass ARGS=-all to emit the full list. +BASE_REF ?= main +.PHONY: print-affected-benchmarks +print-affected-benchmarks: + @$(MAKE) -s print-sharded-benchmarks | $(GO) -C $(TOOLS_MOD_DIR) run ./affectedmods $(if $(strip $(BASE_REF)),-base=$(BASE_REF),) $(ARGS) + .PHONY: golangci-lint golangci-lint-fix golangci-lint-fix: ARGS=--fix golangci-lint-fix: golangci-lint diff --git a/vendor/go.opentelemetry.io/otel/README.md b/vendor/go.opentelemetry.io/otel/README.md index ce97e9a251..9291d1470d 100644 --- a/vendor/go.opentelemetry.io/otel/README.md +++ b/vendor/go.opentelemetry.io/otel/README.md @@ -53,18 +53,25 @@ Currently, this project supports the following environments. | OS | Go Version | Architecture | |----------|------------|--------------| +| Ubuntu | 1.27 | amd64 | | Ubuntu | 1.26 | amd64 | | Ubuntu | 1.25 | amd64 | +| Ubuntu | 1.27 | 386 | | Ubuntu | 1.26 | 386 | | Ubuntu | 1.25 | 386 | +| Ubuntu | 1.27 | arm64 | | Ubuntu | 1.26 | arm64 | | Ubuntu | 1.25 | arm64 | +| macOS | 1.27 | amd64 | | macOS | 1.26 | amd64 | | macOS | 1.25 | amd64 | +| macOS | 1.27 | arm64 | | macOS | 1.26 | arm64 | | macOS | 1.25 | arm64 | +| Windows | 1.27 | amd64 | | Windows | 1.26 | amd64 | | Windows | 1.25 | amd64 | +| Windows | 1.27 | 386 | | Windows | 1.26 | 386 | | Windows | 1.25 | 386 | diff --git a/vendor/go.opentelemetry.io/otel/VERSIONING.md b/vendor/go.opentelemetry.io/otel/VERSIONING.md index b27c9e84f5..56eaa7d220 100644 --- a/vendor/go.opentelemetry.io/otel/VERSIONING.md +++ b/vendor/go.opentelemetry.io/otel/VERSIONING.md @@ -12,6 +12,12 @@ is designed so the following goals can be achieved. * [Semantic import versioning](https://github.com/golang/go/wiki/Modules#semantic-import-versioning) will be used. + * Stable module compatibility is understood in terms of the [Go 1 + compatibility guidelines](https://go.dev/doc/go1compat). Code that + compiled against an older version of a package should continue to compile + against newer versions of that package, subject to the exceptions in the + Go 1 compatibility guidelines and any additional exceptions documented + below. * Versions will comply with [semver 2.0](https://semver.org/spec/v2.0.0.html) with the following exceptions. * New methods may be added to exported API interfaces. All exported diff --git a/vendor/go.opentelemetry.io/otel/attribute/hash.go b/vendor/go.opentelemetry.io/otel/attribute/hash.go index f651eb13d9..54c26e67a0 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/hash.go +++ b/vendor/go.opentelemetry.io/otel/attribute/hash.go @@ -33,21 +33,73 @@ const ( emptyID uint64 = 7305809155345288421 // "__empty_" (little endian) ) -// hashKVs returns a new xxHash64 hash of kvs. -func hashKVs(kvs []KeyValue) uint64 { - h := xxhash.New() - for _, kv := range kvs { - h = hashKV(h, kv) - } - sum := h.Sum64() - // Remap 0 to a non-zero value for non-empty input because hash == 0 is a reserved sentinel (treated as empty/invalid). - const remappedZeroHash uint64 = 1 - if sum == 0 && len(kvs) > 0 { - return remappedZeroHash +// Hasher computes a Distinct value from KeyValue attributes supplied with +// Write. +// +// A Hasher must be obtained from [NewHasher]. The zero value is not usable and +// its methods will panic. +type Hasher struct { + h xxhash.Hash +} + +// NewHasher returns a new Hasher. +func NewHasher() *Hasher { + return &Hasher{h: xxhash.New()} +} + +// Reset resets h to its initial state so it can be reused. +func (h *Hasher) Reset() { + h.h.Reset() +} + +// Write adds kv to the hash. +// +// Write requires attributes to be supplied in ascending key order with no +// duplicate keys. To produce the same Distinct as Set.Equivalent, write +// attributes in ascending key order, with no more than one value for each key. +// If the source contains duplicate keys, retain the last value for each key +// before calling Write. +func (h *Hasher) Write(kv KeyValue) { + // hashKV mutates the digest h.h refers to in place and returns the same + // Hash value it was passed. Discarding the result keeps the digest pointer + // from flowing back into h, which would force the digest to be heap + // allocated for every Hasher. Keeping Write this small also keeps it within + // the inlining budget, which matters because hashKVs calls it per attribute. + _ = hashKV(h.h, kv) +} + +// Distinct returns the identifier for the attributes written to h. When Write +// is called as described above, it returns the same value as [Set.Equivalent]. +func (h *Hasher) Distinct() Distinct { + // No count of written attributes is needed to detect the empty case. The + // sum of a digest with nothing written to it is emptyHash, which is + // non-zero (0xef46db3751d8e999), so it passes through remapZeroHash + // unchanged and matches emptySet.Equivalent. + return Distinct{hash: remapZeroHash(h.h.Sum64())} +} + +// remapZeroHash remaps a 0 sum to a non-zero value, because hash == 0 is a +// reserved sentinel (treated as empty/invalid). +func remapZeroHash(sum uint64) uint64 { + if sum == 0 { + return 1 } return sum } +// hashKVs returns a new xxHash64 hash of kvs. +// +// This routes through [Hasher] so that Set hashing and Hasher cannot disagree: +// there is exactly one implementation of how attributes are mixed and how the +// final sum is framed. +func hashKVs(kvs []KeyValue) uint64 { + h := NewHasher() + for _, kv := range kvs { + h.Write(kv) + } + return h.Distinct().hash +} + // hashKV returns the xxHash64 hash of kv with h as the base. func hashKV(h xxhash.Hash, kv KeyValue) xxhash.Hash { h = h.String(string(kv.Key)) diff --git a/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go b/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go index c851179cca..4fcdd2dbea 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go +++ b/vendor/go.opentelemetry.io/otel/attribute/internal/xxhash/xxhash.go @@ -62,3 +62,8 @@ func (h Hash) String(val string) Hash { func (h Hash) Sum64() uint64 { return h.d.Sum64() } + +// Reset resets the hash to its initial state. +func (h Hash) Reset() { + h.d.Reset() +} diff --git a/vendor/go.opentelemetry.io/otel/attribute/set.go b/vendor/go.opentelemetry.io/otel/attribute/set.go index 87d6f96200..56a70a9f6d 100644 --- a/vendor/go.opentelemetry.io/otel/attribute/set.go +++ b/vendor/go.opentelemetry.io/otel/attribute/set.go @@ -313,6 +313,9 @@ func filteredToFront(slice []KeyValue, keep Filter) int { // Filter returns a filtered copy of this Set. See the documentation for // NewSetWithSortableFiltered for more details. func (l *Set) Filter(re Filter) (Set, []KeyValue) { + if l == nil { + return emptySet, nil + } if re == nil { return *l, nil } diff --git a/vendor/go.opentelemetry.io/otel/renovate.json b/vendor/go.opentelemetry.io/otel/renovate.json index fa5acf2d3b..5008ab2bf1 100644 --- a/vendor/go.opentelemetry.io/otel/renovate.json +++ b/vendor/go.opentelemetry.io/otel/renovate.json @@ -15,6 +15,13 @@ "matchDepTypes": ["indirect"], "enabled": true }, + { + "description": "Disable Go module major updates to v2+", + "matchManagers": ["gomod"], + "matchUpdateTypes": ["major"], + "matchNewValue": "/^v?([2-9]|[1-9][0-9]+)\\./", + "enabled": false + }, { "matchPackageNames": ["go.opentelemetry.io/build-tools/**"], "groupName": "build-tools" diff --git a/vendor/go.opentelemetry.io/otel/version.go b/vendor/go.opentelemetry.io/otel/version.go index ea79030969..f599bf285d 100644 --- a/vendor/go.opentelemetry.io/otel/version.go +++ b/vendor/go.opentelemetry.io/otel/version.go @@ -5,5 +5,5 @@ package otel // Version is the current release version of OpenTelemetry in use. func Version() string { - return "1.45.0" + return "1.46.0" } diff --git a/vendor/go.opentelemetry.io/otel/versions.yaml b/vendor/go.opentelemetry.io/otel/versions.yaml index 3b1ad7301a..f329d171d9 100644 --- a/vendor/go.opentelemetry.io/otel/versions.yaml +++ b/vendor/go.opentelemetry.io/otel/versions.yaml @@ -3,7 +3,7 @@ module-sets: stable-v1: - version: v1.45.0 + version: v1.46.0 modules: - go.opentelemetry.io/otel - go.opentelemetry.io/otel/bridge/opencensus @@ -22,12 +22,12 @@ module-sets: - go.opentelemetry.io/otel/sdk/metric - go.opentelemetry.io/otel/trace experimental-metrics: - version: v0.67.0 + version: v0.68.0 modules: - go.opentelemetry.io/otel/exporters/prometheus - go.opentelemetry.io/otel/metric/x experimental-logs: - version: v0.21.0 + version: v0.22.0 modules: - go.opentelemetry.io/otel/log - go.opentelemetry.io/otel/log/logtest @@ -37,7 +37,7 @@ module-sets: - go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp - go.opentelemetry.io/otel/exporters/stdout/stdoutlog experimental-schema: - version: v0.0.18 + version: v0.0.19 modules: - go.opentelemetry.io/otel/schema excluded-modules: diff --git a/vendor/modules.txt b/vendor/modules.txt index b8a87990e1..c098b7652a 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -2323,7 +2323,7 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/internal/semconv ## explicit; go 1.25.0 go.opentelemetry.io/contrib/zpages go.opentelemetry.io/contrib/zpages/internal -# go.opentelemetry.io/otel v1.45.0 +# go.opentelemetry.io/otel v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel go.opentelemetry.io/otel/attribute @@ -2366,7 +2366,7 @@ go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/counter go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/observ go.opentelemetry.io/otel/exporters/stdout/stdouttrace/internal/x -# go.opentelemetry.io/otel/metric v1.45.0 +# go.opentelemetry.io/otel/metric v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel/metric go.opentelemetry.io/otel/metric/embedded @@ -2382,7 +2382,7 @@ go.opentelemetry.io/otel/sdk/trace go.opentelemetry.io/otel/sdk/trace/internal/env go.opentelemetry.io/otel/sdk/trace/internal/observ go.opentelemetry.io/otel/sdk/trace/tracetest -# go.opentelemetry.io/otel/trace v1.45.0 +# go.opentelemetry.io/otel/trace v1.46.0 ## explicit; go 1.25.0 go.opentelemetry.io/otel/trace go.opentelemetry.io/otel/trace/embedded