The refuse specs use registered analyzers (fulltext is gone), the golden regenerates via UPDATE_GOLDEN, MappingGetResp grew an accessor, and the parity suite passes the new NewBackend signature.
The opensearch-go bump renamed the mapping-get accessor, the schema check
takes a context and a logger now, and the golden bleve mapping carries the
word-broken Name and Title.
- export mapping.SortedUnionKeys and reuse it in bleve.compareKeysExcept
instead of a copied union-of-keys block
- add Classification.AddBreaking to fold engine-specific breaking reasons and
force the verdict, replacing the identical block in the bleve and opensearch
Classify paths
- shorten the multi-line doc comments flagged as too verbose
- add reconcile_test.go: direct unit tests for Reconcile incl. the
persisted-but-errored and classify-error branches (previously only
reached indirectly through the engine integration tests)
- convert the 11 near-identical Classify It blocks to a DescribeTable
- the additive warnings advertise --all-spaces --force-rescan; a plain
walk skips unchanged documents and never backfills the new fields
- Apply checks index existence first again, so a pre-provisioned index
needs no create privilege and odd create-error shapes (string error
bodies, cluster blocks) cannot fail a healthy startup; Create on 404
keeps the typed already-exists swallow as the creation-race backstop
- number_of_replicas drift is not breaking, it is runtime-tunable and
needs no rebuild
- bleve returns the classification alongside post-persist errors and
the server warns before the error check, so the one-time additive
warning is not lost when close or reopen fails
- a golden fixture pins the marshaled bleve mapping so a dependency
bump that changes marshaling fails in CI instead of refusing every
installation in the field
Addresses the two Copilot review comments on the PR: the additive
opensearch log now matches the bleve warning (level and re-index hint),
and the refuse message spells out how to delete the index per engine
(DELETE /<name> vs removing the bleve directory).
Both engines now diff the stored/live index schema against the schema
generated from code when the service starts. A shared recursive
classifier in the mapping package is the single oracle:
- equal: start normally.
- additive (new fields without any indexed data): applied in place.
OpenSearch gets a PUT _mapping with the full code properties, bleve
persists the code mapping into the index (SetInternal + reopen) so
the new fields are properly typed immediately and later startups
classify equal. A startup warning lists the new fields because
documents indexed before the upgrade lack them until re-indexed.
- breaking (changed definitions or analyzers, removed or renamed
fields, or new fields that already contain data of unknown form):
refuse to start with an error describing the rebuild procedure
(delete the index, start, run "opencloud search index --all-spaces")
and the OC_EXCLUDE_RUN_SERVICES=search escape hatch.
PUT _mapping is deliberately only the apply mechanism, never the
judge: its merge semantics cannot see removals or renames and it
accepts in-place updatable param changes with an ack. bleve
additionally checks idx.Fields() so previously dynamically indexed
data (which leaves no schema trace in bleve) is caught, matching by
exact name and by path prefix.
While at it: the OpenSearch startup check runs with a real,
minute-bounded context instead of context.TODO(), bleve indexes are
opened with a 5s bolt_timeout so a second process fails fast instead
of hanging on the file lock, and the reversed errors.Is arguments in
bleve.NewIndex were fixed.
https://github.com/opencloud-eu/opencloud/issues/3092
The engine suites keep what is engine-specific (index setup, health, purge-space batching); every behavior answer lives in the parity suite once. The shared test client learns IndicesCount, and FIELDS-15 pins the Size/Type gate for number queries.
KQL searches case-insensitively, so every keyword field gets its lowercased
search sibling unless it opts out: ids are opaque, paths are POSIX, the mime
type is normalized already. That takes the facets along, artist or camera
model match regardless of case, while the case-preserved base still answers
and aggregates. Which siblings a field carries is decided once, from the
struct and the overrides, and the renderers, the document writer and the
query lowering all read it from there.
A single word finds the names and titles that contain it, `report` finds
Report.txt, on bleve as well as on OpenSearch, which so far only did it by
accident of its dynamic mapping. Modelled like SharePoint's NoWordBreaker:
a keyword field is one whole value unless the override switches that off,
which adds a search-only _words sibling next to _lowercase, analyzed into
lowercased words (a dot is a word boundary, no stemming). The base stays the
whole value for returning and aggregating, wildcards and whole values keep
using _lowercase. Quotes do not change the meaning, a phrase is a phrase
either way, and there is no exact-match operator yet.
Paths act as references (location scoping, deep links): /Foo and /foo
are distinct siblings, so path: matching must be exact. Case-insensitive
folder discovery is served by name: and its lowercase sibling. This also
matches bleve on main, where path queries have always been
case-sensitive.
Dropping the sibling removes its biggest maintenance cost: Path is the
one mutable sibling field, a move rewrites the paths of a whole subtree
and the OpenSearch move script had to rebuild Path_lowercase alongside
the base field.
With paths case-sensitive, the ref path scope moves into the query
itself: bleve as a term/prefix disjunction on the keyword Path,
OpenSearch as a term filter on its path_hierarchy tokens. The
post-filter that used to drop out-of-scope hits after the query ran is
gone; totals and paging now respect the scope instead of being computed
over the whole space, and a wrong-cased scope simply matches nothing.
A leading NOT next to an operator was miscompiled: the bleve compiler left the consumed term in `next`, so `NOT x AND y` dropped `y` and produced a self-contradicting clause; the OpenSearch transpiler checked nextOp==AND before prevOp==NOT, so the negated term landed in `must` instead of `must_not`. NOT is unary and binds to the node directly after it regardless of what follows. This also fixes `mediatype:file AND <term>` (the web Files filter) at the root, so the earlier mediatype:file group workaround is dropped.
mediatype:file expands to a NOT restriction. Spliced inline as `NOT MimeType:httpd/unix-directory`, the bleve compiler's NOT branch left a stale operand, so `mediatype:file AND name:x` dropped `name:x` and matched nothing (the web Files filter). It is now wrapped in a group so the negation stays atomic; verified fixing both bleve and OpenSearch.
mediatype:Folder / mediatype:IMAGE resolved to a literal MimeType search and matched nothing because Expand switched on the raw value. The value is now lowercased in the lowering pass, so categories and literal MIME types match regardless of case, consistently on both backends.
Adds bleve and OpenSearch coverage for category (image), literal MIME (image/svg+xml, with + and /), and raw MimeType: queries. Documents why MimeType skips the bleve escaper: it is not a bug, bleve treats / and + as literals mid-term, so a literal MIME still matches exactly while the category wildcard image/* keeps its *.
bleve compiled a path restriction to a DisjunctionQuery, which mapBinary redistributes as an OR-chain, so `path:/Foo AND name:bar` matched the folder itself unconditionally. It is now a BooleanQuery (should: folder OR descendants), which mapBinary keeps atomic under an enclosing AND.
The OpenSearch full-text branch ran before the wildcard check, so `content:foo*` degraded to a phrase match and diverged from bleve; the wildcard check now comes first.
Adds the missing coverage the review flagged: path AND term, content wildcard, case-insensitive tags (the array sibling branch), and a spaced path with descendants on OpenSearch.
Single-term `content:` built an unanalyzed term query, so once this branch dropped the blanket query-value lowercasing, `content:Foo` missed on OpenSearch (bleve was unaffected, its query analyzes). Fielded full-text queries now use a match query. OpenSearch `Content` also gets a porter stemming analyzer (it used the default standard analyzer and never stemmed), so full-text search matches bleve on both case and stemming.
Keyword and path fields always index their case-preserved base and, when CaseInsensitive is set, an additional <field>_lowercase sibling used only for matching. The KQL lowering marks a restriction case-insensitive; each backend searches the sibling and lowercases the query value the same way the sibling is precomputed at index time (Go strings.ToLower on both sides, so non-ASCII stays consistent).
Search always returns the case-preserved base, so the sibling never has to be read back. In bleve it is indexed but not stored, kept out of _all, and without doc values. In OpenSearch it deliberately stays in _source: excluding it would make every update-by-query script rebuild all siblings from the document via painless toLowerCase, which lowercases differently than Go and would drift from the query side. Keeping it in _source avoids that, and a lowercased copy of a name or path is negligible disk in a cluster.
The OpenSearch move script keeps the base and its sibling in sync by swapping the moved prefix in Path_lowercase and setting Name_lowercase from Go-lowercased params, so case-insensitive search still finds a file after it moves (previously the sibling went stale). bleve re-indexes the whole document on move/delete/restore, so its siblings stay fresh for free.
This also repairs OpenSearch path search (the query value was no longer folded to lowercase, so path:<Foo> returned nothing) and makes bleve path queries match a folder and its descendants like OpenSearch's path_hierarchy. The Path base stays case-preserved so the move/delete descendant update (an exact TermQuery on Path) matches mixed-case folders.
The KQL parser produced its own validation errors but imported them from the
search service's query package. Move them into pkg/kql and let the search
backend consume kql.IsValidationError, so the parser stops depending on a
service package.
Both backends carry a shared search.SchemaVersion in the index name
(OpenSearch <base>-vN) and data path (bleve-vN). A breaking schema change
bumps the version so the service builds a fresh index instead of colliding
with the incompatible previous one; the old index is left in place.
Add a TypeGeopoint field type. The libregraph Location facet is kept as an
object (retrieval / numeric queries) and a sibling <name>_geopoint field
carries the {lat,lon} form for geo-distance / bbox / polygon queries,
uniform across bleve and OpenSearch via the shared mapping. PrepareForIndex
splices the sibling in at write time.
Build the bleve and OpenSearch index mappings from the Go struct via
reflection (json tags + per-field overrides) instead of hand-rolled
mappings and hit deserializers. New mapping package: BleveBuildMapping,
OpenSearchBuildMapping, Deserialize[T], PrepareForIndex; field decoding is
fail-soft. Mtime is typed as a date so mtime ranges are chronological on
both backends. Route CS3 facet parsing through mapping.DeserializeStringMap.
The any-valued (bleve hit) and string-valued (CS3 metadata) deserializers
share one generic fillStruct walker with a per-value setLeaf callback.
both engines now agree on names, titles, tags, paths, types, sizes, dates,
hidden flags, facet values and wildcards. quotes only delimit phrases and
the equals operator matches the whole field value, following the kql spec.
the index name carries a generation so a changed mapping starts on a fresh
index, MIGRATION.md says how to fill it.
Everything the bleve and OpenSearch backend suites checked now runs against
both engines in services/search/pkg/parity. The two cases without a twin
there join it (a facet value keeps its case, batches stay apart), the one
thing only OpenSearch can do, refuse an unhealthy cluster, stays in its
package.
The FIXME pointed at #2632 (dotted keys in KQL property restrictions),
which is now merged. Use audio.artist — the originally intended target
field for this regression — so the test matches its name: a nested
string field that is not on the lowercase allowlist.
Dominik Schmidt