The file expansion is grouped again so its NOT stays atomic next to other terms (OpenSearch turned 'mediatype:file OR x' into '(NOT dir) AND x'), and the bleve compiler no longer re-keys resolved groups (a negated mediatype group targeted the raw 'mediatype' field and matched everything). Pinned as MEDIATYPE-07..10.
From #3408: hidden takes bool words only, type categories map to the stored value in the shared pass, ? counts as a wildcard, a non-suffix wildcard on a word-broken field forgives the extension, = matches the whole value on the lowercased sibling, paths lose their trailing slash. Dead compiler helpers removed.
A single word finds the names and titles that contain it, `report` finds
Report.txt, on bleve as well as on OpenSearch, which so far only did it by
accident of its dynamic mapping. Modelled like SharePoint's NoWordBreaker:
a keyword field is one whole value unless the override switches that off,
which adds a search-only _words sibling next to _lowercase, analyzed into
lowercased words (a dot is a word boundary, no stemming). The base stays the
whole value for returning and aggregating, wildcards and whole values keep
using _lowercase. Quotes do not change the meaning, a phrase is a phrase
either way, and there is no exact-match operator yet.
Paths act as references (location scoping, deep links): /Foo and /foo
are distinct siblings, so path: matching must be exact. Case-insensitive
folder discovery is served by name: and its lowercase sibling. This also
matches bleve on main, where path queries have always been
case-sensitive.
Dropping the sibling removes its biggest maintenance cost: Path is the
one mutable sibling field, a move rewrites the paths of a whole subtree
and the OpenSearch move script had to rebuild Path_lowercase alongside
the base field.
With paths case-sensitive, the ref path scope moves into the query
itself: bleve as a term/prefix disjunction on the keyword Path,
OpenSearch as a term filter on its path_hierarchy tokens. The
post-filter that used to drop out-of-scope hits after the query ran is
gone; totals and paging now respect the scope instead of being computed
over the whole space, and a wrong-cased scope simply matches nothing.
A leading NOT next to an operator was miscompiled: the bleve compiler left the consumed term in `next`, so `NOT x AND y` dropped `y` and produced a self-contradicting clause; the OpenSearch transpiler checked nextOp==AND before prevOp==NOT, so the negated term landed in `must` instead of `must_not`. NOT is unary and binds to the node directly after it regardless of what follows. This also fixes `mediatype:file AND <term>` (the web Files filter) at the root, so the earlier mediatype:file group workaround is dropped.
mediatype:file expands to a NOT restriction. Spliced inline as `NOT MimeType:httpd/unix-directory`, the bleve compiler's NOT branch left a stale operand, so `mediatype:file AND name:x` dropped `name:x` and matched nothing (the web Files filter). It is now wrapped in a group so the negation stays atomic; verified fixing both bleve and OpenSearch.
Adds bleve and OpenSearch coverage for category (image), literal MIME (image/svg+xml, with + and /), and raw MimeType: queries. Documents why MimeType skips the bleve escaper: it is not a bug, bleve treats / and + as literals mid-term, so a literal MIME still matches exactly while the category wildcard image/* keeps its *.
bleve compiled a path restriction to a DisjunctionQuery, which mapBinary redistributes as an OR-chain, so `path:/Foo AND name:bar` matched the folder itself unconditionally. It is now a BooleanQuery (should: folder OR descendants), which mapBinary keeps atomic under an enclosing AND.
The OpenSearch full-text branch ran before the wildcard check, so `content:foo*` degraded to a phrase match and diverged from bleve; the wildcard check now comes first.
Adds the missing coverage the review flagged: path AND term, content wildcard, case-insensitive tags (the array sibling branch), and a spaced path with descendants on OpenSearch.
Keyword and path fields always index their case-preserved base and, when CaseInsensitive is set, an additional <field>_lowercase sibling used only for matching. The KQL lowering marks a restriction case-insensitive; each backend searches the sibling and lowercases the query value the same way the sibling is precomputed at index time (Go strings.ToLower on both sides, so non-ASCII stays consistent).
Search always returns the case-preserved base, so the sibling never has to be read back. In bleve it is indexed but not stored, kept out of _all, and without doc values. In OpenSearch it deliberately stays in _source: excluding it would make every update-by-query script rebuild all siblings from the document via painless toLowerCase, which lowercases differently than Go and would drift from the query side. Keeping it in _source avoids that, and a lowercased copy of a name or path is negligible disk in a cluster.
The OpenSearch move script keeps the base and its sibling in sync by swapping the moved prefix in Path_lowercase and setting Name_lowercase from Go-lowercased params, so case-insensitive search still finds a file after it moves (previously the sibling went stale). bleve re-indexes the whole document on move/delete/restore, so its siblings stay fresh for free.
This also repairs OpenSearch path search (the query value was no longer folded to lowercase, so path:<Foo> returned nothing) and makes bleve path queries match a folder and its descendants like OpenSearch's path_hierarchy. The Path base stays case-preserved so the move/delete descendant update (an exact TermQuery on Path) matches mixed-case folders.
OpenSearch lowercased every KQL query value, so exact-match queries on
case-preserved keyword fields (facet values, ids) never matched their stored
token. Fold the value only for fields with a lowercasing analyzer, mirroring the
bleve backend. The field set is derived once in search.LowercaseValueFields and
shared by both backends (bleve's local buildLowercaseFields is dropped).
Build the bleve and OpenSearch index mappings from the Go struct via
reflection (json tags + per-field overrides) instead of hand-rolled
mappings and hit deserializers. New mapping package: BleveBuildMapping,
OpenSearchBuildMapping, Deserialize[T], PrepareForIndex; field decoding is
fail-soft. Mtime is typed as a date so mtime ranges are chronological on
both backends. Route CS3 facet parsing through mapping.DeserializeStringMap.
The any-valued (bleve hit) and string-valued (CS3 metadata) deserializers
share one generic fillStruct walker with a per-value setLeaf callback.
both engines now agree on names, titles, tags, paths, types, sizes, dates,
hidden flags, facet values and wildcards. quotes only delimit phrases and
the equals operator matches the whole field value, following the kql spec.
the index name carries a generation so a changed mapping starts on a fresh
index, MIGRATION.md says how to fill it.
Bring the membership lookup in line with the existing repo convention
for set types (see services/thumbnails/pkg/thumbnail/mimetypes.go for
the same pattern). Storing struct{} values instead of bool makes the
set semantics explicit and rules out accidental false entries.
Copilot review pointed out that the comment claimed pre-lowercasing
makes non-analyzed query types (wildcard, fuzzy) match for every
allowlisted field. That is true for Name/Tags/Favorites, whose
lowercaseKeyword analyzer emits a single lowercased token, but the
Content analyzer also stems terms — so the guarantee doesn't hold
there. Drop the specific claim and keep the comment to the intent:
stay consistent with the field's analyzer.
The bleve compiler lowercased every query value (except Hidden)
before handing it to the engine. This matched the index tokens
for fields whose analyzer folds case — Name, Tags, Favorites,
Content — but silently broke matching for every other field,
whose default keyword analyzer preserves case. A query like
Title:"Some Title" parsed fine, lowercased to "some title", and
missed the indexed token "Some Title".
Replace the blanket lowercasing with an allowlist of the four
fields whose index mapping actually uses a lowercasing analyzer.
Every other field now passes through unchanged, which keeps
values like "deadmau5" or "Motörhead" intact instead of
normalising them to a case the tag writer didn't choose.
* enhancement: use kql as default search query language
* enhancement: add support for unicode search queries
* fix: escape bleve field query whitespace
* fix: search related acceptance tests
* enhancement: remove legacy search query language
* enhancement: add support for kql dateTime restriction node types
* chore: bump web to v8.0.0-alpha.2
* fix: failing search api test
* enhancement: search bleve query compiler use DateRangeQuery as DateTimeNode counterpart
* enhancement: support for colon operators in dateTime kql queries
* feat(search): introduce search query package
With the increasing complexity of how we organize our resources, the search must also be able to find them using entity properties.
The query package provides the necessary functionality to do this.
This makes it possible to search for resources via KQL, the microsoft spec is largely covered and can be used for this.
In the current state, the legacy query language is still used, in a future update this will be deprecated and KQL will become the standard
Dominik Schmidt