query.Normalize resolves field names (query.ResolveField, from the derived
index + a small alias overlay) and expands media-type restrictions
(mimetype.Expand) once, between parse and backend compilation.
The KQL parser produced its own validation errors but imported them from the
search service's query package. Move them into pkg/kql and let the search
backend consume kql.IsValidationError, so the parser stops depending on a
service package.
OpenSearch lowercased every KQL query value, so exact-match queries on
case-preserved keyword fields (facet values, ids) never matched their stored
token. Fold the value only for fields with a lowercasing analyzer, mirroring the
bleve backend. The field set is derived once in search.LowercaseValueFields and
shared by both backends (bleve's local buildLowercaseFields is dropped).
Build the bleve and OpenSearch index mappings from the Go struct via
reflection (json tags + per-field overrides) instead of hand-rolled
mappings and hit deserializers. New mapping package: BleveBuildMapping,
OpenSearchBuildMapping, Deserialize[T], PrepareForIndex; field decoding is
fail-soft. Mtime is typed as a date so mtime ranges are chronological on
both backends. Route CS3 facet parsing through mapping.DeserializeStringMap.
The any-valued (bleve hit) and string-valued (CS3 metadata) deserializers
share one generic fillStruct walker with a per-value setLeaf callback.
Bring the membership lookup in line with the existing repo convention
for set types (see services/thumbnails/pkg/thumbnail/mimetypes.go for
the same pattern). Storing struct{} values instead of bool makes the
set semantics explicit and rules out accidental false entries.
Copilot review pointed out that the comment claimed pre-lowercasing
makes non-analyzed query types (wildcard, fuzzy) match for every
allowlisted field. That is true for Name/Tags/Favorites, whose
lowercaseKeyword analyzer emits a single lowercased token, but the
Content analyzer also stems terms — so the guarantee doesn't hold
there. Drop the specific claim and keep the comment to the intent:
stay consistent with the field's analyzer.
The bleve compiler lowercased every query value (except Hidden)
before handing it to the engine. This matched the index tokens
for fields whose analyzer folds case — Name, Tags, Favorites,
Content — but silently broke matching for every other field,
whose default keyword analyzer preserves case. A query like
Title:"Some Title" parsed fine, lowercased to "some title", and
missed the indexed token "Some Title".
Replace the blanket lowercasing with an allowlist of the four
fields whose index mapping actually uses a lowercasing analyzer.
Every other field now passes through unchanged, which keeps
values like "deadmau5" or "Motörhead" intact instead of
normalising them to a case the tag writer didn't choose.
* enhancement: add more kql spec tests and simplify ast normalization
* enhancement: kql parser error if query starts with AND
* enhancement: add kql docs and support for date and time only dateTimeRestriction queries
* enhancement: add the ability to decide how kql nodes get connected
connecting nodes (with edges) seem straight forward when not using group, the default connection for nodes with the same node is always OR. THis only applies for first level nodes, for grouped nodes it is defined differently. The KQL docs are saying, nodes inside a grouped node, with the same key are connected by a AND edge.
* enhancement: explicit error handling for falsy group nodes and queries with leading binary operator
* enhancement: use optimized grammar for kql parser and toolify pigeon
* enhancement: simplify error handling
* fix: kql implicit 'AND' and 'OR' follows the ms html spec instead of the pdf spec
* enhancement: add support for natural language kql date queries
* enhancement: structure kql parser tests into logical clusters
* fix: time-range error naming
* enhancement: add more kql spec tests and simplify ast normalization
* enhancement: kql parser error if query starts with AND
* enhancement: add kql docs and support for date and time only dateTimeRestriction queries
* enhancement: add the ability to decide how kql nodes get connected
connecting nodes (with edges) seem straight forward when not using group, the default connection for nodes with the same node is always OR. THis only applies for first level nodes, for grouped nodes it is defined differently. The KQL docs are saying, nodes inside a grouped node, with the same key are connected by a AND edge.
* enhancement: explicit error handling for falsy group nodes and queries with leading binary operator
* enhancement: use optimized grammar for kql parser and toolify pigeon
* enhancement: simplify error handling
* fix: kql implicit 'AND' and 'OR' follows the ms html spec instead of the pdf spec
* enhancement: use kql as default search query language
* enhancement: add support for unicode search queries
* fix: escape bleve field query whitespace
* fix: search related acceptance tests
* enhancement: remove legacy search query language
* enhancement: add support for kql dateTime restriction node types
* chore: bump web to v8.0.0-alpha.2
* fix: failing search api test
* enhancement: search bleve query compiler use DateRangeQuery as DateTimeNode counterpart
* enhancement: support for colon operators in dateTime kql queries
* feat(search): introduce search query package
With the increasing complexity of how we organize our resources, the search must also be able to find them using entity properties.
The query package provides the necessary functionality to do this.
This makes it possible to search for resources via KQL, the microsoft spec is largely covered and can be used for this.
In the current state, the legacy query language is still used, in a future update this will be deprecated and KQL will become the standard
