scope: takes an opaque resource id, which is hostile to hand-written
queries. Accept driveId:"<storage$space>" as a regular KQL field
instead: it resolves to the indexed RootID, and a bare drive id is
completed to the root resource id (a space root's opaque id is its
space id). Full root ids pass through untouched.
Combined with path: this gives a readable location scope without any
token stripping: both are plain fields, so they compose with groups,
OR and NOT like everything else.
* add the ability to disable the gRPC API handler
(POLICIES_GRPC_DISABLED)
* add the ability to disable the Events API handler
(POLICIES_EVENTS_DISABLED)
* add metrics
* add support for specifying rego files via the environment varirable
POLICIES_ENGINE_FILES
* for file paths specified in yaml or in POLICIES_ENGINE_FILES, support
'config:' and 'data:' path prefixes
* fix typos in the documentation, and try to make it more clear
* add metrics to the documentation
* add a section for testing in the documentation
* introduces a new top-level package pkg/metrics/ with utilities for
metrics that are backported from the groupware branch, with unit
tests
The delete-by-query matched the bare Path, so a purge could take same-path documents in other spaces with it; bleve was already RootID-scoped. Pinned as rootscope-04.
The OpenSearch _lowercase siblings are search-only like bleve's (no doc_values, own map instance instead of aliasing the base), and the dead Path_words field is gone: _words exists for keyword fields only, as SearchSiblings declares.
The file expansion is grouped again so its NOT stays atomic next to other terms (OpenSearch turned 'mediatype:file OR x' into '(NOT dir) AND x'), and the bleve compiler no longer re-keys resolved groups (a negated mediatype group targeted the raw 'mediatype' field and matched everything). Pinned as MEDIATYPE-07..10.
The engine suites keep what is engine-specific (index setup, health, purge-space batching); every behavior answer lives in the parity suite once. The shared test client learns IndicesCount, and FIELDS-15 pins the Size/Type gate for number queries.
From #3408: hidden takes bool words only, type categories map to the stored value in the shared pass, ? counts as a wildcard, a non-suffix wildcard on a word-broken field forgives the extension, = matches the whole value on the lowercased sibling, paths lose their trailing slash. Dead compiler helpers removed.
SharePoint's default for a text property is word breaking, so ours is too:
every keyword field gets the _words sibling unless it opts out with
NoWordBreaker, which now carries SharePoint's polarity as well. Artist,
album, camera model and the other facets match by word like name and title;
tags and favorites stay one label, ids, paths and the mime type one value.
KQL searches case-insensitively, so every keyword field gets its lowercased
search sibling unless it opts out: ids are opaque, paths are POSIX, the mime
type is normalized already. That takes the facets along, artist or camera
model match regardless of case, while the case-preserved base still answers
and aggregates. Which siblings a field carries is decided once, from the
struct and the overrides, and the renderers, the document writer and the
query lowering all read it from there.
A single word finds the names and titles that contain it, `report` finds
Report.txt, on bleve as well as on OpenSearch, which so far only did it by
accident of its dynamic mapping. Modelled like SharePoint's NoWordBreaker:
a keyword field is one whole value unless the override switches that off,
which adds a search-only _words sibling next to _lowercase, analyzed into
lowercased words (a dot is a word boundary, no stemming). The base stays the
whole value for returning and aggregating, wildcards and whole values keep
using _lowercase. Quotes do not change the meaning, a phrase is a phrase
either way, and there is no exact-match operator yet.
Paths act as references (location scoping, deep links): /Foo and /foo
are distinct siblings, so path: matching must be exact. Case-insensitive
folder discovery is served by name: and its lowercase sibling. This also
matches bleve on main, where path queries have always been
case-sensitive.
Dropping the sibling removes its biggest maintenance cost: Path is the
one mutable sibling field, a move rewrites the paths of a whole subtree
and the OpenSearch move script had to rebuild Path_lowercase alongside
the base field.
With paths case-sensitive, the ref path scope moves into the query
itself: bleve as a term/prefix disjunction on the keyword Path,
OpenSearch as a term filter on its path_hierarchy tokens. The
post-filter that used to drop out-of-scope hits after the query ran is
gone; totals and paging now respect the scope instead of being computed
over the whole space, and a wrong-cased scope simply matches nothing.
A leading NOT next to an operator was miscompiled: the bleve compiler left the consumed term in `next`, so `NOT x AND y` dropped `y` and produced a self-contradicting clause; the OpenSearch transpiler checked nextOp==AND before prevOp==NOT, so the negated term landed in `must` instead of `must_not`. NOT is unary and binds to the node directly after it regardless of what follows. This also fixes `mediatype:file AND <term>` (the web Files filter) at the root, so the earlier mediatype:file group workaround is dropped.
A path value with spaces went through a match_phrase query, which analyzes
the query with the path_hierarchy analyzer; the resulting "." prefix token
matches every document in the space, breaking descendant matching and the
stale-path check after a move.
The move script rewrote Path/Path_lowercase with painless String.replace, which replaces every occurrence of the old path, not just the leading prefix. OpenCloud paths are ./-prefixed so the full old path only occurs at the start and the result is byte-identical, but startsWith + substring makes the prefix-only intent explicit and robust to any path format. Not a live bug fix, a hardening.
The guard only rejected CaseInsensitive when a non-keyword/path Type was set explicitly. With no Type, isCasedType treated the field as cased, so CaseInsensitive on an inferred numeric/bool/datetime field passed validation but produced no _lowercase sibling, and the query would silently match nothing. Validate now falls back to the inferred Go type.
mediatype:file expands to a NOT restriction. Spliced inline as `NOT MimeType:httpd/unix-directory`, the bleve compiler's NOT branch left a stale operand, so `mediatype:file AND name:x` dropped `name:x` and matched nothing (the web Files filter). It is now wrapped in a group so the negation stays atomic; verified fixing both bleve and OpenSearch.
resolveField marked every anonymous field embedded, so walkFields (mapping, field index, validate) and fillStruct (deserializer) flattened a json-tagged embedded struct, while conversions.To/encoding/json on the write path nests it under the tag, mapping and deserializing it at the wrong path. An anonymous field is now embedded only without a json tag name, matching encoding/json; fillStruct also recurses into a value nested struct. No current type has a tagged embedded struct, so runtime behavior is unchanged; this hardens the reflection walker.
mediatype:Folder / mediatype:IMAGE resolved to a literal MimeType search and matched nothing because Expand switched on the raw value. The value is now lowercased in the lowering pass, so categories and literal MIME types match regardless of case, consistently on both backends.
Adds bleve and OpenSearch coverage for category (image), literal MIME (image/svg+xml, with + and /), and raw MimeType: queries. Documents why MimeType skips the bleve escaper: it is not a bug, bleve treats / and + as literals mid-term, so a literal MIME still matches exactly while the category wildcard image/* keeps its *.
CaseInsensitive routes queries to a <field>_lowercase sibling that is only generated for keyword/path fields, so marking any other type CaseInsensitive would silently match nothing. Validate now rejects it up front.
bleve compiled a path restriction to a DisjunctionQuery, which mapBinary redistributes as an OR-chain, so `path:/Foo AND name:bar` matched the folder itself unconditionally. It is now a BooleanQuery (should: folder OR descendants), which mapBinary keeps atomic under an enclosing AND.
The OpenSearch full-text branch ran before the wildcard check, so `content:foo*` degraded to a phrase match and diverged from bleve; the wildcard check now comes first.
Adds the missing coverage the review flagged: path AND term, content wildcard, case-insensitive tags (the array sibling branch), and a spaced path with descendants on OpenSearch.
Single-term `content:` built an unanalyzed term query, so once this branch dropped the blanket query-value lowercasing, `content:Foo` missed on OpenSearch (bleve was unaffected, its query analyzes). Fielded full-text queries now use a match query. OpenSearch `Content` also gets a porter stemming analyzer (it used the default standard analyzer and never stemmed), so full-text search matches bleve on both case and stemming.
Keyword and path fields always index their case-preserved base and, when CaseInsensitive is set, an additional <field>_lowercase sibling used only for matching. The KQL lowering marks a restriction case-insensitive; each backend searches the sibling and lowercases the query value the same way the sibling is precomputed at index time (Go strings.ToLower on both sides, so non-ASCII stays consistent).
Search always returns the case-preserved base, so the sibling never has to be read back. In bleve it is indexed but not stored, kept out of _all, and without doc values. In OpenSearch it deliberately stays in _source: excluding it would make every update-by-query script rebuild all siblings from the document via painless toLowerCase, which lowercases differently than Go and would drift from the query side. Keeping it in _source avoids that, and a lowercased copy of a name or path is negligible disk in a cluster.
The OpenSearch move script keeps the base and its sibling in sync by swapping the moved prefix in Path_lowercase and setting Name_lowercase from Go-lowercased params, so case-insensitive search still finds a file after it moves (previously the sibling went stale). bleve re-indexes the whole document on move/delete/restore, so its siblings stay fresh for free.
This also repairs OpenSearch path search (the query value was no longer folded to lowercase, so path:<Foo> returned nothing) and makes bleve path queries match a folder and its descendants like OpenSearch's path_hierarchy. The Path base stays case-preserved so the move/delete descendant update (an exact TermQuery on Path) matches mixed-case folders.
query.Normalize resolves field names (query.ResolveField, from the derived
index + a small alias overlay) and expands media-type restrictions
(mimetype.Expand) once, between parse and backend compilation.
The KQL parser produced its own validation errors but imported them from the
search service's query package. Move them into pkg/kql and let the search
backend consume kql.IsValidationError, so the parser stops depending on a
service package.
OpenSearch lowercased every KQL query value, so exact-match queries on
case-preserved keyword fields (facet values, ids) never matched their stored
token. Fold the value only for fields with a lowercasing analyzer, mirroring the
bleve backend. The field set is derived once in search.LowercaseValueFields and
shared by both backends (bleve's local buildLowercaseFields is dropped).
Both backends carry a shared search.SchemaVersion in the index name
(OpenSearch <base>-vN) and data path (bleve-vN). A breaking schema change
bumps the version so the service builds a fresh index instead of colliding
with the incompatible previous one; the old index is left in place.
Dominik Schmidt