query.Normalize resolves field names (query.ResolveField, from the derived
index + a small alias overlay) and expands media-type restrictions
(mimetype.Expand) once, between parse and backend compilation.
The KQL parser produced its own validation errors but imported them from the
search service's query package. Move them into pkg/kql and let the search
backend consume kql.IsValidationError, so the parser stops depending on a
service package.
OpenSearch lowercased every KQL query value, so exact-match queries on
case-preserved keyword fields (facet values, ids) never matched their stored
token. Fold the value only for fields with a lowercasing analyzer, mirroring the
bleve backend. The field set is derived once in search.LowercaseValueFields and
shared by both backends (bleve's local buildLowercaseFields is dropped).
Both backends carry a shared search.SchemaVersion in the index name
(OpenSearch <base>-vN) and data path (bleve-vN). A breaking schema change
bumps the version so the service builds a fresh index instead of colliding
with the incompatible previous one; the old index is left in place.
The Mtime field is mapped as an OpenSearch `date`, which rejects an
empty value with `mapper_parsing_exception: cannot parse empty date`.
The folder and root fixtures had no Mtime, so serializing them to
`"Mtime": ""` made TestEngine_Purge/purge_resource_trees fail when the
document was indexed. Give both a valid RFC3339 Mtime, matching the
file fixture.
Add a TypeGeopoint field type. The libregraph Location facet is kept as an
object (retrieval / numeric queries) and a sibling <name>_geopoint field
carries the {lat,lon} form for geo-distance / bbox / polygon queries,
uniform across bleve and OpenSearch via the shared mapping. PrepareForIndex
splices the sibling in at write time.
Build the bleve and OpenSearch index mappings from the Go struct via
reflection (json tags + per-field overrides) instead of hand-rolled
mappings and hit deserializers. New mapping package: BleveBuildMapping,
OpenSearchBuildMapping, Deserialize[T], PrepareForIndex; field decoding is
fail-soft. Mtime is typed as a date so mtime ranges are chronological on
both backends. Route CS3 facet parsing through mapping.DeserializeStringMap.
The any-valued (bleve hit) and string-valued (CS3 metadata) deserializers
share one generic fillStruct walker with a per-value setLeaf callback.
Bring the membership lookup in line with the existing repo convention
for set types (see services/thumbnails/pkg/thumbnail/mimetypes.go for
the same pattern). Storing struct{} values instead of bool makes the
set semantics explicit and rules out accidental false entries.
The FIXME pointed at #2632 (dotted keys in KQL property restrictions),
which is now merged. Use audio.artist — the originally intended target
field for this regression — so the test matches its name: a nested
string field that is not on the lowercase allowlist.
Copilot review pointed out that the comment claimed pre-lowercasing
makes non-analyzed query types (wildcard, fuzzy) match for every
allowlisted field. That is true for Name/Tags/Favorites, whose
lowercaseKeyword analyzer emits a single lowercased token, but the
Content analyzer also stems terms — so the guarantee doesn't hold
there. Drop the specific claim and keep the comment to the intent:
stay consistent with the field's analyzer.
The bleve compiler lowercased every query value (except Hidden)
before handing it to the engine. This matched the index tokens
for fields whose analyzer folds case — Name, Tags, Favorites,
Content — but silently broke matching for every other field,
whose default keyword analyzer preserves case. A query like
Title:"Some Title" parsed fine, lowercased to "some title", and
missed the indexed token "Some Title".
Replace the blanket lowercasing with an allowlist of the four
fields whose index mapping actually uses a lowercasing analyzer.
Every other field now passes through unchanged, which keeps
values like "deadmau5" or "Motörhead" intact instead of
normalising them to a case the tag writer didn't choose.
Address review feedback: now that the flag is read under its
registered name `force-rescan`, line the local variable up with the
operator-facing vocabulary. The proto field `ForceReindex` is left
untouched so the wire format stays the same.
The `opencloud search index` command registers the flag as
`--force-rescan` (see pflag registration below) but reads it via
`GetBool("force-reindex")`, so the value is always false — passing
`--force-rescan` had no effect and no force rescan was ever triggered.
Read the flag under its registered name.
Address review feedback: a straight int64 cast truncates toward zero,
so Tika values that produce results like 1234.999... millisecond would
land at 1234 ms instead of 1235 ms. Round before casting so durations
are as accurate as float64 allows.
Tika emits xmpDM:duration as seconds in floating-point form (for
example "154.57379150390625"), so strconv.ParseInt rejected every
value and the field was silently dropped — every indexed audio item
ended up without a duration.
Parse the value with strconv.ParseFloat and convert to milliseconds
ourselves. Adjust the existing extractor test to cover the fractional
case.
That can be helpful when the search service configuration has changed,
e.g. by enabling TIKA. Previously files that had already been indexed
were not indexed again and thus were no part of the fulltext index.
Fixes#2285Fixes#2578
