The refuse specs use registered analyzers (fulltext is gone), the golden regenerates via UPDATE_GOLDEN, MappingGetResp grew an accessor, and the parity suite passes the new NewBackend signature.
The opensearch-go bump renamed the mapping-get accessor, the schema check
takes a context and a logger now, and the golden bleve mapping carries the
word-broken Name and Title.
- export mapping.SortedUnionKeys and reuse it in bleve.compareKeysExcept
instead of a copied union-of-keys block
- add Classification.AddBreaking to fold engine-specific breaking reasons and
force the verdict, replacing the identical block in the bleve and opensearch
Classify paths
- shorten the multi-line doc comments flagged as too verbose
- add reconcile_test.go: direct unit tests for Reconcile incl. the
persisted-but-errored and classify-error branches (previously only
reached indirectly through the engine integration tests)
- convert the 11 near-identical Classify It blocks to a DescribeTable
- the additive warnings advertise --all-spaces --force-rescan; a plain
walk skips unchanged documents and never backfills the new fields
- Apply checks index existence first again, so a pre-provisioned index
needs no create privilege and odd create-error shapes (string error
bodies, cluster blocks) cannot fail a healthy startup; Create on 404
keeps the typed already-exists swallow as the creation-race backstop
- number_of_replicas drift is not breaking, it is runtime-tunable and
needs no rebuild
- bleve returns the classification alongside post-persist errors and
the server warns before the error check, so the one-time additive
warning is not lost when close or reopen fails
- a golden fixture pins the marshaled bleve mapping so a dependency
bump that changes marshaling fails in CI instead of refusing every
installation in the field
Addresses the two Copilot review comments on the PR: the additive
opensearch log now matches the bleve warning (level and re-index hint),
and the refuse message spells out how to delete the index per engine
(DELETE /<name> vs removing the bleve directory).
Both engines now diff the stored/live index schema against the schema
generated from code when the service starts. A shared recursive
classifier in the mapping package is the single oracle:
- equal: start normally.
- additive (new fields without any indexed data): applied in place.
OpenSearch gets a PUT _mapping with the full code properties, bleve
persists the code mapping into the index (SetInternal + reopen) so
the new fields are properly typed immediately and later startups
classify equal. A startup warning lists the new fields because
documents indexed before the upgrade lack them until re-indexed.
- breaking (changed definitions or analyzers, removed or renamed
fields, or new fields that already contain data of unknown form):
refuse to start with an error describing the rebuild procedure
(delete the index, start, run "opencloud search index --all-spaces")
and the OC_EXCLUDE_RUN_SERVICES=search escape hatch.
PUT _mapping is deliberately only the apply mechanism, never the
judge: its merge semantics cannot see removals or renames and it
accepts in-place updatable param changes with an ack. bleve
additionally checks idx.Fields() so previously dynamically indexed
data (which leaves no schema trace in bleve) is caught, matching by
exact name and by path prefix.
While at it: the OpenSearch startup check runs with a real,
minute-bounded context instead of context.TODO(), bleve indexes are
opened with a 5s bolt_timeout so a second process fails fast instead
of hanging on the file lock, and the reversed errors.Is arguments in
bleve.NewIndex were fixed.
https://github.com/opencloud-eu/opencloud/issues/3092
* add the ability to disable the gRPC API handler
(POLICIES_GRPC_DISABLED)
* add the ability to disable the Events API handler
(POLICIES_EVENTS_DISABLED)
* add metrics
* add support for specifying rego files via the environment varirable
POLICIES_ENGINE_FILES
* for file paths specified in yaml or in POLICIES_ENGINE_FILES, support
'config:' and 'data:' path prefixes
* fix typos in the documentation, and try to make it more clear
* add metrics to the documentation
* add a section for testing in the documentation
* introduces a new top-level package pkg/metrics/ with utilities for
metrics that are backported from the groupware branch, with unit
tests
The delete-by-query matched the bare Path, so a purge could take same-path documents in other spaces with it; bleve was already RootID-scoped. Pinned as rootscope-04.
The OpenSearch _lowercase siblings are search-only like bleve's (no doc_values, own map instance instead of aliasing the base), and the dead Path_words field is gone: _words exists for keyword fields only, as SearchSiblings declares.
The file expansion is grouped again so its NOT stays atomic next to other terms (OpenSearch turned 'mediatype:file OR x' into '(NOT dir) AND x'), and the bleve compiler no longer re-keys resolved groups (a negated mediatype group targeted the raw 'mediatype' field and matched everything). Pinned as MEDIATYPE-07..10.
The engine suites keep what is engine-specific (index setup, health, purge-space batching); every behavior answer lives in the parity suite once. The shared test client learns IndicesCount, and FIELDS-15 pins the Size/Type gate for number queries.
Dominik Schmidt