- the additive warnings advertise --all-spaces --force-rescan; a plain
walk skips unchanged documents and never backfills the new fields
- Apply checks index existence first again, so a pre-provisioned index
needs no create privilege and odd create-error shapes (string error
bodies, cluster blocks) cannot fail a healthy startup; Create on 404
keeps the typed already-exists swallow as the creation-race backstop
- number_of_replicas drift is not breaking, it is runtime-tunable and
needs no rebuild
- bleve returns the classification alongside post-persist errors and
the server warns before the error check, so the one-time additive
warning is not lost when close or reopen fails
- a golden fixture pins the marshaled bleve mapping so a dependency
bump that changes marshaling fails in CI instead of refusing every
installation in the field
Both engines now diff the stored/live index schema against the schema
generated from code when the service starts. A shared recursive
classifier in the mapping package is the single oracle:
- equal: start normally.
- additive (new fields without any indexed data): applied in place.
OpenSearch gets a PUT _mapping with the full code properties, bleve
persists the code mapping into the index (SetInternal + reopen) so
the new fields are properly typed immediately and later startups
classify equal. A startup warning lists the new fields because
documents indexed before the upgrade lack them until re-indexed.
- breaking (changed definitions or analyzers, removed or renamed
fields, or new fields that already contain data of unknown form):
refuse to start with an error describing the rebuild procedure
(delete the index, start, run "opencloud search index --all-spaces")
and the OC_EXCLUDE_RUN_SERVICES=search escape hatch.
PUT _mapping is deliberately only the apply mechanism, never the
judge: its merge semantics cannot see removals or renames and it
accepts in-place updatable param changes with an ack. bleve
additionally checks idx.Fields() so previously dynamically indexed
data (which leaves no schema trace in bleve) is caught, matching by
exact name and by path prefix.
While at it: the OpenSearch startup check runs with a real,
minute-bounded context instead of context.TODO(), bleve indexes are
opened with a 5s bolt_timeout so a second process fails fast instead
of hanging on the file lock, and the reversed errors.Is arguments in
bleve.NewIndex were fixed.
https://github.com/opencloud-eu/opencloud/issues/3092
Both backends carry a shared search.SchemaVersion in the index name
(OpenSearch <base>-vN) and data path (bleve-vN). A breaking schema change
bumps the version so the service builds a fresh index instead of colliding
with the incompatible previous one; the old index is left in place.
Address review feedback: now that the flag is read under its
registered name `force-rescan`, line the local variable up with the
operator-facing vocabulary. The proto field `ForceReindex` is left
untouched so the wire format stays the same.
The `opencloud search index` command registers the flag as
`--force-rescan` (see pflag registration below) but reads it via
`GetBool("force-reindex")`, so the value is always false — passing
`--force-rescan` had no effect and no force rescan was ever triggered.
Read the flag under its registered name.
That can be helpful when the search service configuration has changed,
e.g. by enabling TIKA. Previously files that had already been indexed
were not indexed again and thus were no part of the fulltext index.
Fixes#2285Fixes#2578
* unmarshal token to filter spaces for current user
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
* Fix tests
---------
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Co-authored-by: André Duffeck <andre.duffeck@firondu.de>
* experimental search backport
fix basic extractor resource name
move escapeQuery regex into global variable
minor pr review changes
rename DebounceDuration env variable
add document title and content when rebuilding bleve resource
Co-authored-by: David Christofas <dchristofas@owncloud.com>
* Introduce TLS Settings for go-micro based grpc services and clients
TLS for the services can be configure by setting the OCIS_MICRO_GRPC_TLS_ENABLED"
"OCIS_MICRO_GRPC_TLS_CERTIFICATE" and "OCIS_MICRO_GRPC_TLS_KEY"
enviroment variables.
TLS for the clients can configured by setting the "OCIS_MICRO_GRPC_CLIENT_TLS_MODE"
and "OCIS_MICRO_GRPC_CLIENT_TLS_CACERT" variables.
By default TLS is disabled.
Co-authored-by: Martin <github@diemattels.at>
* Unify TLS configuration for all grpc services
All grpc service (whether they're based on reva) or go-micro use the
same set of config vars now.
TLS for the services can be configure by setting the OCIS_GRPC_TLS_ENABLED,
OCIS_GRPC_TLS_CERTIFICATE and OCIS_GRPC_TLS_KEY enviroment variables.
TLS for the clients can configured by setting the OCIS_GRPC_CLIENT_TLS_MODE
and OCIS_MICRO_GRPC_CLIENT_TLS_CACERT variables.
There are no individual per service config vars currently. If really
needed, per service tls configurations can be specified via config file.
Co-authored-by: Martin <github@diemattels.at>
Co-authored-by: Martin <github@diemattels.at>
* Increase the timeout for indexing spaces
* Allow for making queries that are not constrained to a RootID
* Use nano precision for the Mtime
* Add a SpaceDebouncer
The debouncer can be used to delay operations on spaces until things
have settled down to avoid doing the same operation multiple times.
* Do not index subtrees until they have changed (i.e. the mtime differs)
* Also pass a user to the space debouncer func
* Trigger a rescan of the according space when an event is received
* Improve wording/logging
* Add changelog
* Get rid of superfluous GetUserByClaim call
* Fix tests
* Fix reindexing triggered by shares by using the event's SpaceOwner
* Bump reva to pull in the space owner events changes
* Fix changelog
* Fix linter issues
* Fall back to the executant if no owner was received from the event
* Bump reva and go-cs3apis
* Fix go.sum
* Bump core
* Adapt expected failures
* Tweak debounce settings
This avoids using a public global variable. It allows us to initialize
the default client a bit later (outside of init()). That way we can e.g.
properly initialize the in-memory registry.
The help flag is configured automatically by default already. We don't
need to redo that for every single service.
This also addresses one of the finding of "go race" (#4088)
