Commit Graph
479 Commits
Author SHA1 Message Date
Andre Duffeck 8f2adf2138 Reindex disabled spaces once they get enabled again (#3579)
Spaces that are disabled while reindexing are skipped. They are now
remembered in a NATS KV bucket (including whether a forced rescan was
requested) and reindexed when the SpaceEnabled event comes in. The
entry is removed when the space gets deleted.
2026-10-05 14:28:58 +02:00
Dominik Schmidt 63c1f15cee fix(search): take the photo iso from exif:IsoSpeedRatings
Base ISO is a Canon maker note tag holding Canon's raw encoded value, not the iso of the shot: other cameras got no iso, Canon photos a wrong one (256 for an ISO 800 shot). exif:IsoSpeedRatings is there on tika 3 and tika 4.
2026-10-03 11:12:18 +02:00
Dominik Schmidt 4006344505 fix(search): read the tika 4 live photo keys
Tika 4 prefixes its metadata keys (mp4:, img:), so the live photo facet was never built there. Live photo support needs tika 4: tika 3 got the video keys only in 3.3.2 and never the still image time. The content key is read through one lookup, marked tika 4 and tika 3 legacy.
2026-10-03 11:12:17 +02:00
Dominik Schmidt b2039f6abe fix(search): ellipsize parity matrix labels by runes, not bytes
The byte-based truncation cut through the two-byte UTF-8 sequence of
Motörhead's ö, committing invalid UTF-8 into the parity README. Shared
rune-based ellipsize helper; matrix regenerated.
2026-10-02 12:48:51 +02:00
André Duffeck b1efb8c616 Pass the query into the bleve descendant walk
Instead of deleted/onlyDeleted bool parameters, the caller now builds
the query and forEachMatch only does the paging.
2026-09-28 11:49:47 +02:00
André Duffeck ad4c8ad112 fix(search): keep trashed and live folders at the same path apart
A trashed folder keeps its path in the index, so a new folder can take
the same one. The descendant lookup behind move, delete, restore and
purge matched on space and path only: purging the trashed folder took
the live folder and its descendants out of the index, and moving the
live folder carried the trashed tree along.

Both engines now also match the root's trash state; purge with
onlyDeleted keeps matching the trashed ones only.
2026-09-25 15:31:31 +02:00
Andre Duffeck 74462d32d7 Merge pull request #3495 from opencloud-eu/feat/search-hierarchy-tokenizer
refactor(search): stream bleve descendants
2026-09-25 15:19:52 +02:00
Andre Duffeck cf1737babe Merge pull request #3493 from opencloud-eu/cleanup-split-changes
align the handler verifier, update docs
2026-09-25 14:27:03 +02:00
Andre Duffeck 65741f00bf Merge pull request #3556 from aduffeck/index-space-auth-session
Use an auth.Session for indexing spaces
2026-09-21 10:27:21 +02:00
André Duffeck be3c7b674e Fix test 2026-09-18 12:01:48 +02:00
André Duffeck 8600cddb8f Switch to jwt v5 2026-09-18 12:01:32 +02:00
André Duffeck f6104a8dc5 Use an auth.Session for indexing spaces
This makes sure that the token does not expire if the indexing process
takes longer than the token lifetime.

This commit also makes sure that the authenticated context is reused
instead of generating a new context for each resource which is much more
efficient.
2026-09-17 09:57:30 +02:00
Pascal Wengerter 002a2e43fb chore: regenerate mockery outputs 2026-09-16 14:57:41 +02:00
André Duffeck fe5fd66cb6 Remove the timeout when reindexing spaces
This was supposed to be removed with #3213 but was either missed or lost
in a rebase.
2026-09-16 10:14:52 +02:00
Dominik Schmidt c24333fb79 refactor(search): stream bleve descendants
Move, delete, restore and purge walk a folder in pages of 20k sorted by id instead of collecting every descendant first: 133 MB -> 19 MB live heap at 100k files, flat over folder size. The walker yields the folder itself, the batch skips it by id.
2026-09-11 11:12:33 +00:00
Dominik Schmidt 9cc7757221 feat(search): hierarchy tokenizer for bleve path fields
Path was a keyword, so the descendant lookup behind delete/move/restore/purge, the scoped search and the KQL path predicate expanded into one term searcher per descendant and OOM-killed the server on large folders (#1269, #3469).

Path is now analyzed into its ancestor prefixes, like path_hierarchy in OpenSearch: ./a/b.txt becomes ., ./a, ./a/b.txt. A folder's descendants are every document carrying the folder's path as a term, so all three call sites are a single term query. Schema 4 -> 5, v4 never shipped.

The same tokenizer with tag_depth is registered as the geohash analyzer, so #3272 can add its geohash field without another schema change.
2026-09-10 18:42:31 +00:00
Viktor Scharf e503c2c5eb add insecure to search reindex (#3505) 2026-09-10 16:29:37 +02:00
Jörn Friedrich Dreyer 1bafa4bd18 align the handler verifier, update docs
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
2026-09-08 17:11:55 +02:00
Dominik Schmidt be83ed19e3 fix(search): take extraction facets from the main document only
The recursive tika response lists the file first, then its embedded
resources (cover art, thumbnails, the clip appended to a motion photo).
The loop applied getImage/getPhoto/getLocation/getAudio/getLivePhoto to
every part, so an mp3's embedded cover art leaked a 200x200 image facet
onto the track (and an embedded EXIF image would leak photo/location).
Read those facets from metas[0] only, like the video facet already does;
the loop now only concatenates title/content and detects the motion
photo clip.
2026-09-08 14:52:35 +02:00
Dominik Schmidt 5995938c34 chore(search): trim the live photo comments 2026-09-03 03:34:33 +02:00
Dominik Schmidt 0a4f3ed786 fix(search): keep livePhoto.contentId out of the search siblings 2026-09-03 03:34:33 +02:00
Dominik Schmidt a7214b3d73 fix(search): drop the live photo facet on an empty pairing id 2026-09-03 03:34:33 +02:00
Dominik Schmidt 81b0df1ad4 feat(search): live photo facet 2026-09-03 03:34:33 +02:00
Dominik Schmidt 3515594787 fix(search): keep the video facet on the file itself 2026-09-03 01:09:41 +02:00
Dominik Schmidt a8318fe03b refactor(search): take the motion photo video size from the video tika extracted 2026-09-03 00:32:07 +02:00
Dominik Schmidt 706aaf02f2 refactor(search): decide the motion photo from the file xmp and its extracted video 2026-09-03 00:32:06 +02:00
Dominik Schmidt 88f21d804b refactor(search): restore cs3 Retrieve, the download split had only one caller left 2026-09-03 00:31:40 +02:00
Dominik Schmidt 8aebf80124 refactor(search): confirm the motion photo video by type, not by name 2026-09-03 00:31:40 +02:00
Dominik Schmidt ae9fd595d9 chore(search): drop the dead legacy tika key 2026-09-03 00:31:40 +02:00
Dominik Schmidt 709962b616 feat(search): confirm the motion photo video via tika instead of reading bytes 2026-09-03 00:31:40 +02:00
Dominik Schmidt a2050d1fb9 feat(search): index and expose the motion photo facet 2026-09-03 00:31:40 +02:00
Dominik Schmidt 0f4b95b9ae feat(search): add RetrieveRange to the content retriever 2026-09-03 00:27:04 +02:00
Dominik Schmidt 42876ca61c refactor(search): let getFirstValue try multiple metadata keys 2026-09-03 00:27:04 +02:00
Jörn Friedrich Dreyer f5a064395a Merge pull request #3201 from opencloud-eu/feat/search-video-facet
feat(search): video facet
2026-09-03 00:20:02 +02:00
amir 67124a6c37 fix(search): ignore invalid exposure denominators 2026-09-01 20:24:44 +03:30
amir db4cee156b fix(search): ignore invalid photo exposure times 2026-09-01 20:24:19 +03:30
Dominik Schmidt 727c986de8 feat(search): index and expose the video facet 2026-09-01 16:26:01 +02:00
Dominik Schmidt df8d0e8fd3 Merge pull request #3198 from opencloud-eu/feat/search-tika-facets
feat(search): extract more data from tika 4 (if available)
2026-09-01 11:45:11 +02:00
Dominik Schmidt 7d4c6d6524 Merge pull request #3424 from opencloud-eu/feat/search-scope
feat(search): scope searches to a drive via the driveId field
2026-09-01 09:36:20 +02:00
Dominik Schmidt 5e4f124e78 fix(search): find the language endpoint on tika 4, keep the legacy path 2026-09-01 09:09:55 +02:00
Dominik Schmidt a1e8aa783d fix(search): reject NaN coordinates and non-positive bitrates 2026-09-01 09:09:55 +02:00
Dominik Schmidt ae9ec4ce70 fix(search): harden facet value parsing 2026-09-01 09:09:55 +02:00
Dominik Schmidt c03b9c7229 fix(search): gate the audio facet inside getAudio, keep facets across meta entries 2026-09-01 09:09:55 +02:00
Dominik Schmidt 1be01ae50c fix(search): read the tika 4 content key, keep the legacy one
Tika 4.0.0 renamed the meta prefix from X-TIKA: to tk:.
2026-09-01 09:09:55 +02:00
Dominik Schmidt 2b4a6fd739 feat(search): populate audio and gps-altitude facets from tika, drop the audio-only gate 2026-09-01 09:09:55 +02:00
Dominik Schmidt 7e0f536b83 refactor(search): split the tika extractor and its tests into per-facet files 2026-09-01 09:09:55 +02:00
Jörn Friedrich Dreyer 8028e3e310 Merge pull request #3435 from fschade/test-search-cjk-parity
test(search): cover chinese search in the parity suite
2026-09-01 08:24:34 +02:00
Dominik Schmidt 7d75b98c69 Merge pull request #3423 from opencloud-eu/test/parity-readme-staleness
test(search): fail the parity suite when the committed matrix is stale
2026-09-01 08:24:15 +02:00
Florian Schade 9e8403818c test(search): cover chinese search in the parity suite 2026-09-01 07:46:03 +02:00
Dominik Schmidt ef41429838 test(search): rename the update flag to UPDATE_SEARCH_PARITY_MATRIX 2026-09-01 07:35:15 +02:00