The Tika preview extractor does a network call; it was using context.Background().
Pass the request context from the gRPC handler through Convert so the unpack call
respects cancellation and deadlines. Non-I/O decoders ignore the context.
Tika sniffs TIFF-based raws (NEF/DNG/ARW/PEF) as generic image/tiff by
content; only the filename extension routes them to the raw parser. Thread
the source name through to the unpack request via Content-Disposition.
Replace the in-process TIFF/BigTIFF walker (rawtiff.go) with a Tika-based
extractor: raw images are sent to a Tika server's /unpack endpoint, which
returns the embedded JPEG previews (Tika's RawTiffParser emits them as embedded
documents). The largest renderable JPEG is decoded through the image pipeline.
This aligns preview extraction with the search service (same Tika, same format
coverage) and drops a large hand-rolled parser. Raw thumbnails now require
THUMBNAILS_TIKA_TIKA_URL; without it, raw types fall back to the default decoder
(no thumbnail).
Audio artwork extraction stays in process for now and moves in a follow-up.
isLongType accepted only LONG/LONG8, so a SubIFDs entry using the dedicated
IFD (13) or, in BigTIFF, IFD8 (18) pointer type (both standards-compliant and
common) was skipped and its embedded preview never discovered. Accept those
pointer types for the SubIFDs tag via isSubIFDType, keeping the stricter
isLongType on length fields so they never accept a pointer type. IFD8 reads at
8-byte width like LONG8.
Review of #3332 surfaced two BigTIFF-only defects in the embedded-preview
walker (attacker-controlled input):
- 64-bit IFD and SubIFD-array offsets were bounds-checked with `off + n > len`,
which wraps for a crafted offset near 2^64, bypassing the guard and slicing
out of range -> panic (recovered by the framework into a 500 + stack log on
every crafted request, defeating the no-panic goal). Now overflow-safe.
- a tag value was always read as an 8-byte Uint64 in BigTIFF; a LONG (4-byte)
offset in a big-endian BigTIFF was thereby shifted. Read it at its declared
type width instead.
Adds tests for the overflow paths (assert ErrNoImageFromRawFile, no panic),
the big-endian LONG offset, and strengthens the truncation test to assert the
error. Trims a few over-long comments.
Add big-endian, largest-candidate, orientation-direction, isRenderableJPEG
classification and maxPreviewLength tests plus a ForType raw-dispatch check;
log a raw file without an embedded preview at debug instead of error; name the
TIFF magic constant and de-enumerate the decoder doc comment.
Reject a preview whose declared length exceeds 100MB rather than serving it;
previews are camera-generated JPEGs, so this bounds the output independently
of the input file size.
Both are TIFF-based camera raw formats handled by the same generic walker;
NRW detection needs the reva mimetype mapping in opencloud-eu/reva#773, SRF is
already mapped. No sample files on hand for these two, added by format
structure alongside their verified siblings.
Cap the IFD queue at maxIFDs so a file packed with huge SubIFD counts can no
longer grow it without bound, and walk the embedded JPEG by a bounded number
of header segments so a preview whose SOF sits past 64KB of leading metadata
is still found instead of being scanned only within a fixed byte window.
Rename RawImageDecoder to RawTiffDecoder and qualify candidates by their
actual stream instead of trusting the tags: CR2 keeps its full-size JPEG
as an IFD0 strip, DNG stores the raw sensor payload as lossless JPEG
(SOF3) which starts with a regular SOI. A JPEG segment walk to the SOF
marker accepts only the DCT processes common decoders render.
* add configuration options to enable the NATS server HTTP monitoring
endpoint:
- NATS_MONITORING_ENABLED (defaults to false)
- NATS_MONITORING_HTTP_HOST (defaults to 127.0.0.1)
- NATS_MONITORING_HTTP_PORT (defaults to 8222)
- NATS_MONITORING_ENABLE_TLS (defaults to false)
Fixes#3294
This config specifies a URL for a yjs server that is used for document
collaboration in web apps (e.g. tiptap editor, excalidraw, ...). It's
optional, not setting this disables this feature in Web.
The command no longer times out after 10 minutes when the processing
hasn't been finished yet. Instead it shows progress and logs successful
and failed spaces. The process can also be aborted using ctrl+c.
Example output:
```
$ bin/opencloud search index --all-spaces --insecure --force-rescan
[1/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$eb6a8a0e-2e62-4b40-bce9-360063236676!eb6a8a0e-2e62-4b40-bce9-360063236676 in 5.501190908s
[2/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$34b7c565-da3b-4f53-8559-825e87db4011!34b7c565-da3b-4f53-8559-825e87db4011 in 5.935075697s
[3/602] indexed space a9033d65-6c13-4556-b923-f321ab33a9ca$e08c6026-24f6-4a9e-a805-ed335b34b8da!e08c6026-24f6-4a9e-a805-ed335b34b8da in 6.016350958s
^Caborted, indexing has been stopped
```
Fixes#2592
This speeds up reindexing all spaces which happend sequentially space
after space until now. The level of concurrency can be configured using
the SEARCH_REINDEX_CONCURRENCY env var (3 by default).
Dominik Schmidt