Add the GEM-X native backend with live pose and root displacement output, bounded duplex WebSocket streaming, session authorization, metrics and usage accounting. Share the embedded bidirectional RPC channel adapter across streaming backends.
Include backend build and gallery integration, API documentation, schemas and tests.
Assisted-by: Codex:gpt-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
(cherry picked from commit 016dc3d71a)
With DOT, EUCLID or MANHATTAN the backend returns Qdrant's raw scores,
not similarity-like values. Router, face and voice stores compare
against cosine thresholds, so log a warning at load when another
metric is configured, and document the behaviour on the stores page.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
StoresSet sent the whole batch in one Upsert under a single
request_timeout_ms, so a large batch could hit the timeout even though
every point was fine. Send at most 500 points per request and apply
the timeout to each request.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A TLS 1.3 floor rejects Qdrant endpoints that only speak TLS 1.2, such
as some managed deployments and proxies. Use TLS 1.2 as the minimum.
When Load fails after the client was created, it closed the client but
left it set, so Free closed it a second time. Clear it after closing.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The sd_embed branch dropped negative_prompt for every pipeline, which
discarded the user's negative prompt for Flux, where no negative embeds
are produced. Only drop it when negative_prompt_embeds is present, and
move the check to a helper that has unit tests.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): preserve chat session state
Newer llama.cpp parsers return template state outside the request JSON.
Pass that state to both inference tasks and response readers so grammar,
tool calls, and reasoning remain available in both chat paths.
Detect the parser API during preparation to preserve older fork builds.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* test(omnivoice): expose ignored duration controls
Compile the real wrapper with a model-free engine fixture. Four failing
assertions expose ignored duration, postprocessing, and validation.
Assisted-by: nib:gpt-6-astra
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(omnivoice): add bounded duration controls
Validate request controls before reference reads or native output.
Preserve engine defaults when controls are omitted. Bound forced
single-shot duration to 30 seconds as wrapper resource policy.
Exercise the real native boundary, streaming rejection, and defaults
with synthetic PCM. Document quantization and postprocessing limits.
Assisted-by: nib:gpt-6-astra
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(omnivoice): require fixture-free ordinary tests
Expose the ordinary suite regression before isolating native fixtures.
Require selected native tests to reject a missing fixture explicitly.
Assisted-by: nib:gpt-6-astra
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(omnivoice): isolate native fixture tests
Ordinary Go tests must run without a compiled native fixture.
Select fixture specs explicitly in the dedicated gate while keeping
parser tests ordinary and native checks mandatory.
Assisted-by: nib:gpt-6-astra
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
test: satisfy lint checks for upscale and diagnostics
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Update stable-diffusion.cpp to its merged Iris support commit and add the original F32 Iris weights with the Qwen3-VL Q4_K_M text encoder to the model gallery.
Assisted-by: nib:gpt-5.6-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(voice-detect): report the encoder family so audio-registered voices are fingerprinted
A voice registered from audio through the voice-detect backend had no
encoder fingerprint, so the parakeet-cpp backend could not tell whether
it was comparable with the loaded speaker model and could only fall back
to the file-name rule.
The voice-detect backend now binds the three new libvoicedetect accessors
with a symbol probe (an older library still loads and reports nothing),
copies the borrowed strings at once and never frees them. It also hashes
the model file once at load. VoiceEmbedResponse gains two optional
fields, encoder_family and encoder_weights ("sha256:<hex>", empty when
the model is not a plain file).
/v1/voice/register stores them as encoder_family and a new
encoder_weights field in the registry entry; model keeps the encoder
name, so the 1:N identify filter by name is unchanged for old entries.
A voice with a family is sent to the backend whatever its file name, and
the backend decides by family. /v1/voice/identify compares the family
when both the stored voice and the probe have one. Old entries load
without the fields and stay unfingerprinted.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(voice-detect): say how a voice with weights but no family is treated
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(voice-detect): mark the model file open as intended for gosec
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update 0xShug0/audio.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(audio-cpp): mirror the turn detection task
The upstream enum adds TurnDetection, so the exhaustive conversion fails
to compile. Extend both conversions and preserve the RPC admission rules.
Test that turn detection cannot route through VAD or another existing RPC.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(diarization): return sound events with include_sounds
A client that wants text, speakers, voice prints and sound events had to
make a diarization call and a separate sound call. Add an include_sounds
request field to /v1/audio/diarization that adds a sounds array of closed
events {start, end, label, confidence}, in seconds.
The parakeet-cpp backend runs a tagger-only scene stream over the clip,
the same stream and thresholds the live path uses, so a clip gives the
same events offline and live. A model with no sound_model companion, or a
backend that does not report sound events, fails with 501 and the stable
code include_sounds_unsupported instead of an empty list. The proto
carries sounds_included so an empty list still means "nothing heard".
The localai-proxy backend forwards the field. Swagger, docs and the
e2e mock backend are updated.
Assisted-by: Claude:claude-sonnet-5-5 [protoc swag go]
* feat(gallery): add parakeet-cpp-multilingual-diarization-speakers-sounds
Same as parakeet-cpp-multilingual-diarization-speakers (TDT 0.6B v3,
Nemotron-3-Diarization, WeSpeaker) plus a CED-Tiny sound_model, so one
model name serves /v1/audio/diarization with include_text,
include_speaker_profiles and include_sounds. It declares the
sound_classification usecase like the realtime scene entries.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
