mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,324 Commits 136 Branches 173 Tags
feat/buun-llama-cpp-backend
Commit Graph

7324 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
localai-org-maint-bot
d902a3381b fix(buun-llama-cpp): match speculative draft split anchor 
The shared gRPC wrapper stores p_split under the draft sub-structure. Match that exact source spelling so the fork-specific patch stage reaches the build on every architecture.

Assisted-by: Codex:gpt-5 [Codex]
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
c438fe8814 fix(buun-llama-cpp): shim cudaMemcpy{To,From}Symbol + WARP_SIZE on fwht128 shuffles 
Two more hipblas-only build failures in buun's fattn.cu, fixed under the
same patches/ infrastructure:

1. cudaMemcpyToSymbol / cudaMemcpyFromSymbol — buun's Q² calibration +
   TCQ codebook upload paths call the symbol variants of cudaMemcpy.
   ggml/src/ggml-cuda/vendors/hip.h aliases every other cudaMemcpy*
   name (cudaMemcpy, cudaMemcpyAsync, cudaMemcpy2DAsync, …) but the
   symbol pair was never added. 15+ "use of undeclared identifier"
   errors across fattn.cu lines 40, 54, 74-76, 94, 100-101, 371, 883,
   905, 954, 976, 1449, 1463. Add the two missing aliases alongside
   the existing memcpy block.

2. __shfl_xor_sync fwht128 calls — same 3-arg omission pattern as the
   earlier argmax top-K fix. Lines 512 (ggml_cuda_fwht128 intra-warp
   butterfly) and 536 (fwht128_store_half neighbor fetch) drop the
   width argument that hip.h:33 requires. Add WARP_SIZE.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
920dec302a fix(buun-llama-cpp): pass WARP_SIZE to argmax __shfl_xor_sync calls 
Two call sites in ggml/src/ggml-cuda/argmax.cu (the top-K intra-warp
merge added by buun) use the 3-arg CUDA form __shfl_xor_sync(mask, var,
laneMask), omitting the optional width parameter. The hipification shim
at ggml/src/ggml-cuda/vendors/hip.h:33 is a function-like macro that
requires all four arguments, so hipcc fails with:

    argmax.cu:265: too few arguments provided to function-like macro
      invocation
    note: macro '__shfl_xor_sync' defined here:
      #define __shfl_xor_sync(mask, var, laneMask, width) \
              __shfl_xor(var, laneMask, width)

Every other call in the same file already passes WARP_SIZE explicitly;
aligning these two with that convention fixes the hipblas build without
changing CUDA codegen (warpSize is the CUDA default).

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
3989c6909a fix(buun-llama-cpp): shim atomicAdd(double*,double) for pre-sm_60 CUDA 
Buun's Q² calibration path in ggml/src/ggml-cuda/fattn.cu calls
atomicAdd with a double* destination. Native double atomicAdd is only
available on CUDA compute capability 6.0 and later — LocalAI's CUDA 12
Docker image builds for the full published arch range (which includes
sm_50/sm_52), so nvcc fails with:

    fattn.cu:812: error: no instance of overloaded function "atomicAdd"
    matches the argument list, argument types are: (double *, double)

Add the canonical CAS-loop shim from the CUDA C Programming Guide
(B.15 Atomic Functions) guarded on __CUDA_ARCH__ < 600. On sm_60+ the
guard is false and nvcc picks up the native intrinsic as before.

Patch file lives under backend/cpp/buun-llama-cpp/patches/ and is
applied to the cloned fork tree by apply-patches.sh (the infrastructure
already put in place for exactly this class of backport).

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
0dae106085 ci(buun-llama-cpp): wire backend into test-extra + build matrix 
Adds the buun-llama-cpp backend to the same CI pipelines that turboquant
and sherpa-onnx already use:

- scripts/changed-backends.js: path resolution for Dockerfile.buun-llama-cpp,
  plus fork-of-fork detection (changes under backend/cpp/llama-cpp/ also
  retrigger the buun pipeline, mirroring how turboquant is handled).
- .github/workflows/test-extra.yml: detect-changes output and a new
  tests-buun-llama-cpp-grpc job that runs make test-extra-backend-buun-llama-cpp
  (turbo3 V-cache, same rationale as tests-turboquant-grpc).
- .github/workflows/backend.yml: 9 matrix entries (CUDA 12/13, L4T CUDA
  13 ARM64, ROCm, SYCL f32/f16, CPU, L4T ARM64, Vulkan) paired with each
  existing turboquant entry so image builds have platform parity.

Also updates .agents/ai-coding-assistants.md to clarify that AI agents
operating under the human submitter's git identity SHOULD emit
Signed-off-by via `git commit -s` (never inventing or guessing another
identity) — documents the workflow this PR is using.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
5e2e0ec17e fix(buun-llama-cpp): drop logit_bias_eog arg from params_from_json_cmpl 
Previous substitution kept the call as 5 args, but buun predates the
upstream refactor that also *added* the logit_bias_eog parameter to
params_from_json_cmpl — buun's signature is still the 4-arg form
  (const llama_vocab*, const common_params&, int, const json&)
and it still derives logit_bias_eog internally from the common_params.

Replace the substitution with a line-delete. Guard matches both the
original call (ctx_server.get_meta().logit_bias_eog) and the previously
substituted form (params_base.sampling.logit_bias_eog) so the script
stays safe across re-runs and whatever state the tree was left in.

Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
3fef509a7f fix(buun-llama-cpp): backport logit_bias_eog field to grpc-server copy 
LocalAI's shared grpc-server.cpp reaches
ctx_server.get_meta().logit_bias_eog twice (the twin params_from_json_cmpl
callsites). That accessor was added to server_context_meta upstream after
buun's 2026-04-05 fork-point, so compiling against buun errors with
  'struct server_context_meta' has no member named 'logit_bias_eog'.

Rewrite the call sites — only in the buun grpc-server.cpp copy — to source
the vector from params_base.sampling.logit_bias_eog instead. That vector is
the underlying data the upstream meta accessor eventually returns (buun
still carries common_params_sampling::logit_bias_eog at common.h:280), so
the substitution yields identical behavior on both trees.

The sed is guarded by a grep for the call site, so this patch is
self-disabling once buun rebases past the upstream refactor.

Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
9a39368b1c test(gallery): extend importer specs to cover buun-llama-cpp 
Two additions that pair with the new backend:
- An Import()-side case that asserts preference buun-llama-cpp produces
  backend: buun-llama-cpp in the emitted YAML (mirrors the existing
  ik-llama-cpp and turboquant cases).
- AdditionalBackends() spec now asserts all three drop-in replacements
  are advertised, and verifies buun-llama-cpp's Modality/Description
  alongside the other two.

Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
Ettore Di Giacinto
f88444f8e3 feat(backend): add buun-llama-cpp fork (DFlash + TCQ KV-cache) 
spiritbuun/buun-llama-cpp is a fork of TheTom/llama-cpp-turboquant that adds
two independent features on top: DFlash block-diffusion speculative decoding
(via a dedicated DFlashDraftModel GGUF arch) and two extra TCQ KV-cache
variants (turbo2_tcq, turbo3_tcq) on top of TurboQuant's turbo2/turbo3/turbo4.

Follows the turboquant thin-wrapper pattern — reuses backend/cpp/llama-cpp
grpc-server sources verbatim, patches only the build copy to extend the KV
allow-list and wire up buun-exclusive tree_budget / draft_topk options.
DraftModel is already wired end-to-end (proto field 39 → params.speculative),
so DFlash activation only needs the existing options passthrough
(spec_type:dflash) plus the drafter path in draft_model.

CacheTypeOptions now surfaces the five turbo* values so the React UI dropdown
shows them — benefits turboquant too (previously users had to type them in
YAML manually).

Assisted-by: Claude:Opus-4.7 [Read] [Edit] [Bash] [WebFetch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:04:29 +00:00
localai-org-maint-bot
1189c6825b feat(tracing): persist bounded trace histories (#11203) 
* feat(tracing): persist bounded trace histories

Retain API and backend traces below the data path, restore them at initialization, and serialize clears with asynchronous consumers.

Assisted-by: Codex:gpt-5

* fix(tracing): satisfy persistence security checks

Document why persisted filenames cannot escape the trace directory and explicitly ignore the best-effort temporary-file cleanup result.

Assisted-by: Codex:gpt-5 [golangci-lint]

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 12:20:01 +02:00
mudler's LocalAI [bot]
cb417464e5 chore(model gallery): 🤖 add 1 new models via gallery agent (#11194) 
chore(model gallery): 🤖 add new models via gallery agent

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-07-30 12:18:11 +02:00
mudler's LocalAI [bot]
c4617265b9 fix(ci): trim two pieces of per-PR work that buy nothing (#11219) 
Measured over the week to 2026-07-30, 97% of CI wall-clock is queueing and 3%
is execution: a median 5-hour queue against a 4-20 minute median job. With the
queue saturated, throughput is concurrency divided by service time, so cutting
execution time raises the drain rate directly. Two steps stood out as paying
nothing for what they cost.

test.yml: drop the free-disk-space step (~3.1min per run, ~22 h/week). That
action exists to make room for docker buildx layers and this job runs no buildx
step. It was also sized for a `make test` that downloaded multi-GB GGUF/whisper
fixtures and built llama-cpp/whisper/stablediffusion-ggml; the test-suite reorg
moved all of that into tests/e2e-backends and tests/e2e-aio, as the Makefile
test target already records. Its tool-cache:true wipe was additionally deleting
/opt/hostedtoolcache, forcing setup-go and setup-node to re-download toolchains
that ship preinstalled on the runner.

build-test.yaml: build only the host target on pull_request. The three-platform
cross-compile (linux/amd64, linux/arm64, darwin/arm64) is the bulk of that job's
~6.6min median, ~47 h/week, and nothing consumes a PR's binaries. goreleaser's
--single-target still runs every before-hook (protogen-go, react-ui, go mod
tidy), so the "is the release build broken" signal is unchanged. master pushes
and tags keep building all three.

Also record why the Linux Go workflows pass cache: false to actions/setup-go,
since it reads as an oversight and is not. Set up Go has a median of 11 seconds
on those runners, so there is nothing to win, and the repo already sits at
GitHub's 10 GB Actions cache ceiling with 31 entries, where each setup-go entry
is 222-375 MB on Linux and up to 1.4 GB on macOS. Re-enabling it would evict
something that is earning its space.


Assisted-by: Claude:opus-5 [claude-code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 12:17:14 +02:00
mudler's LocalAI [bot]
47097041ff fix(vllm): apply Options[] engine flags before engine init (#11147) 
fix(vllm): apply Options[] engine flags before engine init (#11130)

CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.

Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.

Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.

The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.


Assisted-by: Claude:claude-opus-5 golangci-lint

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 12:13:00 +02:00
mudler's LocalAI [bot]
9c85cacfe3 feat(audio-cpp): add the audio.cpp native backend (#11141) 
* backend(audio-cpp): add the native build scaffold

Links 0xShug0/audio.cpp engine_runtime through its public framework headers
and serves Health/Status. Model loading and the audio RPCs follow.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): keep the build-tree rpath at $ORIGIN

Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so
CMake was appending its build-tree library dir to our target and baking an
absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH
on the target so a package that forgets to bundle libggml*.so fails on the
build machine too, instead of only on a user's box.

Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call,
correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment
from repeating the assignment token that bump_deps.sh rewrites, and make
test-engine fail rather than pass when no test is registered.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): parse namespaced model options

Splits option entries on the first colon so path values survive, and routes
load./session. prefixes to the upstream load and session option maps.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): reject out-of-range numeric model options

std::atoi is undefined once the digits exceed long and in practice wraps, so
device:2147483648 was accepted and handed the ggml backend selector a device
index of -2147483648 from a function whose error text promises a non-negative
integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and
on any unconsumed trailing input. The error strings are unchanged.

Name the whole entry in the unknown-key error too: an entry like ':value' has
an empty key and left the user nothing to grep for in their YAML.

Tests look keys up through a helper instead of map::at, so a prefix off-by-one
fails one named check rather than aborting the binary and skipping the rest of
the suite, and cover the overflow, negative and non-numeric paths.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks

Task-major resolution over the family's advertised capability set, with the
voice-reference and instructions signals selecting cloning and voice design,
and a streaming-to-offline fallback for server-streaming transcription only.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): use upstream's 'spk' task name and pin the preference order

The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints
nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec'
would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an
input-only alias, and correct the known-tasks lists.

Three assertions were vacuous because their fixtures advertised a single task,
so reversing a preference order or dropping the RPC name and the attempted
pairs from the capability error all passed. Give them fixtures that can tell
the orderings apart.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): convert sample, time and PCM units

Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the
VAD and diarization messages, and saturating s16le encode so an overshooting
sample cannot wrap to the opposite sign.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): harden seconds_to_samples against NaN and overflow

seconds_to_samples is the one entry point fed by untrusted-shaped input: a
float-seconds timestamp off the wire, or a boundary from a model that diverged.
Its guard covered only the low side, so NaN and out-of-range values fell through
to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely
negative sample index used later as an offset or a length is a wild pointer
rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and
saturate before the cast.

Also round instead of truncating there. These functions exist to cross the float
seconds boundary the VAD and diarize messages use, and truncation lost a sample
about half the time on the samples-to-seconds-and-back round trip, starting at
n=1.

Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero
fraction, and record why the clamp argument order in f32_to_s16le is
load-bearing for NaN.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): map NaN PCM samples to silence explicitly

f32_to_s16le relied on std::min argument order to keep a NaN sample away from
std::lround, whose result is unspecified for NaN. That was too subtle to rest on
a comment, and the comment was itself wrong: it warned against a spelling that
the outer std::max already catches, while three real spellings leak, including
std::clamp, which is the idiomatic C++17 way to write the same clamp and so the
likeliest future edit.

Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a
full-scale click is worse audio than a dropped one, and this unit converts audio
that may have originated off the wire.

Pin it with an exact-value check rather than a range check, since all three
outcomes the plausible spellings produce are finite and inside full scale, plus
an invalid-operation check that fails unless the NaN is diverted before any
ordered comparison. That second check is what catches modernizing the clamp and
dropping the guard together.

Also bound the seconds round-trip comment, which claimed unconditionally what
holds only below roughly 2^23 samples, and document NaN, saturation and that
bound in the header.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): assemble transcripts from runtime spans

The top-level transcript text is TaskResult.text_output verbatim. audio.cpp
carries text nowhere else: speech_segments, speaker_turns and word_timestamps
hold spans and labels only, so deriving the text from them empties the
transcript for any producer that omits word timing, VibeVoice diarized ASR
included. Fixtures cover every observed producer shape.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): keep a nested speaker turn's own label

A segment sourced from speaker_turns re-derived its speaker by greatest
overlap. A turn's overlap with its own span is the largest possible, so a turn
nested inside another speaker's turn could only tie with the container, and the
tie went to whichever came first. sortformer_diar binarizes each speaker
independently and sorts by start sample, so the container always comes first
and the interjecting speaker was silently erased from DiarizeSegment.speaker.
choose_segment_spans now carries the label out with the span.

Also pins the nearest-segment fallback against measuring from either endpoint
or from segment position, which a trailing-only stray word could not do, and
exercises the empty-word guard in join_words. Two fixtures that pin a rule but
do not mirror any pinned family are relabelled defensive.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serialize runs with a wedge-aware guard

audio.cpp sessions are not reentrant and a wedged CUDA call cannot be
cancelled, so a plain mutex would pile every worker thread behind a stuck GPU.
Callers waiting past the configured bound, or arriving while the holder has
already overrun it, fail fast instead.

A caller that queues behind a healthy run deliberately does not stamp the
clock: only the thread that takes the lock does. Stamping on arrival would
restart the wedge clock on every request and hide a stuck run from everyone
behind it, which is the pile-up this guard exists to prevent.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serialize inference through an InferenceLane

One audio.cpp model is loaded per backend process and its sessions are not
reentrant, so concurrent gRPC handlers have to take turns. Serialization alone
is not enough: a wedged GPU call cannot be cancelled from userspace, so an
unbounded queue behind one stuck run would swallow every gRPC worker thread
until the process is useless.

InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies
it for a scope and gives it back on every exit, including an exception, and is
the only way to take the lane at all: occupy/vacate are private with LaneEntry
as the sole friend, so a caller cannot acquire without holding something that
releases. LaneEntry is immovable on purpose, because a moved-from entry would
have to stop releasing while the lane still recorded it as occupied.

A caller either waits indefinitely or brings a millisecond budget. A bounded
caller that cannot get in fails instead of waiting on, and a bounded caller
whose budget is already shorter than the age of the run in the lane fails
immediately, which is what stops a queue forming behind a wedged run. The two
failures carry different text: one names the wait it exhausted, the other states
the measured age of the run without claiming to know why it is long, since a
short budget meeting a legitimately long run lands there too.

The run's age is stamped only after acquisition. A waiter that published itself
as holder would restart the measurement and hide a genuinely stuck holder from
every caller behind it.

Budget negotiation and the overrun decision are pure functions taking their
inputs explicitly, so both are covered without threads or sleeping. The
per-model ceiling arrives as an int of milliseconds; a request may tighten it
and may never loosen it.

Replaces the previous run_guard unit, which was a derivative of an
Apache-2.0 file upstream and could not stay in an MIT tree. Written from a
behaviour contract with no reference to the removed code.

Tests: 65 checks, standard library only, single translation unit, clean under
-Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed
missing coverage and the tests were extended until they died. ThreadSanitizer
clean.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): make the B10 test able to fail, and document LaneEntry

Review of the previous commit found the B10 test could not fail for the reason
it was named. It aged the in-flight run to about 120 ms and then tried two
budgets, 30 ms and 60 ms, both under that age, so both callers took the
fail-fast path. "The two failure modes do not share one message" was comparing
two fail-fast messages that differ only in the budget they print, and the
timeout path was never reached. The second budget is now 400 ms, well over the
run's age, so that caller queues and times out, and a new check asserts which
path each caller took instead of inferring it from inequality. A mutant that
makes the fail-fast path emit the timeout message previously died only on B4 and
B8 checks; it now also dies on B10.

Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does
not detect reentrancy: a second entry on a thread that already holds the lane
surfaces as LaneUnavailable with a positive budget, but parks silently in
unbounded mode, which matters because a handler may hold one across a whole
stream. The immovability note now names the shapes that work, an optional
emplaced in place or a unique_ptr, rather than saying to hold the entry
indirectly without saying how; all three documented forms were compiled before
being written down, which is how the note came to say that an optional of an
immovable type cannot itself be returned.

The header's explanation of why fail-fast exists is reworded. Two clauses traced
back to a specification written after reading the Apache-2.0 upstream header,
and while that was judged de minimis, this unit was rewritten precisely to carry
no upstream expression at all.

The margin table in the report was also wrong about which wall-clock margins are
load-sensitive: there are four, not one, and the tightest is the B3 arrival
check, which is now flagged at the call site. No margin value changed and none
moved across 65 runs.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): gate model loading on the audio.cpp family

Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF
path without an explicit family option, so the model loader's greedy backend
probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287).

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): load models and cache sessions per task

Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per
(task, mode), so the same model serves both the unary and streaming RPCs.
LoadModel derives the family from GGUF metadata or an explicit option and
fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the
backend probe can see.

audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally,
and drift there is silent: every unit still compiles and every test still
passes while the backend runs a different task. Two mechanisms pin it. The
static_asserts in loaded_model.cpp catch an insertion or a reorder, and
-Werror=switch on that one file turns an appended upstream enumerator into a
build failure rather than a warning in a 600 file log.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): stop aborting the process on SIGTERM

The signal handler called grpc::Server::Shutdown directly. Shutdown takes an
absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread
already holding that mutex, and abseil's deadlock detector responds by aborting.
Every SIGTERM therefore ended in exit 134 and a 'dying due to potential
deadlock' stack rather than a drained shutdown.

The handler now sets a lock-free atomic and returns. Server::Wait moves to a
helper thread so the main thread can poll that flag and call Shutdown itself,
outside any signal context. A condition variable would not have helped, because
notifying one from a handler is not async-signal-safe either.

SIGTERM and SIGINT both exit 0 with no stack trace, where both previously
exited 134.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel

An environment fault during session creation was reported as UNIMPLEMENTED. A
missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises
vad/offline but refused to create the session: Failed to initialize CPU
backend', which tells LocalAI the model cannot do this and must never be
retried, and sends an operator hunting a capability bug instead of a packaging
one. A throw from create_task_session is now a plain runtime_error, so it maps
to INTERNAL. Only a null return, where the family genuinely declined, stays a
CapabilityError.

The model.'s task: option was parsed and then dropped: it lived in a local that
died at the end of LoadModel and had no route to RequestShape::pinned_task.
LoadedModel now keeps it and exposes pinned_task().

The global model becomes a shared_ptr reached through snapshot(). An audio RPC
runs for seconds and cannot hold g_model_mu for its duration, so under a
unique_ptr a Free arriving mid-request would destroy the model underneath it.
Handlers now take a counted reference and whichever finishes last does the
teardown, outside the lock.

session_for documents the streaming state contract rather than resetting the
session itself. Resetting on a cache hit was tried first and is not possible:
silero_vad throws 'session prepare() must be called before Silero VAD reset()',
so it would turn an ordinary second fetch into a hard error. start_stream's base
implementation is already a reset, so a caller that runs prepare then
start_stream per stream gets a clean session; a probe against the bundled
silero_vad confirms an identical replay when it does and a carried-over stream
when it does not.

Also: an unknown backend: name is rejected before the model loads rather than
after; MainGPU is parsed instead of passed through std::atoi, which turned
'gpu1' into device 0 silently; and device carries a device_set flag, because 0
is both the default and a real device index, so MainGPU was overriding an
explicit device:0 that the neighbouring threads: handling promises will win.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the VAD and Diarize RPCs

Both emit float seconds, converted from the runtime's sample-index spans, and
both take a counted reference to the loaded model through snapshot() and hold it
for the whole call: a Free arriving mid-request drops only the global's
reference, so whichever request finishes last destroys the model instead of one
of them running on freed weights. An AddressSanitizer build reproduces exactly
that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw
pointer instead, which is why the shape is what it is.

The inference lane is taken before session_for, not after. session_for reads and
writes an unsynchronised session cache and the offline run calls prepare(),
which mutates the session, so both belong inside the lane.

Diarize routes before it reads the input file, so a family that cannot diarize
at all says so rather than complaining about the audio first. Its per-segment
text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker
label only, and nested or overlapping turns are passed through untouched: a
sortformer turn inside another speaker's turn is correct output for overlapped
speech, and LocalAI is overlap-tolerant downstream. Duration counts frames
rather than floats, so a stereo input does not report twice its length.

Verified end to end against upstream's bundled silero_vad, which needs no
download, using the bundled 16 kHz speech asset: a synthetic tone returns
nothing, correctly, because silero detects speech and a sine is not speech.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): enforce ModelIdentity on VAD and Diarize

audio-cpp was the only C++ backend without the model-identity guard, and no
later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity
on exactly these two RPCs, for the reason #10952 records: in distributed mode a
worker can recycle a stopped backend's gRPC port for another model's backend,
and the controller's liveness-only probe cannot tell a stale cached route from a
live one. Without this guard a stale route gets a different model's VAD or
diarization answer back with a 200.

The loaded identity lives on LoadedModel rather than in a separate global, which
is where this differs from llama-cpp. A handler holding the model through
snapshot() then necessarily judges against the identity that model was loaded
with, and a concurrent reload cannot swap one without the other. The refusal is
NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring.

session_for and run_offline now take a const LaneEntry & proof-of-holding
parameter. The rule that both must run under the inference lane was prose, which
is exactly how the plan came to specify the inverted order; it is now a compile
error. Restoring the inverted order fails to build rather than racing on an
unsynchronised session map with a mutating prepare().

Diarize's speaker-hint comment claimed the dropped hints were "not a silent
failure". From the caller's side that is what they are, and backend.proto
documents num_speakers as forcing, so the comment now says plainly that the
forwarding is dead for sortformer and that the family which lands must either
honour num_speakers or refuse it. read_audio_file inspects the error_code from
exists(), so an unsearchable parent directory no longer reports as a missing
file. The VAD handler records the stimulus that actually works, since silero
correctly ignores synthetic tones and the next task would otherwise rediscover
that.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): make the lane and identity guards structural

Two hardenings ahead of the eleven handlers still to be written, both of which
get harder to retrofit later.

The lane proof-of-holding parameter was a const reference, which binds to a
temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such
temporary dies at the end of its own full-expression, releasing the lane between
two calls that must share one: precisely the split the parameter exists to
prevent, and the form a future author is most likely to reach for because it
reads as tidy. A non-const reference requires an lvalue, so the temporary form
now fails to compile while the named-local handlers build unchanged. The header
comment no longer implies the check is total either: it proves a lane was taken,
not that it is this model's lane.

The identity check was two lines each handler had to remember, with nothing
failing if a new one forgot them and no C++ equivalent of
model_identity_modalities_test.go to notice. snapshot() becomes
snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage
carries no ModelIdentity. Handlers go through snapshot_for(), which takes the
counted reference, refuses when nothing is loaded, and runs the identity check
before anything can route. Every handler already has to call something to obtain
the model, so the guarded call is now the shortest path and skipping it means
deliberately typing snapshot_unchecked. A convention that has to be remembered
can rot; this cannot.

Verified: the temporary-argument and inverted-order forms each fail to compile
with the expected diagnostic, the real handlers build, and bypassing the guard in
Diarize alone turns the identity test red on that RPC while VAD stays green.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the AudioTranscription RPC

Adds result_map, the engine-to-proto boundary, and wires the offline
transcription RPC.

The handler branches on the ROUTED task: for Asr the request's prompt is
whisper-style decoding context and becomes a request option, for Alignment
the same field IS the transcript to align and becomes the text input.
Routing has already decided which.

The result text is TaskResult.text_output verbatim and is never derived
from the segments. audio.cpp carries transcript text in text_output and
nowhere else, so deriving it returns an empty transcript for every
producer that reports segments without word timing. transcript_assembly
already enforces that; this commit's job is not to undo it at the proto
boundary, and result_map_ctest pins it there.

read_audio_file now takes the sample rate the caller needs. Both file-fed
speech handlers ask for 16 kHz mono, for two reasons: silero_vad and
sortformer_diar refuse anything else outright, which turned an ordinary
44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in
its own 16 kHz feature domain whatever the input was, so only a 16 kHz
buffer makes the emitted nanoseconds right. Zero keeps the file's native
rate and channels, which is what source separation will need.

LoadedModel::check_can_serve answers a capability refusal before the lane
is taken and before the input file is read. Routing is a pure read of the
immutable capabilities, so a model that cannot serve an RPC no longer
waits out somebody else's run to say so. VAD and Diarize use it too.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): stop linking sentencepiece's vendored protobuf

engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER
builds the protobuf-lite 3.14.0 sources it vendors. The generated
backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both
ended up in the binary: 476 google::protobuf:: symbols came from the
archive, 278 of them also defined by libprotobuf.so, and the archive won,
because once ld pulls a member in for sentencepiece's own code every
reference binds to the definitions that member carries.

The visible symptom is one function.
ParseContext::ParseMessage(MessageLite*, const char*) is what a generated
_InternalParse calls for a submessage field and for nothing else, so flat
messages parsed and nested ones did not: a TranscriptResult carrying
segments serialized to correct bytes that the same process could not read
back, and TranscriptLiveRequest, a oneof of submessages, could not have
been parsed at all. Underneath that, 3.21 generated code was running 3.14
arena, ArenaStringPtr and ExtensionSet code.

-Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in
.dynsym and the parse still fails, because the binding was decided at
static link time and no visibility flag revisits it.

Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points
sentencepiece at the protobuf the generated code was already built
against. Zero google::protobuf:: definitions remain in the executable
afterwards, every nested message round trips, and citrinet_asr, which
parses a SentencePiece ModelProto at load time and would break first if
this were wrong, still tokenizes and transcribes correctly.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): fix the segment text a transcription response is built from

Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.

nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me  call   me  na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.

This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.

Two smaller corrections in the same area:

timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.

The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.

Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the AudioTransform RPC

Covers voice conversion, singing voice conversion, speech to speech and source
separation, the four tasks LocalAI's AudioTransform can represent.

AudioTransformResult carries one dst while htdemucs and mel_band_roformer
produce several named stems from a single run, so inference runs ONCE, every
stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem]
selects which one dst receives, defaulting to vocals and falling back to the
first output. An unknown stem name is INVALID_ARGUMENT listing the real stem
names rather than a silent substitution, and the selection happens before the
first write so a refused request leaves no files behind. params[stem] is
consumed here and is not forwarded into the engine's request options.

The stem decision lives in stem_selection, which is stdlib only and therefore
tested by backend/cpp/run-unit-tests.sh. It also validates the names, because
they come from the model (htdemucs reads them from the GGUF's config.sources)
and each becomes a component of a path this backend writes: a name carrying a
path separator would escape the caller's output directory, and two stems
sharing a name would silently overwrite one another.

Both files are read at their native rate and channel count. Separation forces
it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo
image that separates a centred vocal from a wide mix. The conversion families
all resample internally (seed_vc, vevo2, miocodec, chatterbox were each
checked), so passing the file through unchanged is also strictly better than
band limiting it to 16 kHz first.

Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems
plus dst, dst byte identical to the selected stem, params[stem] selecting a
different one, an unknown stem refused with no files written, and mono input
preserved as mono output. Also against miocodec for the single output path,
where params[stem] is refused rather than ignored.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write

Four fixes from the first review of the AudioTransform RPC.

check_can_serve now returns the resolved route, so params[stem] on a route that
is not source separation is refused from the route instead of after a full
inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far
worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a
separation-routed family that returns no stems anyway. The typo'd-stem-name
case still needs the run, since no framework header publishes the stem names
before one.

Stem names carrying control bytes are refused. GGUF strings are length prefixed
and demucs reads its sources from JSON, so an embedded NUL survives to here:
two names differing only after the NUL are distinct std::strings, so the
duplicate check passes them, and then path::c_str() truncates both and they
open the same file. That is exactly the silent overwrite the duplicate check
exists to prevent, with the .wav lost as well.

A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is
LocalAI's own generated-content directory, not anything the caller named, so a
full disk or a permission fault there is a server fault and is worth retrying,
which is the opposite of what INVALID_ARGUMENT tells a client. An empty output
path stays INVALID_ARGUMENT.

Two comment corrections and one clarification: the separators' required rate is
their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc
resamples with soxr and falls back to sinc-hann, and the "no files left behind"
guarantee covers a refused request, not a write that fails partway through the
loop.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems

Two defects that made source separation unusable through LocalAI's own API,
even though the backend served it correctly over gRPC.

/audio/transform normalized every upload to 16 kHz mono s16 through
utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse
any rate but their checkpoint's own and separate a centred vocal from a wide
mix using the stereo image, so every separation request through the HTTP API
died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000"
while the same call over gRPC worked. The fold is not wrong, it is
backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and
needs the reference in the same shape. So it becomes a declaration,
BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing
else. A backend that declares nothing gets its upload unchanged, which means no
backend has to opt in to work. utils.AudioToWavPreservingShape is the
non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any
rate and channel count, anything else is transcoded to WAV with its rate and
channel layout kept.

The other defect is that the run-once stem design bought nothing. A separation
backend writes every stem beside dst from one inference, but AudioTransformResult
carried only dst, so the other three were files no caller could find and a
caller wanting all four had to run four separations. AudioTransformResult grows
a repeated AudioTransformStem, the backend fills it, core/backend validates that
each path really is inside the generated-content directory it handed over, and
the endpoint publishes them as an X-Audio-Stems JSON header beside the existing
X-Audio-Input-Url. JSON because a stem name is the model's own string and could
contain any separator a hand-rolled format would use.

Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz
stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable
through /generated-audio/, body byte identical to the selected stem, and
params[stem]=drums returning a different one. The same upload sent to a model
whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed
both by the engine's own rate refusal and by the persisted input file.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst

Four fixes from the second review, plus one bug they made visible.

isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at
the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed
through untouched where the old fold would have transcoded it. audio.cpp's WAV
reader accepts 16-bit only when the tag is 1, so such a file died with
"unsupported WAV encoding". Extensible is what many DAWs and Windows tools
write and music files are this endpoint's new headline input, so it is a
first-contact failure rather than a corner. The check now requires tag 1, with a
spec that fails against the old implementation.

Stem URLs are percent-escaped. A stem name is the model's own string and legally
contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL
before the request is even sent. The name field keeps the raw name.

sample_rate and response_format are applied to the stems as well as to dst.
Applying beat documenting: dst IS one of those stems, so leaving them alone
broke the "dst duplicates the selected stem" invariant the whole design rests
on, and both conversions are no-ops when unset. A stem whose conversion fails is
dropped from the header rather than advertised in the wrong shape.

Verifying that turned up why it had never been noticed: the two fields were
never bound at all. The request arrives as multipart/form-data and echo's binder
falls back to the FIELD NAME without a form tag, matching only
case-insensitively, so "SampleRate" never matched "sample_rate" and "Format"
never matched "response_format". Both were documented in the endpoint table and
silently ignored. Two form tags fix it, and with them the conversion is
observable end to end.

Docs: audio-transform.md now documents what LocalAI does to an upload before the
backend sees it, which backend gets the 16 kHz mono fold and why, params[stem],
and the X-Audio-Stems header with a worked example.

Also records the known limitation that the fold lookup is on the bare backend
name, so pinned variants (vulkan-localvqe) do not match, and points at
IsLlamaCppBackend as the suffix-tolerant precedent.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the TTS and SoundGeneration RPCs

TTSRequest.voice is treated as a speaker reference clip when it names an
existing regular file, which makes routing prefer VoiceCloning, and as a named
preset otherwise, in which case it travels as VoiceReference::cached_voice_id.
Both the clip and SoundGenerationRequest.src are read at the file's own rate and
channel count: upstream's own CLI and server do exactly that, every consuming
family resamples internally and mostly with a better resampler than ours, and
ace_step and stable_audio resample their input per channel, so a downmix here
would delete the stereo image they are built to consume.

The request builders live in their own unit rather than in grpc-server.cpp's
anonymous namespace so they can be tested; grpc-server.cpp has a main() and
cannot be linked into a test binary. The option keys are the whole point of
these functions, so each one was grepped against the pinned upstream and the
accounting is written down beside it. instructions maps to "instruct", which is
what upstream's own server maps the OpenAI field to and what qwen3_tts and
omnivoice read, and to "caption" for irodori_tts; the style tag is spelled
"instruct" too, because "instructions" is looked up nowhere. duration maps to
"duration_seconds", read by all three generation families, with the proto's own
name kept only as a forward-tolerant alias. Keys that no family reads say so.

Both handlers answer a capability refusal before taking the lane and before any
file read, so a model that cannot synthesise does not queue behind somebody
else's run to be told no.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): stop emitting an empty style language, and name the missing clip

StyleCondition::language was set whenever has_language() was true, with no
!empty() guard, while the language option twelve lines below had one.
core/backend/tts.go sets Language unconditionally, so has_language() is true on
every request LocalAI sends and carries "" when the caller named none. An
engaged-but-empty style language is worse than an absent one: supertonic reads
text_input->language behind its own !empty() guard and then overrides it from
style->language with no guard at all, so "" replaced its "en" default and its
tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request
that set instructions and no language would have been an INTERNAL against a
supertonic model. A plain request never saw it, because the style condition only
exists when instructions are non-empty, which is why the chatterbox end to end
run did not catch it.

TTS also stops discarding the Route that check_can_serve already returns. A
family routed to voice cloning without a reference clip used to be refused from
inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the
field to set; chatterbox advertises clon and no tts, so that was every
preset-only request to it. It is now an INVALID_ARGUMENT naming
TTSRequest.voice, answered in about 4 ms, and it cannot misfire because
has_voice_reference is what selected cloning in the first place. Reading
CapabilitySet::supports_speaker_reference to generalise this stays a follow-up.

The src read carries a written caveat rather than a family blocklist, because
ace_step's editing routes legitimately need src: setting src on a stable_audio
model corrupts the heap and aborts the process in the pinned upstream, and the
only thing keeping that off the network is that
schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that
Go schema would know why, so the reason is recorded where the field is read.

build_tts_shape is extracted so TTSStream cannot describe the same request
differently, and it arrived untested: two mutations of it survived until a
test_tts_shape case was added.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs

TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching
the convention backend/go/vibevoice-cpp established, so an HTTP client can start
playback before the full PCM exists. Its chunks are read from
StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and
voxcpm2 all put their streamed audio there and leave audio_output empty until
the very end, so reading the obvious field yields a stream with no audio in it.
The finish_stream result is the family's own merged whole rather than a tail, so
it is emitted only when nothing was streamed.

Streaming transcription sends incremental deltas and degrades to a single delta
plus the final result on families that offer no streaming ASR, which is the same
message sequence with fewer deltas. The four streaming ASR families disagree on
what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report
incremental fragments while voxtral_realtime reports the whole hypothesis and
reports it twice, so the reconciliation lives in one tested unit rather than in
the handler. nemotron_asr reports only through the stream event sink, and only
from inside finalize, so the audio driver installs one and clears it again
before returning: the session is cached and a sink left holding the caller's
frame is a use after free waiting for the next stream.

begin_stream is now the only implementation of the streaming state obligation,
prepare then start_stream. Streaming sessions are cached, and what clears the
previous stream is start_stream's reset; a family override that dropped it would
break every call site with no compile error, so there is one call site.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort

TranscriptStreamResponse.delta is a proto3 string, whose wire format requires
valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw
token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between
two consecutive reports is eventually a lone continuation byte, and the C++
runtime serializes that with only a logged warning while the Go runtime refuses
to unmarshal it: the client loses the remaining deltas AND the final_result.
Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to
unmarshal and every accented character was lost. TranscriptDeltaTracker now
holds back an incomplete trailing sequence and merges it into the next fragment;
reconcile flushes it, which it always can because the final text is complete.
The same trace now unmarshals in full with zero failures.

A streaming buffer whose float count is not a whole number of frames is refused
rather than truncated. The integer division dropped the tail floats from the fed
audio and therefore from the transcript, with no diagnostic; vibevoice_asr
refuses the same thing from the other side of the call.

A supertonic GGUF whose weights are not f32 is refused at load. It reaches
ggml_concat with mismatched operand types and ggml_abort takes the whole backend
process down on the first request, so nothing downstream can report it: the
model loads, then every request kills the process. Attributed rather than
assumed, the unary TTS path aborts identically, and upstream records that
package as untested. The refusal names the orig package and says what to run
before deleting the guard.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): stop a repeated lead byte from orphaning the next delta

The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment
the known text already starts with, and when that fragment is the LEAD BYTE of a
new character it looks exactly like a repeat of an older character beginning
with the same byte. It was discarded rather than held, its continuation bytes
then arrived alone and began the next delta, and utf8_complete_prefix_length
only ever inspected the trailing sequence, so a delta invalid at the FRONT went
out whole. Through a real Go proto.Unmarshal the review's four-character repro
gave 3 deltas, 2 unmarshal failures and a lost transcript.

Reachable from the incremental families, not only from voxtral: nemotron_asr's
decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares
bytes, so both split characters. Measured over 30,000 randomized incremental
traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one
delta the Go runtime refuses.

Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the
lead byte is held instead of swallowed and the character survives intact; the
cost is a few duplicated bytes in a shrinking cumulative report, which no pinned
family produces. release() additionally drops leading orphan continuation bytes,
so no delta can begin mid-character whatever the rules above it decide. Losing a
byte keeps the stream alive; emitting one ends the RPC and takes the
final_result with it.

Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative
streams plus both pure-ASCII incremental streams are byte-identical to the
previous commit, so nothing changed for the families already working.

The weight-dtype allow list moves to family_gate, where it is stdlib-only and
pinned by a test rather than only by a comment. Two comment citations corrected.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): read only an exact repeat as a repeat, not any prefix

Rule 2 discarded any partial the known text merely started with. For a cumulative
family that is a duplicate; for an incremental family it is an ordinary short
fragment that happens to coincide with the start of the transcript, and it was
dropped, silently corrupting the text. Pure ASCII, no multi-byte character
anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the
client holding "pure ascii transcrit". Over 5,000 randomized traces per
transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client
holding something other than final_result.text, with a 200 and no diagnostic.
Both incremental families emit fragments that small routinely, since nemotron_asr
cuts at a byte offset and vibevoice_asr at a common prefix.

Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts
and changes no cumulative stream at all: 30,000 randomized cumulative traces are
byte-identical to the previous commit.

What rule 2 guarded was established from upstream rather than from its own
comment. The only duplicate any pinned family produces is voxtral_realtime's,
where process_available_stream_chunks feeds each event to the sink from inside
its loop and returns the last of the batch, so that event arrives twice with
byte-equal text. A duplicate is an exact repeat, so equality still covers it. The
case given up is a cumulative report that SHRINKS, which no pinned family can
produce: voxtral decodes a token vector that is only push_back'ed and cleared by
reset(), so within a stream it can only grow.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): serve the AudioTranscriptionLive RPC

The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.

The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.

Three things the handler had to get right and one it cannot:

  - The audio contract. A live request carries no samples, but nemotron_asr's
    streaming prepare() throws without an audio contract, and
    build_preparation_request derives it from TaskRequest::audio_input, so that
    field is an EMPTY buffer holding only the rate and the channel count.
  - 16 kHz or a refusal. The families express their spans in their own 16 kHz
    feature domain whatever the input was, and live frames cannot be resampled
    on the way in the way a file can, so an 8 kHz session would return
    timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
  - A mid-stream Config is refused. backend.proto calls it a decoder reset, but
    deltas already on the wire cannot be retracted, so a reset would leave the
    final text contradicting the transcript the client assembled. Ignoring the
    message would hand a client that believes it reset the decoder a transcript
    that silently continues the audio it thought it discarded.
  - The stale-route identity check cannot run here: TranscriptLiveRequest
    carries no ModelIdentity in either arm of its oneof, so snapshot_for does
    not instantiate for it. snapshot_unchecked's comment now names that as a
    second legitimate class of caller and says the fix is a proto change.

eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.

The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.

Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.

Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): bound the lane a live stream can hold

AudioTranscriptionLive holds the model's inference lane for the whole stream,
which is correct (the streaming session is stateful and a concurrent run would
interleave two callers' audio) and newly dangerous. Every other RPC holds the
lane across compute, or across a write to a slow reader, and both of those
terminate on their own. A live stream instead blocks in a client-driven read,
and a peer that goes silent WITHOUT closing the stream never terminates
anything: the lane stays taken and every other request against that model queues
behind a client that stopped speaking.

live_watchdog is a one-shot idle timer that ends the stream when no frame has
arrived inside a window. It is standard library only, so it is unit tested
without an engine. gRPC's synchronous Read has no timeout and cannot be given
one, so the only way to unblock it is ServerContext::TryCancel, which decides
the wire status itself: the client sees CANCELLED rather than the
DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming
back is the point. When it fires the read loop throws rather than reporting
end-of-input, so the driver does not go on to finalize a decode nobody is
waiting for.

It is armed only after the lane is taken and disarmed as soon as the read side
closes, and both ends matter. Arming earlier would cover acquire(), which
legitimately blocks while another live stream runs, so a queued caller would be
cancelled for waiting its turn. Disarming later would cover our own decode,
where a window overrun is not a peer going quiet and cancelling would throw away
the transcript the client is waiting for.

The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no
limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds
every tick that produced new audio while a turn is open, so 30 s of silence is a
hundred ticks that delivered nothing. It is also longer than any pause a speaker
takes mid-utterance, which is the case that must never be cut off, and
backend.proto lets one stream span many utterances, so a client that pauses
longer between them raises the option rather than discovering it.

Two smaller corrections in the same handler:

  - check_can_serve now runs BEFORE the sample rate check.
    pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and
    on nothing else, so a live-incapable model asked at a wrong rate was
    answering INVALID_ARGUMENT and costing the caller its fallback.
  - a negative sample rate is refused instead of silently becoming 16000. Zero
    still means 16000, which is what the proto documents; -1 is malformed rather
    than absent and gets the same refusal every other bad rate gets.

And one thing recorded rather than changed, at the handler: "live" here means
incremental INPUT, not low latency, and with the pinned families it does not yet
mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends
to its buffer, so its whole decode and every delta happen inside finalize(),
after the client closes its send side. The policy-window buffering is inert for
that family and matters only for vibevoice_asr and higgs_audio_stt.

Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at
371 ms and was cancelled at 3.371 s; a second live stream opened one second
later received its ack 2.37 s in, i.e. at the instant the first was cancelled,
and then transcribed successfully on the same cached session. Without the
watchdog it would still be waiting. Re-ran the live transcription (ready first,
59 incremental deltas, concat equal to the final text, word timestamps in
nanoseconds, eou and eob false), the citrinet refusal at both a right and a
wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream
on nemotron_asr, which is unchanged.

Mutation testing the watchdog found a weakness in its own test: the destructor
test slept past the window inside the watched scope, so a destructor that
DETACHED the thread instead of joining it passed unnoticed. The test now uses a
window longer than the scope, which kills that mutant, and says why.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): refuse the unsupported RPCs with a reason

AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and
VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns
UNIMPLEMENTED naming the loaded family, what that family does support, and the
upstream limitation, instead of the generated base class's bare status. The
reasons live in a table in capability_routing.cpp so they are data rather than
literals copied into five handlers, and so a test can assert every one of them.

The five claims this was planned against were re-read at the pinned upstream
e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp
streams tts and asr only" is false: silero_vad advertises vad with
RunMode::Streaming. The refusal stands on the narrower claim that survives, that
no family advertises streaming for any task AudioTransform routes to, and a test
asserts the refuted wording does not come back.

VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs
the #10952 check before answering: a stale route must get NOT_FOUND and the
router's sentinel, not "audio.cpp cannot embed speakers" about a model that is
not loaded here. It cannot use snapshot_for, whose no-model branch would tell
the caller to load a model when no model can help, so it takes the reference
through snapshot_unchecked and checks identity itself. That function's comment
now names three classes of caller instead of two.

The two bidirectional surfaces refuse without reading their stream, verified
with a client that writes a config and eight frames first and gets the status
rather than hanging.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): correct the vevo2 clause, and assert the absences

Review found a false clause in the AudioToAudioStream refusal. It said s2s is
"offline voice conversion ... which converts one clip into another speaker's
voice", which is true of miocodec and false of vevo2: vevo2's s2s route is
`editing` and only `editing` (default_route_for_task and route_matches_task in
src/models/vevo2/session.cpp), documented as "Edit source speech into new target
text while using the target voice" and requiring --target-text, so it rewrites
what was said. vevo2's voice conversion is its separate vc task. It now reads
"offline clip-to-clip processing against a target voice, declared only by
miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the
miscast cannot come back. The conclusion is unchanged: neither family converses.

That defect was undetectable on the wire, since vevo2 does not load here, which
is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely
to interrogate make_default_registry() and asserts the five premises the refusal
reasons rest on: no codec task kind, no family advertising spk, no streaming for
sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by
exactly miocodec and vevo2. The last two are exact sets, so an addition fails
here rather than leaving a message stale. A positive control proves the registry
is populated and the query works before any absence is believed, and every
assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION
bump from "remember to re-read five prose paragraphs" into a test failure.

unsupported_surface now switches over UnsupportedRpc with no default label, so
-Wswitch reports a sixth enumerator added without a row at build time; the
runtime bounds guard it replaces is deleted.

The AudioTransformStream reason had a true premise and an overreaching
conclusion: an offline sep family could be buffered into a stream, as other
LocalAI backends do. It now says this backend declines to offer a buffered
offline call in disguise, rather than implying impossibility.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): make the missing-switch-case diagnostic fatal

unsupported_surface() switches UnsupportedRpc onto the table row that explains
it, with no default label, so -Wswitch reports an enumerator nobody handled. As
a warning that is not enough: adding a sixth enumerator and building the shipping
target gives exit 0, a binary and one warning, and the trailing
`return surfaces[0];` then answers the new RPC with AudioEncode's codec reason.
That is a confident, specific and false statement about audio.cpp on the wire, on
the one code path whose entire job is to be truthful about what this backend
cannot do, and it is worse than the runtime fallback it replaced, which at least
named itself as a bug in this file.

capability_routing.cpp therefore joins loaded_model.cpp on the existing
-Werror=switch pin, whose comment already made this argument for the engine enum.
The comment now covers both files. The pin stays per-file rather than
project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled
-Wswitch cases of its own.

Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no
binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so
the two pins fire independently; both reverted clean.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): package the backend image

Bundles the dependency closure for the from-scratch image, the dlopened ggml
CPU-variant shared objects that ldd cannot see, and upstream's bundled
silero_vad and marblenet_vad assets so VAD works with no download.

The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh
execs it, which makes /proc/self/exe name the loader, and this backend has two
consumers of that path: ggml discovers the libggml-cpu-*.so by listing
dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the
same directory. Rooting the loader makes the binary, the ggml objects and
assets/ share the one directory all three resolution mechanisms agree on.
llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well.

The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike
Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and
the base-grpc-* images supply vendors protobuf 26, which pulls abseil into
message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with
sentencepiece's vendored mini-abseil and every absl::internal reference becomes
ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the
pair every earlier verification of this backend ran against.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): exempt the driver libraries from the packaging gate

package.sh already left libcuda.so* and libnvidia-* to the host when copying,
because the driver has to match the kernel module on whatever host runs the
image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas
ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED
libcuda.so.1 and the gate would have rejected the very absence the copy loop
created, failing every cublas build in CI. One regex now feeds both.

Building a control for that found a second defect: ld.so --list refuses to trace
an object with an unresolvable dependency at all, exiting 127 without emitting a
per-library line, so the "=> not found" rule was dead code and no exemption could
have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and
LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also
what run.sh does at run time.

Adds a layout assertion so a future move of the loader into lib/ fails the build
instead of shipping a package that resolves bundled: models into lib/assets and
finds no ggml CPU backend, and records for Task 16 that the Darwin script must
not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh
and would silently drop assets/.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): register the backend with CI and the gallery

Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the
manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps
later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the
bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile,
the gallery meta plus its -development variant and the image entries for every
variant, and the Makefile docker-build wiring.

The matrix entries carry base-image only, with no builder-base-image, unlike
the llama-cpp and privacy-filter blocks they sit next to. The prebuilt
quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose
protobuf v26 depends on abseil, and this backend's sentencepiece is built with
SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116::
internal alongside its own vendored plain absl::internal and every
absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64
fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is
ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself,
which is also the pair every unit and end-to-end run of this backend has been
verified against, and the CUDA toolkit therefore has to come from base-image.

No Darwin matrix entry and no metal gallery entries: the Metal build needs
scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and
a routing step in backend_build_darwin.yml, none of which exist yet, so an
entry added now would be routed to build-darwin-go-backend and look for
backend/go/audio-cpp/. The inferBackendPathDarwin case and the
DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the
entry later is a one-line change that cannot be claimed by the generic Go path.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant

Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target
whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/
linux.md says so outright. ggml's own default does not rescue it: it
list(APPEND)s in the ggml subdirectory scope, which never reaches the root
scope where the engine_runtime property is decided. No CI runner has a GPU for
`native` to enumerate, so both cublas entries would have gone red on the very
commit that first turns a CUDA build on.

Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION,
which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously
discarding. The values are copied from ggml's own version guards rather than
invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the
Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them
and adds 121a-real. The `a` suffix is used rather than `f` because the latter
needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake
3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as
the rejected control.

Drop the vulkan matrix entry, its two gallery entries, the vulkan capability
key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa
ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then
bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and
glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs
a vulkan image against real hardware, so that would have passed green and failed
in users' hands. BUILD_TYPE=vulkan stays supported for local builds.

Also note on the cublas entries that cuda-major-version now selects the
architecture list and that cuda-minor-version and the base-image tag encode the
same toolkit, and correct the stale entry counts on matrixEntryKey.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): build for Darwin Metal

Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix
entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the
metal image entries plus metal/metal-darwin-arm64 capability keys in the
backend gallery.

The build script deliberately does NOT reassemble the package the way
privacy-filter-darwin.sh does. It runs the backend's own `make package` and
copies the result, so the Darwin package keeps the root-level layout the Linux
one has: grpc-server, run.sh, the ggml objects and assets/ in one directory,
with lib/ for the dylib closure. Hand-assembling would drop assets/, and
assets/ is what makes the bundled: model paths resolve with nothing downloaded.
The dylib walk is a full transitive closure rather than the single level ds4
and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares
and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a
package that only works on a machine that already has Homebrew grpc.

Two fixes folded in, both in the backend Makefile:

  - an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list,
    which contains 120a-real and so needs nvcc >= 12.8. A local
    BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where
    upstream's documented default (native) worked. EMPTY now maps to native,
    12 and 13 keep their lists, and any other non-empty value is an error on
    cublas builds. CI always passes a major, so CI is unaffected.

  - the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang
    ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so
    FindOpenMP finds neither the library nor the header, and audio.cpp calls
    find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without
    the hint the macOS build would have died at configure time. If the keg is
    absent the build disables OpenMP instead of failing.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete

Review follow-up on the Darwin Metal build.

The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty,
CI produced a green Metal package with 108 #pragma omp directives across ~30
files compiled out, and clang says nothing about an ignored omp pragma without
-Wsource-uses-openmp, so the only trace was one absent flag inside a set -x
cmake line. That regression would have been blamed on Metal. It now warns.

The @rpath arm of the dylib walk had no live candidate when it was written, on
the reasoning that a Metal build links ggml statically. The OpenMP fix in the
same commit made libomp.dylib one, and whether Homebrew records it as an
absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The
walk now expands @rpath, @loader_path and @executable_path against the object's
own LC_RPATH entries, and only fails when nothing on disk answers, printing the
rpath list with the error so a failure on a machine nobody can attach to
explains itself.

Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they
are deduplicated and their own dependencies bundled; build/darwin/lib is
created explicitly instead of relying on package.sh pre-creating it; the libomp
probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and
would otherwise expand empty and take the OFF branch on an older make; and
-DOpenMP_ROOT is quoted like its CUDA sibling.

Verified with a Linux harness that runs the script verbatim against a stubbed
otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH,
and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates,
system libraries are skipped, the packaged tree has assets/ at the root beside
grpc-server with the dylibs in lib/, and both failure paths exit non-zero.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): make bundled: reachable from a model YAML

resolve_model_path() tested the bundled: prefix on `candidate`, which prefers
ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath
onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and
only sets it from a managed artifact otherwise, so a model YAML saying
`model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad"
and Model "bundled:silero_vad". The prefix therefore never matched through the
normal load path: it matched only for a hand-written LoadModel call that left
ModelFile empty, which is exactly how task 15 verified it, and every model YAML
using the form failed with "model path does not exist:
/models/bundled:silero_vad".

Both fields are now checked, Model first, so the zero-download VAD path the
package ships assets for is reachable the way it is documented. A caller that
puts the form in ModelFile still works, so task 15's verification stands.

Compiled clean; the runtime check could not run on this host, whose system
libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no
longer resolves its libraries either), so it wants a container run.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): advertise the backend and document its options

Registers audio-cpp as preference-only in /backends/known: the family lives in
GGUF metadata that an importer cannot read from a remote repo, and one repo
hosts thirty families, so there is no honest auto-detect signal. Modality is a
single string and the import form chips on a fixed key set, so it registers as
tts with the other modalities named in the description rather than under an
invented key the UI would bucket as "other".

Adds a features page covering the option namespacing, the routing table per
endpoint, the RPCs this backend declines and why, the bundled VAD path, the
separation stem behaviour, and the family gotchas (supertonic needs the orig
package; chatterbox advertises cloning and no plain tts; nemotron_asr defers
its whole decode to finalize so live transcription emits nothing until the
client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option
name and family capability in it was read off the pinned upstream checkout.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(audio-cpp): test resolve_model_path, and correct the family names

The bundled: fix in 842443cd7 shipped without a test, which is how the bug got
there: task 15 verified the form with a hand-written LoadModel that left
ModelFile empty, and that is the one shape the server never produces. Four cases
in streaming_driver_ctest, which already links loaded_model.cpp, pin the
PRODUCTION shapes instead. The first fails against the pre-fix source (returns
the joined /models/bundled:silero_vad); the other three are the branches the
bundled: lookup now runs in front of and must fall through for.

Three family names in the docs were the source directory rather than the
registered family, on pages whose whole argument is that these names cannot be
guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is
mel_band_roformer (roformer/assets.h:15), and moss is TWO families,
moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to
match, here and in the compatibility table.

The supertonic dtype note claimed more than the evidence carries. The f16 abort
is a local observation, identical through TTS and TTSStream; upstream's
docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No
(unsupported weight dtype)", which says unusable rather than fatal. Both are
still refused, because the allow list is what the family can run. Corrected in
family_gate.h, family_gate.cpp and the docs together, since the docs inherited
the wording from the code.

The importers tripwire says in the file that it is a tripwire: it exercises no
audio-cpp behaviour, and the registration assertion lives in backend_test.go.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* gallery: add audio.cpp models covering every served RPC

One representative model per RPC group of the audio-cpp backend, plus the two
bundled VAD models, which need no download at all because the assets ship
inside the backend package.

Every hash was computed with sha256sum on the downloaded file. Quantizations
come from upstream's tested-status table in docs/gguf.md rather than a default
of q8_0: supertonic ships the orig package (its q8_0 is recorded as an
unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr
and htdemucs ship f16 because their q8_0 builds are recorded with drift while
16-bit is a clean pass.

Diarization and separation use the diarization and audio_transform usecases,
not transcript: /v1/audio/diarization and /audio/transform filter the default
model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a
transcript flag would have hidden both models from their own endpoints. The
forced aligner sets parameters.language, which the transcription endpoint uses
as the fallback when no language form field is sent, because the family
requires both a transcript and a language.

All ten entries were run twice: once against the raw gRPC server, and once
installed with local-ai models install and called through the HTTP endpoint.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* gallery: correct the audio.cpp entries' licenses

Swept all ten entries against the real upstream named in audio.cpp's
tools/model_manager.py rather than against the audio.cpp repo's own license.
Three were wrong:

  supertonic   apache-2.0 -> openrail      weights come from
                                           mlx-community/supertonic-3-mlx, and
                                           both it and Supertone/supertonic are
                                           openrail
  citrinet     apache-2.0 -> other         pulled from NGC
                                           nvidia/nemo/stt_en_citrinet_256,
                                           governed by the NGC Terms of Use
  sortformer   other -> cc-by-nc-4.0       nvidia/diar_sortformer_4spk-v1 is
                                           CC BY-NC 4.0, and the gallery already
                                           uses that exact string, so there is no
                                           reason to obscure a non-commercial bar

The license field is one word, so citrinet and sortformer also gained a
sentence saying why they are restricted. The other seven were confirmed
correct against their sources.

Also drops an unverified claim from the nemotron description. It said the
model drives the realtime transcription session; that endpoint actually calls
TranscribeStream, and the live RPC reaches LocalAI only through
realtime_semantic_vad.go. Neither path was exercised here, so the description
now states only the two calls that were.

MarbleNet gains the NeMo upstream under urls: for parity with silero.

No sha256, quantization, usecase or model choice changed.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(audio-transform): bound sample_rate, keep same-named uploads apart

Four defects the whole-branch review found on the Go side, plus two comment
corrections.

sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that
makes the field bind for the first time, so the resample path went from dead to
live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar
with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes
20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second,
into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per
separation stem. Clamped to 8000..192000 in the handler, before the temp dir and
before the model is touched, and rejected with a 400 outside it.

The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte
header with no audio behind it, whose declared data size still claims 70 bytes,
so go-audio parses it as a 35 SECOND file and a size check does not see it. The
guard therefore compares the declared data chunk against the bytes actually on
disk, and AudioResample now fails rather than returning a WAV carrying nothing.

Both parts of a transform request land in one temp dir, and the raw copy was
named only after the client's basename, so `-F audio=@mic/clip.wav
-F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since
AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it,
the reference part's os.Create truncated the inode audio.wav pointed at: mic and
reference came out identical, which makes an echo canceller null everything and
return near-silence with a 200. The raw copy now carries the form field name.

audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil
before it ever consulted the model's tts.voice_cloning override and every
`voice: "profile:<id>"` request was refused with a 400, on a backend that ships
audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered
with its RPCs, usecases and the reference-audio contract, and deliberately
without the 16 kHz mono fold, which its separation families cannot survive.

GetBackendCapability was exact-match only, so every pinned gallery variant read
as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be
unconditional and started failing inside LocalVQE, and the usecase gate does not
stand in for it because BuildFilteredFirstAvailableDefaultModel returns early
once the client names a model. Lookup now falls back to the meta name by
stripping the gallery's hardware prefix and release-channel suffix, exact match
first so nothing can be shadowed. Same class as #10945.

Also corrected: the AudioTransformRequest comment claimed echo's binder falls
back to the field name, which it does not in either direction (bindData binds
ONLY tagged fields and `continue`s otherwise; `model` arrives from
setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap
corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go
developer who would add the field can see it, instead of only in C++.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane

The model's `task:` option is copied into the request shape by all nine
handlers, which is correct, but resolve_route then replaced the RPC's candidate
list with the pin WHOLESALE and never asked whether the pin was something that
RPC routes to. One pin therefore bled across all nine surfaces, and because the
family still supported the pinned task the result was a wrong 200 rather than an
error. Reproduced live: nemotron with task:asr made Vad return 200 with zero
segments after a full ASR decode, so 14 seconds of speech was reported as
silence, and Diarize did the same; silero_vad with task:vad made
AudioTranscription return 200 with empty text and four segments whose spans were
VAD segments, which combined with response_format in {text,srt,vtt,lrc} building
the body solely from Segments[].Text yields a well formed SRT of four timed
EMPTY cues. It also contradicted the documented contract, that a family which
cannot serve a request is refused rather than rerouted.

A pin is now checked against the RPC's admissible task set before it is adopted,
and the refusal names both the pin and the RPC. The set is derived from
task_candidates with every shape flag set rather than restated, so a task added
to an RPC's candidates cannot become inadmissible by omission. Every legitimate
pin survives, and the test asserts all fifteen of them alongside the eight
crossings that must not.

The live watchdog was defeated by empty frames. idle.touch() ran on ANY message,
before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or
zero-length frames faster than the window held the lane indefinitely while
feeding the decoder nothing. There is one lane per model and one model per
process, so that is a single client denying the whole backend, which is what the
watchdog exists to prevent, and the thrown text already said "no audio frame
arrived". The touch moved below the filters, which are now a named predicate so
the distinction is testable rather than a call order nobody can see.

Three comments corrected against measurement rather than reasoning:

- CMakeLists claimed zero google::protobuf:: definitions remain in the
  executable. nm -C --defined-only reports 2515, and that is expected: they are
  generated code, sentencepiece::ModelProto's own _InternalParse among them. The
  claim that holds, and the one the ABI fix is actually about, is that no
  vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is
  UNDEFINED in the executable, resolving to libprotobuf.so.
- refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning
  backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip,
  which is the case being caught; chatterbox, which ships in the gallery,
  advertises clon and no tts at all, so every voice-less request lands there.
- audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min.
  2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and
  the note is now a per-rate table.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk

run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today
only by luck: upstream's 44 tests all put "test" at the FRONT of the filename
(17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of
them, and nothing enforces that. This gate runs on every PR for every backend
and compiles each match as a standalone translation unit with nothing but
nlohmann/json on the include path, so the day upstream adds or renames one test
the gate goes red repo-wide on an Apache-2.0 file nobody here wrote.

audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH
list unconditionally, before the walk. Both awk filters in that script assume a
column layout nobody working on this can observe, since it runs only on the CI
Mac, and a green first Darwin run proves nothing about the assumption: an awk
that silently matched nothing yields an empty dependency list, which reads
exactly like "no non-system dependencies" and packages happily. Both filters
otherwise feed process substitutions, so their input never reached the log.

It also lists every symlink in the package and fails on one that cannot resolve
inside the image. A dangling link does not fail anything else here, because
every assertion tests with -e, which follows links; it fails at dlopen on a
user's Mac. Links are NOT banned outright, which the review suggested but which
would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists
to preserve. What is banned is a link that resolves on the build host and will
not resolve in the image: a broken one, or an absolute one pointing outside the
package.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* style(audio-cpp): drop em dashes from the audio-cpp capability entry

Follow-up to a84b3c4b9, no behaviour change.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(config): key the voice-cloning model rule on the resolved backend

Making GetBackendCapability strip the gallery hardware prefix and release
channel fixed pinned variants of /audio/transform, but VoiceCloningForModel
kept keying its per-backend switch on the caller's spelling. A pinned name
therefore resolved the capability by stripping and then missed every case in
the switch, falling through to the permissive default: cuda12-vibevoice-cpp
advertised voice cloning for the realtime 0.5B model, metal-coqui for
tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for
CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech
accepted a profile: voice it had to fail on inside the backend rather than
rejecting it with a 400, and the UI advertised the capability too.

resolveBackendCapability now returns the key the entry was found under, and
callers that branch on backend identity use that key instead of the name they
were handed. The exact-match-first order is unchanged, so a backend genuinely
registered under a variant-looking name still keys on its own name.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* gallery(audio-cpp): declare audio_transform on the chatterbox entry

Chatterbox advertises VoiceCloning AND VoiceConversion
(src/models/chatterbox), and the entry's own description already said so, but
known_usecases listed only tts. /audio/transform selects its default model by
FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the
model explicitly and was invisible to every usecase-driven surface. It is the
one audio.cpp task with a shipped gallery model and no way to find it.

Verified against the real model rather than inferred from the capability list:
AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip
as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero
stems, which is the single-output shape voice conversion should have.

The description now says which endpoint reaches that half and warns that
installing this next to a source-separation model gives /audio/transform two
candidates, so the model should be named rather than defaulted.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* gallery(audio-cpp): add voice-design and singing-voice-conversion entries

Two of the three audio.cpp task kinds that had no gallery model now have one.
Both were driven end to end against the real weights through the backend
before being written, not inferred from the capability tables.

audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes
to the vdes task, so the voice is described in words rather than supplied as a
clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle
tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop
citrinet pass recovers the sentence with the accent drift expected from a
Japanese-first model read by an English recogniser.

audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else
can reach it. seed_vc advertises svc and ordinary voice conversion, no request
signal means "this input is singing", and auto-routing resolves the tie to
voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output
whose closed-loop citrinet transcription is exact.

s2s deliberately has no entry, and the reason is not effort. miocodec is the
only upstream family whose speech-to-speech route needs no text, and it
returned audio with correct duration and level but no recoverable speech in
four independent attempts: the stale build, v2 q8_0, v2 orig (the variant
upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on
both sides. vevo2's route refuses with "Vevo2 text/prosody route requires
text_input or target_text", and session.cpp:897 fills target_text only from
request.text_input, which AudioTransform has no field to carry. The same
vevo2 weights convert voice correctly through the default route with an exact
ASR round trip, so the model and the plumbing are both healthy; it is the s2s
route specifically that this RPC cannot express.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* backend(audio-cpp): carry transform text through params, add the s2s entry

AudioTransform is audio-in / audio-out and its proto message has no text
field, but not every task it routes to is audio-only. vevo2's speech-to-speech
route is a text and prosody route: session.cpp:897 fills refs.target_text from
request.text_input and nowhere else, and the run refuses without one with
"Vevo2 text/prosody route requires text_input or target_text". The params map
is the only channel this RPC has that reaches the engine, so the text travels
through it and apply_transform_text_input unpacks it after the params have
been copied into task.options.

Before this, s2s was not awkward to reach through /audio/transform, it was
unreachable, and it was the last audio.cpp task kind with a real model and no
way to get to it.

target_text is canonical and text is its alias, the order vevo2's own option
table declares them in, so a request setting both gets the canonical one
rather than whichever the map happened to store first. An empty value falls
through to the next candidate instead of ending the search. language rides
along only when a text was found: on its own it conditions nothing, and
manufacturing a text_input for it would route a plain separation request
carrying a language hint through the text path. The keys are left in
task.options rather than erased, because vevo2's loader advertises target_text
as a request option and a family reading it there keeps working.

Nine tests, all confirmed failing on behaviour against a stub that returned
false before the implementation was written. Verified end to end afterwards:
vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose
closed-loop citrinet transcription is exact, and htdemucs separation with no
text param still returns its four stems, with and without params[stem].

audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind
with a loadable family now has a gallery entry; spk remains the only gap and
has no family upstream at all.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* docs(audio-cpp): document params[text] and the pinned transform tasks

The text channel and the two task pins are both invisible from the endpoint
contract alone: nothing in the AudioTransform form tells a reader that a
speech-to-speech model needs the line it is resynthesising, and nothing says
that asking for singing voice conversion without task:svc silently gets plain
voice conversion instead. Both are the kind of thing a user only discovers
from a refusal or, worse, from output that looks right and is not.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* fix(utils): annotate the two G304 sites this branch introduced

gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go
are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav
opens the exact path it is about to hand ffmpeg as input, which in the upload
path is a server-created temp file named from path.Base of the client name so
no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name
this package derives from src and has just had ffmpeg write.

Annotated in the repo's existing style rather than restructured, with the
reason spelled out, because a bare suppression is worth nothing to the next
reader. The three other G304 sites in this file, in passthroughWAV and
isTargetWav, predate the branch and are left untouched.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants

The arm64 CPU image failed to build:

  cc1: error: invalid feature modifier 'sme' in
       '-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme'

ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and
Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the
table has to compile even though a host only ever dlopens the one its own CPU
supports, so a single unbuildable variant fails the whole image. gcc-14 accepts
it, which is exactly the fix llama-cpp already carries in
.docker/llama-cpp-compile.sh; this is the same problem reached by a different
Dockerfile.

Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that
differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU
image because its GPU builds run llama-cpp-fallback, which builds no variant
table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for
every non-Darwin build, GPU included, so an arm64 GPU image would hit the
identical error. The matrix has no arm64 GPU entry today, which is precisely
why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds
the first one.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 12:11:56 +02:00
mudler's LocalAI [bot]
7d8e0bac18 fix(model): deterministic, type-filtered backend auto-detection (#9287) (#10286) 
* fix(model): deterministic, file-type-filtered backend auto-detect (#9287)

When a model config declares no explicit `backend:`, Load() fell into a
trial loop built by ranging the external-backends Go map (random order)
with no filtering, returning the first backend whose gRPC LoadModel
succeeded. An unrelated installed backend - e.g. the "opus" audio codec -
could therefore win a GGUF/LLM model load, so a model that should run on
llama.cpp wrongly tried to use opus.

Extract the candidate selection into a pure, testable function
SelectAutoLoadBackends that:

  - sorts the candidate list deterministically (no more map-order
    nondeterminism), and
  - for a `.gguf` model, filters to LLM-capable backends (via
    core/config.BackendCapabilities) and puts llama-cpp first, so an
    incompatible audio/codec/image backend can never win the trial loop.

If filtering would leave zero candidates, the full sorted set is returned
unchanged, so a previously-loadable model is never made unloadable.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(model): break core/config <-> pkg/model import cycle in backend auto-detect

The #9287 auto-detect change made pkg/model/autoload.go import core/config
for the backend capability table. core/config already imports pkg/model
(runtime_settings_registry.go uses model.DefaultWatchdogInterval), so this
closed a core/config -> pkg/model -> core/config import cycle and broke the
build and golangci-lint.

Invert the dependency so the lower-level pkg/model no longer imports the
higher-level core/config. pkg/model exposes RegisterLLMCapableBackendFunc and
uses the registered predicate; core/config (which owns the capability table)
registers it from an init(). The deterministic, GGUF-type-filtered selection
behaviour is unchanged. When the predicate is unwired the GGUF filter is
skipped, preserving the existing zero-candidate fallback.

The unit test now injects a fake capability predicate so SelectAutoLoadBackends
is exercised independently of the core/config table.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 12:07:47 +02:00
Tai An
37f2087f97 fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash (#11020) (#11041) 
* fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash

JSONSchemaConverter.visit resolved $ref entries by recursively calling
itself with no cycle detection. A client-supplied grammar_json_functions
schema whose $defs contains a self- or mutually-referential $ref (e.g.
{"A": {"$ref": "#/$defs/A"}}) made visit recurse until the goroutine
stack was exhausted, producing a fatal "stack overflow" that kills the
whole process rather than failing the single request. The schema is
converted synchronously in the /v1/chat/completions handler before any
backend call, so this is an unauthenticated remote crash. Fixes #11020.

Track the $ref targets currently on the recursion stack and error out
when one is re-entered, while popping after each descent so sibling
(non-cyclic) reuse of the same $ref is still allowed.

Signed-off-by: Tai An <antai12232931@outlook.com>

* fix(grammars): add a bounded recursion depth and cover llama31 $ref cycles

Addresses the review on #11041. The stack-set approach catches cyclic
$ref chains, but a deeply nested yet acyclic client schema (thousands of
nested arrays/objects) can still recurse through visit until the
goroutine stack is exhausted, which is the same unauthenticated remote
crash surface as #11020.

- Add a bounded depth counter to JSONSchemaConverter.visit (incremented
  with a defer-based cleanup, capped at maxSchemaDepth = 256, far above
  any realistic schema) so an over-deep schema fails the request with an
  ordinary error instead of crashing the process.
- Apply the same cyclic-$ref guard and depth bound to
  LLama31SchemaConverter.visit, the other production grammar entry point
  named in #11020, which previously had no cycle detection at all.
- Regression tests: a deeply nested acyclic schema is rejected while a
  moderately nested one still builds, plus direct/indirect $ref cycle
  and depth tests for the llama31 converter.

Signed-off-by: Tai An <antai12232931@outlook.com>

* test(grammars): make llama31 cycle fixtures valid function-call shapes

The two new llama31 $ref-cycle specs asserted on "cyclic $ref" but the
converter requires each top-level oneOf alternative to carry its
function-name property before descending, so both fixtures failed
earlier with "no function name found in the schema" and never reached
the cycle guard.

Give each fixture a valid llama31 shape: construct the converter with
NewLLama31SchemaConverter("function"), put "function": {"const": "test"}
on the top-level alternative, and hang the cyclic $ref under an
arguments property, so all 29 grammar specs pass and the assertions
genuinely observe the cyclic $ref error.

Signed-off-by: Tai An <antai12232931@outlook.com>

---------

Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 12:06:17 +02:00
mudler's LocalAI [bot]
c5ae41a29d chore: ⬆️ Update ikawrakow/ik_llama.cpp to 6647db9c27760044950fd6f99060456ae3d15df3 (#11204) 
⬆️ Update ikawrakow/ik_llama.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-07-30 11:23:57 +02:00
mudler's LocalAI [bot]
d225e15f0f fix(ci): skip the image and Go PR workflows on content they cannot see (#11218) 
backend_pr.yml and test-extra.yml already filter themselves, so a gallery-only
or docs-only PR costs them about one job each. The image and Go workflows had no
filter of any kind, so a one-line gallery/index.yaml edit queued 20 jobs: 7
container image builds, 3 GoReleaser/darwin launcher builds, 3 unit test jobs, 2
golangci-lint, 1 e2e, 1 yamllint, plus the 3 that correctly stop after their
detect step. A docs-only PR queued the same.

This matters more than the job count suggests. Measured over the week to
2026-07-30, 97% of CI wall-clock is queueing and 3% is execution: a median
5-hour queue against a 4-20 minute median job. Cutting job count is the only
lever that shortens feedback time. The volume is there to cut, too: 13
gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs
opened were bot-generated.

Add paths-ignore for gallery/**, docs/**, examples/** and **/*.md to the
pull_request trigger of image-pr.yml, build-test.yaml and tests-e2e.yml, and
add gallery/** to lint.yml, which already excluded the rest. That drops 13 of
the 20 jobs. None of the four can observe such a diff: gallery metadata is
parsed at runtime and never copied into an image, docs and markdown never enter
one at all, GoReleaser and the launcher take no such input, the e2e suite drives
backends over gRPC directly, and golangci-lint runs new-from-merge-base so a
diff with no touched Go lines is a no-op. The build-test exclusion also frees
macOS capacity, which is the scarcest runner class.

The two checks that do validate the gallery are deliberately left alone.
test.yml still runs core/gallery/variants_lint_test.go, which reads the real
gallery/index.yaml and asserts the index invariants, and yaml-check.yml still
lints the syntax.

paths-ignore skips a run only when every changed file matches, so a PR touching
the gallery and Go code still runs everything. master carries no branch
protection and no rulesets, so a skipped workflow reports no status and nothing
waits on it; .agents/ci-caching.md records that constraint for whenever required
status checks are introduced.

image.yml on master push is left unfiltered on purpose: skipping it would stop
the master and latest tags being republished for a gallery commit, which is a
publishing decision rather than a cost one.


Assisted-by: Claude:opus-5 [claude-code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 11:22:44 +02:00
localai-org-maint-bot
d6d9f899d6 gallery: add Nanbeige4.2 3B GGUF variants (#11170) 
* gallery: add Nanbeige4.2 3B GGUF variants

Add Q4_K_M and Q8_0 builds of the compact Nanbeige4.2 agentic and reasoning model, grouped as install-time variants.

Assisted-by: Codex:gpt-5

* gallery: simplify Nanbeige4.2 model name

Apply the maintainer-requested canonical model name while retaining the quantization variants under the entry.\n\nAssisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 11:07:32 +02:00
localai-org-maint-bot
c5a7d394a5 gallery: add Laguna XS 2.1 GGUF variants (#11202) 
Add the official Q4_K_M build and seven APEX quality and size variants for llama.cpp.

Assisted-by: Codex:gpt-5 [Hugging Face API]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 11:05:32 +02:00
localai-org-maint-bot
4b917936ef gallery: add Mellum2 Instruct GGUF variants (#11211) 
Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 11:04:17 +02:00
mudler's LocalAI [bot]
aaec1d695e chore(model-gallery): ⬆️ update checksum (#11210) 
⬆️ Checksum updates in gallery/index.yaml

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-07-30 10:33:59 +02:00
Dimitris Karakasilis
c6c347ce13 feat(stablediffusion-ggml): make VAE tiling configurable (#11216) 
GenerateImage hardcoded TilingParamsSetEnabled(vaep, false), so tiled VAE
decoding was unreachable from a model config even though all four upstream
setters were already bound in main.go.

Sampling runs in latent space, but the final VAE decode expands to full
resolution and needs one large compute buffer. At 1024x1024 that buffer
exceeds 8GB, which fails on two kinds of device: cards without the VRAM
for a full-frame decode, and drivers that cap a single allocation
regardless of how much memory is free. Mesa RADV reports a 4GiB
maxMemoryAllocationSize, so a Radeon 8060S with 74GiB of device-local
heap still cannot serve that decode:

    [INFO ] sampling completed, taking 251.82s
    [INFO ] decoding 1 latents
    ggml_vulkan: Requested buffer size exceeds device buffer size limit:
                 ErrorOutOfDeviceMemory
    [ERROR] vae: failed to allocate the compute buffer
    [ERROR] decode_first_stage failed for latent 1

Every sampling step completes and then the run is discarded at the last
stage, so the whole generation is wasted.

Add three options, parsed in Load and applied per generation:

    vae_tiling:true            enable tiled decoding (bare flag also works)
    vae_tile_size:512          tile size, or 512x384 for a rectangle
    vae_tile_overlap:0.25      overlap between tiles

Tiling stays off unless requested, so existing models are unaffected. Tile
size and overlap only reach the library when the operator set them, which
keeps upstream's defaults rather than pushing a zero, and an unparseable
value is treated as absent for the same reason.

Truthy spellings match what load_model already accepts for its own bool
options, and the bare-flag form matches diffusion_model, so no new
convention is introduced.

Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
 2026-07-30 10:33:31 +02:00
localai-org-maint-bot
c10460d4de gallery: add Fara 1.5 27B GGUF variants (#11217) 
Assisted-by: Codex:gpt-5 [web]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 10:23:22 +02:00
Ching
43b6ed2018 feat: add CAJAL gallery model (#9879) 
Add the CAJAL GGUF gallery template and gallery index entry for local llama-cpp installs.

Assisted-by: Codex:gpt-5

Signed-off-by: Ching Kao <0980124jim@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 09:57:31 +02:00
ResearchForumOnline
7a7ebb5c2f add OpenZero Zero GGUF models to gallery (#11138) 
Signed-off-by: ResearchForumOnline <116322650+ResearchForumOnline@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 09:47:38 +02:00
ghshhf
5b9aa02900 fix(ci): skip security scan on forks to avoid SARIF upload permission error (#10323) 
The Security Scan workflow was failing on fork PRs because the workflow
does not have permission to upload SARIF files to the GitHub Security tab
when running from a fork.

This change adds '!github.repository.fork' checks to all steps
to prevent the workflow from running on fork repositories.

This fix should be applied to the main repository so that
all forks inherit the correct configuration.

Fixes #10322, #10318, #10320, #10321

Co-authored-by: ghshhf <ghshhf@users.noreply.github.com>
 2026-07-30 09:42:48 +02:00
localai-org-maint-bot
5f055a407c gallery: add POCKET-35B GGUF variants (#11197) 
Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-30 09:02:44 +02:00
Dimitris Karakasilis
d27c5e82ea Fix use case for video model (#11214) 
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
 2026-07-30 08:57:09 +02:00
Tai An
efb43776ba fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads (fixes #11070) (#11074) 
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads

The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:

- torch/torchaudio: unlike requirements-cublas13.txt and
  requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
  a wheel whose CUDA runtime (cu130) is newer than the host driver
  supports ("NVIDIA driver on your system is too old"). Add the cu124
  index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
  that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
  bare try/except and silently sets PerthImplicitWatermarker=None,
  making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
  Cap to <81 in requirements.txt.

Fixes #11070

Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 00:23:45 +02:00
mudler's LocalAI [bot]
d8a1e3c2e4 fix(realtime): echo response.metadata on response.created and response.done (#11198) 
response.create accepts a metadata map and ResponseCreateParams has carried
the field all along, but triggerResponse never copied it onto the Response it
emits, so both terminals went out with metadata omitted.

That field is the only thing tying a terminal event back to the
response.create that asked for it. Our own doc comment on ResponseCreateEvent
says so — "the metadata field is a good way to disambiguate multiple
simultaneous Responses" — and it is what makes an out-of-band response
(conversation: "none") usable at all: a client running one alongside the
spoken conversation has no way to tell its own answer from the conversation's,
so it waits for a reply it already received and gave away.

Found from the client side: a headless text turn injected into a live session
was answered correctly in about a second, and the caller still blocked until
its own two-minute timeout because it could not recognise the answer.

Carry the map on liveResponse so all three terminals (in_progress, cancelled,
completed) report it, and leave it omitted when response.create sent none.

Assisted-by: Claude:claude-opus-5 gofmt

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-29 23:17:31 +02:00
localai-org-maint-bot
9bfd71387b feat(stores): add Valkey Search vector store backend (#11196) 
* feat: add Valkey Search vector store backend

Add a new built-in Go gRPC store backend 'valkey-store' that implements the
four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*)
using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the
existing per-request 'backend' field on /stores, so there is no proto or HTTP
API change, and it mirrors the in-memory local-store while adding persistence
across restarts and opt-in HNSW.

Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is
created lazily on first Set (FLAT+COSINE by default), cosine similarity is
derived as 1-distance, and namespaces get a collision-resistant token. Includes
unit tests (valkey-go mock) and env-gated integration tests against
valkey/valkey-bundle, plus build/matrix/gallery wiring and docs.

Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* Address review feedback: recover persisted index dimension, harden Find

- Load now recovers the persisted vector DIM from FT.INFO (not just index
  existence), so a post-restart Set/Find validates against the real DIM
  instead of silently re-learning a wrong one and dropping mismatched
  vectors from the index. This also restores Find's dimension check after
  a restart.
- StoresFind treats a dropped/missing index as an empty store (empty
  result, no error) and clears the stale indexCreated flag, matching
  local-store's empty-store behaviour.
- StoresSet reuses checkDims for its per-key length check so the four RPCs
  share one dimension-guard implementation.
- Add unit tests for FT.INFO dimension recovery, loadIndexState, and the
  dropped-index Find path.

Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast

Addresses external review comments on the valkey-store backend:

- StoresFind now rejects a nil/empty query Key before dereferencing it,
  so a malformed gRPC request can no longer panic the backend.
- TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate
  verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT
  (custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs.
- Config integer parsing now fails fast on a malformed value (e.g.
  VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the
  fail-fast behaviour of the index-algo/distance-metric validation.
- Add VALKEY_DB (SELECT n) support for logical-DB isolation.
- Cap the human-readable part of a namespace token at 64 chars so a very
  long model name cannot produce an unbounded key prefix / index name
  (the appended short hash keeps distinct namespaces collision-free).
- Document the KNN-query injection-safety invariant (fields are constants)
  and why StoresGet uses a single aggregate DoMulti deadline for reads.
- Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing,
  and VALKEY_DB parsing/validation; docs + .env updated for the new vars.

Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* Address review feedback: configure valkey-store via model config

richiejp asked that the valkey-store backend take its configuration from
a model config rather than process-wide VALKEY_* environment variables,
so multiple stores can each have their own Valkey config within one
LocalAI process. This removes every env access from the backend and
routes config through the model-config seam every other backend uses.

- config.go: loadConfig(opts *pb.ModelOptions) now parses the model
  config `options:` list (key:value strings, split on the first ':')
  instead of os.Getenv. Option keys mirror the old VALKEY_* names without
  the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast
  validation and the mandatory client name are unchanged.
- store.go: Load threads opts into loadConfig; TLS comments/errors renamed
  off the VALKEY_* names.
- core/backend/stores.go: StoreBackend and NewVectorStore take a
  *config.ModelConfigLoader, resolve the per-store ModelConfig by store
  name, and pass its Options (and Backend when unset) to the backend via
  WithLoadGRPCLoadModelOpts. No config -> default backend + built-in
  defaults, preserving the zero-config experience.
- Endpoints/routes/application: thread the config loader to StoreBackend.
- Unit + integration tests: configure via options; the integration test
  passes addr through the model-config path (VALKEY_ADDR is now only the
  test harness locating the server).
- docs + .env: document the model-config options, drop the env var table.

Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md.

Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts.

Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation.

Signed-off-by: Daria Korenieva <daric2612@gmail.com>

* fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md

Signed-off-by: Daria Korenieva <daric2612@gmail.com>

---------

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
Co-authored-by: Daria Korenieva <daric2612@gmail.com>
 2026-07-29 20:12:29 +02:00
walcz-de
2f33d6dee0 docs(gpu): add ROCm 7.x and RDNA 3.5 / Strix Halo (gfx1151) to GPU acceleration guide (#9229) 
* docs(gpu): add gfx1151 / ROCm 7.x and fix ROCm section

- Fix typo: "deditated" → "dedicated", "ROCm6" → "ROCm"
- Add ROCm 7.x to requirements (alongside ROCm 6.x)
- Add Ubuntu 24.04 to tested OS list
- Add AMD Strix Halo / gfx1151 section with kernel params,
  required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT),
  and Docker Compose example
- Add gfx1151 to the list of compiled GPU targets
- Add ROCm version column to verified devices table
- Add gfx1151 / Radeon 8060S (ROCm 7.11.0) as verified device

* fix(docs/gpu): correct gfx1151 section — env vars, image tag, safety warning

- Add all 4 required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT,
  HSA_XNACK=1, HSA_ENABLE_SDMA=0) with descriptions in a table
- Fix Docker Compose example to use the ROCm 7.x image tag (-gpu-hipblas-rocm7),
  not the ROCm 6.x image
- Add explicit warning: GGML_CUDA_ENABLE_UNIFIED_MEMORY must NOT be set
  (even =0 activates hipMallocManaged due to getenv != nullptr check)
- Add --force-recreate note (docker restart does not update container env)
- Add tested hardware note (Geekom A9 Mega / Ryzen AI MAX+ 395)

* docs(gpu): single ROCm image — drop -rocm7 tag suffix

Per maintainer feedback on PR #9229: there is only one ROCm/hipblas
main image, and it ships with ROCm 7.x by default — no separate
-rocm7 tag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-07-29 20:09:52 +02:00
localai-org-maint-bot
ecdb32193d docs(proxy): cover long inference timeouts (#11065) 
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog.

Assisted-by: Codex:gpt-5 [Codex]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-29 16:37:03 +02:00
Richard Palethorpe
9058a2bb46 feat: Add 3d generation UI/API and trellis2cpp backend (#10979) 
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint

Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:

- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
  glb dst, seed/step/cfg_scale/texture_steps, quality and background
  enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
  the backend interfaces; connection-evicting and distributed-node
  wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
  backend), '3d' canonical usecase string mapped to the Generate3D
  method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
  OpenAIResponse with a /generated-3d URL or b64_json; conditioning
  image accepted as URL, base64, or data URI; quality/background
  validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend

Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:

- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
  asserted at startup), eager pipeline load with model-set validation
  (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
  exactly like the upstream demo), Generate3D via t2_generate +
  t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
  cover resolution/validation/param mapping — CI never downloads the
  multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
  libggml sonames collide across variants, unlike sd-ggml's flat
  renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
  amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
  latest/master image entries, bump_deps tracking of the pbr-textures
  branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
  llama-cpp so the .gguf match isn't stolen) and expands any trellis
  URI to the full 10-file component set spanning the three LocalAI-io
  HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
  trellis2-4b-geometry (512 untextured) with verified sha256s.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(ui): 3D generation page with native GLB viewer and IndexedDB history

Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:

- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
  renderer (quaternion trackball, metallic-roughness PBR, ACES,
  hidden-line wireframe with a bounded index budget) and pairs it with
  a minimal GLB parser for the two forms t2_bake_glb emits — dense
  vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
  normalized integers) and the opt-in UV-atlas textured form. Parsing
  happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
  the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
  GLBs are multi-MB binaries localStorage can't hold — and the page
  offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
  /api/models/capabilities serves), threeDApi, router entries, Studio
  tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
  one-triangle vertex-PBR GLB through the parser/viewer and exercises
  IndexedDB persistence, selection, deletion, and API errors.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(3d): address API correctness and UX issues

Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.

Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(3d): add previewable print remeshing

Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.

Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* build(trellis2cpp): centralize remesh dependency pins

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(kokoros): implement Generate3D stub for new proto RPC

The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.

Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 16:15:04 +02:00
mudler's LocalAI [bot]
8089b2bf09 fix(ci): only rebuild the full backend matrix on breaking backend.proto edits (#11192) 
backend/backend.proto is consumed by every language, so its SHARED_BUILD_INPUTS
rule could only ever be always/always: 417 Linux plus 56 Darwin builds. It fires
on ~1.3% of commits (10 of 767 over six months), which made it the single
largest CI cost driver in the repo.

On 2026-07-29 the queue reached 2178 jobs against 8 concurrent runners. Four
runs totalling 935 of those jobs were triggered by nothing but a proto edit. The
largest, 378 jobs on master, came from PR #11158, whose entire proto diff was
six lines adding `bool cache_prompt = 8;` to one message. No backend that does
not read that field behaves any differently for it.

Make the rule content-aware. changed-backends.js resolves backend.proto at the
base revision (the contents-API pattern already used for backend-matrix.yml) and
hands both texts to protoChangeIsAdditive(), which compares them structurally so
a comment reflow, reindent or field reorder does not read as a change. An
additive-only edit (new field with an unused number, new message, new enum
value, new RPC) suppresses the rule and rebuilds nothing; a removed, renumbered,
retyped or renamed field, a dropped RPC or a changed option still rebuilds
everything, as does an unresolvable base revision.

Every other matched rule is untouched, so a PR that edits the proto and
scripts/build/ is still a full rebuild, and the weekly full-matrix cron remains
the backstop for stale wheels.

Verified against all ten proto commits of the preceding six months: the nine
with a resolvable parent all classify as additive, and controls covering a
retyped-and-renumbered field, a deleted RPC, identical revisions and a
reindent-plus-comment-reflow all classify correctly.


Assisted-by: Claude:opus-5 [claude-code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-29 15:58:27 +02:00
localai-org-maint-bot
89ee62b2af gallery: add KAT-Coder V2.5 Dev GGUF variants (#11186) 
* gallery: add KAT-Coder V2.5 Dev GGUF variants

Add Q4_K_M and Q8_0 builds of the newly released KAT-Coder-V2.5-Dev agentic coding model.

Assisted-by: Codex:gpt-5 [Hugging Face API]

* gallery: add KAT-Coder APEX variants

Assisted-by: Codex:gpt-5 [web]

* gallery: add KAT-Coder APEX checksums

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-29 15:35:51 +02:00
Richard Palethorpe
49ef40a187 feat(classifier/VAD): support voice control on low power devices (#10804) 
* feat(llama-cpp): route Score through the slot loop

Score previously bypassed the slot loop with a direct llama_decode: a
conflict guard aborted the whole process if scoring raced generation, the
config validator had to reject score alongside chat/completion/embeddings,
and every candidate re-decoded the full shared prompt.

Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks
are scheduled like any other slot work: generation and scoring serialize
naturally, the shared prompt is decoded once per call, and the slot's
prompt cache carries the conversation prefix across calls. Context
checkpoints at the score boundary and at the cache-divergence point keep
SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole
prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s
to ~0.5s on a desktop CPU.

The conflict guard and the validation split are removed; declaring score
with generation usecases on one config is now supported and shares the
slot cache.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier wire types and pipeline config

Wire types and YAML config for realtime classifier mode: sessions carry a
localai_classifier extension (options with canned replies/tool calls,
softmax threshold, normalization, history trimming, fallback modes, and a
deterministic wake-word address gate), mirrored by pipeline.classifier in
the model YAML and surfaced in the config-meta registry. The
localai.classifier.result server event reports the full score distribution
per turn.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier response flow

Classifier-mode responses: instead of autoregressive generation, each user
turn is prefill-scored against the option list (router.ScoreClassifier
prompt/candidate shapes over the Score primitive) and the winning option's
canned reply and tool call are emitted through the existing response
machinery. Below-threshold turns take the configured fallback (none /
canned reply / generate); empty transcripts and unaddressed turns (wake
word not mentioned) skip scoring entirely. The scoring probe defaults to
the latest user message only — small scorers echo canned replies from
prior turns back as the top option otherwise.

Built for hardware that can afford prompt processing but not decode: with
slot-based Score the option list stays KV-cached across turns, so a turn
costs roughly one forward pass over the new words.

session_update_error events now carry the validation cause instead of a
generic message.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): bound the VAD tick's scan window and buffer retention

The VAD tick loop re-scanned the entire input buffer every 300ms and only
trimmed it on zero-segment ticks or commits. Audio that keeps producing
segments without a committing pause (steady noise a mic pipeline lets
through, music, continuous speech) grew the buffer toward the 100MB cap
with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero
per buffered second: past ~90s retained, ticks run back to back and pin
~4 cores until the stream stops.

Silero's recurrent state only carries a few hundred ms of context, so
rescanning old audio buys nothing. Clip the slice handed to the VAD to the
largest silence the commit test can need to measure (server_vad silence
window or the semantic eagerness fallback) plus a warm-up margin, and
rebase the returned segment times so every downstream consumer keeps
whole-buffer coordinates. An open turn whose clipped window is all silence
now commits (the silence outran the window) instead of being discarded as
no-speech. Independently, retain at most 90s of raw buffer, rebasing the
live-feed and EOU cursors on trim — this also bounds the previously
unbounded VAD-error path. Turn boundaries are otherwise unchanged: no
forced commits, no new coordinator states.

pipeline.turn_detection.vad_window_sec can widen the scan window; values
below the automatic floor are ignored. The tick body is extracted into
vadTick so specs can drive turn detection synchronously (same shape as
classifySoundWindow); the babble reproduction that pinned 4 cores now
plateaus under 10% of one core.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(backend): let per-model threads override the global default

ModelOptions overrode a set per-model threads value with the app-level
--threads whenever the latter was non-zero — and WithThreads defaults it
to the physical core count, so it always was. The YAML threads: knob has
been dead config: a tiny VAD model could never opt down from the global
pool size.

SetDefaults already fills an unset per-model value from the app config,
which is the intended precedence; resolve threads through a helper that
honors it (explicit threads: 0 still means unset).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(gallery): single-thread the silero VAD

Silero is a ~2MB recurrent model with no exploitable graph parallelism:
measured per-call latency is identical at 1 and 10 ORT threads, while
every extra pool thread just spin-waits between the realtime loop's
frequent tiny inferences.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(realtime): classifier mode, VAD scan window, threads precedence

Document the realtime classifier mode (options, threshold guidance,
wake-word address gate, empty-transcript handling), the VAD scan window
and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the
per-model threads precedence, and the M3 classifier note in the realtime
state-machine design doc.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): score all candidates in one batched decode

One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot
decodes the shared prefix (prompt + longest common candidate token
prefix) once, then forks one sequence per candidate off it
(metadata-only for the unified KV cache, copy-on-write for recurrent
state) and decodes every candidate's unique tail in one llama_decode.
Previously each candidate was its own task that restored the boundary
checkpoint and re-decoded its full tail sequentially, paying
per-candidate task and decode overhead.

The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and
recurrent-state cells) beyond the parallel slots via the new
common_params::n_seq_score_forks. Forking requires the unified KV cache
(already this backend's default) since per-sequence streams would shrink
n_ctx_seq; an explicit kv_unified:false disables forking and Score calls
that need it fail cleanly. Candidates beyond the fork/output budget
decode in successive chunks.

Wire contract and scores are unchanged: per-token logprobs are stitched
from the shared region and the forked tails. Verified bitwise
deterministic call-to-call and independent of candidate order (no
cross-fork leakage via equal-length candidate swap); ranking matches the
per-candidate implementation on the drone battery (winner softmax
0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and
empty candidates all pass.

Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm
realtime classifier turns 196-303ms. The 9-candidate drone turn decodes
~17 unique tail tokens in one batch instead of nine sequential ~220ms
checkpoint-restore tasks.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): gate scoring capacity by model usecase

Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity.

Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step

The builder-prebuilt path installs gcc-14 with apt directly and ignored
the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already
honors, so an ubuntu mirror outage broke every arm64 backend build. Pass
the args into the stage and run apt-mirror.sh (already in the build
context via COPY . /LocalAI) before the apt step.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier argument slots via constrained completion

Hybrid classify-then-complete: a classifier option's canned tool call can
declare typed argument slots (number | enum | string, with defaults and
prompt hints) referenced as "{{name}}" in the arguments template. When
the option wins, the slots are filled by a short grammar-constrained
completion that continues the exact scoring prompt — rendered by the same
cached ScoreClassifier, so the llama.cpp prompt cache is already warm —
with the chosen route JSON re-opened at the first slot field. A GBNF
grammar pins the field skeleton and frees only the values; temperature 0,
a couple dozen tokens at most (~300ms on a desktop CPU for two slots).

Slot declarations and hints ride the option descriptions in the shared
system prompt, informing scoring and the fill alike at no per-turn token
cost. The localai.classifier.result event carries the final arguments and
a fill_latency_ms. On inference failure the slots' defaults apply; a slot
without a default fails the response (or falls through with
fallback.mode: generate). Slot filling requires completion alongside
score in the scoring model's known_usecases.

Verified end-to-end on the Pi drone demo: "fly forward three meters" in
distance mode classifies forward and infers {"distance": 3, "units":
"meters"} in ~310ms, and the drone flies exactly 3 units.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): splice filled slot values into classifier replies

A classifier option's spoken reply can now reference its tool's argument
slots ("Going forward {{distance}} {{units}}."): the values inferred by
the slot-fill completion — or the recovery defaults — are spliced into
the reply as plain text before it is emitted, so what the assistant says
confirms what it actually inferred. Placeholders without a value stay
literal, and options without slots are untouched.

FillToolArguments now returns the raw slot values alongside the spliced
arguments JSON to make the reply templating possible.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): harden classifier slot completion

Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): prewarm the classifier scoring prompt on registration

Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.

Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.

Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix

Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new
small models are headed) cannot rewind their state, so any prompt-cache
reuse that needs a rewind falls back to a full re-prefill. For classifier
scoring that meant every probe change re-processed the whole option-list
prompt: the server's checkpoints were placed reactively (at wherever the
previous task happened to diverge), so a checkpoint past the next
divergence was erased rather than restored — measured as intermittent
2-10s turns on prompts with a 95%+ common prefix.

The classifier now computes the probe-invariant prompt prefix once (the
byte-wise common prefix of two synthetic probe renders) and declares its
length with every Score request; the server maps it to a token boundary
and forces a KV checkpoint exactly there on each score prefill. That
checkpoint sits at or before every future divergence under the same
option list, so it always survives and always restores — repeat scoring
costs probe+candidates regardless of how the probe changes.

Also:
- prewarm reruns on every option-list registration instead of memoizing
  per list: with boundary checkpoints a redundant rewarm costs two
  probe-sized decodes, while skipping one after a slot eviction (three
  lists sharing fewer slots evict in LRU cascades) silently moves a full
  re-prefill onto the user's next turn
- new llama.cpp backend option rs_seq:N exposes bounded recurrent-state
  rollback outside speculative decoding; measured impractical for
  deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap
  insurance for small-state models
- docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot
  similarity threshold funnels distinct lists onto one slot)

Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady
state: every turn 285-421ms including mode switches, vs 2.4s post-switch
and intermittent 1.3-2.9s re-prefills before.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): align classifier cache guidance

Document the single-score prewarm behavior and clean the vendored score patch formatting.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(llama-cpp): guard score task for fork backends

TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields.

Assisted-by: Codex:gpt-5 [gh]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(dev): generate gRPC code before commit lint

The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-07-29 12:50:22 +02:00
localai-org-maint-bot
bc21f832aa gallery: add Laguna S 2.1 GGUF variants (#11188) 
Add the official Q4_K_M and Q8_0 builds plus the DFlash speculative-decoding pairing for llama.cpp.

Assisted-by: Codex:gpt-5 [Hugging Face API]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-29 10:27:09 +02:00
mudler's LocalAI [bot]
967becb365 chore: ⬆️ Update ikawrakow/ik_llama.cpp to b054a8b983827c01aec59d4dc273a27c492c51c4 (#11175) 
⬆️ Update ikawrakow/ik_llama.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 09:57:23 +02:00
mudler's LocalAI [bot]
0569bb30a2 chore: ⬆️ Update ggml-org/whisper.cpp to 97c56f1dc1d1100a9d859c865a20c82d22f823ed (#11182) 
⬆️ Update ggml-org/whisper.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 09:57:12 +02:00
mudler's LocalAI [bot]
550545c03a chore: ⬆️ Update mudler/parakeet.cpp to e747acdaee69b916cef62263ae5f718bda9ff3f3 (#11181) 
⬆️ Update mudler/parakeet.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 09:57:01 +02:00
mudler's LocalAI [bot]
c5e5141010 fix(ci): unbreak the sglang and darwin nemo backend builds (#11168) 
* fix(sglang): keep nvidia-modelopt on a stable release

Every cublas sglang image currently fails to build:

  Failed to build `nvidia-modelopt==0.46.0rc0`
  Call to `wheel_stub.buildapi.build_wheel` failed
  ModuleNotFoundError: No module named 'wheel_stub'

sglang[all] pulls nvidia-modelopt in through its `diffusion` extra with no
version bound of its own, and install.sh adds a GLOBAL --prerelease=allow so
that flash-attn-4, which only ships 4.0.0b* wheels, can resolve. Unbounded plus
prereleases-allowed picks 0.46.0rc0, whose build backend imports wheel_stub
without declaring it in build-system.requires. EXTRA_PIP_INSTALL_FLAGS also
starts with --no-build-isolation, so nothing installs wheel_stub and the build
dies. Latest stable is 0.45.0 and resolves cleanly.

Bounding this one package rather than dropping the global flag, because the
flag is load-bearing for flash-attn-4 and this is the narrower change with the
smaller blast radius. Raise the bound when 0.46.0 final ships.

This is invisible on master because the backend build is path-filtered: sglang
is only rebuilt when sglang changes. It surfaces on any PR touching a shared
build input such as backend/backend.proto, which rebuilds the whole matrix.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* fix(nemo): build the darwin venv on Python 3.12

The darwin nemo image fails to build:

  ModuleNotFoundError: No module named 'maturin'

nemo_toolkit pulls in text2num, a Rust extension built with maturin, whose
macOS arm64 wheels start at cp311: 3.0.2 publishes cp311, cp312, cp313 and
cp314 and no cp310. libbackend.sh defaults PYTHON_VERSION to 3.10, so pip finds
no wheel, falls back to the sdist, and dies in the PEP 517 hook because
EXTRA_PIP_INSTALL_FLAGS carries --no-build-isolation and nothing installs the
build backend. Taking the prebuilt wheel avoids the source build entirely, so
the runner needs no Rust toolchain.

Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
for the same package and have no reason to move. The override is set after
libbackend.sh is sourced and before installRequirements, the same shape
sglang's install.sh already uses for its l4t13 profile.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* fix(nemo): pin the darwin portable-Python patch level too

The 3.12 bump alone traded one failure for another:

  curl: (56) The requested URL returned error: 404
  make[1]: *** [nemo-asr] Error 56

libbackend builds the portable-Python URL from
cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}, and
PYTHON_PATCH defaults to 18 because the default interpreter is 3.10.18. Setting
only PYTHON_VERSION asked for a 3.12.18 that was never released.

Patch 11, not the 12 that sglang/install.sh pairs with 3.12 for l4t13: at the
20250818 tag python-build-standalone published 3.12.12 for linux aarch64 but
not for aarch64-apple-darwin, where 3.12.11 is the newest. Both URLs were
checked against the release assets rather than assumed to match.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 09:56:50 +02:00
mudler's LocalAI [bot]
47c48e9409 chore: ⬆️ Update antirez/ds4 to 54b36ed9ba42da31b24f2d1a5feb075c2475dbb1 (#11178) 
⬆️ Update antirez/ds4

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 08:45:31 +02:00
mudler's LocalAI [bot]
6aaf7db5f0 chore: ⬆️ Update mudler/depth-anything.cpp to 2028b47ac75a8659c6a9aa617baf09be193eb55f (#11179) 
⬆️ Update mudler/depth-anything.cpp

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 08:45:18 +02:00
mudler's LocalAI [bot]
193d49001b chore: ⬆️ Update CrispStrobe/CrispASR to 754b67289cf1137e3ed722885705f94132fc614f (#11180) 
⬆️ Update CrispStrobe/CrispASR

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-29 08:44:45 +02:00
localai-org-maint-bot
8f9184fbb2 feat(cli): support systemd socket activation (#11169) 
* feat(cli): support systemd socket activation

Serve the API from a single stream listener inherited through the systemd activation protocol while retaining the existing address bind path when no listener is provided. Validate activation metadata, preserve the public-bind safety check, and document an on-demand systemd setup.

Assisted-by: Codex:gpt-5

* fix(cli): satisfy listener cleanup lint

Make the best-effort close explicit so errcheck accepts the deferred systemd listener cleanup.

Assisted-by: Codex:gpt-5 [golangci-lint]

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-29 01:44:24 +00:00
mudler's LocalAI [bot]
84972cb745 chore(model-gallery): ⬆️ update checksum (#11176) 
⬆️ Checksum updates in gallery/index.yaml

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
 2026-07-29 00:11:10 +02:00
localai-org-maint-bot
034df6ceb1 fix(worker): report RAM alongside GPU memory (#11167) 
* fix(worker): report RAM alongside GPU memory

Assisted-by: Codex:gpt-5

* feat(ui): show worker RAM on node views

Assisted-by: Codex:gpt-5

---------

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-28 23:55:35 +02:00
localai-org-maint-bot
996bdcecdc fix(mlx-vlm): install torch dependencies on Metal (#11164) 
Some Transformers processors used by MLX-VLM, including Qwen vision models, import both PyTorch and Torchvision. Include them in the Metal backend environment so model loading does not fail with missing-library errors.

Assisted-by: Codex:gpt-5

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-28 19:45:07 +02:00