mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-19 03:32:29 -04:00
211aa0a536aa294301ac8576fb13be2aa60aff2a
1822
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler's LocalAI [bot]andmudler
|
211aa0a536 |
chore: ⬆️ Update mudler/vllm.cpp to a42b8187caff02c570c28e19e4dc2b1d7f55ed14 (#11174)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
c86b3b207b |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 60389410a1ff01f9d37dcc6261db33b3183bdea2 (#11331)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
62316e52a9 |
chore: ⬆️ Update 0xShug0/audio.cpp to 4e3aea2fd99aeaa5924e71c51eb2793846045332 (#11332)
⬆️ Update 0xShug0/audio.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
3090101156 |
chore: ⬆️ Update CrispStrobe/CrispASR to fe3caf8e363b27572dbdd1a9d37083f25e6decda (#11334)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
8b667cd1ce |
chore: ⬆️ Update ggml-org/whisper.cpp to 64d57d3df5c8dacee098577257edcaa154bf5ef3 (#11326)
⬆️ Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
f447faf08d |
chore: ⬆️ Update ggml-org/llama.cpp to 221f0f6356efe2260023208365705ec5d5a7c8f5 (#11303)
⬆️ Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
b6d2e94153 |
fix(sglang): bound cuda-tile below the 1.6 prereleases (#11320)
Every CUDA sglang image failed in the 2026-08-02 full-matrix rebuild:
-gpu-nvidia-cuda-12-sglang, -gpu-nvidia-cuda-13-sglang and
-nvidia-l4t-cuda-13-arm64-sglang, all with the same build error.
Building cuda-tile==1.6.0rc3
x Failed to build `cuda-tile==1.6.0rc3`
ModuleNotFoundError: No module named 'wheel_stub'
hint: `cuda-tile` (v1.6.0rc3) was included because `sglang` (v0.5.16)
depends on `flashinfer-python` (v0.6.14) which depends on `cuda-tile`
This is the failure mode requirements-cublas1{2,3}-after.txt already
carries an nvidia-modelopt bound for, arriving through a different
package. install.sh passes a global --prerelease=allow, which is
load-bearing for flash-attn-4, so an unbounded dependency resolves to a
prerelease; cuda-tile 1.6.0rc3's build backend imports wheel_stub without
declaring it in build-system.requires; --no-build-isolation means nothing
provides it, and the build dies.
Nothing in this repo changed. cuda-tile published 1.6.0rc1 and rc3 and
the weekly cron picked them up, which is the drift that job exists to
catch.
Bound the one package rather than dropping the global flag, matching the
existing precedent. 1.5.0 is the newest stable release, so <1.6 takes the
last good one. l4t13 gets the same bound: it installs plain sglang rather
than sglang[all], but flashinfer-python is a dependency of both.
NOT VERIFIED LOCALLY: reproducing this needs a CUDA docker build, which
this machine cannot run. The diagnosis is from the CI log and the
resolver's own hint, and the change follows a fix already proven in these
same files. CI on this PR is the check that matters.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0f7faaa2a |
fix(sycl): stop building the ggml CPU variant matrix with icpx (#11321)
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS matrix, so a partial offload uses the host's SIMD kernels. That works everywhere except SYCL, where the Makefile compiles the whole tree with icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at -march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped at that translation unit and sat there for 5h30m with a single compile in flight until GitHub killed the job at its 6h limit, and turboquant's f16 job lost its runner outright. gcc compiles the same file in seconds in the vulkan and CPU jobs of the same run, so the CPU variant matrix is only unbuildable under icpx. Route SYCL back to the portable fallback binary, which is what these images shipped before #11255. run.sh already prefers *-cpu-all when present and falls back otherwise, so nothing else has to change. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Peteandlocalai-org-maint-bot
|
8a68f3571c |
feat(api): add POST /v1/images/upscale endpoint (#10227)
* feat(api): add POST /v1/images/upscale endpoint Add a new image upscaling endpoint that accepts a source image and returns an upscaled version. Supports selectable upscaler models (e.g. realesrgan) and a configurable scale factor (2x or 4x). - backend.proto: add UpscaleImage RPC and UpscaleImageRequest message - pkg/grpc: implement UpscaleImage in Backend interface, client, server and embed shim - core/backend/upscale.go: new backend helper (mirrors ImageGeneration) - core/http/endpoints/openai/upscale.go: new multipart/form-data handler - core/http/routes/openai.go: register POST /v1/images/upscale - core/http/auth/features.go: gate upscale routes under FeatureImages - backend/python/diffusers/backend.py: implement UpscaleImage — uses diffusers upscale pipeline when loaded, falls back to Lanczos resize * fix(grpc): add UpscaleImage stub to Base backend All Go backends embedding Base now satisfy the AIModel interface without needing to implement UpscaleImage explicitly. * fix(images): complete upscale endpoint integration Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
cd890b6a26 |
chore: ⬆️ Update leejet/stable-diffusion.cpp to db99efdd6d2a43c7937fd55b3359206c680a75b0 (#11299)
⬆️ Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
1c0380ad44 |
chore: ⬆️ Update 0xShug0/audio.cpp to 5a8312ef7b8aa7cf14e9a24ac568cabd8725d68a (#11302)
⬆️ Update 0xShug0/audio.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
cb6e4d4391 |
chore: ⬆️ Update CrispStrobe/CrispASR to fcb79282a6bc52e13d858026c42b24fb6e63c97a (#11304)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
45cd47cb99 |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to cb9147fd0d9c08a9a84eee5ac405a73f4e10e3e1 (#11300)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
8a80830f33 |
chore: ⬆️ Update ggml-org/llama.cpp to a7a6d0d269c896218b6c78e0933bd6a17519d3f6 (#11283)
⬆️ Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
0990be35b7 |
chore: ⬆️ Update PrismML-Eng/llama.cpp to 9ca265a57f85f2117942490f421f64a226dd9847 (#11280)
⬆️ Update PrismML-Eng/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
a49f115b0d |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 0be97a7a5ad113f33e08729261649ccea2cdc5ff (#11282)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
0d6b38e709 |
chore: ⬆️ Update 0xShug0/audio.cpp to 545e29a6f2fde24298cb3b0f07baab4352987ac9 (#11281)
⬆️ Update 0xShug0/audio.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
bef30732cd |
chore: ⬆️ Update CrispStrobe/CrispASR to 66ac7843e319b588f5410051c575affd19424fb3 (#11279)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
21ecc799e5 |
fix(qwen3-tts-cpp): hold qwentts.cpp at 35ebe537, upstream master hangs in synthesis (#11286)
tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.
goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
qwen3-tts-cpp.(*Qwen3TtsCpp).TTS goqwen3ttscpp.go:154
qwen3-tts-cpp.init.func2.4 e2e_test.go:90
Not a flake: reproduced on master and again on an explicit re-run.
Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.
35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.
Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
9fe1165f61 |
fix(turboquant): retain CPU variants in GPU builds (#11276)
Select the CPU_ALL_VARIANTS target for x86 GPU images so partial offload uses runtime-selected host kernels. Keep GPU arm64 builds on the portable fallback until their toolchains consistently provide gcc-14. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
|
ad2be8a856 |
chore: ⬆️ Update ggml-org/llama.cpp to 876a4321163249c43ca4e986818fab5ab081f282 (#11177)
* ⬆️ Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(llama-cpp): drop merged MiniMax-M3 patch The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch. Assisted-by: Codex:gpt-5 [systematic-debugging] --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
cedcbf97a9 |
fix(llama-cpp): retain CPU variants in GPU builds (#11255)
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
7e4a60c701 |
chore: ⬆️ Update TheTom/llama-cpp-turboquant to 8a891f4b566efdbd3cea92fafee3227a0a267683 (#11258)
⬆️ Update TheTom/llama-cpp-turboquant Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
a7440f032d |
chore: ⬆️ Update PrismML-Eng/llama.cpp to 4dd165625bb6c020285eec8b342af25cf60233dd (#11259)
⬆️ Update PrismML-Eng/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
4a6cd227a3 |
chore: ⬆️ Update 0xShug0/audio.cpp to f78227c52736a4792a50aa3f82ead7e7385c891b (#11261)
⬆️ Update 0xShug0/audio.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
740d8684b5 |
chore: ⬆️ Update ggml-org/whisper.cpp to 2ca53bb45e38748d07b310eeb36245a7157ac882 (#11263)
⬆️ Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
cb432c4c99 |
chore: ⬆️ Update CrispStrobe/CrispASR to b5211ac635489049ee8ce86a82d69faa18e8d8da (#11264)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
a4cd387100 |
chore: ⬆️ Update localai-org/rf-detr.cpp to 98d0f381b832ef08a608b65c7dd78db066ed8b9a (#11260)
⬆️ Update localai-org/rf-detr.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
Dimitris Karakasilisandlocalai-org-maint-bot
|
c089caf320 |
feat(sycl): make the intel llama.cpp backend self-contained on any host (#10991)
* feat(sycl): make the intel llama.cpp backend self-contained on any host The SYCL backend shipped an incomplete oneAPI runtime AND relied on a host-provided GPU driver, so it only ran inside the build container. On a bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open shared object file", and even with the host's Intel driver installed it SIGSEGV'd during SYCL init when the host driver was built against a newer glibc than the backend's bundled loader (rolling-release distros). package_intel_libs now bundles the complete, coherent oneAPI runtime (the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd UR adapters, plus a sweep of the backend binaries' own direct deps) and the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm) with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled driver, and install-base-deps.sh installs it in the SYCL build image. Bundling the driver is safe across kernels because it talks to the host i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked userspace). Validated on Arch (glibc 2.43, i915): the backend loads and runs on an Iris Xe with no host Intel packages installed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): install a driver that exists, and let the user choose their own The driver install added earlier in this branch asked apt for intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails outright on an unknown name, so neither driver was installed, nothing was there to copy, and the images carried no driver at all. It now comes from Intel's own repository, which has 25.18 for this Ubuntu release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver does not know any card released since, so a machine with a recent Intel GPU would end up carrying a driver that cannot drive it. Anything that goes wrong during that install fails the build on purpose: an unreachable repository is a passing problem that a retry fixes, while quietly carrying a different driver, or none, is a difference nobody would notice until a user reports an idle GPU. run.sh used to overwrite whatever driver the user had chosen. Level Zero uses only the driver it is given, so on a machine with a card too new for the carried driver, the GPU would go unused with no way back. Both that setting and the OpenCL one are now left alone when already set, and the docs say how to point a backend at the machine's own driver. The OpenCL setting also used to be applied whenever the backend held a driver list, even when the driver it named had not been copied, which leaves OpenCL with nothing instead of falling back to the machine's own driver. It now requires the copied driver to be present, and the packaging leaves out the list entry of any driver it did not copy. The oneAPI images list a processor-only OpenCL library, which was being carried with nothing behind it. Two more corrections in the packaging. The scan for libraries a program is linked against only looked at files named llama-cpp-*, so turboquant and bonsai, which are also built for Intel GPUs, were left with the incomplete set of libraries this branch set out to fix; it now looks at every program in the directory. And a build that should carry a driver but ends up without one now says so, which is what a stale prebuilt base image looks like: such a backend still runs on a machine that has its own driver, so nothing fails and the only other symptom is a user reporting an idle GPU. Backends now also ask the driver to report how much graphics memory is free, without which llama.cpp reads zero on an integrated GPU, since such a chip shares the system memory instead of having its own. turboquant and bonsai get the same run.sh handling as llama.cpp. The driver is only carried by the builds that start through run.sh, because run.sh is what points Level Zero and OpenCL at it. The Python backends for Intel GPUs start differently and would never load it, so they keep using the machine's own driver rather than carrying several hundred megabytes they cannot use. Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with the files where the packaging expects them, an unreachable repository fails the build, and the copied set resolves on its own once the machine's Intel packages are moved away. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(ci): rebuild every Linux backend when the GPU packaging script changes scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an image. The filter that builds the backend matrix listed it as an input of the Python images only, so changing it rebuilt no Go and no C++ backend, even though those run it from their own package.sh. A packaging fix aimed at the Intel llama.cpp backend could merge and reach no image, which is the same failure this rule was written to prevent. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): carry only the driver Level Zero uses, not the OpenCL one llama.cpp reaches an Intel GPU through Level Zero, which hands the driver programs that are already compiled and so needs only the back end of the graphics compiler. The OpenCL driver can be handed source code instead, so it needs the compiler's front end as well, and that arrives with its own copy of clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and took the carried set from 123 MB to 261 MB. Nothing here takes that path. No LocalAI code selects an OpenCL device, each backend image holds one backend, and the documentation never described OpenCL as a way to run models: the only mentions are a stale clblas row in the BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no build matrix entry uses, and the sycl-ls troubleshooting hint. Before this branch the packaging carried the OpenCL loader and adapter but no driver, so the path could not work in a released image either. There is nobody to keep working. The driver list that OpenCL reads is no longer carried, and run.sh no longer sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the image provides rather than being pointed at a directory with no driver in it. Checked in a container against the real 25.18 driver: the carried set is 123 MB with nothing unresolved, and Level Zero still reports the GPU with the machine's own Intel packages moved out of the way. Neither the Level Zero driver nor the compiler back end names the front end or clang among the libraries it opens by name, so the leaner set is complete for this path. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> --------- Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andmudler
|
51c9cc1934 |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 3f53a059024039358e9fef75b5dc0c99dbcb40f9 (#11262)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
b60b01d783 |
chore: ⬆️ Update 0xShug0/audio.cpp to f32876cfb45732dd4f43264e9104d229e95b0bc3 (#11233)
⬆️ Update 0xShug0/audio.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
5d461ec7d2 |
chore: ⬆️ Update CrispStrobe/CrispASR to 677e95d0e60010f10636c3a0b1ba215b38a4a943 (#11234)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
25a8a73b35 |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 9992f6b515ee63c7d6f7beee6b8414b0a6d1dd43 (#11235)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
4076b32d42 |
chore: ⬆️ Update leejet/stable-diffusion.cpp to e31a86ce9110b11a98bd5990c329093244c2d1e3 (#11237)
⬆️ Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
d2be530d14 |
chore: ⬆️ Update ggml-org/whisper.cpp to 4523d0ce373ee4b2176b3251fff29fd4864fcf38 (#11240)
⬆️ Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
735420c216 |
chore: ⬆️ Update ServeurpersoCom/qwentts.cpp to abab6b3bf317cfa1b788efce1d25f4f9239395ad (#11241)
⬆️ Update ServeurpersoCom/qwentts.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
704b87dc8e |
chore: ⬆️ Update leejet/stable-diffusion.cpp to e92e86fb11b3028ac9edaf63d93709801d106b12 (#11206)
⬆️ Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
Leoyandlocalai-org-maint-bot
|
632c4b6db2 |
refactor(backends): extract package-system-libs.sh from 31 package.sh (#11095)
refactor(backends): extract shared package-system-libs.sh from package.sh
The arch-detect-and-copy-system-libs block (Darwin rpath / x86_64 / aarch64
loader + libc/libstdc++/libgcc_s/libm/libgomp/libdl/librt/libpthread) was
inlined verbatim in 31 backend package.sh scripts. Extract it into a single
sourced scripts/build/package-system-libs.sh, the CPU-side counterpart to
scripts/build/package-gpu-libs.sh and its sourcing contract.
Consolidating the copies fixes three drift classes that had crept in:
- libgcc_s.so.1 and libstdc++.so.6 were listed twice in 9 backends
(acestep-cpp, crispasr, moss-tts-cpp, omnivoice-cpp, piper,
qwen3-tts-cpp, silero-vad, stablediffusion-ggml, whisper); the shared
script copies each once.
- libgomp.so.1 was omitted from opus. OpenMP consumers dlopen it rather
than link it, so the missing copy only failed at runtime; the shared
script always includes it.
- the Darwin @loader_path/lib rpath was applied only in piper and
silero-vad; both now pass their packaged binary to the shared script,
preserving that behavior. Every other backend passes an empty binary
path so no rpath is added, preserving its current behavior.
Each backend's pre/post packaging steps (binary copy, run.sh, ldd closure
walks, ggml variant bundling, espeak/OpenBLAS extras, the ds4 validate step)
are preserved verbatim; only the inline if/elif/else arch block is replaced
by a single source line.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
|
||
|
Adiraandlocalai-org-maint-bot
|
ef724a3c9d |
feat(api): add /v1/detokenize endpoint (#9620)
* feat(api): add /v1/detokenize endpoint Closes #1649. Mirror of the existing /v1/tokenize path, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing. - Add Detokenize gRPC RPC with DetokenizeRequest{tokens} / DetokenizeResponse{content} messages. - Implement in the llama.cpp backend using common_token_to_piece, the same primitive TokenizeString already uses internally. - Other backends inherit the default Unimplemented from base.Base, in line with how Detect, Rerank, etc. are gated per-backend. - Wire up the Go gRPC interface, server, client, and in-process embed wrapper alongside their TokenizeString counterparts. - Add the schema types, ModelDetokenize wrapper, HTTP handler, route registration, RouteFeatureRegistry entry (gated by FeatureTokenize so no new feature flag is needed), and the discovery map entry under ai_functions. - Regenerated swagger reflects the new endpoint and types. - Update authentication.md to list /v1/detokenize alongside /v1/tokenize. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * test(e2e): add mock backend tests for /v1/detokenize Add Detokenize to the mock gRPC backend and wire up two e2e tests in the MockBackend suite: one that posts known token IDs and asserts a non-empty content response, and a round-trip that tokenizes first then detokenizes the returned IDs. Addresses reviewer feedback on #9620. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * fix(kokoros): implement detokenize in the Rust backend service The Detokenize RPC added in this PR grows the tonic-generated Backend trait. Unlike the other languages there is nothing to inherit a default from — Rust trait impls must list every method — so backend/rust/kokoros failed to compile: error[E0046]: not all trait items implemented, missing: `detokenize` --> src/service.rs:72:1 72 | impl Backend for KokorosService { Go backends pick up the Unimplemented default from base.Base, and the generated C++/Python servicer bases default to UNIMPLEMENTED, which is why the Rust backend was the only one that broke. kokoros is the sole Rust crate in the tree, so this is the full extent of the fallout. Return Status::unimplemented("Not supported"), matching how this same file already gates tokenize_string and ~20 other unsupported RPCs. Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros) failures on the previous head. Assisted-by: Claude:claude-opus-5 cargo Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> --------- Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andmudler
|
965180581b |
chore: ⬆️ Update ggml-org/whisper.cpp to a630b35c6fc02c8879f751ec3f39a61327f01dc7 (#11205)
⬆️ Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
5e541894df |
fix(llama-cpp): preserve GPU layers during option passthrough (#11193)
Stage the negative GPU-layer sentinels expected by the upstream argument parser, then restore LocalAI resolved values unless a passthrough flag explicitly overrides them. This avoids the parser assertion that terminated the backend for any generic option. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
79e88581c8 |
chore: ⬆️ Update localai-org/trellis2cpp to 2f3e6e26edbbaaf8ce93d092f16f46968a366a6a (#11208)
⬆️ Update localai-org/trellis2cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
052ccb5e00 |
chore: ⬆️ Update mudler/parakeet.cpp to 1bfbebfaaf493866f49597cd3b7901959d395c60 (#11209)
⬆️ Update mudler/parakeet.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
aa5a17d452 |
chore: ⬆️ Update CrispStrobe/CrispASR to 4e863bae52aa76a875e4aca57db54ae6d4145c5c (#11207)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
cbc916aec2 |
fix(ci): build the native engine in a layer the registry cache can restore (#11221)
Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA and compiles it once per SIMD variant (depth-anything-cpp builds four: avx, avx2, avx512, fallback), and those variant targets depend only on the clone. They cannot observe a change anywhere else in the LocalAI tree. The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and recompiled C++ that had not changed. Move it above that COPY, behind a copy of only the backend's own directory. This lands the compile in the part of the image the registry cache already restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008 and 90551028904, both fresh runners): 13 of 17 layers CACHED from quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the git-config RUN and the build RUN. Putting the engine above the COPY moves it from the uncached tail into the cached region. Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file outside the backend: master 78s, 216 C++ objects compiled this change 25s, 0 C++ objects compiled (67% faster) Note what this deliberately is not. An earlier attempt wired a --mount=type=cache ccache into the same RUN. BuildKit does not export cache mounts to a registry cache, so that measured well locally and is a no-op in CI (see the ccache section of .agents/ci-caching.md). This change relies only on ordinary layer caching, which the 13-of-17 figure above shows already works here. The layer copies the backend's whole directory rather than just the Makefile: the CMake targets also need CMakeLists.txt and the file list differs per backend. The cost is that editing a backend's own Go sources invalidates its engine layer. The expensive cases are unaffected, since a shared-build-input or backend.proto change, the weekly full-matrix cron and a tag push all rebuild every backend while touching none of their directories. Scoped to one backend for now: only depth-anything-cpp gains the `engine` target. The other 27 fall through the `make -n engine` guard and build exactly as before, verified against local-store and silero-vad. Rolling the target out to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this is confirmed against the registry cache on master. One caveat on merge: inserting layers shifts the cache keys, so the first build of each entry after this lands is a full miss. It pays for itself on the second. Assisted-by: Claude:opus-5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
47097041ff |
fix(vllm): apply Options[] engine flags before engine init (#11147)
fix(vllm): apply Options[] engine flags before engine init (#11130) CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`, `--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could still reach the engine. Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine is constructed. Names are normalized the way vLLM's CLI spells them (`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to the target field's type (bare flag -> True for booleans), and unknown or uncoercible flags warn and are skipped instead of failing the load, since Options[] is a bag shared with backend-level settings. Field types come from the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not mistaken for a float. Precedence is typed proto fields -> `options:` -> `engine_args:`. The production engine_args defaults seeded in hooks_vllm.go therefore skip any key the user already set as an option, otherwise the later engine_args pass would silently override it. Parser lookups now accept both spellings, so `--reasoning-parser:qwen3` selects LocalAI's parser as well. The helper's tests are stdlib-only and run in the lint workflow's dependency-light job via `make test-python-helpers`. Assisted-by: Claude:claude-opus-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9c85cacfe3 |
feat(audio-cpp): add the audio.cpp native backend (#11141)
* backend(audio-cpp): add the native build scaffold Links 0xShug0/audio.cpp engine_runtime through its public framework headers and serves Health/Status. Model loading and the audio RPCs follow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep the build-tree rpath at $ORIGIN Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so CMake was appending its build-tree library dir to our target and baking an absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH on the target so a package that forgets to bundle libggml*.so fails on the build machine too, instead of only on a user's box. Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call, correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment from repeating the assignment token that bump_deps.sh rewrites, and make test-engine fail rather than pass when no test is registered. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): parse namespaced model options Splits option entries on the first colon so path values survive, and routes load./session. prefixes to the upstream load and session option maps. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): reject out-of-range numeric model options std::atoi is undefined once the digits exceed long and in practice wraps, so device:2147483648 was accepted and handed the ggml backend selector a device index of -2147483648 from a function whose error text promises a non-negative integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and on any unconsumed trailing input. The error strings are unchanged. Name the whole entry in the unknown-key error too: an entry like ':value' has an empty key and left the user nothing to grep for in their YAML. Tests look keys up through a helper instead of map::at, so a prefix off-by-one fails one named check rather than aborting the binary and skipping the rest of the suite, and cover the overflow, negative and non-numeric paths. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks Task-major resolution over the family's advertised capability set, with the voice-reference and instructions signals selecting cloning and voice design, and a streaming-to-offline fallback for server-streaming transcription only. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): use upstream's 'spk' task name and pin the preference order The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec' would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an input-only alias, and correct the known-tasks lists. Three assertions were vacuous because their fixtures advertised a single task, so reversing a preference order or dropping the RPC name and the attempted pairs from the capability error all passed. Give them fixtures that can tell the orderings apart. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): convert sample, time and PCM units Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the VAD and diarization messages, and saturating s16le encode so an overshooting sample cannot wrap to the opposite sign. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): harden seconds_to_samples against NaN and overflow seconds_to_samples is the one entry point fed by untrusted-shaped input: a float-seconds timestamp off the wire, or a boundary from a model that diverged. Its guard covered only the low side, so NaN and out-of-range values fell through to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely negative sample index used later as an offset or a length is a wild pointer rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and saturate before the cast. Also round instead of truncating there. These functions exist to cross the float seconds boundary the VAD and diarize messages use, and truncation lost a sample about half the time on the samples-to-seconds-and-back round trip, starting at n=1. Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero fraction, and record why the clamp argument order in f32_to_s16le is load-bearing for NaN. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): map NaN PCM samples to silence explicitly f32_to_s16le relied on std::min argument order to keep a NaN sample away from std::lround, whose result is unspecified for NaN. That was too subtle to rest on a comment, and the comment was itself wrong: it warned against a spelling that the outer std::max already catches, while three real spellings leak, including std::clamp, which is the idiomatic C++17 way to write the same clamp and so the likeliest future edit. Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a full-scale click is worse audio than a dropped one, and this unit converts audio that may have originated off the wire. Pin it with an exact-value check rather than a range check, since all three outcomes the plausible spellings produce are finite and inside full scale, plus an invalid-operation check that fails unless the NaN is diverted before any ordered comparison. That second check is what catches modernizing the clamp and dropping the guard together. Also bound the seconds round-trip comment, which claimed unconditionally what holds only below roughly 2^23 samples, and document NaN, saturation and that bound in the header. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): assemble transcripts from runtime spans The top-level transcript text is TaskResult.text_output verbatim. audio.cpp carries text nowhere else: speech_segments, speaker_turns and word_timestamps hold spans and labels only, so deriving the text from them empties the transcript for any producer that omits word timing, VibeVoice diarized ASR included. Fixtures cover every observed producer shape. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep a nested speaker turn's own label A segment sourced from speaker_turns re-derived its speaker by greatest overlap. A turn's overlap with its own span is the largest possible, so a turn nested inside another speaker's turn could only tie with the container, and the tie went to whichever came first. sortformer_diar binarizes each speaker independently and sorts by start sample, so the container always comes first and the interjecting speaker was silently erased from DiarizeSegment.speaker. choose_segment_spans now carries the label out with the span. Also pins the nearest-segment fallback against measuring from either endpoint or from segment position, which a trailing-only stray word could not do, and exercises the empty-word guard in join_words. Two fixtures that pin a rule but do not mirror any pinned family are relabelled defensive. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize runs with a wedge-aware guard audio.cpp sessions are not reentrant and a wedged CUDA call cannot be cancelled, so a plain mutex would pile every worker thread behind a stuck GPU. Callers waiting past the configured bound, or arriving while the holder has already overrun it, fail fast instead. A caller that queues behind a healthy run deliberately does not stamp the clock: only the thread that takes the lock does. Stamping on arrival would restart the wedge clock on every request and hide a stuck run from everyone behind it, which is the pile-up this guard exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serialize inference through an InferenceLane One audio.cpp model is loaded per backend process and its sessions are not reentrant, so concurrent gRPC handlers have to take turns. Serialization alone is not enough: a wedged GPU call cannot be cancelled from userspace, so an unbounded queue behind one stuck run would swallow every gRPC worker thread until the process is useless. InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies it for a scope and gives it back on every exit, including an exception, and is the only way to take the lane at all: occupy/vacate are private with LaneEntry as the sole friend, so a caller cannot acquire without holding something that releases. LaneEntry is immovable on purpose, because a moved-from entry would have to stop releasing while the lane still recorded it as occupied. A caller either waits indefinitely or brings a millisecond budget. A bounded caller that cannot get in fails instead of waiting on, and a bounded caller whose budget is already shorter than the age of the run in the lane fails immediately, which is what stops a queue forming behind a wedged run. The two failures carry different text: one names the wait it exhausted, the other states the measured age of the run without claiming to know why it is long, since a short budget meeting a legitimately long run lands there too. The run's age is stamped only after acquisition. A waiter that published itself as holder would restart the measurement and hide a genuinely stuck holder from every caller behind it. Budget negotiation and the overrun decision are pure functions taking their inputs explicitly, so both are covered without threads or sleeping. The per-model ceiling arrives as an int of milliseconds; a request may tighten it and may never loosen it. Replaces the previous run_guard unit, which was a derivative of an Apache-2.0 file upstream and could not stay in an MIT tree. Written from a behaviour contract with no reference to the removed code. Tests: 65 checks, standard library only, single translation unit, clean under -Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed missing coverage and the tests were extended until they died. ThreadSanitizer clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the B10 test able to fail, and document LaneEntry Review of the previous commit found the B10 test could not fail for the reason it was named. It aged the in-flight run to about 120 ms and then tried two budgets, 30 ms and 60 ms, both under that age, so both callers took the fail-fast path. "The two failure modes do not share one message" was comparing two fail-fast messages that differ only in the budget they print, and the timeout path was never reached. The second budget is now 400 ms, well over the run's age, so that caller queues and times out, and a new check asserts which path each caller took instead of inferring it from inequality. A mutant that makes the fail-fast path emit the timeout message previously died only on B4 and B8 checks; it now also dies on B10. Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does not detect reentrancy: a second entry on a thread that already holds the lane surfaces as LaneUnavailable with a positive budget, but parks silently in unbounded mode, which matters because a handler may hold one across a whole stream. The immovability note now names the shapes that work, an optional emplaced in place or a unique_ptr, rather than saying to hold the entry indirectly without saying how; all three documented forms were compiled before being written down, which is how the note came to say that an optional of an immovable type cannot itself be returned. The header's explanation of why fail-fast exists is reworded. Two clauses traced back to a specification written after reading the Apache-2.0 upstream header, and while that was judged de minimis, this unit was rewritten precisely to carry no upstream expression at all. The margin table in the report was also wrong about which wall-clock margins are load-sensitive: there are four, not one, and the tightest is the B3 arrival check, which is now flagged at the call site. No margin value changed and none moved across 65 runs. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): gate model loading on the audio.cpp family Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF path without an explicit family option, so the model loader's greedy backend probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287). Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): load models and cache sessions per task Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per (task, mode), so the same model serves both the unary and streaming RPCs. LoadModel derives the family from GGUF metadata or an explicit option and fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the backend probe can see. audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally, and drift there is silent: every unit still compiles and every test still passes while the backend runs a different task. Two mechanisms pin it. The static_asserts in loaded_model.cpp catch an insertion or a reorder, and -Werror=switch on that one file turns an appended upstream enumerator into a build failure rather than a warning in a 600 file log. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop aborting the process on SIGTERM The signal handler called grpc::Server::Shutdown directly. Shutdown takes an absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread already holding that mutex, and abseil's deadlock detector responds by aborting. Every SIGTERM therefore ended in exit 134 and a 'dying due to potential deadlock' stack rather than a drained shutdown. The handler now sets a lock-free atomic and returns. Server::Wait moves to a helper thread so the main thread can poll that flag and call Shutdown itself, outside any signal context. A condition variable would not have helped, because notifying one from a handler is not async-signal-safe either. SIGTERM and SIGINT both exit 0 with no stack trace, where both previously exited 134. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel An environment fault during session creation was reported as UNIMPLEMENTED. A missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises vad/offline but refused to create the session: Failed to initialize CPU backend', which tells LocalAI the model cannot do this and must never be retried, and sends an operator hunting a capability bug instead of a packaging one. A throw from create_task_session is now a plain runtime_error, so it maps to INTERNAL. Only a null return, where the family genuinely declined, stays a CapabilityError. The model.'s task: option was parsed and then dropped: it lived in a local that died at the end of LoadModel and had no route to RequestShape::pinned_task. LoadedModel now keeps it and exposes pinned_task(). The global model becomes a shared_ptr reached through snapshot(). An audio RPC runs for seconds and cannot hold g_model_mu for its duration, so under a unique_ptr a Free arriving mid-request would destroy the model underneath it. Handlers now take a counted reference and whichever finishes last does the teardown, outside the lock. session_for documents the streaming state contract rather than resetting the session itself. Resetting on a cache hit was tried first and is not possible: silero_vad throws 'session prepare() must be called before Silero VAD reset()', so it would turn an ordinary second fetch into a hard error. start_stream's base implementation is already a reset, so a caller that runs prepare then start_stream per stream gets a clean session; a probe against the bundled silero_vad confirms an identical replay when it does and a carried-over stream when it does not. Also: an unknown backend: name is rejected before the model loads rather than after; MainGPU is parsed instead of passed through std::atoi, which turned 'gpu1' into device 0 silently; and device carries a device_set flag, because 0 is both the default and a real device index, so MainGPU was overriding an explicit device:0 that the neighbouring threads: handling promises will win. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the VAD and Diarize RPCs Both emit float seconds, converted from the runtime's sample-index spans, and both take a counted reference to the loaded model through snapshot() and hold it for the whole call: a Free arriving mid-request drops only the global's reference, so whichever request finishes last destroys the model instead of one of them running on freed weights. An AddressSanitizer build reproduces exactly that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw pointer instead, which is why the shape is what it is. The inference lane is taken before session_for, not after. session_for reads and writes an unsynchronised session cache and the offline run calls prepare(), which mutates the session, so both belong inside the lane. Diarize routes before it reads the input file, so a family that cannot diarize at all says so rather than complaining about the audio first. Its per-segment text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker label only, and nested or overlapping turns are passed through untouched: a sortformer turn inside another speaker's turn is correct output for overlapped speech, and LocalAI is overlap-tolerant downstream. Duration counts frames rather than floats, so a stereo input does not report twice its length. Verified end to end against upstream's bundled silero_vad, which needs no download, using the bundled 16 kHz speech asset: a synthetic tone returns nothing, correctly, because silero detects speech and a sine is not speech. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): enforce ModelIdentity on VAD and Diarize audio-cpp was the only C++ backend without the model-identity guard, and no later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity on exactly these two RPCs, for the reason #10952 records: in distributed mode a worker can recycle a stopped backend's gRPC port for another model's backend, and the controller's liveness-only probe cannot tell a stale cached route from a live one. Without this guard a stale route gets a different model's VAD or diarization answer back with a 200. The loaded identity lives on LoadedModel rather than in a separate global, which is where this differs from llama-cpp. A handler holding the model through snapshot() then necessarily judges against the identity that model was loaded with, and a concurrent reload cannot swap one without the other. The refusal is NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring. session_for and run_offline now take a const LaneEntry & proof-of-holding parameter. The rule that both must run under the inference lane was prose, which is exactly how the plan came to specify the inverted order; it is now a compile error. Restoring the inverted order fails to build rather than racing on an unsynchronised session map with a mutating prepare(). Diarize's speaker-hint comment claimed the dropped hints were "not a silent failure". From the caller's side that is what they are, and backend.proto documents num_speakers as forcing, so the comment now says plainly that the forwarding is dead for sortformer and that the family which lands must either honour num_speakers or refuse it. read_audio_file inspects the error_code from exists(), so an unsearchable parent directory no longer reports as a missing file. The VAD handler records the stimulus that actually works, since silero correctly ignores synthetic tones and the next task would otherwise rediscover that. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the lane and identity guards structural Two hardenings ahead of the eleven handlers still to be written, both of which get harder to retrofit later. The lane proof-of-holding parameter was a const reference, which binds to a temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such temporary dies at the end of its own full-expression, releasing the lane between two calls that must share one: precisely the split the parameter exists to prevent, and the form a future author is most likely to reach for because it reads as tidy. A non-const reference requires an lvalue, so the temporary form now fails to compile while the named-local handlers build unchanged. The header comment no longer implies the check is total either: it proves a lane was taken, not that it is this model's lane. The identity check was two lines each handler had to remember, with nothing failing if a new one forgot them and no C++ equivalent of model_identity_modalities_test.go to notice. snapshot() becomes snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage carries no ModelIdentity. Handlers go through snapshot_for(), which takes the counted reference, refuses when nothing is loaded, and runs the identity check before anything can route. Every handler already has to call something to obtain the model, so the guarded call is now the shortest path and skipping it means deliberately typing snapshot_unchecked. A convention that has to be remembered can rot; this cannot. Verified: the temporary-argument and inverted-order forms each fail to compile with the expected diagnostic, the real handlers build, and bypassing the guard in Diarize alone turns the identity test red on that RPC while VAD stays green. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscription RPC Adds result_map, the engine-to-proto boundary, and wires the offline transcription RPC. The handler branches on the ROUTED task: for Asr the request's prompt is whisper-style decoding context and becomes a request option, for Alignment the same field IS the transcript to align and becomes the text input. Routing has already decided which. The result text is TaskResult.text_output verbatim and is never derived from the segments. audio.cpp carries transcript text in text_output and nowhere else, so deriving it returns an empty transcript for every producer that reports segments without word timing. transcript_assembly already enforces that; this commit's job is not to undo it at the proto boundary, and result_map_ctest pins it there. read_audio_file now takes the sample rate the caller needs. Both file-fed speech handlers ask for 16 kHz mono, for two reasons: silero_vad and sortformer_diar refuse anything else outright, which turned an ordinary 44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in its own 16 kHz feature domain whatever the input was, so only a 16 kHz buffer makes the emitted nanoseconds right. Zero keeps the file's native rate and channels, which is what source separation will need. LoadedModel::check_can_serve answers a capability refusal before the lane is taken and before the input file is read. Routing is a pure read of the immutable capabilities, so a model that cannot serve an RPC no longer waits out somebody else's run to say so. VAD and Diarize use it too. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop linking sentencepiece's vendored protobuf engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER builds the protobuf-lite 3.14.0 sources it vendors. The generated backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both ended up in the binary: 476 google::protobuf:: symbols came from the archive, 278 of them also defined by libprotobuf.so, and the archive won, because once ld pulls a member in for sentencepiece's own code every reference binds to the definitions that member carries. The visible symptom is one function. ParseContext::ParseMessage(MessageLite*, const char*) is what a generated _InternalParse calls for a submessage field and for nothing else, so flat messages parsed and nested ones did not: a TranscriptResult carrying segments serialized to correct bytes that the same process could not read back, and TranscriptLiveRequest, a oneof of submessages, could not have been parsed at all. Underneath that, 3.21 generated code was running 3.14 arena, ArenaStringPtr and ExtensionSet code. -Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in .dynsym and the parse still fails, because the binding was decided at static link time and no visibility flag revisits it. Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points sentencepiece at the protobuf the generated code was already built against. Zero google::protobuf:: definitions remain in the executable afterwards, every nested message round trips, and citrinet_asr, which parses a SentencePiece ModelProto at load time and would break first if this were wrong, still tokenizes and transcribes correctly. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): fix the segment text a transcription response is built from Segment text is not decoration. core/http/endpoints/openai/transcription.go routes response_format text, srt, vtt and lrc through schema.TranscriptionResponse, which builds the entire body out of Segments[].Text and never reads the top-level text. So for those four formats the segment text IS the response. nemotron_asr emits one word_timestamp per SentencePiece token, and the word boundary is carried as a LEADING SPACE on the piece ("So", "me", " call"). join_words inserted a space unconditionally, so response_format=text returned "So me call me na ture ," while the correct sentence sat unread in the top-level field. The separator is now chosen from the words themselves: whole words are space-joined, subword pieces are concatenated, and one leading space anywhere selects the latter. Concatenating the real nemotron pieces reproduces text_output exactly, verified end to end. This does not touch the top-level text, which is still text_output verbatim. The rule that forbids deriving the transcript from the segments is about the direction segments -> text; segment text has no source other than its words. Two smaller corrections in the same area: timestamp_granularities ["word"] set only "word_timestamps", a key no family in the pinned upstream reads. It now sets "return_timestamps", which qwen3_asr does read and which both runs its forced aligner and shortens its chunk window, so asking for word granularity no longer silently returns nothing. The request-option comment claimed more than it delivered. prompt, translate and temperature are read by no ASR family, and are forwarded only so a family adopting them works unchanged; the comment now says so per key, and gives TranscriptRequest.diarize the same explicit treatment threads already had. Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it never did, so "the build is clean" starts meaning something; and fill_transcript_result no longer swallows a null response pointer, since answering OK with an empty transcript is the one failure mode this unit exists to prevent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTransform RPC Covers voice conversion, singing voice conversion, speech to speech and source separation, the four tasks LocalAI's AudioTransform can represent. AudioTransformResult carries one dst while htdemucs and mel_band_roformer produce several named stems from a single run, so inference runs ONCE, every stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem] selects which one dst receives, defaulting to vocals and falling back to the first output. An unknown stem name is INVALID_ARGUMENT listing the real stem names rather than a silent substitution, and the selection happens before the first write so a refused request leaves no files behind. params[stem] is consumed here and is not forwarded into the engine's request options. The stem decision lives in stem_selection, which is stdlib only and therefore tested by backend/cpp/run-unit-tests.sh. It also validates the names, because they come from the model (htdemucs reads them from the GGUF's config.sources) and each becomes a component of a path this backend writes: a name carrying a path separator would escape the caller's output directory, and two stems sharing a name would silently overwrite one another. Both files are read at their native rate and channel count. Separation forces it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo image that separates a centred vocal from a wide mix. The conversion families all resample internally (seed_vc, vevo2, miocodec, chatterbox were each checked), so passing the file through unchanged is also strictly better than band limiting it to 16 kHz first. Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems plus dst, dst byte identical to the selected stem, params[stem] selecting a different one, an unknown stem refused with no files written, and mono input preserved as mono output. Also against miocodec for the single output path, where params[stem] is refused rather than ignored. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write Four fixes from the first review of the AudioTransform RPC. check_can_serve now returns the resolved route, so params[stem] on a route that is not source separation is refused from the route instead of after a full inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a separation-routed family that returns no stems anyway. The typo'd-stem-name case still needs the run, since no framework header publishes the stem names before one. Stem names carrying control bytes are refused. GGUF strings are length prefixed and demucs reads its sources from JSON, so an embedded NUL survives to here: two names differing only after the NUL are distinct std::strings, so the duplicate check passes them, and then path::c_str() truncates both and they open the same file. That is exactly the silent overwrite the duplicate check exists to prevent, with the .wav lost as well. A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is LocalAI's own generated-content directory, not anything the caller named, so a full disk or a permission fault there is a server fault and is worth retrying, which is the opposite of what INVALID_ARGUMENT tells a client. An empty output path stays INVALID_ARGUMENT. Two comment corrections and one clarification: the separators' required rate is their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc resamples with soxr and falls back to sinc-hann, and the "no files left behind" guarantee covers a refused request, not a write that fails partway through the loop. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems Two defects that made source separation unusable through LocalAI's own API, even though the backend served it correctly over gRPC. /audio/transform normalized every upload to 16 kHz mono s16 through utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse any rate but their checkpoint's own and separate a centred vocal from a wide mix using the stereo image, so every separation request through the HTTP API died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000" while the same call over gRPC worked. The fold is not wrong, it is backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and needs the reference in the same shape. So it becomes a declaration, BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing else. A backend that declares nothing gets its upload unchanged, which means no backend has to opt in to work. utils.AudioToWavPreservingShape is the non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any rate and channel count, anything else is transcoded to WAV with its rate and channel layout kept. The other defect is that the run-once stem design bought nothing. A separation backend writes every stem beside dst from one inference, but AudioTransformResult carried only dst, so the other three were files no caller could find and a caller wanting all four had to run four separations. AudioTransformResult grows a repeated AudioTransformStem, the backend fills it, core/backend validates that each path really is inside the generated-content directory it handed over, and the endpoint publishes them as an X-Audio-Stems JSON header beside the existing X-Audio-Input-Url. JSON because a stem name is the model's own string and could contain any separator a hand-rolled format would use. Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable through /generated-audio/, body byte identical to the selected stem, and params[stem]=drums returning a different one. The same upload sent to a model whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed both by the engine's own rate refusal and by the persisted input file. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst Four fixes from the second review, plus one bug they made visible. isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed through untouched where the old fold would have transcoded it. audio.cpp's WAV reader accepts 16-bit only when the tag is 1, so such a file died with "unsupported WAV encoding". Extensible is what many DAWs and Windows tools write and music files are this endpoint's new headline input, so it is a first-contact failure rather than a corner. The check now requires tag 1, with a spec that fails against the old implementation. Stem URLs are percent-escaped. A stem name is the model's own string and legally contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL before the request is even sent. The name field keeps the raw name. sample_rate and response_format are applied to the stems as well as to dst. Applying beat documenting: dst IS one of those stems, so leaving them alone broke the "dst duplicates the selected stem" invariant the whole design rests on, and both conversions are no-ops when unset. A stem whose conversion fails is dropped from the header rather than advertised in the wrong shape. Verifying that turned up why it had never been noticed: the two fields were never bound at all. The request arrives as multipart/form-data and echo's binder falls back to the FIELD NAME without a form tag, matching only case-insensitively, so "SampleRate" never matched "sample_rate" and "Format" never matched "response_format". Both were documented in the endpoint table and silently ignored. Two form tags fix it, and with them the conversion is observable end to end. Docs: audio-transform.md now documents what LocalAI does to an upload before the backend sees it, which backend gets the 16 kHz mono fold and why, params[stem], and the X-Audio-Stems header with a worked example. Also records the known limitation that the fold lookup is on the bare backend name, so pinned variants (vulkan-localvqe) do not match, and points at IsLlamaCppBackend as the suffix-tolerant precedent. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTS and SoundGeneration RPCs TTSRequest.voice is treated as a speaker reference clip when it names an existing regular file, which makes routing prefer VoiceCloning, and as a named preset otherwise, in which case it travels as VoiceReference::cached_voice_id. Both the clip and SoundGenerationRequest.src are read at the file's own rate and channel count: upstream's own CLI and server do exactly that, every consuming family resamples internally and mostly with a better resampler than ours, and ace_step and stable_audio resample their input per channel, so a downmix here would delete the stereo image they are built to consume. The request builders live in their own unit rather than in grpc-server.cpp's anonymous namespace so they can be tested; grpc-server.cpp has a main() and cannot be linked into a test binary. The option keys are the whole point of these functions, so each one was grepped against the pinned upstream and the accounting is written down beside it. instructions maps to "instruct", which is what upstream's own server maps the OpenAI field to and what qwen3_tts and omnivoice read, and to "caption" for irodori_tts; the style tag is spelled "instruct" too, because "instructions" is looked up nowhere. duration maps to "duration_seconds", read by all three generation families, with the proto's own name kept only as a forward-tolerant alias. Keys that no family reads say so. Both handlers answer a capability refusal before taking the lane and before any file read, so a model that cannot synthesise does not queue behind somebody else's run to be told no. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop emitting an empty style language, and name the missing clip StyleCondition::language was set whenever has_language() was true, with no !empty() guard, while the language option twelve lines below had one. core/backend/tts.go sets Language unconditionally, so has_language() is true on every request LocalAI sends and carries "" when the caller named none. An engaged-but-empty style language is worse than an absent one: supertonic reads text_input->language behind its own !empty() guard and then overrides it from style->language with no guard at all, so "" replaced its "en" default and its tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request that set instructions and no language would have been an INTERNAL against a supertonic model. A plain request never saw it, because the style condition only exists when instructions are non-empty, which is why the chatterbox end to end run did not catch it. TTS also stops discarding the Route that check_can_serve already returns. A family routed to voice cloning without a reference clip used to be refused from inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the field to set; chatterbox advertises clon and no tts, so that was every preset-only request to it. It is now an INVALID_ARGUMENT naming TTSRequest.voice, answered in about 4 ms, and it cannot misfire because has_voice_reference is what selected cloning in the first place. Reading CapabilitySet::supports_speaker_reference to generalise this stays a follow-up. The src read carries a written caveat rather than a family blocklist, because ace_step's editing routes legitimately need src: setting src on a stable_audio model corrupts the heap and aborts the process in the pinned upstream, and the only thing keeping that off the network is that schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that Go schema would know why, so the reason is recorded where the field is read. build_tts_shape is extracted so TTSStream cannot describe the same request differently, and it arrived untested: two mutations of it survived until a test_tts_shape case was added. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching the convention backend/go/vibevoice-cpp established, so an HTTP client can start playback before the full PCM exists. Its chunks are read from StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and voxcpm2 all put their streamed audio there and leave audio_output empty until the very end, so reading the obvious field yields a stream with no audio in it. The finish_stream result is the family's own merged whole rather than a tail, so it is emitted only when nothing was streamed. Streaming transcription sends incremental deltas and degrades to a single delta plus the final result on families that offer no streaming ASR, which is the same message sequence with fewer deltas. The four streaming ASR families disagree on what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report incremental fragments while voxtral_realtime reports the whole hypothesis and reports it twice, so the reconciliation lives in one tested unit rather than in the handler. nemotron_asr reports only through the stream event sink, and only from inside finalize, so the audio driver installs one and clears it again before returning: the session is cached and a sink left holding the caller's frame is a use after free waiting for the next stream. begin_stream is now the only implementation of the streaming state obligation, prepare then start_stream. Streaming sessions are cached, and what clears the previous stream is start_stream's reset; a family override that dropped it would break every call site with no compile error, so there is one call site. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort TranscriptStreamResponse.delta is a proto3 string, whose wire format requires valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between two consecutive reports is eventually a lone continuation byte, and the C++ runtime serializes that with only a logged warning while the Go runtime refuses to unmarshal it: the client loses the remaining deltas AND the final_result. Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to unmarshal and every accented character was lost. TranscriptDeltaTracker now holds back an incomplete trailing sequence and merges it into the next fragment; reconcile flushes it, which it always can because the final text is complete. The same trace now unmarshals in full with zero failures. A streaming buffer whose float count is not a whole number of frames is refused rather than truncated. The integer division dropped the tail floats from the fed audio and therefore from the transcript, with no diagnostic; vibevoice_asr refuses the same thing from the other side of the call. A supertonic GGUF whose weights are not f32 is refused at load. It reaches ggml_concat with mismatched operand types and ggml_abort takes the whole backend process down on the first request, so nothing downstream can report it: the model loads, then every request kills the process. Attributed rather than assumed, the unary TTS path aborts identically, and upstream records that package as untested. The refusal names the orig package and says what to run before deleting the guard. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): stop a repeated lead byte from orphaning the next delta The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment the known text already starts with, and when that fragment is the LEAD BYTE of a new character it looks exactly like a repeat of an older character beginning with the same byte. It was discarded rather than held, its continuation bytes then arrived alone and began the next delta, and utf8_complete_prefix_length only ever inspected the trailing sequence, so a delta invalid at the FRONT went out whole. Through a real Go proto.Unmarshal the review's four-character repro gave 3 deltas, 2 unmarshal failures and a lost transcript. Reachable from the incremental families, not only from voxtral: nemotron_asr's decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares bytes, so both split characters. Measured over 30,000 randomized incremental traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one delta the Go runtime refuses. Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the lead byte is held instead of swallowed and the character survives intact; the cost is a few duplicated bytes in a shrinking cumulative report, which no pinned family produces. release() additionally drops leading orphan continuation bytes, so no delta can begin mid-character whatever the rules above it decide. Losing a byte keeps the stream alive; emitting one ends the RPC and takes the final_result with it. Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative streams plus both pure-ASCII incremental streams are byte-identical to the previous commit, so nothing changed for the families already working. The weight-dtype allow list moves to family_gate, where it is stdlib-only and pinned by a test rather than only by a comment. Two comment citations corrected. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): read only an exact repeat as a repeat, not any prefix Rule 2 discarded any partial the known text merely started with. For a cumulative family that is a duplicate; for an incremental family it is an ordinary short fragment that happens to coincide with the start of the transcript, and it was dropped, silently corrupting the text. Pure ASCII, no multi-byte character anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the client holding "pure ascii transcrit". Over 5,000 randomized traces per transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client holding something other than final_result.text, with a 200 and no diagnostic. Both incremental families emit fragments that small routinely, since nemotron_asr cuts at a byte offset and vibevoice_asr at a common prefix. Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts and changes no cumulative stream at all: 30,000 randomized cumulative traces are byte-identical to the previous commit. What rule 2 guarded was established from upstream rather than from its own comment. The only duplicate any pinned family produces is voxtral_realtime's, where process_available_stream_chunks feeds each event to the sink from inside its loop and returns the last of the batch, so that event arrives twice with byte-equal text. A duplicate is an exact repeat, so equality still covers it. The case given up is a cumulative report that SHRINKS, which no pinned family can produce: voxtral decodes a token vector that is only push_back'ed and cleared by reset(), so within a stream it can only grow. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): serve the AudioTranscriptionLive RPC The one bidirectional stream this backend serves. The client sends a TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges with ready, emits deltas as the audio arrives, and sends final_result once the read side closes. There is no offline fallback: live transcription has to consume audio incrementally, so a family with no streaming ASR is refused rather than served a batch run, which is what this RPC's Streaming-only mode_candidates list already says. The driver is a new sibling of run_streaming_audio, run_streaming_live, because the audio does not exist yet: instead of slicing a buffer it pulls frames from the caller until the read side closes. It installs the same ScopedStreamSink in the same order, which is not optional, since nemotron_asr returns a bare event from process_audio_chunk and reports every partial through the sink from inside finalize(). It buffers the wire's frames up to the family's own preferred window rather than feeding whatever size the client's audio callback produced, and it does not call finish_stream at all when no audio arrived, because nemotron_asr throws "finalize requires streamed audio" and an empty transcript is the truthful answer to transcribing nothing. Three things the handler had to get right and one it cannot: - The audio contract. A live request carries no samples, but nemotron_asr's streaming prepare() throws without an audio contract, and build_preparation_request derives it from TaskRequest::audio_input, so that field is an EMPTY buffer holding only the rate and the channel count. - 16 kHz or a refusal. The families express their spans in their own 16 kHz feature domain whatever the input was, and live frames cannot be resampled on the way in the way a file can, so an 8 kHz session would return timestamps 2x off with a 200. core/backend hardcodes 16000 anyway. - A mid-stream Config is refused. backend.proto calls it a decoder reset, but deltas already on the wire cannot be retracted, so a reset would leave the final text contradicting the transcript the client assembled. Ignoring the message would hand a client that believes it reset the decoder a transcript that silently continues the audio it thought it discarded. - The stale-route identity check cannot run here: TranscriptLiveRequest carries no ModelIdentity in either arm of its oneof, so snapshot_for does not instantiate for it. snapshot_unchecked's comment now names that as a second legitimate class of caller and says the fix is a proto change. eou and eob stay false. They exist for cache-aware models that emit end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no equivalent signal, and a client uses eou to decide the speaker yielded the turn, so a guess inferred from silence cuts people off mid-sentence. The lane is held for the whole stream, which is as long as the user keeps talking: the streaming session is stateful and cached, so a concurrent run would interleave two callers' audio and corrupt both transcripts. Verified against nemotron_asr over a real connection with a 14 s WAV in 512-sample frames: ready first, 59 incremental deltas with no repeated prefix, concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing asr/offline. A config followed by a close returns an empty final_result rather than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two concurrent streams both return the complete transcript. Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator is now asserted at compile time: the reported out-of-bounds read did not exist, the single entry does terminate, but the loops have no other bound and any edit that widened an entry would walk off the end. And the dtype guard now short-circuits on "is there a table entry" through a new predicate rather than on the emptiness of the description string, which would have skipped the check on an entry with an empty allow list, i.e. on precisely the entry that refuses every dtype. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): bound the lane a live stream can hold AudioTranscriptionLive holds the model's inference lane for the whole stream, which is correct (the streaming session is stateful and a concurrent run would interleave two callers' audio) and newly dangerous. Every other RPC holds the lane across compute, or across a write to a slow reader, and both of those terminate on their own. A live stream instead blocks in a client-driven read, and a peer that goes silent WITHOUT closing the stream never terminates anything: the lane stays taken and every other request against that model queues behind a client that stopped speaking. live_watchdog is a one-shot idle timer that ends the stream when no frame has arrived inside a window. It is standard library only, so it is unit tested without an engine. gRPC's synchronous Read has no timeout and cannot be given one, so the only way to unblock it is ServerContext::TryCancel, which decides the wire status itself: the client sees CANCELLED rather than the DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming back is the point. When it fires the read loop throws rather than reporting end-of-input, so the driver does not go on to finalize a decode nobody is waiting for. It is armed only after the lane is taken and disarmed as soon as the read side closes, and both ends matter. Arming earlier would cover acquire(), which legitimately blocks while another live stream runs, so a queued caller would be cancelled for waiting its turn. Disarming later would cover our own decode, where a window overrun is not a peer going quiet and cancelling would throw away the transcript the client is waiting for. The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds every tick that produced new audio while a turn is open, so 30 s of silence is a hundred ticks that delivered nothing. It is also longer than any pause a speaker takes mid-utterance, which is the case that must never be cut off, and backend.proto lets one stream span many utterances, so a client that pauses longer between them raises the option rather than discovering it. Two smaller corrections in the same handler: - check_can_serve now runs BEFORE the sample rate check. pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and on nothing else, so a live-incapable model asked at a wrong rate was answering INVALID_ARGUMENT and costing the caller its fallback. - a negative sample rate is refused instead of silently becoming 16000. Zero still means 16000, which is what the proto documents; -1 is malformed rather than absent and gets the same refusal every other bad rate gets. And one thing recorded rather than changed, at the handler: "live" here means incremental INPUT, not low latency, and with the pinned families it does not yet mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends to its buffer, so its whole decode and every delta happen inside finalize(), after the client closes its send side. The policy-window buffering is inert for that family and matters only for vibevoice_asr and higgs_audio_stt. Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at 371 ms and was cancelled at 3.371 s; a second live stream opened one second later received its ack 2.37 s in, i.e. at the instant the first was cancelled, and then transcribed successfully on the same cached session. Without the watchdog it would still be waiting. Re-ran the live transcription (ready first, 59 incremental deltas, concat equal to the final text, word timestamps in nanoseconds, eou and eob false), the citrinet refusal at both a right and a wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream on nemotron_asr, which is unchanged. Mutation testing the watchdog found a weakness in its own test: the destructor test slept past the window inside the watched scope, so a destructor that DETACHED the thread instead of joining it passed unnoticed. The test now uses a window longer than the scope, which kills that mutant, and says why. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): refuse the unsupported RPCs with a reason AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns UNIMPLEMENTED naming the loaded family, what that family does support, and the upstream limitation, instead of the generated base class's bare status. The reasons live in a table in capability_routing.cpp so they are data rather than literals copied into five handlers, and so a test can assert every one of them. The five claims this was planned against were re-read at the pinned upstream e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp streams tts and asr only" is false: silero_vad advertises vad with RunMode::Streaming. The refusal stands on the narrower claim that survives, that no family advertises streaming for any task AudioTransform routes to, and a test asserts the refuted wording does not come back. VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs the #10952 check before answering: a stale route must get NOT_FOUND and the router's sentinel, not "audio.cpp cannot embed speakers" about a model that is not loaded here. It cannot use snapshot_for, whose no-model branch would tell the caller to load a model when no model can help, so it takes the reference through snapshot_unchecked and checks identity itself. That function's comment now names three classes of caller instead of two. The two bidirectional surfaces refuse without reading their stream, verified with a client that writes a config and eight frames first and gets the status rather than hanging. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): correct the vevo2 clause, and assert the absences Review found a false clause in the AudioToAudioStream refusal. It said s2s is "offline voice conversion ... which converts one clip into another speaker's voice", which is true of miocodec and false of vevo2: vevo2's s2s route is `editing` and only `editing` (default_route_for_task and route_matches_task in src/models/vevo2/session.cpp), documented as "Edit source speech into new target text while using the target voice" and requiring --target-text, so it rewrites what was said. vevo2's voice conversion is its separate vc task. It now reads "offline clip-to-clip processing against a target voice, declared only by miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the miscast cannot come back. The conclusion is unchanged: neither family converses. That defect was undetectable on the wire, since vevo2 does not load here, which is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely to interrogate make_default_registry() and asserts the five premises the refusal reasons rest on: no codec task kind, no family advertising spk, no streaming for sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by exactly miocodec and vevo2. The last two are exact sets, so an addition fails here rather than leaving a message stale. A positive control proves the registry is populated and the query works before any absence is believed, and every assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION bump from "remember to re-read five prose paragraphs" into a test failure. unsupported_surface now switches over UnsupportedRpc with no default label, so -Wswitch reports a sixth enumerator added without a row at build time; the runtime bounds guard it replaces is deleted. The AudioTransformStream reason had a true premise and an overreaching conclusion: an offline sep family could be buffered into a stream, as other LocalAI backends do. It now says this backend declines to offer a buffered offline call in disguise, rather than implying impossibility. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the missing-switch-case diagnostic fatal unsupported_surface() switches UnsupportedRpc onto the table row that explains it, with no default label, so -Wswitch reports an enumerator nobody handled. As a warning that is not enough: adding a sixth enumerator and building the shipping target gives exit 0, a binary and one warning, and the trailing `return surfaces[0];` then answers the new RPC with AudioEncode's codec reason. That is a confident, specific and false statement about audio.cpp on the wire, on the one code path whose entire job is to be truthful about what this backend cannot do, and it is worse than the runtime fallback it replaced, which at least named itself as a bug in this file. capability_routing.cpp therefore joins loaded_model.cpp on the existing -Werror=switch pin, whose comment already made this argument for the engine enum. The comment now covers both files. The pin stays per-file rather than project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled -Wswitch cases of its own. Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so the two pins fire independently; both reverted clean. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): package the backend image Bundles the dependency closure for the from-scratch image, the dlopened ggml CPU-variant shared objects that ldd cannot see, and upstream's bundled silero_vad and marblenet_vad assets so VAD works with no download. The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh execs it, which makes /proc/self/exe name the loader, and this backend has two consumers of that path: ggml discovers the libggml-cpu-*.so by listing dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the same directory. Rooting the loader makes the binary, the ggml objects and assets/ share the one directory all three resolution mechanisms agree on. llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well. The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and the base-grpc-* images supply vendors protobuf 26, which pulls abseil into message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with sentencepiece's vendored mini-abseil and every absl::internal reference becomes ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the pair every earlier verification of this backend ran against. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): exempt the driver libraries from the packaging gate package.sh already left libcuda.so* and libnvidia-* to the host when copying, because the driver has to match the kernel module on whatever host runs the image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED libcuda.so.1 and the gate would have rejected the very absence the copy loop created, failing every cublas build in CI. One regex now feeds both. Building a control for that found a second defect: ld.so --list refuses to trace an object with an unresolvable dependency at all, exiting 127 without emitting a per-library line, so the "=> not found" rule was dead code and no exemption could have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also what run.sh does at run time. Adds a layout assertion so a future move of the loader into lib/ fails the build instead of shipping a package that resolves bundled: models into lib/assets and finds no ggml CPU backend, and records for Task 16 that the Darwin script must not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh and would silently drop assets/. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): register the backend with CI and the gallery Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile, the gallery meta plus its -development variant and the image entries for every variant, and the Makefile docker-build wiring. The matrix entries carry base-image only, with no builder-base-image, unlike the llama-cpp and privacy-filter blocks they sit next to. The prebuilt quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose protobuf v26 depends on abseil, and this backend's sentencepiece is built with SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116:: internal alongside its own vendored plain absl::internal and every absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64 fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself, which is also the pair every unit and end-to-end run of this backend has been verified against, and the CUDA toolkit therefore has to come from base-image. No Darwin matrix entry and no metal gallery entries: the Metal build needs scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and a routing step in backend_build_darwin.yml, none of which exist yet, so an entry added now would be routed to build-darwin-go-backend and look for backend/go/audio-cpp/. The inferBackendPathDarwin case and the DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the entry later is a one-line change that cannot be claimed by the generic Go path. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/ linux.md says so outright. ggml's own default does not rescue it: it list(APPEND)s in the ggml subdirectory scope, which never reaches the root scope where the engine_runtime property is decided. No CI runner has a GPU for `native` to enumerate, so both cublas entries would have gone red on the very commit that first turns a CUDA build on. Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION, which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously discarding. The values are copied from ggml's own version guards rather than invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them and adds 121a-real. The `a` suffix is used rather than `f` because the latter needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake 3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as the rejected control. Drop the vulkan matrix entry, its two gallery entries, the vulkan capability key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs a vulkan image against real hardware, so that would have passed green and failed in users' hands. BUILD_TYPE=vulkan stays supported for local builds. Also note on the cublas entries that cuda-major-version now selects the architecture list and that cuda-minor-version and the base-image tag encode the same toolkit, and correct the stale entry counts on matrixEntryKey. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): build for Darwin Metal Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the metal image entries plus metal/metal-darwin-arm64 capability keys in the backend gallery. The build script deliberately does NOT reassemble the package the way privacy-filter-darwin.sh does. It runs the backend's own `make package` and copies the result, so the Darwin package keeps the root-level layout the Linux one has: grpc-server, run.sh, the ggml objects and assets/ in one directory, with lib/ for the dylib closure. Hand-assembling would drop assets/, and assets/ is what makes the bundled: model paths resolve with nothing downloaded. The dylib walk is a full transitive closure rather than the single level ds4 and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a package that only works on a machine that already has Homebrew grpc. Two fixes folded in, both in the backend Makefile: - an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list, which contains 120a-real and so needs nvcc >= 12.8. A local BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where upstream's documented default (native) worked. EMPTY now maps to native, 12 and 13 keep their lists, and any other non-empty value is an error on cublas builds. CI always passes a major, so CI is unaffected. - the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so FindOpenMP finds neither the library nor the header, and audio.cpp calls find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without the hint the macOS build would have died at configure time. If the keg is absent the build disables OpenMP instead of failing. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete Review follow-up on the Darwin Metal build. The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty, CI produced a green Metal package with 108 #pragma omp directives across ~30 files compiled out, and clang says nothing about an ignored omp pragma without -Wsource-uses-openmp, so the only trace was one absent flag inside a set -x cmake line. That regression would have been blamed on Metal. It now warns. The @rpath arm of the dylib walk had no live candidate when it was written, on the reasoning that a Metal build links ggml statically. The OpenMP fix in the same commit made libomp.dylib one, and whether Homebrew records it as an absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The walk now expands @rpath, @loader_path and @executable_path against the object's own LC_RPATH entries, and only fails when nothing on disk answers, printing the rpath list with the error so a failure on a machine nobody can attach to explains itself. Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they are deduplicated and their own dependencies bundled; build/darwin/lib is created explicitly instead of relying on package.sh pre-creating it; the libomp probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and would otherwise expand empty and take the OFF branch on an older make; and -DOpenMP_ROOT is quoted like its CUDA sibling. Verified with a Linux harness that runs the script verbatim against a stubbed otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH, and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates, system libraries are skipped, the packaged tree has assets/ at the root beside grpc-server with the dylibs in lib/, and both failure paths exit non-zero. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): make bundled: reachable from a model YAML resolve_model_path() tested the bundled: prefix on `candidate`, which prefers ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and only sets it from a managed artifact otherwise, so a model YAML saying `model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad" and Model "bundled:silero_vad". The prefix therefore never matched through the normal load path: it matched only for a hand-written LoadModel call that left ModelFile empty, which is exactly how task 15 verified it, and every model YAML using the form failed with "model path does not exist: /models/bundled:silero_vad". Both fields are now checked, Model first, so the zero-download VAD path the package ships assets for is reachable the way it is documented. A caller that puts the form in ModelFile still works, so task 15's verification stands. Compiled clean; the runtime check could not run on this host, whose system libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no longer resolves its libraries either), so it wants a container run. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): advertise the backend and document its options Registers audio-cpp as preference-only in /backends/known: the family lives in GGUF metadata that an importer cannot read from a remote repo, and one repo hosts thirty families, so there is no honest auto-detect signal. Modality is a single string and the import form chips on a fixed key set, so it registers as tts with the other modalities named in the description rather than under an invented key the UI would bucket as "other". Adds a features page covering the option namespacing, the routing table per endpoint, the RPCs this backend declines and why, the bundled VAD path, the separation stem behaviour, and the family gotchas (supertonic needs the orig package; chatterbox advertises cloning and no plain tts; nemotron_asr defers its whole decode to finalize so live transcription emits nothing until the client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option name and family capability in it was read off the pinned upstream checkout. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * backend(audio-cpp): test resolve_model_path, and correct the family names The bundled: fix in |
||
|
mudler's LocalAI [bot]andmudler
|
c5ae41a29d |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 6647db9c27760044950fd6f99060456ae3d15df3 (#11204)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
Dimitris Karakasilis
|
c6c347ce13 |
feat(stablediffusion-ggml): make VAE tiling configurable (#11216)
GenerateImage hardcoded TilingParamsSetEnabled(vaep, false), so tiled VAE
decoding was unreachable from a model config even though all four upstream
setters were already bound in main.go.
Sampling runs in latent space, but the final VAE decode expands to full
resolution and needs one large compute buffer. At 1024x1024 that buffer
exceeds 8GB, which fails on two kinds of device: cards without the VRAM
for a full-frame decode, and drivers that cap a single allocation
regardless of how much memory is free. Mesa RADV reports a 4GiB
maxMemoryAllocationSize, so a Radeon 8060S with 74GiB of device-local
heap still cannot serve that decode:
[INFO ] sampling completed, taking 251.82s
[INFO ] decoding 1 latents
ggml_vulkan: Requested buffer size exceeds device buffer size limit:
ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] decode_first_stage failed for latent 1
Every sampling step completes and then the run is discarded at the last
stage, so the whole generation is wasted.
Add three options, parsed in Load and applied per generation:
vae_tiling:true enable tiled decoding (bare flag also works)
vae_tile_size:512 tile size, or 512x384 for a rectangle
vae_tile_overlap:0.25 overlap between tiles
Tiling stays off unless requested, so existing models are unaffected. Tile
size and overlap only reach the library when the operator set them, which
keeps upstream's defaults rather than pushing a zero, and an unparseable
value is treated as absent for the same reason.
Truthy spellings match what load_model already accepts for its own bool
options, and the bare-flag form matches diffusion_model, so no new
convention is introduced.
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
|
||
|
|
efb43776ba |
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads (fixes #11070) (#11074)
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes #11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
|