mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,265 Commits 136 Branches 173 Tags
fix/vllm-cpp-l4t-cuda12-fallback
Commit Graph

717 Commits

Author SHA1 Message Date
mudler's LocalAI [bot]
856b0ea951 fix(ci): build the CUDA 13 image on Ubuntu 24.04 (#11143) 
* fix(ci): build the CUDA 13 image on Ubuntu 24.04

The amd64 `-gpu-nvidia-cuda-13` image is the only runtime image still
built FROM ubuntu:22.04. The Ubuntu 24.04 migration (#7769) bumped its
`ubuntu-version` to 2404 but left `base-image` on jammy, so the image
ships glibc 2.35 while adding the noble CUDA apt repository, and every
backend it unpacks is built on noble.

Backends therefore cannot dlopen the libraries they bundle. The vLLM
backend dies at import time with:

  OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not
  found (required by /backends/cuda13-vllm/lib/libnuma.so.1)

and torchcodec finds no usable libavutil because jammy ships ffmpeg 4.x
(libavutil.so.56) while torchcodec looks for .so.57 through .so.60.

Add a spec over the build matrices that fails when a base image and the
`ubuntu-version`/`ubuntu-codename` it is paired with disagree, or when
the runtime images are split across Ubuntu releases. Entries whose base
image does not name a release (JetPack) are left alone.

The `base-grpc-cuda-13-amd64` builder base stays on jammy: it only
compiles backends, and a lower glibc floor in a builder is safe.

Fixes #11059

Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ci): drop the build matrix invariant spec

Per review, the CI matrix guard does not belong in the tree. Only the
base image bump remains.

Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-27 18:49:02 +02:00
dependabot[bot]
ac9352ef54 chore(deps): bump actions/setup-node from 4 to 7 (#11080) 
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 7.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v7)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-26 23:05:07 +02:00
mudler's LocalAI [bot]
4baa36ddd8 feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100) 
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)

Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.

Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
  (anything else is refused, satisfying the greedy-probe rule); context_size
  maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
  KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
  with the per-delta C callback bridged into the gRPC stream. The backend
  embeds base.Base (not SingleThread): concurrent requests batch continuously
  in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
  grammar-constrained tool calling at parity with llama-cpp; the ABI also
  exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
  (unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
  runtime dispatch), so no avx/avx2/avx512 variant builds.

Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
  amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
  the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
  cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
  VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
  specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
  advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
  without tokenizer templating (the C ABI takes the FINAL prompt; templating
  and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
  compatibility table.

Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)

The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.

- With use_tokenizer_template + structured Messages, PredictOptions lowers to
  ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
  stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
  auto lowers engine-side to a LAZY structural-tag decode constraint - free
  text until the model emits the tool trigger, then the call is
  grammar-constrained; required/named force a call. Tool output is parsed by
  the engine's streaming Hermes-style parser; each chat.completion.chunk maps
  onto ChatDeltas (content / reasoning_content / tool_calls) which the host
  already prefers over Go-side tag extraction. Without structured messages the
  plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
  metadata (or tokenizer_config.json); templates beyond its minja subset -
  e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
  Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
  a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
  (use_tokenizer_template: true, no-grammar autoparser flow); only the
  llama-cpp-specific use_jinja option and the vllm-python parser options are
  dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).

Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning

Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.

Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
  layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
  to the engine; unset means template auto-detection (18-row tool marker
  table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
  disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
  reasoning_content), flowing into ChatDelta.ReasoningContent which the host
  already prefers.

Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).

Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out

mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.

Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): add the vllm-cpp-development gallery meta

The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.

Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton

Two distinct failures took down all five vllm-cpp backend builds on the PR:

1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
   engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
   false positive through a staging std::optional's raw storage. Fixed
   upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
   verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
   diagnostics exactly, fixed compiles clean) with the engine's behavior
   suites green. Pin bumped to that sha.

2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
   cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
   multi-arch (120a;121a) fat build unless pinned to one tree, which would be
   unsound for the other arch. Triton is now enabled only on the single-arch
   arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
   engine's non-AOT GDN path.

Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only

The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.

Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain

Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:

- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
  cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
  to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
  older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
  fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
  arm64/l4t keeps 121a with the Triton cubins.

Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.

Assisted-by: Claude Code:claude-fable-5

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine

chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.

Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix

The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent

The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.

Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-26 23:04:48 +02:00
mudler's LocalAI [bot]
2d889e61a6 feat(backend): add magpie-tts-cpp text-to-speech backend (#11115) 
* feat(backend): add magpie-tts-cpp text-to-speech backend

Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).

The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).

Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.

Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* gallery: add magpie-tts-cpp model entries (q8_0 + f16)

Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA

Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-25 08:38:48 +02:00
mudler's LocalAI [bot]
8eb8376596 fix(ci): dedup the three workflows that stack runs on every PR push (#11058) 
build-test.yaml, yaml-check.yml and secscan.yaml had no concurrency block at
all, so every push to a PR stacked another full batch instead of superseding
the previous one. build-test carries a macos-latest job, the scarcest runner
class we use, and secscan fires on every push to every branch because its
`push:` trigger is unfiltered.

build-test and yaml-check use the same group idiom as lint.yml and the other
eleven workflows that already have one: key on the PR number so pushes to a PR
share a group, and cancel only on pull_request. On a master push the key falls
back to github.sha and cancel-in-progress is false, so master runs never cancel
each other -- that is deliberate, since backend.yml builds only the backends a
given commit touched and superseding would drop those builds.

secscan needs a different key: it has no pull_request trigger, so the shared
idiom would fall back to the unique-per-commit sha and dedup nothing. It groups
on github.ref instead, and excludes master from cancellation for the same
per-commit reason. Cancelling a superseded feature-branch scan is safe because
the only output is a SARIF upload and code scanning keeps the latest result
per ref.

No behaviour change on master for any of the three.


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-22 22:51:11 +02:00
localai-org-maint-bot
47c0e06198 fix(ci): authenticate the nightly dependency-bump API calls (#11042) 
The "Bump Backend dependencies" workflow has failed every night for the
last two weeks. #11012 fixed one cause (repos renamed under localai-org);
what is left is rate limiting.

bump_deps.sh fans out to ~25 parallel matrix jobs that each query
api.github.com anonymously. Anonymous calls are capped at 60/hour per
source IP and GitHub-hosted runners egress through shared NAT addresses,
so a random handful of jobs draw HTTP 403 and die at curl exit 22 with an
empty response. Last night that hit ggml-org/whisper.cpp and
mudler/depth-anything.cpp -- both public and resolvable, nothing wrong
with either pin.

Route every bump script through a shared gh_curl helper that sends
GITHUB_TOKEN when present (1000/hour instead of 60) and retries transient
failures, including the 403s that plain --retry ignores. The helper
suppresses xtrace around the call so the Authorization header cannot land
in a public job log.

bump_docs.sh had a sharper version of the same bug: it piped an
unchecked response into `jq -r .tag_name`, so a throttled request
resolved to the string "null" and would have been published as the docs
version. It now refuses to write anything it cannot resolve to a tag.

Verified locally by running all four scripts end to end against their
real upstreams: correct SHAs/tags written, exit 0; a nonexistent repo now
fails with a named diagnostic instead of a bare exit 22 and leaves the
pinned file untouched; the token is absent from the xtrace output; and
the scripts still work unauthenticated.

Assisted-by: Claude:opus-4.8 [Claude Code]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-22 08:25:05 +02:00
mudler's LocalAI [bot]
5c96e097ba feat(gallery): fix stale DFlash drafters and add the APEX families as variant ladders (#11027) 
* fix(gallery): repoint qwen3-4b/qwen3.5-9b dflash drafters at post-rename GGUFs

The drafters both entries referenced were converted from the pre-merge DFlash
PR branch and carry dflash.target_layer_ids. llama.cpp reads dflash.target_layers
and refuses the load. The stored values are offset by +1 relative to the HF-side
field, so the files cannot be repaired by renaming the key and must be replaced.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): add apexentries HuggingFace client

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apexentries): build the HF client via pkg/httpclient

The apexentries HuggingFace client was constructed as a raw
&http.Client{Timeout: 60s}. The repo convention (documented in
.golangci.yml, which cannot express this as a forbidigo pattern) is that
all outbound HTTP goes through pkg/httpclient, which refuses redirects by
default and sets a TLS 1.2 floor. The std client follows redirects and
forwards custom credential headers to the redirect target on a cross-host
hop (GHSA-3mj3-57v2-4636). Only a User-Agent is sent today, but this
calls an external API and an HF_TOKEN header added later would leak.

Switch to httpclient.NewWithTimeout, preserving the 60 second timeout.
No behaviour change for the current header set.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): discover APEX tiers by filename suffix

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): resolve unsloth counterparts and sharded quants

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): render APEX child entries with the dflash/mtp tag rule

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apexentries): set backend, known_usecases and cross-repo drafters

RenderChild left three gaps against the hand-written gallery entries.

The generated entries reference gallery/virtual.yaml, which supplies no
backend, so every generated entry named no engine at all. All comparable
hand-written entries set backend: llama-cpp in overrides; do the same.
Set known_usecases to [chat] alongside it: LocalAI falls back to the
backend defaults when it is absent, so this is convention rather than
breakage, but generated entries should not read differently from their
neighbours.

The drafter was also assumed to live in the repo publishing the weights.
Speculative pairings routinely cross repos, and a drafter URI built from
the weights repo 404s at install time. Add ChildInput.DraftRepo, used for
both the drafter URI and its local path, falling back to Repo when empty
so pairings that do ship the drafter alongside the weights are unchanged.

The dflash/mtp tagging rule is untouched: the tag still follows SpecType
and nothing else.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): dedupe generated entries against the existing gallery

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* apexentries: canonicalize HF URIs and dedup the generated batch

Merge exists to stop a second gallery entry being added for weights the
gallery already ships, but two gaps let duplicates through on a bulk run.

The URI key was compared as an exact string while render.go only ever emits
https://huggingface.co/{repo}/resolve/main/{file} and the gallery records
1038 of its URIs in huggingface://{repo}/{file} shorthand. A generated
unsloth rung whose weights are already shipped in shorthand was therefore
not recognised. canonicalURI reduces both spellings to one key and is
applied on both sides, taking care that the repo is exactly the first two
path segments so sharded quants in a subdirectory still match. A URI in
neither form is returned untouched so other hosts dedup on their literal
string.

Merge also never accounted for entries it had just accepted, so two
generated entries sharing a name or a primary URI both landed in add.
Several APEX repos share one base model and resolve to the same unsloth
counterpart, so the identical rungs are generated twice under the same
name. Batch state is tracked locally rather than written back into the
caller's ExistingIndex, which a caller may reasonably reuse.

Name is still checked before URI: a name collision must block the add
regardless of the weights.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): verify variant and tagging invariants in the gallery index

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ci): scope the apex-entries verifier to what it can actually judge

The verifier reported 60 problems against the real gallery, 57 of which were
llama.cpp assumptions meeting entries from other backends. A gate that is wrong
57 times out of 60 cannot gate anything.

- The weight-count check catches a quant label collision in llama-cpp quant
  discovery, so it now runs only for overrides.backend: llama-cpp. Entries with
  no declared backend are skipped because their weights are declared in the
  referenced url: template, which the verifier never reads.
- The dflash/mtp tag check now implements the per-backend table in
  .agents/adding-gallery-models.md instead of assuming llama.cpp's spec_type:
  vocabulary. ds4 declares mtp_path:/mtp_draft:; sglang declares
  speculative_algorithm: in a file this verifier cannot follow, so sglang
  entries are not judged in either direction. The check stays bidirectional
  within the backends it does judge.
- sha256 is now required on .gguf files only, since every non-GGUF asset in the
  index belongs to a hand-curated entry outside this generator's scope.

Against the current gallery this leaves exactly the three genuine problems:
two entries setting spec_type:draft-mtp without the mtp tag, and one entry
whose overrides.mmproj names a file it does not download.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apexentries): anchor quant matching and invert the sha256 rule

UnaccountedQuants matched files to wanted quants with strings.Contains, which
reproduces the substring collision it was written to warn about: Q8_0 is a
substring of UD-Q8_0, so a repo publishing only UD-Q8_0 was reported as
publishing an unbuilt Q8_0. Subdirectory-sharded UD quants are the normal
unsloth layout for large repos, so this fired on realistic input.

Match on the quant label as an anchored token instead, the way
DiscoverUnslothQuants does, so the diagnostic and the discovery it audits
cannot disagree about what a file is. Root-level shards, the layout the
diagnostic mainly exists to catch, stay detected.

The sha256 requirement was scoped to .gguf, which exempted seven real model
weights: wan_2.1_vae.safetensors and clip_vision_h.safetensors across the
wan-2.1-*-ggml entries, both load-bearing weights named by gallery/wan-ggml.yaml.
Invert the rule so a checksum is required on everything except metadata
extensions, which keeps a future weight format covered by default rather than
silently exempt.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): wire the apexentries command

Adds the generation path to the apexentries command: list the mudler APEX
repos, discover each one's quality ladder and its unsloth counterpart's quant
rungs from the filenames actually published, render a child entry per build
plus a family parent carrying the variants list, dedup against the gallery,
and write the additions to -out or append them with -apply.

Discovery shortfalls are reported at discovery time rather than left to the
verifier. A quant or a tier that discovery drops leaves no trace in a finished
gallery file, and because an empty imatrix ladder falls back to the plain one,
a repo whose imatrix filenames all fail to match downgrades the whole family
silently instead of erroring.

Merge's single reused map is split into two reported categories. A URI match
means the gallery already ships exactly these weights and referencing the
existing entry is correct; a name collision means an unrelated entry owns the
name and referencing it would substitute a different build.

Multimodal children now declare known_usecases [chat, vision]. An explicit
known_usecases suppresses the backend-default fallback, so a chat-only entry
carrying an mmproj never matches the vision or multimodal gallery filters.

.github/ci is invisible to go list ./..., so a workflow names both generator
packages explicitly and their specs finally run on pull requests.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ci): gather APEX builds under the base model entry

The hub for a family is the BASE model entry, never a generated *-apex
parent. Somebody looking for qwen3.6-35b-a3b has to find every build of
those weights under that one name, so a competing qwen3.6-35b-a3b-apex
hub would split the family and leave half of it invisible.

When the gallery already ships the base entry, a variants block is
spliced into it textually, leaving its description, icon, tags,
overrides and files untouched. Only a family whose base model the
gallery does not ship gets a new hub, still named for the base model and
carrying one of the discovered builds as its own payload so it declares
a backend the verifier can judge.

The line editing is factored into .github/ci/galleryedit, shared with
the variantproposals job, so the two cannot drift apart on where a
variants block belongs.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(apexentries): treat an unreadable optional counterpart repo as absent

HuggingFace answers 401 Unauthorized, not 404, for a repository that does
not exist when the request carries no credentials. FetchRepoFiles treated
only 404 as absence, so probing for the OPTIONAL unsloth counterpart hard
failed for every family that legitimately has none: 27 of the 45 APEX
families are community merges that will never have an unsloth build, and a
full run failed all of them.

Split the fetch so the two call sites can apply different policies to the
same response. The APEX repo itself stays strict: a 401 or 403 on a repo
the run requires is a real failure and still errors. Only the optional
probe tolerates it, because without a token 401 cannot be told apart from
absence.

That collapse is lossy in one direction, since a private or gated repo also
answers 401, so the skipped candidates are named in the run summary
alongside the other silent-shortfall counters instead of being dropped in
silence.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apexentries): report full-precision sources as a known exclusion

The 45 APEX repos publish their unquantized F16 sources next to the
imatrix ladder, flat or sharded. Discovery correctly emits nothing for
them, but they were landing in the unclassified total, leaving a
permanent baseline of 24 benign lines on every run.

That baseline is what the unclassified check exists to prevent: a
standing count of known-benign files is exactly what hides the one file
that ever genuinely matters. Count full-precision sources separately and
give them their own summary line, so unclassified returns to 0 and stays
loud when something really is an unknown shape.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(apexentries): namespace local paths by owner and enable MTP builds

localPath namespaced downloads by the repo basename alone, so two repos
publishing the same filename under different owners collapsed to one local
path. LiquidAI/LFM2.5-8B-A1B-GGUF and unsloth/LFM2.5-8B-A1B-GGUF collided that
way, and both were offered from the same hub, so installing the second either
overwrote the first model's weights or was skipped as already present while
recording a sha256 that did not match the bytes on disk. The owner is now its
own path segment: owner/repo is globally unique on HuggingFace and neither half
can contain a separator, so uniqueness holds by construction.

Verify gains a check for the whole class, that no local filename may map to two
different upstream URIs. It surfaces seven pre-existing collisions in the
gallery, which are left alone here.

Entries built from the *-APEX-MTP-GGUF repos now configure MTP rather than
shipping the heads inert, matching the pattern the hand-written MTP entries
already use: spec_type:draft-mtp with spec_n_max and spec_p_min, tagged mtp, and
no draft_model because the heads live in the weights. RenderChild no longer
requires a separate drafter file before it will configure a spec type, while the
cross-repo drafter path is unchanged.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add the APEX GGUF families as variant ladders

Adds the imatrix quality ladder from each mudler/*-APEX-GGUF repo, a fixed
subset of unsloth quant rungs where a counterpart repo exists, and the MTP
builds, then attaches them to the base model entry so one entry offers every
build of the same weights and LocalAI picks the one that fits the hardware.

Ten existing base model entries gain a variants list; twenty-seven families that
the gallery had no base entry for get one. Builds are discovered from the
filenames each repo actually publishes rather than derived from its name, since
six repos ship a stem that differs from their repo name. Every file carries a
sha256 taken from the HuggingFace API.

Assisted-by: Claude Opus 4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-21 21:40:51 +02:00
localai-org-maint-bot
1e0baec2a7 fix(ci): repair nightly backend dep bumps for renamed localai-org repos (#11012) 
The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.

bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.

Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.

Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-07-21 09:40:10 +02:00
mudler's LocalAI [bot]
0cdd781c2d ci(gallery): propose variant groupings for review instead of letting them decay (#10992) 
ci(gallery): propose variant groupings for review on a schedule

A gallery entry may declare `variants:`, references to other entries that
are alternative builds of the same weights, and auto-selection then installs
the best build for the host. Those families exist only because humans curated
them in two manual sweeps.

The gallery agent dedupes on the HuggingFace repo URL and picks one
quantization per model, so it never adds a second build of a repo it already
has, and consequently never creates a family and never joins one. A model
published across two repos lands as two unrelated standalone entries. The
grouping decays as the gallery grows and nothing notices.

Add a scheduled job, in the same shape as the checksum checker: compute
offline, edit the index textually, open a pull request against ci-forks. It
proposes and never decides. Grouping is a judgement call that has gone wrong
in both directions, so the value is catching drift and surfacing candidates
with their evidence.

Three grouping signals, taken from the manual sweeps: same name once
quantization markers are stripped, the `:` config-suffix convention, and the
same primary weight filename once quantization markers are stripped. The
third requires the same upstream repository. Excluding auxiliary files is not
enough on its own: bert-embeddings, an ultravox audio model and a roleplay
finetune all declare a primary file called llama-3.2-1b-instruct-q4_k_m.gguf,
and grouping on that is the same error that linked four wan-2.1 entries and
Z-Image-Turbo to qwen3-4b.

Add gallery/variant-exclusions.yaml, a checked-in rejection ledger. A job
that re-proposes declined candidates every night becomes noise and gets
ignored. Declining a proposal is one flow-mapping line a reviewer adds inside
the proposal pull request itself. Seeded with the six -abliterated pairs whose
base is also in the gallery, the mistral-small multimodal pair, the whisper-1
alias, the kokoros language set, and the recurring finetune tokens. qat and
apex are deliberately not on it: they are quantization techniques.

Proposals refuse to nest, to let two parents claim one target, to target an
entry that installs nothing, and to touch a merge anchor, since a variants key
added to an anchor is inherited by every merging child. The anchor refusal
names every entry that would inherit, which is the worklist a human needs.

Run against the pre-sweep gallery, the job rediscovers 12 of the 19 groupings
the second manual sweep made, with no false positives. The rest it reports as
refusals or ledger declines rather than missing silently.

Assisted-by: Claude:claude-opus-4-8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-20 23:08:00 +02:00
mudler's LocalAI [bot]
6e52d0c2ef fix(ci): rebuild backends when shared build inputs change (#10975) 
The backend matrix path filter only matched files under a backend's own
directory, so a change to shared build infrastructure rebuilt nothing at
all: an empty matrix, every job green, and the change reaching no image.

PR #10946 fixed scripts/build/package-gpu-libs.sh shipping a partial
4-of-8 cuDNN library set, which mixed versions with the venv's pip cuDNN
and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time.
It merged 1h48m after the weekly full-matrix cron had already run, so no
backend image ever received the fix and nothing signalled that it had
been un-shipped.

Add a SHARED_BUILD_INPUTS table mapping each shared path to the narrowest
set of matrix entries it can honestly invalidate, plus a generic rule for
backend/Dockerfile.<x> (which each entry already names). A full matrix is
417 Linux + 56 Darwin builds, so package-gpu-libs.sh now rebuilds the 176
Python entries rather than everything. Unclassified files under
scripts/build/ fall back to a full rebuild deliberately: over-building is
recoverable, silently shipping nothing is not.

Extract the filtering logic to scripts/lib/backend-filter.mjs so it can be
unit-tested without bun, js-yaml or a GitHub API round-trip, and run those
tests from the existing lint workflow via `make test-ci-scripts`.


Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-20 13:48:12 +02:00
mudler's LocalAI [bot]
963c637130 fix(gpu-libs): bundle cuDNN only where it is used, and complete it when it is (#10946) 
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.

On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.

  longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
  in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
  DT_RUNPATH, so the bundle won and the rest still came from the venv:
  CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.

  Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
  libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
  out of the runtime image and the process ran bundled 9.22.0 against system
  9.23.2.

Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.

Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.

So bundle per backend, by what that backend actually needs:

  - venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
    set, which is the one its torch was built against            (longcat-video)
  - venv has no pip cuDNN         -> bundle the complete family. Stays
    conservative rather than detecting consumers: for a Python backend they sit
    inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
    look                                                                 (vllm)
  - no venv, nothing references cuDNN -> bundle nothing    (llama-cpp, whisper,
                                     rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
  - no venv, something references it   -> bundle the complete family
                                                  (face-detect, voice-detect)

The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.

Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.

verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.

Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.

The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.

Fixes #10905


Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-19 07:48:51 +00:00
dependabot[bot]
036eccc32d chore(deps): bump actions/setup-node from 6 to 7 (#10915) 
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6 to 7.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-18 22:36:37 +02:00
localai-org-maint-bot
bc653c9b09 ci(dependabot): ignore torch/transformers for diffusers to fix Jetson-index auth failure (#10913) 
The weekly "Dependabot Updates" pip job for /backend/python/diffusers has been
failing with `private_source_authentication_failure` against the Jetson pip
index (https://pypi.jetson-ai-lab.io/jp6/cu129/), referenced by that backend's
requirements-l4t12.txt. diffusers is the only dependabot-configured pip
directory that pulls from that private index, so it is the only update job that
fails; the other backends update cleanly.

torch and transformers are deliberately pinned in this backend for
reproducibility (see backend/python/diffusers/requirements-*.txt and #9979), so
we do not want dependabot bumping them anyway. Ignoring both dependencies for
this directory stops dependabot from resolving them against the unreachable
Jetson index and keeps the weekly update job green, without removing update
coverage for the rest of the backend's dependencies.


Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-18 08:36:23 +02:00
mudler's LocalAI [bot]
40d35c0385 docs: onboarding overhaul, dedup, and error docs (#7711) (#10895) 
* docs: fix CPU image tag (latest, not latest-cpu)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: use canonical localai/localai registry in models guide

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: replace dead llama-stable backend with llama-cpp

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: correct mitm-proxy intercept config and redaction tier

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fix text-to-audio endpoint and broken notice block

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fix VAD example, stale FAQ, broken link, CLI list, whats-new dump

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: render advanced/reference section indexes (consolidate _index)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove duplicate getting-started build/kubernetes pages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fold container image reference into installation/containers

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove stale advanced fine-tuning page (superseded by features/fine-tuning)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fold distribution/longcat/sound pages into their parents

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: make getting-started index accurate and complete

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: carry one concrete model through the getting-started path

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add end-to-end 'build your first agent' walkthrough

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add runtime errors reference; consolidate troubleshooting from FAQ

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add agent actions catalog

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: agent-scoped MCP, skills walkthrough, agentic disambiguation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add concrete gallery install lines to media feature pages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: merge installation into getting-started (URLs preserved via aliases)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add Operations section; move operator pages and P2P API reference

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: journey-ordered top nav and grouped feature sections

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add docs-with-code process gate (PR template + agent instructions)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove em/en dashes from documentation prose

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-17 22:08:20 +02:00
LocalAI [bot]
3bb0d1cb49 feat(backend): add moss-tts-cpp text-to-speech backend (#10860) 
* feat(backend): add moss-tts-cpp text-to-speech backend

Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.

Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines

Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release

moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta

The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-17 09:26:12 +02:00
LocalAI [bot]
bbe018c1a0 feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834) 
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models

Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.

Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).

Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.

Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-16 10:09:14 +02:00
Richard Palethorpe
b9d6d49e31 fix(cloud-proxy): publish backend gallery entries (#10858) 
Add stable and development gallery variants for Linux and Darwin, and wire the backend build matrix so the referenced images are published.

Assisted-by: Codex:gpt-5 [yq]

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-07-16 08:36:23 +02:00
dependabot[bot]
cdb6702ca6 chore(deps): bump actions/stale from 10.3.0 to 10.4.0 (#10807) 
Bumps [actions/stale](https://github.com/actions/stale) from 10.3.0 to 10.4.0.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](eb5cf3af3a...1e223db275)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-version: 10.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-13 22:50:01 +02:00
LocalAI [bot]
b00422e45f feat(backends): add LongCat video and avatar generation (#10792) 
* feat(backends): add LongCat video and avatar generation

Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]

* refactor(config): declare model I/O modalities

Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.

Assisted-by: Codex:GPT-5 [web]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-12 23:58:46 +02:00
LocalAI [bot]
94bdc825dc feat(backend): add moss-transcribe-cpp backend (MOSS-Transcribe-Diarize) (#10756) 
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).

Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.

Assisted-by: Claude:claude-opus-4-8 golangci-lint

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-09 23:27:11 +02:00
dependabot[bot]
fa0622604a chore(deps): bump actions/cache from 4 to 6 (#10704) 
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-06 21:39:37 +02:00
LocalAI [bot]
29db4ab414 fix(ci): shard single-arch backend matrix under GitHub's 256-job limit (#10703) 
GitHub Actions refuses to instantiate a matrix that would generate more
than 256 jobs. It does so silently: the job hangs forever at "Waiting for
pending jobs" and the whole run is marked `failure` while every other job
stays green. This is exactly what happened on the v4.6.1 tag build
(run 28786533892): the single-arch build matrix had grown to 268 entries,
so `backend-jobs-singlearch` (and its downstream merge) never produced a
single job, and the release build "failed" with no failing job to point at.

The single-arch list is the one that grows unbounded as backends are added,
so shard it across a fixed number of matrix jobs (SINGLEARCH_SHARDS=4,
~67 entries each today, headroom to ~1020 backends). Each merge shard
`needs:` only its matching build shard, preserving the "merge waits only on
its own build" property that keeps slow CUDA/ROCm builds from gating
multi-arch manifest assembly.

changed-backends.js now emits per-shard matrix/has-* outputs and throws
loudly if a shard ever reaches the 256 limit (telling the maintainer to
bump SINGLEARCH_SHARDS and add matching job blocks) instead of letting
GitHub drop the overflow silently. backend.yml and backend_pr.yml define
the four build + four merge shard jobs; multi-arch and darwin groups are
untouched.


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-06 21:38:12 +02:00
Richard Palethorpe
5d0c43ec6e feat(realtime): Semantic VAD EOU token (#10444) 
* feat(realtime): EOU-driven semantic_vad turn detection

Add a `semantic_vad` turn-detection mode to the realtime API that feeds
the transcription model live and decides "the user finished speaking"
from the `<EOU>` end-of-utterance token rather than from silence alone.
When EOU fires the turn commits immediately (~0.3s); otherwise it falls
back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s).

Plumbing, bottom to top:

- proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof,
  mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus
  `TranscriptResult.eou` for the unary retranscribe gate.
- pkg/grpc: client/server/base/embed scaffolding for the bidi stream,
  modeled on AudioTransformStream; release stream conns on terminal Recv.
- parakeet-cpp: live transcription RPC with per-C-call engine locking
  (one live stream per turn, finalize+free at commit); bump parakeet.cpp
  to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel
  recompute that delayed EOU on long turns) and the <EOU>/<EOB> split;
  strip the literal <EOU>/<EOB> from offline text and set Eou.
- core/backend: LiveTranscriptionSession wrapper + pipeline
  `turn_detection:` config block (type/eagerness/retranscribe).
- realtime: semantic_vad integration — live input captions streamed as
  transcription deltas while the user speaks, EOU-immediate commit with
  eagerness fallback, optional retranscribe gate (batch re-decode must
  also end in <EOU> to confirm), clause synthesis off the LLM token
  callback, and per-turn live-transcription / model_load telemetry.
- UI: show the realtime pipeline components as a vertical list.

Docs and tests included; opt-in via the pipeline YAML or per-session
`session.update`. Non-streaming STT backends degrade to silence-only.

Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash]
Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): explicit formally-verified state machines + parakeet streaming driver

The realtime API had several implicit state machines whose state was inferred
from scattered booleans, channels, and five separate mutexes, leaving
illegal/inconsistent states reachable. Make them explicit and keep the
implementation in step with a formal design; rework the parakeet streaming
backend along the same lines.

Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect
with a total, pure Next(state,event)->(state,[]effect) behind a single-writer
Coordinator:

  M1 conncoord    connection lifecycle: VAD toggle + once-only teardown
                  (replaces vadServerStarted + a `done` channel closed from
                  two sites).
  M2 turncoord    turn detection: collapses speechStarted and the live-stream
                  "turn open" flag into one state, so discardTurn can no longer
                  desync them and suppress the next onset.
  M3 respcoord    response coordination: serializes the dual-writer
                  start/cancel so at most one response is live; one
                  response.done per response.create.
  M4 compactcoord conversation compaction: single-flight (replaces the
                  `compacting atomic.Bool` CAS).
  M5 ttscoord     TTS pipeline: open->closing->closed, idempotent wait(),
                  rejects enqueue-after-close (was a silent drop).

The Coordinator/Sink/Next plumbing — only the sealed types and Next differed
per machine — is extracted once into core/http/endpoints/openai/coordinator as
a generic Coordinator[S,E,F]; each machine keeps its public API via type
aliases, so no sink, call-site, or test moved.

Hierarchy. session_lifecycle.fizz models M1 as the parent region with its
children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn
torn => all children terminal, none start after teardown). respcoord and
compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's
teardown drives the children terminal. This closes a compaction teardown gap: a
fire-and-forget compaction could outlive a torn session — compactionSink now
takes a session-scoped cancellable context + WaitGroup and joins the in-flight
summarize+evict on shutdown.

Formal verification. formal-verification/ holds one authoritative FizzBee spec
per machine plus the composition spec, each with an always-assertion and a
documented one-line edit that makes the checker fail (verified non-vacuous).
scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under
-race AND a model-check of every .fizz spec; a missing FizzBee is a hard error
(only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI).
FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into
.tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow,
and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the
repo's forbidigo lint): transition tables + fixed-seed property walks +
concurrent/-race specs, no rapid dependency. Design map:
docs/design/realtime-state-machines.md.

Parakeet streaming backend. The same treatment applied to the parakeet-cpp
streaming paths:
- AudioTranscriptionStream returns codes.Unimplemented for non-streaming models
  instead of decoding offline and emitting it as one delta + final. A client
  that asked for streaming learns the model cannot stream rather than receiving
  a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported
  carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it
  as an SSE error event. Mirrors AudioTranscriptionLive, which already did this.
- utteranceBoundary (boundary.go): a single definition of the end-of-utterance
  latch, replacing three open-coded finalEou toggles. Modelled as a two-valued
  type so illegal states are unrepresentable.
- Shared decode driver (driver.go): streamFeedResult (one per-feed event) +
  feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail.
  The feed loop is written once.
- AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed
  {delta,eou,eob,words} the realtime turn detector consumes and a terminal
  FinalResult carrying only Text. Segments/duration/eou are offline-only and no
  longer produced (nor read) on the live path; liveTraceState drops the terminal
  eou and keeps the per-feed eou_events count.
- AudioTranscriptionStream + streamJSON merge into one driver-based function;
  streamSegmenter is generalized to the unified event with a text-only fallback
  that preserves the legacy (no-words) library's per-utterance segmentation.

Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and
parakeet packages under -race, the fail-closed conformance gate green, and
make test-realtime (12 e2e WS+WebRTC).

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-06-30 09:01:22 +02:00
dependabot[bot]
036f950b1b chore(deps): bump actions/cache from 4 to 6 (#10593) 
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-29 22:31:10 +02:00
LocalAI [bot]
de2ec2f136 feat(backends): add voice-detect + face-detect ggml backends (replace Python insightface/speaker-recognition) (#10441) 
* feat(voice-detect): add Go purego backend for voice-detect.cpp

Add backend/go/voice-detect implementing the Backend gRPC voice subset
(VoiceEmbed/VoiceVerify/VoiceAnalyze) over libvoicedetect.so via purego,
mirroring the parakeet-cpp / omnivoice-cpp backends.

The flat voicedetect_capi C ABI is dlopen'd cgo-less; malloc'd string and
float-vector returns are owned by Go and released through the matching capi
free functions, with the per-ctx last error surfaced into Go errors. Calls are
serialized via base.SingleThread since the C context is not reentrant.

Proto field mapping:
- VoiceEmbed: VoiceEmbedRequest.audio (path) -> embed_path -> Embedding+Model.
- VoiceVerify: audio1/audio2 + threshold (<=0 falls back to the
  verify_threshold option, default 0.25) -> verify_paths -> verified/distance/
  threshold/confidence/model/processing_time_ms.
- VoiceAnalyze: audio (path) -> analyze_path_json; the JSON age/gender/emotion
  document maps to a single VoiceAnalysis segment (start/end 0; gender "label"
  -> dominant_gender with the remaining float scores as the gender map; emotion
  label/scores -> dominant_emotion/emotion).

The Makefile pins voice-detect.cpp to 47546430, clones+builds libvoicedetect.so
with ggml static-linked (PIC, GGML_NATIVE off) so dlopen needs no external
libggml/libvoicedetect; ldd on the artifact shows only system libs. Ginkgo
tests cover option parsing and analyze-JSON mapping; embed/verify smoke specs
gate on VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(voice-detect): wire backend into index, gallery and build

Register the voice-detect.cpp speaker-recognition + voice-analysis
backend (added in Voice-INT-A) into LocalAI's distribution surfaces,
mirroring the ced backend (the closest mudler C++/ggml audio analogue):

- backend/index.yaml: add the &voicedetect meta-backend (capabilities
  platform map, no top-level uri) plus the full set of concrete per-arch
  image entries (cpu/cuda12/cuda13/metal/rocm/sycl/vulkan/l4t and the
  -development variants). Referential integrity audited - every alias
  target resolves.
- gallery/index.yaml: add 5 model entries on backend voice-detect -
  ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++ and the
  wav2vec2 age/gender/emotion analyze model. The engine architecture is
  read from GGUF metadata (voicedetect.arch) at load. GGUF artifacts are
  not yet published: each files: entry points at the intended
  mudler/voice-detect-gguf location with a TODO to fill sha256 after
  upload (no fabricated hashes).
- .github/backend-matrix.yml: add the linux build matrix block + the
  darwin metal entry mirroring ced.
- .github/workflows/bump_deps.yaml: track mudler/voice-detect.cpp via
  VOICEDETECT_VERSION (pin 47546430, = 4754643).
- core/config/backend_capabilities.go: register voice-detect in the
  backend capability map (VoiceVerify/VoiceEmbed/VoiceAnalyze ->
  speaker_recognition), mirroring speaker-recognition.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(face-detect): add purego Go backend for face-detect.cpp

Add the LocalAI Go backend that dlopens libfacedetect.so (the flat
facedetect_capi_* C-ABI) via purego, mirroring the sibling voice-detect
backend. Implements the Face subset of the Backend gRPC service:

- Embeddings(PredictOptions): Images[0] base64 -> temp file -> embed_path
  -> L2-normalized ArcFace embedding.
- Detect(DetectOptions): src -> detect_path_json -> Detection boxes
  (class_name "face", [x1,y1,x2,y2] -> x/y/w/h).
- FaceVerify(FaceVerifyRequest): two images + threshold + anti_spoof ->
  verify_paths; best-effort img areas via detect.
- FaceAnalyze(FaceAnalyzeRequest): img -> analyze_path_json -> per-face
  age + gender ("M"/"F" normalized to "Man"/"Woman").

The Makefile pins face-detect.cpp to 636a1963 and builds the shared lib
with ggml + vendored libjpeg-turbo static (PIC), so the .so is
ldd-clean (no libggml) and exports only facedetect_capi_* (no jpeg_
symbols). Gated Ginkgo e2e mirrors voice-detect.

Note for the gallery-wiring task: backend registration (index.yaml,
gallery, core/config/backend_capabilities.go) is intentionally not
touched here.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* fix(voice-detect): replace em dashes in net-new descriptions

Project style forbids em/en dashes. Replace the three U+2014 chars
introduced by the voice-detect gallery/index wiring with `-`/`:`.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(face-detect): wire backend into index, gallery and build

Register the face-detect.cpp face detection / embedding / verification /
analysis backend (added in Face-INT-A) into LocalAI's distribution
surfaces, mirroring the voice-detect wiring (the closest mudler C++/ggml
recognition analogue):

- backend/index.yaml: add the &facedetect meta-backend (capabilities
  platform map, no top-level uri to avoid the meta-backend gotcha) plus
  the full set of concrete per-arch image entries (cpu/cuda12/cuda13/
  metal/rocm/sycl-f16/sycl-f32/vulkan/l4t and the -development variants),
  22 entries. Referential integrity audited: every alias target resolves.
- gallery/index.yaml: add 4 model entries on backend face-detect -
  face-detect-buffalo-l/m/s (insightface SCRFD + ArcFace/MBF, NON-COMMERCIAL)
  and face-detect-yunet-sface (OpenCV-Zoo YuNet + SFace, APACHE-2.0, the
  commercial-friendly alternative). The detector/embedder architecture is
  read from GGUF metadata (facedetect.arch) at load; only the real
  verify_threshold option is set (0.35 buffalo, 0.363 sface). GGUF
  artifacts are not yet published: each files: entry points at the
  intended mudler/face-detect-gguf location with a TODO to fill sha256
  after upload (no fabricated hashes).
- core/config/backend_capabilities.go: register face-detect in the
  backend capability map (Embedding/Detect/FaceVerify/FaceAnalyze ->
  face_recognition), mirroring insightface.
- .github/backend-matrix.yml: add the linux build matrix block + the
  darwin metal entry mirroring voice-detect.
- .github/workflows/bump_deps.yaml: track mudler/face-detect.cpp via
  FACEDETECT_VERSION (pin 636a1963).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* fix(recon): voice-detect metal build branch + face-detect gallery usecases

Add the missing metal BUILD_TYPE branch to the voice-detect Makefile
forwarding -DVOICEDETECT_GGML_METAL=ON, mirroring face-detect, so the
darwin metal CI artifact is built with the Metal backend instead of
CPU-only.

Expand the 4 face-detect gallery models' known_usecases to
[face_recognition, detection, embeddings] to match the backend
capabilities map and the mirrored insightface-buffalo entries, so
auto-selection for /v1/detect and /embeddings works.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* docs(recon): document voice-detect and face-detect ggml backends

Document the new standalone C++/ggml biometric backends as the
recommended/default option for face and voice recognition, keeping the
existing Python insightface / speaker-recognition backends framed as the
legacy path.

- features/face-recognition.md: add a face-detect (ggml) backend section
  with the gallery entries (buffalo-l/m/s non-commercial, yunet-sface
  Apache-2.0), licensing, and verify/detect/analyze quickstart.
- features/voice-recognition.md: add a voice-detect (ggml) backend
  section with the gallery entries (ecapa-tdnn, wespeaker-resnet34,
  eres2net, campplus speaker recognizers; emotion-wav2vec2 non-commercial
  analyze head) and quickstart.
- reference/compatibility-table.md: add face-detect.cpp and
  voice-detect.cpp rows to the Vision, Detection & Recognition table.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(gallery): publish recon backend GGUF uris + sha256

Fill in the published HuggingFace GGUF uris and verified sha256 for the
9 recon gallery entries (voice-detect-* and face-detect-*), and remove
the TODO publish markers. Correct the eres2net, campplus, and
emotion-wav2vec2 uris to the actual published filenames.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(gallery): re-embed buffalo anti-spoof + add audeering age/gender voice model

Update the 3 buffalo face-detect GGUF sha256 (anti-spoof ensemble now
embedded and re-uploaded under the same filenames/uris) and note the
FaceVerify anti_spoof request flag in each description. Add a new
voice-detect-age-gender-wav2vec2 gallery entry mirroring the emotion
model.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(gallery): add face-detect-buffalo-sc and antelopev2 packs

Add gallery entries for two newly-published insightface face packs on
the face-detect backend: buffalo_sc (smallest pack, SCRFD-500M + small
ArcFace) and antelopev2 (higher-accuracy, SCRFD-10G + ArcFace glint360k
R100, 512-d). Both are non-commercial research-only.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(recon): honor LocalAI per-model threads in voice/face-detect backends

LocalAI spawns one backend process per model and serves requests
concurrently, so the engines' own min(hardware_concurrency, 8) default
can oversubscribe cores. Forward the per-model Threads value from the
gRPC LoadModel options into the engine via VOICEDETECT_THREADS /
FACEDETECT_THREADS (read at backend construction) before the capi load.
A non-positive Threads is treated as unset, leaving the engine default.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump backend pins to CPU-optimized engine commits

voice-detect.cpp -> 0d9c1b3 (radix-2 FFT FBank, threads, flash attn + cached
pos-conv); face-detect.cpp -> 523aee1 (thread-gated direct conv, threads).
Brings the CPU optimizations into the LocalAI backend builds. GGUF format and
parity unchanged, so the published HF GGUFs remain valid.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump backend pins to round-2 CPU-optimized engines

voice-detect.cpp -> fe7e6a3 (ERes2Net 1x1->mul_mat, CAM++ layout+context,
wav2vec2 conv-LN, ECAPA capture-drop, AVX512 dispatch opt-in); face-detect.cpp
-> 9c8adb7 (AVX2 Winograd F(2x2,3x3) for SCRFD/ArcFace 3x3 convs, ArcFace
BN-fold). Parity unchanged (cosine=1.0); GGUF format unchanged, HF GGUFs valid.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump backend pins to round-3 Winograd engines

voice-detect.cpp -> 45122ec (Winograd F(2x2,3x3) for WeSpeaker/ERes2Net 3x3
convs, -22%/-20% @8t); face-detect.cpp -> cd5c962 (Winograd F(4x4,3x3) for
SCRFD large maps, -22% @1t on top of F(2x2), more load-stable). Parity held
(cosine=1.0); GGUF format unchanged, HF GGUFs valid.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump backend pins to round-4 Winograd engines (CPU opt complete)

voice-detect.cpp -> d2839ca (CAM++ FCM 2D convs through Winograd, -15.5%/-10.3%);
face-detect.cpp -> c1db23d (AVX2-vectorized Winograd tile transforms, SCRFD
detect -14%/-9.6%). Final CPU optimization round; the conv-kernel lever class is
now exhausted (parity held cosine=1.0; GGUF/parity unchanged, HF GGUFs valid).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump face-detect pin to deep-kernel engine (7ae5c4d)

face-detect.cpp -> 7ae5c4d: register-blocked winograd-domain GEMM microkernel
(2.8x isolated GFLOP/s), AVX-512 zmm evolution behind runtime CPUID dispatch
(ship-safe, AVX2 fallback bit-identical), bias/relu fused into the winograd
output transform, and SFace Conv+BN fold + bias/PReLU fusion. SCRFD detect
~1.4x faster end-to-end vs the round-4 baseline; parity bit-exact; portable
single binary (function-multiversioned, no global -mavx512f).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump voice-detect pin to ECAPA operand-order win (e9c56ae)

voice-detect.cpp -> e9c56ae: weight-as-src0 mul_mat order in ECAPA's F32
conv1d_same (routes through tinyBLAS sgemm); ECAPA embed 1.67x @1t / ~1.3x @8t,
parity cosine=1.0. Isolated to encoder.cpp (ECAPA-only); ERes2Net/CAM++/WeSpeaker
do not call conv1d_same so are provably unaffected.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to FMA-throughput engines (voice f7b9f89, face 2d2d5f0)

face -> 2d2d5f0: route ArcFace 3x3 body convs through the AVX-512 winograd
microkernel (kWinoMinSize 80->14); ArcFace 1.62x @1t, SCRFD detect to 0.966 of
MLAS @1t, no regression. voice -> f7b9f89: runtime-CPUID-dispatched AVX-512
winograd-GEMM microkernel (ship-safe, AVX2 fallback bit-identical); WeSpeaker
1.90x @1t. Parity cosine=1.0 throughout; portable single binaries.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to MLAS-class direct-conv engines (voice 7ecfd07, face be22d67)

Hand-tuned nChw16c AVX-512 register-tiled direct-conv microkernel (~263 GFLOP/s,
within 6-7% of MLAS per-op efficiency), runtime-CPUID-dispatched + AVX2 fallback,
fused bias/relu. voice 7ecfd07: default 3x3-s1 kernel for WeSpeaker (+37%/+32%)
+ ERes2Net, CAM++ pinned to Winograd. face be22d67: shape-gated to the ArcFace
recognizer body (+25-27% @8t); SCRFD detector stays on Winograd (no regression).
Parity cosine=1.0 / detect <=1px on AVX-512 + AVX2 paths. Portable single binaries.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump voice pin to Phase-A blocked backbone (f4e7eef)

WeSpeaker ResNet34 runs as one nChw16c blocked island (2 reorders/forward vs
~60) on AVX-512, default; per-conv directconv fallback on AVX2. +2.9% @1t /
+17-19% @8t vs per-conv directconv, parity cosine=1.0. The conv microkernel is
already FMA-bound near peak (~0.86-0.98x MLAS-implied); residual to MLAS is
sub-peak edge + non-conv tail, documented in docs/cpu-optimization.md.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to breadth blocked-backbone (voice 7f66871, face d80092b)

voice 7f66871: AVX2-vectorized (ymm) blocked island - AVX2-only hosts now run
the blocked backbone for WeSpeaker (2.3x over per-conv-AVX2, cosine=1.0);
ERes2Net stays per-conv (blocked regresses, opt-in only); CAM++ Winograd-pinned.
face d80092b: ArcFace recognizer blocked island, AVX-512 default (-13% @8t, ~0.90x
MLAS, the closest conv result), auto per-conv on AVX2; SCRFD untouched on Winograd
(0 island invocations during detect). Parity cosine=1.0 / detect <=1px throughout.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to small-spatial + stem conv kernels (voice 99b1804, face 47fdab6)

Measured-gap-driven conv kernels: small-spatial (fill the register tile when
output width <= tile width) + small-IC stem + strided-1x1/downsample recovery.
ArcFace recognizer 0.57 -> 0.70x MLAS @1t (the closest conv model), WeSpeaker
0.65 -> 0.79x @1t. Parity cosine=1.0 / detect <=1px. The OC-block-sharing lever
was a measured dead-end (deep stride-1 is L3-weight-bandwidth bound, not
read-port bound) and was NOT shipped. Kernel ceiling reached; further gap needs
an algorithm-class change (cache-blocked weight-stationary GEMM, or q8 weights).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to GPU persistent-graph + multi-model-safe cache (voice 45d2e6b, face 0a4799a)

GPU wins (CUDA/ggml backend, no CPU-path change): persistent per-shape graph+context
cache in Backend::compute() eliminates the per-call cudaGraph re-instantiation churn
-> wav2vec2 emotion+age-gender now AT GPU parity with torch-cuDNN on GB10 (0.97-0.98x),
CAM++ -5.7ms; bit-identical parity. Cache hardened multi-model-safe (invalidate-on-free
keyed by the ModelLoader weights buffer) so LocalAI multi-model hosting cannot stale-hit.
Conv models still trail cuDNN (im2col-materialization-bound) - cuDNN implicit-GEMM lever next.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump pins to cuDNN-conv-capable engines (voice b6e4356, face 6107a24)

Adds the opt-in cuDNN implicit-GEMM conv path (VOICEDETECT_GGML_CUDNN /
FACEDETECT_GGML_CUDNN, DEFAULT OFF -> zero build/runtime dep until enabled).
On GPU it kills the im2col-materialization bottleneck and reaches torch-cuDNN
parity on the spill-bound convs: SCRFD detect 14.8->6.4ms (2.3x, ~parity),
WeSpeaker ~parity, ERes2Net beats torch (1.10x); ArcFace/CAM++ neutral (no
spill). Parity exact (SCRFD <=1px, cosine=1.0). To USE it in LocalAI, the CUDA
backend build must enable the flag AND bundle libcudnn - deferred until a
cuDNN-bundled GPU image; flag stays OFF here.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* feat(recon): enable cuDNN conv path on arm64+CUDA13 recon backends

The voice-detect.cpp / face-detect.cpp engines have an opt-in cuDNN
implicit-GEMM conv path behind VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN
(default OFF) that kills im2col on the GPU and reaches torch-cuDNN parity
(SCRFD 2.3x, WeSpeaker/ERes2Net parity), measured on the GB10
(arm64, CUDA 13, sm_121a).

Enable it for the CUDA build, but only where cuDNN actually ships: the
arm64 + CUDA 13 image (GB10/Jetson/L4T). x86 CUDA images carry no cuDNN,
so flipping it on globally for BUILD_TYPE=cublas would be a link failure.
The Makefiles gate on CUDA_MAJOR_VERSION=13 + arch (TARGETARCH from the
matrix/Docker build, uname -m fallback for local builds).

backend/Dockerfile.golang already installs the runtime libcudnn9-cuda-13
in the arm64+CUDA13 apt block; add the matching libcudnn9-dev-cuda-13 so
the build-time link resolves.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): bump voice-detect pin to ERes2Net blocked-default (30beecd)

Defaults VD_ERES2NET_BLOCKED ON: routes the ERes2Net Res2Net body through the
blocked nChw16c AVX-512 directconv island instead of the 1x1 mul_mat fast path
(CONT-transpose + skinny low-K GEMM). On the shipped GGML_NATIVE=OFF build (ggml
mul_mat is AVX2-only) this wins ~2x at every thread count (2.07x@1t, 2.2x@4t,
2.05x@8t); pure-AVX2 fallback still 1.3-1.62x. Parity exact (cosine=1.000000 vs
golden), so registered voices + verify/identify thresholds are unaffected. The
prior default-OFF rested on a stale comment whose 23pct regression only held on
the non-shipping GGML_NATIVE=ON build.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* docs(readme): announce native voice-detect + face-detect backends in Latest News

Add a Latest News entry for the new from-scratch C++/ggml biometric backends
(voice-detect.cpp + face-detect.cpp) that replace the Python insightface and
speaker-recognition backends: no Python/onnxruntime at inference, self-contained
GGUF, bit-exact parity, GPU cuDNN parity. Mirrors the parakeet.cpp /
locate-anything.cpp native-backend news entries. Refs PR #10441.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* chore(recon): re-pin to the squashed engine release commits

The voice-detect.cpp and face-detect.cpp histories were squashed to a single
release commit, which orphaned the previous pins (voice 30beecd, face 6107a24).
Re-pin to the new single-commit SHAs (voice 3d51077, face 06914b0); the tree is
identical, so the backend build is unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-28 09:29:08 +02:00
LocalAI [bot]
fdff114701 ci(vibevoice): skip the ASR transcription e2e on release tag builds (#10567) 
The `tests-vibevoice-cpp-grpc-transcription` job downloads the vibevoice ASR
model (`vibevoice-asr-q4_k.gguf`, ~10 GB) and decodes it through the
e2e-backends harness. On release tag pushes the detect step forces the full
matrix (run-all=true), so this job runs and consistently times out: the inner
`go test -timeout 30m` cannot pull a 10 GB file from HuggingFace's throttled
Xet CDN within budget (curl --max-time 600 x5 retries overruns the deadline),
leaving an orphaned curl and a 30m panic. It has been red on every release
(v4.5.3/4/5).

Guard the job's `if` with `!startsWith(github.ref, 'refs/tags/')` so it no
longer runs on tag/release builds. It still runs on PRs and branch pushes that
touch vibevoice-cpp, so real regressions are caught off the release path. A
proper fix (a small ASR test GGUF) can re-enable it on tags later.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-28 00:40:21 +02:00
LocalAI [bot]
8aba4fdba3 chore(fish-speech): drop the darwin/metal build target (#10561) 
The fish-speech metal-darwin-arm64 backend build has been failing on every
release (v4.5.3, v4.5.4, v4.5.5) and is a standing red on the darwin backend
matrix. fish-speech pulls `tokenizers` transitively from its upstream source
(`pip install -e fish-speech-src`), and on darwin/arm64 there is no prebuilt
wheel for the pinned old `tokenizers` version, so pip builds it from source.
Modern rustc rejects that old crate as a hard error:

    error: casting `&T` to `&mut T` is undefined behavior ...
       --> tokenizers-lib/src/models/bpe/trainer.rs:517:47
       = note: `#[deny(invalid_reference_casting)]` on by default
    error: could not compile `tokenizers` (lib) due to 1 previous error

This is deterministic, not a flake, and there is no clean fix that does not
either pin a stale Rust toolchain or downgrade a soundness lint guarding real
UB. Until upstream fish-speech moves to a tokenizers version that compiles on
current toolchains, drop darwin support so the release backend build stays
green. The Linux/CUDA/ROCm/Intel/L4T variants are unaffected.

Removes:
- the `-metal-darwin-arm64-fish-speech` entry from `includeDarwin` in
  backend-matrix.yml
- the `metal:` capability mappings and the concrete `metal-fish-speech` /
  `metal-fish-speech-development` gallery entries in backend/index.yaml
- the now-unused darwin-only requirements-mps.txt

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-27 23:24:21 +02:00
LocalAI [bot]
f0d0bff232 fix(llama-cpp): stop reinterpreting plain-string message content as JSON (#10524) (#10538) 
The llama-cpp gRPC backend reconstructs OpenAI messages from proto for the
tokenizer-template path and blindly json::parse'd each message's content
string. LocalAI's Go layer always flattens content to a plain string, so a
user prompt that merely looks like JSON (e.g. mealie's ingredient array
["1/4 cup brown sugar", ...]) was reinterpreted as structured content parts and
rejected by oaicompat_chat_params_parse with "unsupported content[].type".

Normalize content per role instead: user/system/developer content is opaque
text and is never JSON-sniffed; assistant/tool content still collapses a literal
JSON null/object (tool-call bookkeeping) to a string, but a plain string is
never turned into an array/scalar. The array defense is role-independent, so the
role gate only governs the benign null/object case.

While here, extract the duplicated per-message reconstruction and the
pre-template content sanitization into shared, unit-tested helpers
(message_content.h) so the streaming (PredictStream) and non-streaming (Predict)
paths cannot drift. This removes ~490 lines of copy-pasted defensive code, the
dead tool-role parse branches, and the redundant Predict-only tool_calls branch,
while preserving the prior #7324 (null content -> "") and #7528 (tool array
content -> string) fixes.

Tests:
- backend/cpp/llama-cpp/message_content_test.cpp: standalone C++ unit tests for
  all three helpers (#10524, #7324, #7528, multimodal), discovered and run by
  `make test-backend-cpp` and a new generic tests-backend-cpp CI job. Also wired
  as an opt-in CMake/ctest target (-DLLAMA_GRPC_BUILD_TESTS=ON).
- core/schema/message_test.go: Go regression pinning that ToProto flattens a
  JSON-array-looking text part to the verbatim string.
- prepare.sh now copies message_content.h into the build tree.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-27 01:42:05 +02:00
LocalAI [bot]
5b3572f8b8 feat(macos): sign and notarize the DMG, app, and server binary (#10510) 
Produce a Gatekeeper-clean macOS distribution with no user workaround:

- Launcher DMG + the LocalAI.app inside it are built via fyne, codesigned
  with the Developer ID under the hardened runtime, then the DMG is signed,
  notarized (notarytool) and stapled. Replaces macos-dmg-creator (which had
  no signing hook) with fyne package + hdiutil so we control the .app before
  packaging.
- The bare local-ai darwin server binary is signed + notarized via
  GoReleaser's native notarize block (quill backend, runs on Linux).
- All signing is gated on secrets being present, so forks/PRs/local builds
  stay unsigned and green (contrib/macos/sign-and-notarize.sh no-ops).
- Add hardened-runtime entitlements and FyneApp.toml for deterministic
  packaging; update macOS install docs to drop the quarantine workaround.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-26 12:45:51 +02:00
LocalAI [bot]
6afe127cd4 fix(backends): make the opus backend build and package on macOS/Darwin (#10523) 
The opus Go backend (WebRTC audio codec) never built on macOS, so the
published master-metal-darwin-arm64-opus image shipped source only — no
opus binary and no libopusshim — because every step assumed Linux.

- Makefile: hardcoded libopusshim.so with no OS handling. Mirror
  sherpa-onnx: SHIM_EXT=so / dylib on Darwin and build
  libopusshim.$(SHIM_EXT). On Darwin link the shim with
  -undefined dynamic_lookup so it resolves opus_encoder_ctl from the
  already globally-loaded libopus (codec.go dlopens it RTLD_GLOBAL
  first) instead of baking an absolute Homebrew path into the dylib,
  keeping the packaged shim relocatable.
- run.sh: hardcoded LD_LIBRARY_PATH + libopusshim.so even on macOS. Add
  a Darwin branch exporting DYLD_LIBRARY_PATH and the .dylib shim, like
  sherpa-onnx/run.sh.
- package.sh: bundle libopusshim.$(SHIM_EXT) and libopus*.dylib (not
  just .so) into package/lib so the OCI image (which ships package/.)
  is self-contained on a runtime with no Homebrew; add a Darwin arch
  branch so it doesn't warn/skip.
- backend_build_darwin.yml: install + link opus and pkg-config via brew
  so the Makefile's `pkg-config opus` resolves on the macOS runner, and
  cache opus' Cellar dir.

Go code is unchanged; darwin build is validated in CI.

Assisted-by: Claude:claude-opus-4-8

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-26 11:19:50 +02:00
LocalAI [bot]
d388f874de feat(backends): darwin/Metal build for the privacy-filter backend (#10513) 
* feat(backends): darwin/Metal build for the privacy-filter backend (timeboxed try)

The privacy-filter.cpp engine is already Metal-capable on Apple Silicon: it pulls
ggml and never forces GGML_METAL=OFF, and ggml defaults Metal ON on Apple, so a
plain Darwin build is Metal-enabled. grpc++/protobuf resolve from Homebrew via
find_package(... CONFIG). It just had no darwin build path - the existing
package.sh and run.sh are Linux-only and there was no make target / workflow step.

Adds the bespoke darwin path, modeled on the ds4 one:
- scripts/build/privacy-filter-darwin.sh: native make grpc-server, otool -L dylib
  bundling, create-oci-image (no Linux package.sh).
- Makefile: backends/privacy-filter-darwin target (+ .NOTPARALLEL).
- .github/workflows/backend_build_darwin.yml: gated build step for privacy-filter.
- scripts/changed-backends.js: inferBackendPathDarwin special-case -> backend/cpp.
- .github/backend-matrix.yml: includeDarwin entry (lang go, like ds4/llama-cpp).
- backend/index.yaml: metal: capability + metal-privacy-filter(-development) entries.
- backend/cpp/privacy-filter/run.sh: DYLD_LIBRARY_PATH branch on Darwin.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* fix(privacy-filter): macOS proto include + bundle ggml dylibs

Validated natively on an M4 (the build/package/load chain now works with Metal):

- CMakeLists.txt: hw_grpc_proto compiles the generated proto/grpc sources but
  only linked the binary dir, so on macOS it could not find protobuf's headers
  (runtime_version.h) - Homebrew puts them under /opt/homebrew, not /usr/include.
  Link protobuf::libprotobuf + gRPC::grpc++ so their include dirs propagate. No-op
  on Linux (apt headers are already on the default search path).
- privacy-filter-darwin.sh: bundle the ggml shared libs the binary @rpath-links
  (libggml{,-base,-cpu,-blas,-metal}); the otool -L walk only catches on-disk
  absolute deps and missed them. Resolved at runtime by run.sh's DYLD_LIBRARY_PATH.

M4 check: arm64 grpc-server links @rpath/libggml-metal.0.dylib; with the 15 ggml
dylibs + grpc/protobuf bundled, it loads clean (no dyld errors) and prints usage.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-26 01:18:41 +02:00
LocalAI [bot]
f2ed63e39a docs(backends): make OS coverage explicit + require darwin support for new backends (#10516) 
docs(backends): make OS coverage explicit + require darwin for new backends

The backend matrix is the source of truth for which OS a backend ships on, but
that was never written down, so backends were landing Linux-only by default even
when the engine builds fine on macOS.

- .github/backend-matrix.yml: header block documenting the two matrices
  (include = Linux, includeDarwin = macOS/Apple Silicon) and the policy that new
  backends target every OS they can build for.
- .agents/adding-backends.md: a 'Cover every OS' subsection in step 2 (full darwin
  wiring: includeDarwin entry, index.yaml metal: + metal-<backend> entries,
  run.sh DYLD branch + inferBackendPathDarwin case for C++ backends, the
  hw_grpc_proto protobuf/grpc link gotcha, and the path-filter touch) plus a
  verification-checklist item.
- AGENTS.md (CLAUDE.md): Quick Reference pointer so it surfaces every session.


Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-25 23:26:39 +02:00
LocalAI [bot]
286c508ce0 feat(backends): darwin build for the localvqe backend (acoustic echo cancellation) (#10512) 
feat(backends): darwin build for the localvqe backend

LocalVQE (acoustic echo cancellation / noise suppression / dereverberation)
already builds on Darwin - its Makefile takes the OS=Darwin branch with
GGML_METAL=OFF (upstream is CPU + Vulkan only), producing a native arm64 CPU
image. It was just never wired into CI.

- .github/backend-matrix.yml: add localvqe to includeDarwin (build-type metal,
  lang go) - the darwin/arm64 build profile; the backend itself stays CPU.
- backend/index.yaml: metal: capability + concrete metal-localvqe(-development)
  entries pointing at the -metal-darwin-arm64-localvqe images.
- backend/go/localvqe/Makefile: note on the existing Darwin branch (also the
  per-backend change the CI path filter needs to build it here).


Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-25 22:54:36 +02:00
LocalAI [bot]
d1a9d59917 feat(backends): darwin/Metal builds for vision C++/ggml backends (depth-anything, locate-anything, rfdetr-cpp, sam3-cpp) (#10511) 
feat(backends): darwin/Metal builds for the vision C++/ggml backends

depth-anything-cpp, locate-anything-cpp, rfdetr-cpp and sam3-cpp already carry
a Darwin/Metal path in their Makefiles (GGML_METAL=ON when build-type=metal),
but were never wired into CI, so no Metal image was published and Apple Silicon
could not install them.

- .github/backend-matrix.yml: add the four to includeDarwin (build-type metal,
  lang go), matching the other go+ggml *-cpp Metal entries.
- backend/index.yaml: add metal: to each backend's capabilities map (main and
  -development) plus concrete metal-<backend>(-development) entries pointing at
  the latest/master -metal-darwin-arm64-<backend> images.
- backend/go/*/Makefile: a one-line note on the existing Darwin branch (also
  the per-backend change the CI path filter needs to actually build them here).


Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-25 22:07:56 +02:00
LocalAI [bot]
3a87d9e48f feat(vllm): macOS/Metal support via vllm-metal (MLX) (#10489) 
* feat(vllm): macOS/Metal support via vllm-metal (MLX)

Add an additive Apple-Silicon path to the existing vllm Python backend so
vLLM runs on macOS via vllm-metal (github.com/vllm-project/vllm-metal).

Spike outcome (proven on a real M4 / macOS 26.5, Qwen3-0.6B):
- vllm-metal registers through vLLM's platform-plugin entry point
  (metal -> vllm_metal:register); MetalPlatform activates and runs on the
  GPU through MLX.
- LocalAI's backend.py is UNCHANGED: AsyncEngineArgs(...) ->
  AsyncLLMEngine.from_engine_args transparently resolves to vLLM 0.23's v1
  AsyncLLM MLX engine, and async generate produced correct output.
- backend.py is NOT touched: its only empty_cache() call is CUDA-only
  (guarded by torch.cuda.is_available()), so the benign shutdown-only
  "Allocator for mps is not a DeviceAllocator" noise comes from vLLM's
  internal EngineCore teardown, not from our code.

Changes (all gated behind a darwin condition; Linux/CUDA/ROCm/Intel paths
are byte-for-byte unchanged):
- install.sh: darwin branch forces PYTHON_VERSION=3.12 (vllm-metal
  requirement), creates/activates LocalAI's managed venv via ensureVenv,
  then reproduces vllm-metal's installer INTO that venv (build vLLM 0.23.0
  from the release source tarball against requirements/cpu.txt, then install
  the prebuilt vllm-metal wheel from its latest GitHub release), and runs
  runProtogen. installRequirements is skipped on darwin.
- backend-matrix.yml: add a vllm includeDarwin entry (mps, python).
- index.yaml: add metal capability + concrete metal-vllm /
  metal-vllm-development child entries mirroring the metal-kitten-tts
  template.

Version coupling: vllm-metal pins vLLM 0.23.0, equal to LocalAI's current
vllm pin. Bumping vllm must be coordinated with a supporting vllm-metal
release; documented in install.sh and requirements-cublas13-after.txt.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* chore(vllm): track the darwin vllm-metal pin via the autobumper

The Apple Silicon build pinned vLLM 0.23.0 as a hidden string in install.sh
while floating the vllm-metal wheel on releases/latest - the two could drift
apart silently. Make both a tracked, reproducible pair (VLLM_METAL_VERSION +
VLLM_VERSION), fetch the wheel by tag, and add .github/bump_vllm_metal.sh wired
into bump_deps.yaml. It tracks vllm-project/vllm-metal (not vllm/vllm latest),
reading the coupled vLLM source version from vllm-metal's own installer, and
opens a bump PR - mirroring the existing bump_vllm_wheel.sh for the cu130 wheel.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* chore(vllm): derive the darwin vLLM version, drop the second pin

Follow-up: VLLM_VERSION was still a hardcoded string duplicating what
VLLM_METAL_VERSION already determines. Derive it at install time from
vllm-metal's own installer (vllm_v=) at the pinned tag - one source of truth,
no second value to drift. The bumper now touches only VLLM_METAL_VERSION;
the derivation is immutable per tag, so builds stay reproducible.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* fix(vllm): fetch the vllm-metal wheel without the GitHub API

The darwin build resolved the wheel URL via api.github.com, whose
unauthenticated rate limit (60/hr per IP) 403s on shared macOS runners
(observed after the 9-min vLLM source build). Construct the release-asset
download URL deterministically from the pinned tag and the cp312/arm64 wheel
name instead - no API call, no rate limit. Verified the URL resolves (200).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* fix(vllm): fail Score cleanly when the engine returns no prompt_logprobs

Audit of the Score path against vllm-metal (MLX on macOS): the engine accepts
SamplingParams(prompt_logprobs=1) but returns an all-None prompt_logprobs list
rather than computing it, so scoring is not supported there. The old guard
treated the truthy [None] list as valid and silently scored every candidate as
0. Detect the all-None case and return UNIMPLEMENTED instead. No-op on
Linux/CUDA, which populate real entries.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-25 15:46:19 +02:00
LocalAI [bot]
a7fec9a49d feat(backends): add darwin/metal (MPS) build for trl (#10487) 
* feat(backends): add darwin/metal (MPS) build for trl

Authors backend/python/trl/requirements-mps.txt and wires trl into the
darwin CI matrix and gallery so the MPS training path can be built and
validated on Apple Silicon. The MPS variant installs plain PyPI torch
wheels (MPS-capable on macOS arm64) and the trl training stack; bitsandbytes
is omitted as it is a CUDA-only dependency with poor Apple Silicon support.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* fix(trl): guard uv-only --index-strategy for the pip/darwin path

The darwin/MPS build installs with pip (USE_PIP=true), which rejects the
uv-only --index-strategy flag and failed the darwin backend build. Add it
only on the uv path; Linux/CUDA resolution is unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-25 08:09:36 +02:00
LocalAI [bot]
62b14fd635 feat(backends): add darwin/metal build for liquid-audio (#10486) 
* feat(backends): add darwin/metal build for liquid-audio

Wire the already-MPS-ready liquid-audio backend (it ships
requirements-mps.txt) into the darwin CI matrix and the gallery so
metal-darwin-arm64 images are built and selectable.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* ci(liquid-audio): trigger darwin build via requirements-mps note

The changed-backends path filter only builds a backend when a file under
its directory changes. The metal wiring lived in index.yaml + the matrix,
so the darwin job was skipped. Add a documenting comment to the MPS
requirements so CI actually exercises the darwin build.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

* fix(liquid-audio): guard uv-only --index-strategy for the pip/darwin path

Same fix as trl: the darwin/MPS build installs with pip (USE_PIP=true), which
rejects the uv-only --index-strategy flag and failed the darwin backend build.
Add it only on the uv path; Linux/CUDA resolution is unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-24 23:16:27 +02:00
LocalAI [bot]
193d0e6aef fix(backends): darwin/metal support for supertonic (#10488) 
The supertonic Go TTS backend dlopens ONNX Runtime, but its runtime and
packaging scripts were Linux-only: run.sh exported LD_LIBRARY_PATH, pointed
ONNXRUNTIME_LIB_PATH at libonnxruntime.so, and always tried the ld.so exec
path, while package.sh hard-failed on any non-Linux host. On macOS dyld has
no ld.so loader, uses DYLD_LIBRARY_PATH, and ONNX Runtime ships as a .dylib.

This applies the same purego .dylib/DYLD_LIBRARY_PATH fix that PR #10481
landed for 15 other ONNX/purego backends (sherpa-onnx, silero-vad, etc.) but
which omitted supertonic:

- run.sh: on darwin export DYLD_LIBRARY_PATH and point ONNXRUNTIME_LIB_PATH
  at libonnxruntime.dylib; guard the ld.so exec path to Linux only.
- package.sh: recognize Darwin instead of erroring out; the bundled .dylib is
  resolved via DYLD_LIBRARY_PATH, no glibc/ld.so to bundle.
- helper.go: platform-native default library extension (dylib on darwin) for
  the last-resort dlopen fallback.

It also wires the darwin CI build and gallery entries, resolving the
inconsistency where backend/index.yaml advertised metal for supertonic but no
includeDarwin matrix entry built the image:

- .github/backend-matrix.yml: add the -metal-darwin-arm64-supertonic Go entry.
- backend/index.yaml: declare metal capabilities and add the concrete
  metal-supertonic / metal-supertonic-development child entries.

The Makefile already detects Darwin/osx/arm64 and stages the per-OS ONNX
Runtime tarball, mirroring sherpa-onnx, so no Makefile change is required.


Assisted-by: Claude:opus-4.8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-24 22:19:03 +02:00
dependabot[bot]
10184b5e28 chore(deps): bump actions/checkout from 6 to 7 (#10451) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-22 21:38:37 +02:00
Richard Palethorpe
63bcbf6c12 fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier (#10401) 
* fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier

Follow-up to the NER tier engine (#10360), already on master. This carries
only the incremental review fixes and tests that postdate that merge — the
feature itself is not re-introduced.

Review fixes:
- openai_completion.go: remove the dead `elem >= 0` conjunct in applyAnyText
  (the `elem < 0` guard above already returns).
- application.go: collapse ResolvePIIPolicy's inline re-implementation of
  PIIIsEnabled to a single cfg.PIIIsEnabled() call (sole source of the
  "explicit pii.enabled wins, else cloud-proxy default" rule) and return true
  past the !enabled guard where it is provable.
- pattern.go: hoist the triple `appConfig != nil && EnableTracing` check in
  patternDetector.Detect into one local.
- grammar.go: MaxQuantifier was 4096, but Go's regexp/syntax rejects repeat
  bounds above 1000 at Parse time, so walk()'s {n,m} guard could never fire —
  dead code shadowed by the parser. Lower it to 512 so a bound in (512,1000]
  is rejected here with an actionable error; >1000 still fails closed via
  Parse. Specs pin the relationship so the guard can't silently revert.
- PatternListEditor.jsx: clamp a directly-typed negative min_len to >=0 and
  force the DOM value back when clamping (min={0} only constrained the spinner,
  so a negative reached saved config and silently disabled the length filter).

Tests:
- piipattern_test.go: MaxQuantifier guard specs (must stay live, not dead).
- model-config.spec.js: assert the min_len clamp, and that entity_actions
  collapses a duplicate group to a single row (map semantics; regression guard
  against emitting an array that drops a row on save).
- tests/e2e-backends: token_classify capability driving the TokenClassify gRPC
  RPC against the backend image, asserting byte-correct, UTF-8 rune-aligned
  spans (entity.Text == text[start:end]) at threshold 0. Verified on CPU via
  `make test-extra-backend-privacy-filter` (3/3 specs).
- Makefile: test-extra-backend-privacy-filter wrapper.
- tests/e2e: e2e_pii_ner_test.go drives /api/pii/analyze + /api/pii/redact
  (mask + block) through the full HTTP -> detector -> redactor path; gated on
  PII_NER_MODEL_GGUF so the default suite is unaffected.
- .github/workflows/tests-pii-ner-e2e.yml: path-filtered / nightly CI job
  running the container harness on CPU.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(gallery): add privacy-filter-nemotron (f16 + q8)

GGUF conversions of OpenMed/privacy-filter-nemotron — a fine-grained English
PII token-classifier (55 categories / 221 BIOES classes), fine-tuned from
openai/privacy-filter on NVIDIA's Nemotron-PII dataset. Sibling to the existing
privacy-filter-multilingual entry, trading language breadth for category depth.

- privacy-filter-nemotron: F16 reference artifact (~2.8 GB).
- privacy-filter-nemotron-q8: Q8_0 quant (~1.64 GB) for RAM-constrained / edge
  use; description notes the size/speed tradeoff and to validate on your own
  data (a single dropped span is a PII leak).

Both run on the privacy-filter backend with known_usecases [token_classify] and
a default mask policy (min_score 0.5); operators add per-category entity_actions
as needed. sha256s taken from the HF repo's LFS object ids.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-06-22 18:26:19 +02:00
LocalAI [bot]
600dafd20b feat(ced): sound-event classification backend (CED audio tagger) (#10425) 
* feat(ced): sketch sound-classification backend (CED audio tagger)

Wires ced.cpp (CED, 527-class AudioSet sound-event tagger; baby cry,
footsteps, glass, alarms, dog bark) into LocalAI as a Go/purego backend.

SKETCH (backend skeleton real; core REST wiring + CI/gallery is a checklist
in DESIGN.md):
- backend/backend.proto: new SoundDetection rpc + SoundClass messages
  (run `make protogen-go` to regenerate pkg/grpc/proto).
- backend/go/ced: main.go (purego dlopen libced.so + ced_capi.h),
  goced.go (Ced gRPC backend: Load + SoundDetection), Makefile
  (clone-at-pin CED_VERSION, ggml static-PIC shared build), run.sh,
  package.sh, .gitignore.
- DESIGN.md: REST /v1/audio/classification wiring (handler/route/capability
  registration checklist), gallery/index + CI registration, and a scoping
  note for the realtime/websocket live-recognition path (sliding-window
  classify over the existing ws transport + voicegate; the ced C-API
  per-PCM entry point is already window-friendly).

Backend code does not compile until protogen-go regenerates the pb types
and a libced.so is built (Makefile clones+builds it).

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): REST /v1/audio/classification endpoint + capability registration

Wires the ced sound-event classification backend (AudioSet audio tagger)
end to end through the REST surface, mirroring the transcription path.

- Handler: core/http/endpoints/openai/sound_classification.go parses the
  multipart audio upload, temp-files it, resolves the model config and
  calls the SoundDetection RPC; returns {model, detections[]} JSON.
- Backend wrapper: core/backend/sound_classification.go (ModelSoundDetection)
  loads the model and normalizes the proto response into schema types.
- Schema: core/schema/sound_classification.go (SoundClassificationResult).
- gRPC layer: SoundDetection wired through the LocalAI wrapper (interface,
  Backend client, Client, embed, server, base default) so the loader-typed
  client exposes the RPC; proto regenerated via make protogen-go.
- Route: POST /v1/audio/classification (+ /audio/classification alias) with
  the audio/multipart default-model middleware in routes/openai.go.
- Capability surfaces: swagger @Tags/@Router on the handler; FLAG_SOUND_
  CLASSIFICATION usecase flag + UsecaseSoundClassification + UsecaseInfoMap +
  GuessUsecases + ModalityGroups + GetAllModelConfigUsecases; meta usecase
  option; /api/instructions audio area updated; auth RouteFeatureRegistry +
  FeatureAudioClassification (APIFeatures, default ON) + FeatureMetas; UI
  usecaseFilters, capabilities.js CAP_SOUND_CLASSIFICATION, Models.jsx filter
  + i18n; docs page features/audio-classification.md + whats-new + crosslink.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): realtime sound-event detection over the websocket API

When a realtime pipeline configures a sound-classification model, each
VAD-committed utterance (the same window the transcription path produces)
is also run through the CED sound-event classifier and the scored AudioSet
tags are emitted as a new server event. No new backend rpc is needed: the
SoundDetection gRPC method already exists on this branch.

- config: add Pipeline.SoundDetection (yaml/json sound_detection,omitempty)
  beside Transcription/VAD.
- realtime: add Model.SoundDetection(ctx, audio, topK, threshold) to the
  ModelInterface; implement it on wrappedModel and transcriptOnlyModel by
  calling backend.ModelSoundDetection with the session's sound-classification
  model config (mirrors how Transcribe dispatches). Load the optional config
  in newModel / newTranscriptionOnlyModel; nil config keeps it additive.
- types: add ConversationItemSoundDetectionEvent (item_id, content_index,
  detections[]{label,score,index}) with type conversation.item.sound_detection,
  its ServerEventType constant and MarshalJSON, mirroring the transcription
  completed event.
- realtime: add emitSoundDetection (unary path: classify the committed window,
  build the event, t.SendEvent) and wire it at the utterance-commit hook right
  after emitTranscription; gated on session.SoundDetectionEnabled (resolved
  from Pipeline.SoundDetection at session setup, defaults top_k=5, threshold=0).
  Its error is logged via xlog but never aborts the turn.
- test: Ginkgo specs for emitSoundDetection (tags emitted, empty detections,
  classifier error) plus a SoundDetection method on the fakeModel double.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ced): implement SoundDetection in nodes backend test doubles

The SoundDetection method added to the grpc backend interface left two
test doubles (fakeBackendClient, fakeGRPCBackend) incomplete, so
core/services/nodes failed to compile under `go vet`/`go test` (go build
missed it: the doubles live in _test.go). Add the method to both,
mirroring their existing Detect mock. Repairs CI for the nodes package.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): decouple realtime sound detection from VAD (sound-only sessions)

Sound-event detection must activate on sounds, not speech, so it no longer
runs through the voice VAD/transcription path. A sound-detection-only
pipeline (sound_detection set, no transcription/LLM) now:

- is accepted by prepareRealtimeConfig (sound_detection counts as a pipeline
  stage),
- builds a lightweight model via newSoundDetectionOnlyModel (no VAD/STT/LLM/TTS
  loaded), and
- defaults the session to turn_detection none (no VAD) with no transcription
  stage, so the client drives windowing via input_audio_buffer.commit
  (option A: client-side sliding window). The per-PCM C-API already supports
  arbitrary windows.

commitUtterance gains a sound-only branch: it emits the
conversation.item.sound_detection event (scored AudioSet tags) and stops -
no transcription, no LLM response. generateResponse is now guarded on a
transcription stage being present, so a sound-only turn never invokes the LLM.

Existing transcription/VAD sessions are unchanged (additive). Added a
commitUtterance sound-only Ginkgo spec asserting it emits the sound event and
neither transcribes nor generates a response. go vet + golangci-lint
(new-from-merge-base) clean; openai suite green.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): register sound-classification backend in gallery + CI

Mechanical backend-image registration for the ced sound-event classifier,
mirroring the parakeet-cpp Go/purego backend everywhere it is wired up.

- .github/backend-matrix.yml: add the ced build matrix, field-for-field copies
  of the parakeet-cpp entries (cpu amd64/arm64, cublas cuda 12/13 amd64,
  l4t cuda-13 arm64, l4t-jetpack cuda-12 arm64, sycl f32/f16, vulkan
  amd64/arm64, rocm hipblas, and the metal darwin entry), changing only
  backend and tag-suffix. dockerfile stays ./backend/Dockerfile.golang.
- backend/index.yaml: add the &ced meta anchor (capabilities map per platform)
  plus ced-development and the per-arch image entries, each uri/mirror
  tag-suffix matching the matrix exactly. The model gallery (GGUF) entry is
  intentionally deferred pending the HuggingFace publish (TODO note inline).
- scripts/changed-backends.js: add an explicit item.backend === "ced" branch in
  inferBackendPath mapping to backend/go/ced/, same mechanism and ordering as
  the parakeet-cpp branch (before the generic golang fallthrough).
- .github/workflows/bump_deps.yaml: register mudler/ced.cpp -> CED_VERSION in
  backend/go/ced/Makefile so the daily bot bumps the pin.
- swagger/{docs.go,swagger.json,swagger.yaml}: regenerated via make swagger so
  the existing /v1/audio/classification annotations land in the generated spec.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): server-side windowing for realtime sound detection (option B)

Adds an optional server-driven sliding-window classifier so a sound-only
realtime client only has to stream audio (no input_audio_buffer.commit):

- Pipeline.sound_detection_window_ms / sound_detection_hop_ms config knobs.
  When both > 0 on a sound-only session, the server classifies the last
  window of streamed audio every hop and emits a conversation.item.sound_
  detection event; the input buffer is trimmed to one window so a long
  stream stays bounded. When unset, the session stays client-driven
  (option A). Runs independent of VAD (sound events are not speech).
- handleSoundWindow (ticker) + classifySoundWindow (one tick, extracted so
  it is unit-testable) + writeWindowWAV, which declares the true
  InputSampleRate (NewWAVHeaderWithRate) so the classifier resamples
  correctly. Goroutine is started after toggleVAD and torn down with the
  session (close + wg.Wait).
- Register pipeline.sound_detection (+window_ms/hop_ms) in the config meta
  registry; the earlier realtime commit added pipeline.sound_detection
  without a registry entry, failing TestAllFieldsHaveRegistryEntries. This
  fixes that and covers the two new knobs.

Tests: classifySoundWindow emits an event + trims the buffer to one window,
no-ops on too-little audio; writeWindowWAV declares the given sample rate.
go build/vet + golangci-lint (new-from-merge-base) clean; config + openai
suites green.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): add ced-base GGUF model gallery entries (f16 + q8_0)

The ced-base weights are now published at mudler/ced-base-gguf (Apache-2.0,
converted from mispeech/ced-base). Adds gallery/ced.yaml (backend: ced +
known_usecases: sound_classification) and two gallery/index.yaml entries
(ced-base-f16 default, ced-base-q8 smallest) with sha256-pinned files, and
removes the now-resolved TODO from backend/index.yaml.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(ced): add tiny/mini/small GGUF model gallery entries

Publishes the rest of the CED family (same architecture, metadata-driven port
verified end-to-end on ced-tiny) to mudler/ced-{tiny,mini,small}-gguf and adds
their f16 + q8_0 gallery entries:

  ced-tiny  (5.5M, edge/Pi-class)  f16 11MB / q8_0 6MB
  ced-mini  (9.6M)                 f16 19MB / q8_0 11MB
  ced-small (22M)                  f16 42MB / q8_0 23MB

All sha256-pinned. ced-base remains the accuracy default.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): point gallery entries at the consolidated mudler/ced-gguf repo

All CED quantizations (tiny/mini/small/base, f16/q8_0) now live in a single
HuggingFace repo, mudler/ced-gguf, instead of per-model repos. Repoint the 8
gallery model entries' urls + file uris accordingly. sha256 and filenames are
unchanged.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): bump CED_VERSION to the short-clip fix

Pin the ced backend to ced.cpp 99c6ed3, which fixes a crash on any clip
shorter than target_length (~10.11s): time_pos_embed was added at its full
63-frame grid instead of being sliced to the clip's actual time grid, tripping
ggml_can_repeat in ggml_add. Surfaced by the live realtime e2e (sub-10s
windows) and gated with a short-clip parity test upstream.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(ced): list ced.cpp as a LocalAI-team engine + backend-guide directive

- README.md: add ced.cpp to the "native C/C++/GGML engines developed and
  maintained by the LocalAI project" table.
- docs/content/features/backends.md: add a Sound Classification backend
  category (sound-event classification / audio tagging) listing ced.cpp.
- .agents/adding-backends.md: add a "Documenting the backend" section and two
  verification-checklist items requiring new backends to be documented in the
  backends.md category list, and in-house native engines to be added to the
  README maintained-engines table. This directive was missing.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(ced): repin CED_VERSION to the v0.1.0 release commit

ced.cpp history was squashed into a single release commit (tagged v0.1.0), so
the previous pin (99c6ed3) no longer exists upstream. Pin to c04ac14, the
v0.1.0 release commit, so the backend builds against a commit that exists.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(ced): silence gosec G304/G103 + govet unsafeptr on audited paths

- sound_classification.go: os.Create(dst) where dst = temp dir + path.Base of
  the upload (no traversal). #nosec G304, matching the depth-anything-cpp handler.
- goced.go: reading a NUL-terminated C string from a libced-owned buffer.
  #nosec G103 (gosec) + //nolint:govet (golangci-lint's unsafeptr check), since
  the uintptr is a C-owned malloc'd buffer, not Go-GC memory.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-22 01:00:28 +02:00
Richard Palethorpe
3fa7b2955c feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360) 
Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see
backup/pii-ner-tier-engine-prerebase). Net change:

- privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter
  PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan).
  TokenClassify moves off the patched llama.cpp path onto this backend.
- PII filter reworked to be NER-centric (encoder/NER detection tier scanning
  whole conversations as one document), with a recreated bounded restricted-
  regex secret-matching pattern detector tier alongside it (per-model
  pii_detection.builtins / .patterns + core/services/routing/piipattern).
- Detection labelled by source (ner vs pattern); backend trace / confidence /
  debug observability; analyze/redact exposed as a synchronous API.
- Instance-wide default detector policy + per-usecase default-on; request
  filtering extended to completions, embeddings, edits & Ollama.
- React UI: NER-centric PII editor, detector-models table, pattern/builtins
  editor, middleware default-policy UI.
- Gallery: privacy-filter-multilingual token-classify model + NER install
  filter; token_classify known_usecase; batch sized to context for NER models.
  privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13
  meta + image entries with a capabilities map) matching its CI matrix jobs,
  and an /import-model auto-detect importer (PrivacyFilterImporter, narrow
  privacy-filter GGUF detection) replacing the prior pref-only registration.

Reconciled against master's independent evolution:

- Dropped master's PIIPatternOverrides feature (global-pattern runtime
  overrides + /api/pii/patterns API + runtime_settings.json persistence). The
  per-model NER + pattern-detector design supersedes it; it was built on the
  global redactor pattern set this branch replaced.
- Reverted the llama.cpp Score carry-patch (0006-server-task-type-score):
  removed the patch and restored master's grpc-server.cpp Score RPC (direct
  llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's
  model_config validation forbidding score + chat/completion/embeddings on
  llama-cpp. token_classify is unaffected (it runs on the privacy-filter
  backend, not llama-cpp).

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-06-18 11:45:22 +01:00
LocalAI [bot]
294170d3ed feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery (#10352) 
* feat(backend): add depth-anything (Depth Anything 3) C++/ggml backend + gallery

Mirrors the locate-anything-cpp backend to register a new depth-anything
backend that wraps the Depth Anything 3 ggml port (depth-anything.cpp) via
purego (cgo-less, no Python at inference).

- backend/go/depth-anything-cpp/: gRPC backend (Load + Predict + GenerateImage),
  purego binding to the da_capi_* C ABI, CMake/Makefile/run/package/test scripts
  building depth-anything.cpp's DA_SHARED static .so per CPU variant.
- backend/index.yaml: depth-anything backend meta + all hardware-variant
  capability entries (cpu/cuda12/cuda13/intel-sycl-f32+f16/vulkan/nvidia-l4t).
- gallery/index.yaml: 8 Depth Anything 3 GGUF models (base q4_k/q8_0/f16/f32,
  small, large, giant, mono-large).
- .github/backend-matrix.yml: one build entry per hardware variant.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(depth): typed Depth RPC + REST endpoint exposing full DA3 data

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(depth): pin depth-anything.cpp to e0b6814 (ABI 3 dense C-API)

The Depth RPC handler calls da_capi_depth_dense / da_capi_points (C-API ABI 3);
pin the native build to the commit that exports them.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(depth): pin depth-anything.cpp to v0.1.0 release (b515c31)

Repoint the native version from the now-orphaned e0b6814 to the
b515c31 release commit, kept alive by the upstream v0.1.0 tag.
C-API is unchanged (da_capi_abi_version == 3).

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(depth): wire depth-anything-cpp into build, CI bump, and importer

The backend dir, gallery index, and CI build-matrix were present but the
backend was never wired into the integration points that adding-backends.md
requires:

- root Makefile: add to .NOTPARALLEL, the test-extra chain, a BACKEND_*
  definition, the docker-build target eval, and docker-build-backends
  (mirrors parakeet-cpp; the backend's own Makefile already documented that
  its `test` target is driven by test-extra).
- bump_deps.yaml: register the DEPTHANYTHING_VERSION pin so the daily
  auto-bump bot tracks mudler/depth-anything.cpp master (it cannot see an
  unregistered Makefile pin).
- import form: add a preference-only KnownBackend entry so depth-anything is
  selectable at /import-model (mirrors sam3-cpp; no reliable GGUF auto-detect
  signal, so pref-only per the doc's default).

changed-backends.js needs no entry: the generic golang suffix branch already
resolves backend/go/depth-anything-cpp/.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(depth): auto-detect importer for depth-anything GGUFs

Replace the preference-only entry with a real auto-detect importer
(mirrors parakeet-cpp / locate-anything):

- DepthAnythingImporter matches a .gguf whose name carries a
  depth-anything token (depth-anything-<size>-<quant>.gguf), so
  /import-model recognises mudler/depth-anything.cpp-gguf repos and direct
  GGUF URLs without an explicit backend preference. preferences.backend=
  "depth-anything" still forces it.
- Registered before LlamaCPPImporter so its GGUF bundles aren't claimed by
  the generic .gguf importer; the narrow name match means it cannot claim
  arbitrary llama GGUFs or the upstream safetensors PyTorch repos.
- Multi-quant repos pick the smallest quant by default (q4_k -> ... -> f32,
  depth stays >0.998 corr even at q4_k); quantizations preference overrides.
- Drops the now-redundant knownPrefOnlyBackends entry (importer-backed
  backends are not listed there, matching parakeet-cpp).
- Table-driven Ginkgo test covers detection, negative cases (llama GGUF,
  upstream safetensors), default/override/fallback quant pick, and direct
  URL import. 10/10 specs pass.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(depth): check conn.Close error in grpc Depth client (errcheck)

The new Depth() client method used a bare `defer conn.Close()`. golangci-lint
runs with new-from-merge-base, so although the 39 sibling methods use the same
bare form (grandfathered), the newly added line trips errcheck. Drop the result
explicitly to satisfy the linter.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8

* fix(depth): bump depth-anything.cpp to v0.1.1 (embeddable CMake)

v0.1.0 (b515c31) used ${CMAKE_SOURCE_DIR} for its include dirs, which
points at the parent project when built via add_subdirectory() as this
backend does, so the container build failed with missing stb_image.h /
da_gguf_keys.h. v0.1.1 (2d42897) switches to project-relative paths.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8

* fix(depth): resolve gosec findings in the backend wrapper

The code-scanning gate flagged three new failure-level alerts in
godepthanythingcpp.go (gosec runs with -no-fail; GitHub gates on new alerts):

- G301: export dirs were created with 0o755. Tighten to 0o750 (no world
  access needed for backend-written export output).
- G304: writeDepthPNG creates req.GetDst(). That path is chosen by the
  LocalAI core as the intended output destination (same pattern every
  image backend uses), not attacker input, so annotate with #nosec G304
  and document why.

The remaining G103 "audit unsafe" notes on the unsafe.Slice C-buffer copies
are warning-level (the same purego interop whisper/parakeet use) and do not
gate the check, per the supertonic exclusion precedent in secscan.yaml.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8

* fix(depth): bump depth-anything.cpp to v0.1.2 (CUDA cross-build arch)

v0.1.1 forced CMAKE_CUDA_ARCHITECTURES=native, which breaks the GPU-less
l4t/cublas CI builds (nvcc "Unsupported gpu architecture 'compute_'" on
CMake 3.22). v0.1.2 (442eea4) drops the override and lets ggml pick its
default cross-build arch list.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-16 16:28:28 +02:00
LocalAI [bot]
6b9f1bd4b3 chore: ⬆️ Update antirez/ds4 to e34a8086693ba7ca5cfabd2b9028ee52f0bfac2e (#10350) 
* ⬆️ Update antirez/ds4

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix(ds4): add Homebrew include/lib prefix for Darwin grpc-proto build

The darwin/metal ds4 backend job runs for the first time on this bump
(it was skipped on prior ds4 PRs) and fails compiling backend.pb.cc with
'google/protobuf/runtime_version.h' file not found.

hw_grpc_proto links neither protobuf::libprotobuf nor gRPC::grpc++, so
the generated proto sources rely on default system include paths. That
works on Linux (/usr/include) but not on macOS, where Homebrew installs
under /opt/homebrew. Add the Homebrew prefix to include/link dirs on
Darwin, mirroring the llama-cpp backend that already builds on Darwin CI.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* fix(ds4): install nlohmann-json on Darwin CI for ds4 backend

After the protobuf include-path fix the ds4 darwin build advances to
compiling dsml_renderer.cpp, which includes <nlohmann/json.hpp> and
#errors when absent. On Linux the header comes from apt nlohmann-json3-dev
in the build image; the macOS runner had no equivalent. Add the
header-only nlohmann-json formula to the shared Darwin backend brew
install/link list and Homebrew cache, alongside the existing deps.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* fix(ds4): build proper OCI image tar for Darwin backend

The darwin packaging referenced scripts/build/oci-pack.sh, which was
never added to the tree, so it fell back to a plain 'tar' that omits
manifest.json. 'local-ai backends install' then rejects the tarball
with 'file manifest.json not found in tar'.

Use './local-ai util create-oci-image' (already built by the 'build'
prerequisite of the backends/ds4-darwin target), mirroring
llama-cpp-darwin.sh, to emit a real OCI image the installer accepts.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

---------

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-16 09:59:50 +02:00
LocalAI [bot]
40cc549882 fix(ci): track ServeurpersoCom/qwentts.cpp for QWEN3TTS_CPP_VERSION bumps (#10356) 
The qwen3-tts backend migrated from predict-woo/qwen3-tts.cpp to
ServeurpersoCom/qwentts.cpp (the Makefile QWEN3TTS_REPO already points
there), but the bump_deps matrix still tracked the old repo. That made
the nightly bumper open PRs (e.g. #10334) against the wrong upstream.
Point the matrix entry at the new repo and its master branch.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-16 09:04:52 +02:00
LocalAI [bot]
2df2876db2 feat(supertonic): add Supertonic ONNX TTS backend (CPU) (#10342) 
* feat(supertonic): vendor upstream Go TTS pipeline (helper.go)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(supertonic): add gRPC backend (Load/TTS/TTSStream, CPU)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(supertonic): satisfy unused linter (use onnxProvider; exclude vendored helper.go)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(supertonic): unit tests for resolvers + gated end-to-end synthesis

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* style(supertonic): gofmt backend.go comment block

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(supertonic): add Makefile, run.sh, package.sh (CPU build)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* build(supertonic): wire backend into root Makefile

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(supertonic): check ort.DestroyEnvironment return (errcheck)

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(supertonic): resolve voice_styles as sibling of onnx dir; guard trim; test voice

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(supertonic): add CPU build matrix + gallery index entries

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(supertonic): expose as pref-only importable backend

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(supertonic): add Supertonic/supertonic-3 TTS model to the gallery

16 files (4 onnx + tts.json + unicode_indexer.json + 10 voice styles)
from HF Supertone/supertonic-3, served via the supertonic backend.
Defaults to voice F1; onnx/ + sibling voice_styles/ layout matches the
backend's resolveVoicesDir.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(meta): register pipeline.max_history_items config field

Pre-existing on master: the field was added without a registry entry,
failing TestAllFieldsHaveRegistryEntries (core/config/meta). Add the
entry so it renders properly in the model-config UI.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(secscan): exclude vendored supertonic backend from gosec

helper.go is vendored from supertone-inc/supertonic; its G304/G404/G104
findings are inherent to upstream and the math/rand use is correct for
flow-matching noise (crypto/rand would be wrong).

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-15 16:54:11 +02:00
LocalAI [bot]
0854932a25 feat(omnivoice-cpp): add OmniVoice TTS backend (file + streaming, voice cloning + voice design) (#10310) 
* feat(omnivoice-cpp): add C wrapper + CMake/Makefile build over OmniVoice ov_* ABI

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(omnivoice-cpp): add option/language parsing + WAV framing helpers with tests

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(omnivoice-cpp): wire purego binding with TTS + streaming TTSStream

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* build(omnivoice-cpp): wire backend into root Makefile

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(omnivoice-cpp): add build matrix entries + dep-bump registration

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(omnivoice-cpp): register backend meta + image entries

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(omnivoice-cpp): expose as preference-only importable backend

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add omnivoice-cpp TTS models (Q8_0 default + BF16 HQ)

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(omnivoice-cpp): document the OmniVoice TTS backend

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(omnivoice-cpp): add env-gated e2e for TTS + streaming

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(omnivoice-cpp): honor tts.audio_path/tts.voice config as default cloning reference

The model config tts.audio_path (ModelOptions.AudioPath) and tts.voice now
provide a default voice-cloning reference used when a request omits Voice, so a
cloned voice can be pinned in the model YAML instead of passed per request. A
per-request voice still overrides. Paths resolve relative to the model dir.

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(omnivoice-cpp): add missing omnivoice-cpp-development backend meta

Mirrors the whisper/vibevoice convention: a -development meta aggregating the
master-tagged image variants (the production meta and per-variant prod+dev image
entries already existed; only the development meta aggregator was missing).

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-13 21:28:46 +02:00
LocalAI [bot]
d7162b9f89 ci(darwin): build the ds4 backend for darwin/arm64 (metal) (#10303) 
The gallery has metal-ds4 / metal-ds4-development entries, and the build
recipe exists (make backends/ds4-darwin, special-cased in
backend_build_darwin.yml), but ds4 was never listed in the darwin matrix,
so no metal-darwin-arm64-ds4 image was ever published and the entries
dangled.

- Add ds4 to the darwin matrix (includeDarwin), mirroring the llama-cpp
  form (the reusable workflow builds it via 'make backends/ds4-darwin').
- Fix inferBackendPathDarwin in scripts/changed-backends.js to map ds4 to
  backend/cpp/ds4/ (like llama-cpp): ds4 is C++ but the matrix entry carries
  lang=go, so without this its darwin build would only ever run on a release
  (FORCE_ALL), never incrementally when backend/cpp/ds4 changes.

sherpa-onnx and speaker-recognition are already in the darwin matrix on
master and are not changed here.

Assisted-by: claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-13 11:02:32 +02:00
LocalAI [bot]
60facc7252 fix(darwin): publish sherpa-onnx and speaker-recognition images for darwin/arm64 (#10275) 
Neither the sherpa-onnx nor the speaker-recognition backend had a
darwin/arm64 image, so `local-ai backends install` failed with "no child
with platform darwin/arm64" on macOS. This left /v1/audio/diarization (the
sherpa-onnx path) and /v1/voice/embed without any usable backend on Apple
Silicon.

Both backends build on darwin/arm64:
- sherpa-onnx (Go) already fetches the onnxruntime osx-arm64 runtime in its
  Makefile; it only needed a darwin matrix entry (build-type metal, lang go,
  like whisper and silero-vad).
- speaker-recognition (Python) needed a requirements-mps.txt so the mps build
  installs plain onnxruntime (which ships a macOS arm64 wheel) instead of the
  onnxruntime-gpu pulled by its base requirements (which does not).

Add both to the includeDarwin build matrix, wire the metal capability and
metal image aliases into the gallery, and add the speaker-recognition
requirements-mps.txt.

Fixes #10268


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-06-12 22:32:42 +02:00
Richard Palethorpe
085fc53bbc fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104) 
* fix(router): score classifier production-readiness

Conversation trimming runs through the classifier model's chat template
and trims by exact token count, sized to the model's n_batch which is
now scaled to context so long probes can't crash the backend. Missing
chat_message templates are a hard error at router build time. Router-
facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve
ModelConfig per call so a model installed post-startup doesn't bind a
stub Backend="" config and silently fall into the loader's auto-
iterate path.

New 'vector_store' backend trace recorded inside localVectorStore on
every Search/Insert — including the backend-load-failure path that
previously vanished into an xlog.Warn — with outcome tagging
(hit/miss/empty_store/backend_load_error/find_error/insert_error/ok).
Companion cleanup drops misleading similarity:0 and input_tokens_count:0
from non-hit and text-mode traces.

Gallery local-store-development aliases to 'local-store' so the master
image satisfies pkg/model.LocalStoreBackend lookups from the embedding
cache.

Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key
(the original bug); ModelTokenize nil-guard; non-fatal mitm proxy
startup; PII 'route_local' renamed to 'allow' with docs/UI in sync;
model-editor footer no longer eats the edit area on small screens;
several config-editor template/dropdown/section fixes.

Tests: e2e router specs (casual/code-hint + long-conversation trim),
vector_store trace specs, lazy-factory specs, gallery dev-alias
resolution, Playwright trace badge + scroll regression.

Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(backend): auto-size batch to context for embedding and rerank models

Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins.

Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse.

Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch.

Assisted-by: claude-code:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(gallery): raise arch-router scoring output cap via parallel:64

Scoring decodes the whole prompt+candidate in a single llama_decode and
reads one logit row per candidate token. The vendored llama.cpp server
caps causal output rows at n_parallel, so the default of 1 aborts with
GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route
labels. Set options: [parallel:64] on both arch-router quant entries to
lift the cap; kv_unified (the grpc-server default) keeps the full context
per sequence, so this does not split the KV cache.

Assisted-by: claude-code:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-06-12 16:21:15 +02:00