* feat(sound): expand simple prompts with chat models
Allow callers to select an installed chat model for sound prompt
expansion. Validate its JSON before audio generation and preserve
explicit instrumental and language controls. Keep backend defaults
when expansion is omitted, including native Python ACE-Step sampling.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(sound): map expansion by loaded audio family
Map neutral expanded fields where the resolved audio family is known.
Keep unmarked requests unchanged and reject unsupported expansion before
inference. Clear YuE2 and HeartMuLa transcripts to prevent lyric fallback.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(sound): choose a prompt expansion model
Require an explicit Simple choice and remember it separately from chat.
Clear unavailable choices instead of silently switching language models.
Preserve the choice in history and restore it when editing a past result.
Keep Backend default and Advanced requests compatible with existing APIs.
Document family support and align the published optional request schema.
Verify selection, request fields, persistence and history in the browser.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(sound): authorize both generation models
Check the resolved audio model and explicit expansion model before
loading either. Permission lookup failures stop generation.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(sound): enforce JSON during prompt expansion
Constrain expansion output with the existing JSON grammar before decoding.
Require a llama-cpp chat model so unsupported backends cannot ignore it.
Keep strict field validation and fail before generating audio.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pocket TTS ships one model per language, but LoadModel called
TTSModel.load_model() with no arguments, so the library always fell back
to its English model whatever the model config said. Read a `language`
option and pass it to load_model, keeping the library default when it is
not set. Document the option and add unit tests that mock the loader.
Fixes#12759
Assisted-by: Claude:claude-opus-5-5
* feat(tts): stream Qwen text through audio pipelines
Feed cleaned LLM deltas into one persistent Qwen synthesis session.
Return audio before input ends and preserve the live RPC through model
cache, staging and in-flight wrappers. Stop generation on sink failures
or cancellation without replaying speech through a fallback.
Pin the incremental native fork and verify early audio, late text,
finalization and cancellation with lifecycle and real-model checks.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(tts): use the organization Qwen fork
Fetch the reviewed incremental synthesis commit from localai-org so
backend builds use the organization-maintained fork.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add a Qwen streaming assistant preset
Expose an English pipeline using existing VAD, ASR, LLM and named-voice
TTS models. Enable incremental text and audio delivery, and document
separate component installs because the gallery has no dependency
installer.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): use Parakeet TDT v3 for the assistant
Select the existing Parakeet TDT 0.6B v3 transcription entry and update
its prerequisite install command. Keep the preset's English voice and
incremental Qwen synthesis configuration.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(kokoros): reject unsupported backend RPCs
Implement the MotionStream and SlotCache methods required by the shared
protobuf service. Return Unimplemented so Kokoros compiles without
advertising support for either operation.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): mark cleanup and logging results ignored
Explicitly ignore connection shutdown, test server shutdown and
Ginkgo diagnostic writer errors. These operations do not replace the
stream's inference or sink errors, and errcheck requires their return
values to be acknowledged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream buffered clauses in realtime pipelines
Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(tts): bound synthetic voice clone generation
The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): require explicit clause buffering opt-in
Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add VITS and Piper streaming assistants
Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
stable-diffusion.cpp PR #2148 added MOSS-Image support, which needs an
external tokenizer file passed via the --tokenizer CLI arg. The C++ binding
(gosd.cpp) never parsed a tokenizer option nor set ctx_params.tokenizer, so
MOSS-Image could not load through LocalAI.
- Wire the "tokenizer" option in gosd.cpp (parse + assign to ctx_params)
- Resolve "tokenizer:..." paths against the model directory in gosd.go,
the same way *_path options are resolved
- Bump the vendored stable-diffusion.cpp to 68285df (moss-image merge)
- Add moss-image-base and moss-image-sft gallery entries (int8_convrot)
FLUX.3 Image Klein is already in the gallery (added in #12736).
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): add lazy_mode model option
Expose upstream --lazy-mode (on, auto, off) as a model option, for
example `lazy_mode:on`. It controls on-demand reading of large tensors
such as per-layer embeddings; `on` requires mmap.
The option is also accepted as `lazy-mode` or `lzm`. Invalid values log
a warning and keep the upstream default (auto). The pinned llama.cpp
already provides the lazy_mode field in common_params.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump llama.cpp to c0fd78626405
Update the default llama.cpp pin to the current upstream master. The
changes since the old pin that the backend touches are additive: a new
common_decision_type value, new mtmd audio-stream and download helper
declarations. The two server patches still apply in order, and the
common_params fields and server headers the backend uses are unchanged.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(llama-cpp): add slot prompt-cache save and restore API
Add POST /v1/slots/save and POST /v1/slots/restore. They write one
llama.cpp slot's prompt cache (KV state) to a file and load it back,
so a client can warm a long system prompt in a few hundred
milliseconds instead of re-running prefill. Refs #12685.
The new SlotCache gRPC call posts the same slot save and restore
tasks the llama-server /slots endpoint uses. It is off unless the
model sets the new slot_save_path option, so existing models are not
affected. The /slots monitoring endpoint is not needed here: the
backend has no HTTP server.
The file name comes from the caller, so it is checked twice, in Go
and again in the backend. It must be one path component: no empty
name, "/", "\", "..", NUL, control characters or absolute path. The
resolved path, with symbolic links followed, must be a direct child
of slot_save_path. The routes are admin only because they write
files on the backend host.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): answer 501 when a backend lacks slot cache
A backend that does not implement the SlotCache rpc returns gRPC
UNIMPLEMENTED. The API passed that text through, so the client saw
"unknown method SlotCache". Map the status to a typed error in
core/backend and return 501 with a fixed message instead. Every other
error path is unchanged.
Make the SlotCache comments in backend.proto backend-neutral: slot is
the backend's state slot, filename must stay inside the save directory,
and an empty model_identity means the backend does not check it. Only
comments change, no field or name.
Add tests that the identity sent is the configured model, and for the
error mapping.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): issue slot cache IDs on the server
The slot save and restore API let the client pick the file name.
Restore could load any file in slot_save_path, and the symlink check
ran before llama.cpp opened the file.
The backend now issues the name. A save returns a random 128-bit ID
(32 lowercase hex characters, read from /dev/urandom) and never
overwrites an existing file. Files live in a 0700 directory per model,
named from the SHA-256 of the model identity, with a manifest.json
that records each save. The manifest is written atomically and a
corrupt one stops both save and restore.
A restore takes the ID, finds it in the manifest, refuses an entry of
another model, builds the path from the entry and checks with lstat
that it is a plain file of the recorded size. The directory mode is
the main defence against another local user, since llama.cpp still
opens the path itself.
Each model keeps at most 32 saves and the oldest is deleted first. A
file over 8 GiB is deleted after the save, because llama.cpp does not
report the size before it writes. Responses carry no host path.
The HTTP bodies change: save takes {model, slot}, restore takes
{model, slot, id}, and an unknown ID returns 404. In the proto,
SlotCacheRequest.filename and SlotCacheResponse.filename become id.
SlotCache no longer calls checkModelIdentity, which reads a field this
request does not have.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): copy slot_cache files in prepare.sh
grpc-server.cpp includes slot_cache.h, but prepare.sh copies the helper
headers into llama.cpp/tools/grpc-server/ one by one and skipped it. The
llama-cpp, turboquant and bonsai builds share this step, so they all
failed with "slot_cache.h: No such file or directory".
Copy slot_cache.h and slot_cache_test.cpp, which CMakeLists.txt builds
when LLAMA_GRPC_BUILD_TESTS is on.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump LocalAGI to pick up the scheduler start/stop fix
LocalAGI's scheduler could hit a nil pointer in run when Start and Stop
raced. That made the core/services/agentpool tests panic at random and
turned tests-linux red.
Move to mudler/LocalAGI@7aa0c8fea0, the merge of
mudler/LocalAGI#519 ("make Start and Stop safe to call concurrently").
The new module version also adds a few indirect requirements to go.mod
and go.sum.
Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
* chore(vllm-cpp): bump MLX pin to 0.32.4
The macOS runner's pip index no longer offers mlx==0.29.4 (it lists only
0.32.4), so the darwin vllm-cpp build fails at the MLX venv step. Move the
pin to 0.32.4 and update the README default to match.
Not verified on macOS: the Makefile was only dry-run on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX 0.32.0, the version the engine builds against
The macOS build fails against mlx 0.32.4:
metal_mlx_provider.mm:147:5: error: no matching member function
for call to 'set_data'
The vllm.cpp Metal provider calls the five-argument
array::set_data(Buffer, size, strides, flags, Deleter). The 0.32.4
headers only declare set_data(allocator::Data) and the Data overload
with size, strides, flags and offset; 0.32.0 still has the Buffer one.
The engine's own CI installs 'mlx==0.32.0' (ci.yml line 1798,
release.yml line 360), so pin that and update the README default.
Not verified on macOS: only the Makefile dry-run and the wheel
headers were checked on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): install the MLX 0.32.0 wheel by URL and hash
The macOS runner's pip index offers only mlx 0.32.4, so
'pip install mlx==0.32.0' fails with "Could not find a version that
satisfies the requirement". 0.32.4 cannot be used: its headers lack the
five-argument array::set_data the engine calls.
Install the 0.32.0 wheel directly by URL with a sha256 check. The wheel
is mlx-metal (py3-none-macosx_14_0_arm64) from PyPI, sha256
5b64b20ac24b0c401f489de01e8209edc4d372125201f19314e6f39e385322aa. It
holds mlx/include, mlx/lib/libmlx.dylib and mlx/lib/mlx.metallib, which
is all the build and package.sh use, and it is not tied to a CPython
version. The URL and hash are Makefile variables; the stamp stays keyed
on MLX_VERSION.
Not verified on macOS: only the Makefile dry-run, the wheel contents and
its sha256 were checked on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio): use captions as the default YuE2 style
Studio sends a caption, but YuE2 requires a style request option.
Default audio-cpp style from caption while preserving explicit options
and the caller's parameter map. Document the Studio fields.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-cpp): map captions to style in backend
Move the YuE2 caption fallback into the C++ request builder as requested.
Preserve explicit style options, including empty values, and move the
regression cases from the Go layer into the backend suite.
Assisted-by: Codex:GPT-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Bump LLAMA_VERSION to b76b13379d3b (upstream 0.7.0 plus later
commits). This is the same one-line change as the bot PR #12725.
Add the moe_cache_mib model option. It sets upstream's
moe_cache_size, a GPU cache for MoE experts kept in host memory,
from a size in MiB. It is off by default. Invalid, negative or
non-numeric values are ignored with a log line. Forks that predate
the field ignore the option, using the same prepare.sh probe as the
other fork-skew switches.
Document the option, and cover the MiB to bytes conversion with a
standalone unit test.
Assisted-by: Claude:claude-sonnet-5-5 g++
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): read the rendered chat prompt through str()
llama.cpp now returns the chat prompt as a common_chat_prompt instead of
a std::string. Convert it with a small helper that still accepts the plain
string used by older forks that share this source.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(stablediffusion-ggml): run MiniMax-H3 video with reference conditioning
Bump stable-diffusion.cpp to 2988060 and follow its tiling rename
(tile_size_x/y -> tile_size_w/h, rel_size_x/y -> rel_size_w/h).
GenerateVideo now passes params["ref_image"] and the request audio to
the MiniMax-H3 Ref2VA inputs (ref_images, ref_audios). The audio must
be a WAV file. params["ref_video"] is refused, since the proto cannot
carry a frame sequence yet.
stable-diffusion.cpp cannot tell the FL2VA and REF2VA DiTs apart, and a
reference input on FL2VA renders a wrong video without an error. The
backend now reads the video_partition option, or the model file name,
and fails early on that mix. Keyframes plus references are refused too.
Add gallery entries for the FL2VA and REF2VA Q4_K_M sets. They do not
enable any bf16 dequantisation.
Assisted-by: Claude:claude-sonnet-5-5 go vet gofmt
* feat(stablediffusion-ggml): take several reference images, videos and audio clips
Phase 1 passed one reference image and one reference audio clip to
stable-diffusion.cpp and refused ref_video. Add repeated reference fields
to GenerateVideoRequest (ref_images, ref_videos, ref_audios) so the
MiniMax-H3 Ref2VA path can use as many references as upstream accepts. A
reference video is a directory of frames with an fps and an optional
paired audio clip.
The /video endpoint stages the new inputs through the same path as
start_image, end_image and audio. Frames of one reference video are
written to a private directory under zero-padded names so the backend can
read them in order. The input count and frame count per request are
bounded.
The backend merges the new fields with params.ref_image, params.ref_video
and audio, which keep working. The FL2VA partition refuses any reference
input, REF2VA accepts all of them, and keyframes stay exclusive with
references. Reference audio is still WAV only.
Assisted-by: Claude:claude-sonnet-5-5 [Bash]
* feat(gallery): offer MiniMax-H3 by partition, DiT layout and format
Name each MiniMax-H3 entry after its partition (fl2va or ref2va), DiT
layout (unpruned or pruned) and format. Add the vllm.cpp builds that its
model page lists: pruned Q8_0 (renders), pruned Q6_K, Q5_0, Q4_K, Q3_K and
Q2_K (load only), and the pruned FL2VA NVFP4 file (loads only). The
earlier names minimax-h3-fl2va-q4 and minimax-h3-ref2va-q4 stay as
aliases. The two stable-diffusion.cpp entries become
minimax-h3-fl2va-unpruned-q4-ggml and minimax-h3-ref2va-pruned-q4-ggml,
and the latter now lists video input.
Fix two bad sources. The stable-diffusion.cpp entries took the VAEs from
a path that does not exist in the GGUF repository; they now use the
Comfy-Org copies. The vllm.cpp entries paired the fp16 video VAE weights
with a config from a different directory; they now use the official
weights and config from FL2VA/video_vae/source/.
Drop video_dequant_bf16 from every vllm.cpp entry. Expanding a quantised
DiT to bf16 is the suspected cause of a host out-of-memory error, so the
quantised weights stay resident. The model page records its renders with
the expansion on, and each description says so.
Assisted-by: Claude:claude-sonnet-5-5 [Bash]
* feat(gallery): add pruned FL2VA Q4 MiniMax-H3 for sd.cpp
Add minimax-h3-fl2va-pruned-q4-ggml, the 11.4 GB pruned FL2VA
counterpart of the existing pruned REF2VA entry.
Assisted-by: Claude Code:claude-sonnet-5-5 [rc]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Remove a stray blank line in backend.proto and group the imports in
pkg/motion/validation.go. Decide the motion usecase guess from the
backend capability registry instead of a hardcoded backend name, so
variant builds of the backend resolve too. Declare the weights licence
in the gallery entry, as stated on the model card.
Assisted-by: Claude Code:claude-sonnet-5-5
Add the GEM-X native backend with live pose and root displacement output, bounded duplex WebSocket streaming, session authorization, metrics and usage accounting. Share the embedded bidirectional RPC channel adapter across streaming backends.
Include backend build and gallery integration, API documentation, schemas and tests.
Assisted-by: Codex:gpt-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
(cherry picked from commit 016dc3d71a)
With DOT, EUCLID or MANHATTAN the backend returns Qdrant's raw scores,
not similarity-like values. Router, face and voice stores compare
against cosine thresholds, so log a warning at load when another
metric is configured, and document the behaviour on the stores page.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
StoresSet sent the whole batch in one Upsert under a single
request_timeout_ms, so a large batch could hit the timeout even though
every point was fine. Send at most 500 points per request and apply
the timeout to each request.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A TLS 1.3 floor rejects Qdrant endpoints that only speak TLS 1.2, such
as some managed deployments and proxies. Use TLS 1.2 as the minimum.
When Load fails after the client was created, it closed the client but
left it set, so Free closed it a second time. Clear it after closing.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The sd_embed branch dropped negative_prompt for every pipeline, which
discarded the user's negative prompt for Flux, where no negative embeds
are produced. Only drop it when negative_prompt_embeds is present, and
move the check to a helper that has unit tests.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
