* feat(proxy): accept direct upstream API keys
Support direct credentials for models created in the UI. Keep env and
file references as mutually exclusive alternatives. Use model env for
frontend probes and omit credentials from structured config logs.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): isolate model environments in worker processes
Forward complete model environment snapshots through backend installation.
Restart worker children when overrides change or are removed, and include
keyed snapshot identities in frontend install coalescing.
Cover wire compatibility, routing, real worker children, and option copying;
document rollout requirements and control-transport security.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(worker): serialize installs by model replica
Different backend names can target the same process slot. Hold a slot
lock through startup, readiness, and operation attachment so concurrent
installs cannot reuse another configuration or an unready child.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nodes): update endpoint stub for model env
Keep the endpoint command stub aligned with the environment-aware install
interface. Assert interface conformance so future changes fail at the
stub declaration rather than at endpoint call sites.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update NVIDIA/NeMo-Speech.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(nemo-speech-cpp): use upstream llama.cpp ggml patching
Initialize ggml through the llama.cpp submodule and let upstream CMake apply the patch series with fail-closed error handling.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): recognize upstream OpenFST ownership fix
Accept both WrapUnique assignments unchanged while preserving the legacy header patch and rejecting unknown or incomplete forms.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): bundle shared text normalization runtime
Stage the normalization DSO when ITN is present and include it in the package alongside its dependency closure. Fail staging if the required DSO is missing.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): pin the Queue declaration fix
Advance vllm.cpp to the upstream fix for the mismatched Queue tag.
This fixes the strict Clang compilation failure without local patches
or warning suppression.
Focused declaration probes fail before and pass after the upstream fix.
Full suites and builds are skipped at the owner’s request.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(sd): adopt owning generation results
The sd.cpp bump replaces raw image and video outputs with owning results.
Adapt image generation, video generation and upscaling to the new API.
Use upstream destructors to release all pixels, audio and metadata.
Add model-free native checks for result ownership and failure paths.
Assisted-by: nib:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(parakeet-cpp): bump parakeet.cpp to load Phonon-2.5 models
Move the pin to the current parakeet.cpp master. It includes the
dense and packed CPU paths for the Phonon-2.5 checkpoints, which the
previous pin cannot load.
After the bump, run `make purge` in backend/go/parakeet-cpp so the
next build clones the new revision.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
* feat(gallery): add Phonon-2.5 parakeet-cpp models
Add the multilingual Phonon-2.5 (25 European languages) and the
English-only Phonon-2.5-English checkpoints as parakeet-cpp entries.
Each comes in four builds: F16, Q8_0, Q4_K and packed. The packed
build runs on the CPU only. The files are in
mudler/parakeet-cpp-gguf, pinned by SHA-256. The weights are
CC-BY-4.0.
The entries need the parakeet.cpp pin from the previous commit.
Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(grpc): add a Forecast RPC for time-series models
Add Forecast to the backend service with columnar request and response
messages: flat row-major arrays, NaN for missing values, and one message
per series so a caller can send series of different shapes in one call.
The response keeps the request order.
Wire the RPC through the server, client, embedded client, connection
evicting client, in-flight tracking client and the base backend, which
answers unimplemented. core/backend gains a Forecast helper that stamps
the model identity and records a trace, like the other single-purpose
RPCs.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(config): add the forecast usecase
FLAG_FORECAST marks a model as a time-series forecaster. The timestral-cpp
backend declares it as its default usecase, so a config with that backend
needs no extra lines. The backend is also listed as a non-text backend, so
a forecast model is never offered as a chat model. It shows up in the
gallery filter and in the capability list.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(api): add POST /v1/forecast
Take one or more numeric series and return quantile trajectories for the
next prediction_length steps. Targets are [variates][steps] or a flat
array for one variate; JSON null marks a missing value. Past and known
covariates and categorical codes are checked for length before the
backend sees them. Quantiles default to 0.1, 0.5 and 0.9 in the backend.
The route is gated by a new default-on forecast API feature. The endpoint
is listed in the swagger spec, the /api/instructions index, the UI model
filter and the capability symbols.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(backends): add the timestral-cpp forecast backend
Wrap the timestral.cpp C ABI with purego, the way the other ggml
backends do: libtimestral is built from a pinned commit with ggml
embedded, and the Go binary loads it at start.
Forecast validates the counts against the array lengths, groups the
series by shape, makes one library call per group and puts the results
back in the order the caller sent them. Quantiles default to 0.1, 0.5
and 0.9, and a level the model does not have is refused with the list of
valid ones. A mutex guards the handle because the library is not
reentrant. Request size is bounded by two model options.
The unit specs use a fake library. The integration specs and a gRPC
spec run against the 5M F32 model that test.sh downloads and checks
against a fixed SHA-256.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* build(backends): register timestral-cpp in the build and the library
Add the CPU image to the backend matrix for amd64 and arm64, plus an
Apple silicon entry that is not built or tested yet. The image has no GPU
keys because the engine runs on the CPU only. Wire the make targets, the
path filter and the dependency bump bot.
The library entries were generated with backendcatalog. The images are
not published yet, so the release entries are unpinned until a
maintainer runs --repin. The gallery reader test now counts the names in
the library instead of a fixed number, so adding a backend does not
break it.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
* feat(gallery): add the Timestral forecast models
Add Timestral 5M, 28M, 123M and 209M as F32 entries, each with its F16
build as a variant. Q8_0 and the K-quants are left out until the quality
question is settled. Each entry runs on the timestral-cpp backend, on
the CPU only.
The weights use Mistral's Modified MIT licence, which bars use by
companies above a monthly revenue threshold. The first paragraph after
the summary says so, the licence file is installed next to the weights,
and the revenue-restricted-license tag makes the entries filterable.
The memory each build needs has not been measured, so no size is set.
Assisted-by: Claude:claude-sonnet-5-5
* docs(forecast): document /v1/forecast and the Timestral models
Add a feature page with the request and response shapes, the error
cases, the model options and the licence note. List the engine in the
README, the compatibility table and the backend categories.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream Qwen text through audio pipelines
Feed cleaned LLM deltas into one persistent Qwen synthesis session.
Return audio before input ends and preserve the live RPC through model
cache, staging and in-flight wrappers. Stop generation on sink failures
or cancellation without replaying speech through a fallback.
Pin the incremental native fork and verify early audio, late text,
finalization and cancellation with lifecycle and real-model checks.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(tts): use the organization Qwen fork
Fetch the reviewed incremental synthesis commit from localai-org so
backend builds use the organization-maintained fork.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add a Qwen streaming assistant preset
Expose an English pipeline using existing VAD, ASR, LLM and named-voice
TTS models. Enable incremental text and audio delivery, and document
separate component installs because the gallery has no dependency
installer.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): use Parakeet TDT v3 for the assistant
Select the existing Parakeet TDT 0.6B v3 transcription entry and update
its prerequisite install command. Keep the preset's English voice and
incremental Qwen synthesis configuration.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(kokoros): reject unsupported backend RPCs
Implement the MotionStream and SlotCache methods required by the shared
protobuf service. Return Unimplemented so Kokoros compiles without
advertising support for either operation.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): mark cleanup and logging results ignored
Explicitly ignore connection shutdown, test server shutdown and
Ginkgo diagnostic writer errors. These operations do not replace the
stream's inference or sink errors, and errcheck requires their return
values to be acknowledged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(tts): stream buffered clauses in realtime pipelines
Start clause synthesis while LLM text is still arriving when both streaming flags are enabled. Keep explicit clause opt-out and native Qwen sessions, bound automatic queued work, and choose streaming support on resolved failover targets.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(tts): bound synthetic voice clone generation
The sine-wave reference checks speaker conditioning and WAV output, but does not provide a meaningful speech EOS target. Limit that smoke request to 128 frames and verify the request limit reaches native synthesis. Leave production sampling defaults unchanged.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): require explicit clause buffering opt-in
Keep whole-reply synthesis for nonnative engines when clause_chunking is unset or false. Use the bounded cancellable worker when explicitly enabled, with unary clauses when TTS streaming is disabled. Preserve native Qwen precedence for every clause setting.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add VITS and Piper streaming assistants
Provide comparable full pipeline configurations with shared VAD, Parakeet transcription, and Qwen chat components. Enable clause buffering explicitly and document the separate prerequisite model installs and Talk selection.
Assisted-by: Codex:gpt-6.1-sol
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
stable-diffusion.cpp PR #2148 added MOSS-Image support, which needs an
external tokenizer file passed via the --tokenizer CLI arg. The C++ binding
(gosd.cpp) never parsed a tokenizer option nor set ctx_params.tokenizer, so
MOSS-Image could not load through LocalAI.
- Wire the "tokenizer" option in gosd.cpp (parse + assign to ctx_params)
- Resolve "tokenizer:..." paths against the model directory in gosd.go,
the same way *_path options are resolved
- Bump the vendored stable-diffusion.cpp to 68285df (moss-image merge)
- Add moss-image-base and moss-image-sft gallery entries (int8_convrot)
FLUX.3 Image Klein is already in the gallery (added in #12736).
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump MLX pin to 0.32.4
The macOS runner's pip index no longer offers mlx==0.29.4 (it lists only
0.32.4), so the darwin vllm-cpp build fails at the MLX venv step. Move the
pin to 0.32.4 and update the README default to match.
Not verified on macOS: the Makefile was only dry-run on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX 0.32.0, the version the engine builds against
The macOS build fails against mlx 0.32.4:
metal_mlx_provider.mm:147:5: error: no matching member function
for call to 'set_data'
The vllm.cpp Metal provider calls the five-argument
array::set_data(Buffer, size, strides, flags, Deleter). The 0.32.4
headers only declare set_data(allocator::Data) and the Data overload
with size, strides, flags and offset; 0.32.0 still has the Buffer one.
The engine's own CI installs 'mlx==0.32.0' (ci.yml line 1798,
release.yml line 360), so pin that and update the README default.
Not verified on macOS: only the Makefile dry-run and the wheel
headers were checked on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): install the MLX 0.32.0 wheel by URL and hash
The macOS runner's pip index offers only mlx 0.32.4, so
'pip install mlx==0.32.0' fails with "Could not find a version that
satisfies the requirement". 0.32.4 cannot be used: its headers lack the
five-argument array::set_data the engine calls.
Install the 0.32.0 wheel directly by URL with a sha256 check. The wheel
is mlx-metal (py3-none-macosx_14_0_arm64) from PyPI, sha256
5b64b20ac24b0c401f489de01e8209edc4d372125201f19314e6f39e385322aa. It
holds mlx/include, mlx/lib/libmlx.dylib and mlx/lib/mlx.metallib, which
is all the build and package.sh use, and it is not tied to a CPython
version. The URL and hash are Makefile variables; the stamp stays keyed
on MLX_VERSION.
Not verified on macOS: only the Makefile dry-run, the wheel contents and
its sha256 were checked on Linux.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(stablediffusion-ggml): run MiniMax-H3 video with reference conditioning
Bump stable-diffusion.cpp to 2988060 and follow its tiling rename
(tile_size_x/y -> tile_size_w/h, rel_size_x/y -> rel_size_w/h).
GenerateVideo now passes params["ref_image"] and the request audio to
the MiniMax-H3 Ref2VA inputs (ref_images, ref_audios). The audio must
be a WAV file. params["ref_video"] is refused, since the proto cannot
carry a frame sequence yet.
stable-diffusion.cpp cannot tell the FL2VA and REF2VA DiTs apart, and a
reference input on FL2VA renders a wrong video without an error. The
backend now reads the video_partition option, or the model file name,
and fails early on that mix. Keyframes plus references are refused too.
Add gallery entries for the FL2VA and REF2VA Q4_K_M sets. They do not
enable any bf16 dequantisation.
Assisted-by: Claude:claude-sonnet-5-5 go vet gofmt
* feat(stablediffusion-ggml): take several reference images, videos and audio clips
Phase 1 passed one reference image and one reference audio clip to
stable-diffusion.cpp and refused ref_video. Add repeated reference fields
to GenerateVideoRequest (ref_images, ref_videos, ref_audios) so the
MiniMax-H3 Ref2VA path can use as many references as upstream accepts. A
reference video is a directory of frames with an fps and an optional
paired audio clip.
The /video endpoint stages the new inputs through the same path as
start_image, end_image and audio. Frames of one reference video are
written to a private directory under zero-padded names so the backend can
read them in order. The input count and frame count per request are
bounded.
The backend merges the new fields with params.ref_image, params.ref_video
and audio, which keep working. The FL2VA partition refuses any reference
input, REF2VA accepts all of them, and keyframes stay exclusive with
references. Reference audio is still WAV only.
Assisted-by: Claude:claude-sonnet-5-5 [Bash]
* feat(gallery): offer MiniMax-H3 by partition, DiT layout and format
Name each MiniMax-H3 entry after its partition (fl2va or ref2va), DiT
layout (unpruned or pruned) and format. Add the vllm.cpp builds that its
model page lists: pruned Q8_0 (renders), pruned Q6_K, Q5_0, Q4_K, Q3_K and
Q2_K (load only), and the pruned FL2VA NVFP4 file (loads only). The
earlier names minimax-h3-fl2va-q4 and minimax-h3-ref2va-q4 stay as
aliases. The two stable-diffusion.cpp entries become
minimax-h3-fl2va-unpruned-q4-ggml and minimax-h3-ref2va-pruned-q4-ggml,
and the latter now lists video input.
Fix two bad sources. The stable-diffusion.cpp entries took the VAEs from
a path that does not exist in the GGUF repository; they now use the
Comfy-Org copies. The vllm.cpp entries paired the fp16 video VAE weights
with a config from a different directory; they now use the official
weights and config from FL2VA/video_vae/source/.
Drop video_dequant_bf16 from every vllm.cpp entry. Expanding a quantised
DiT to bf16 is the suspected cause of a host out-of-memory error, so the
quantised weights stay resident. The model page records its renders with
the expansion on, and each description says so.
Assisted-by: Claude:claude-sonnet-5-5 [Bash]
* feat(gallery): add pruned FL2VA Q4 MiniMax-H3 for sd.cpp
Add minimax-h3-fl2va-pruned-q4-ggml, the 11.4 GB pruned FL2VA
counterpart of the existing pruned REF2VA entry.
Assisted-by: Claude Code:claude-sonnet-5-5 [rc]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the GEM-X native backend with live pose and root displacement output, bounded duplex WebSocket streaming, session authorization, metrics and usage accounting. Share the embedded bidirectional RPC channel adapter across streaming backends.
Include backend build and gallery integration, API documentation, schemas and tests.
Assisted-by: Codex:gpt-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
(cherry picked from commit 016dc3d71a)
With DOT, EUCLID or MANHATTAN the backend returns Qdrant's raw scores,
not similarity-like values. Router, face and voice stores compare
against cosine thresholds, so log a warning at load when another
metric is configured, and document the behaviour on the stores page.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
