* chore(llama-cpp): update upstream revision
Assisted-by: Codex:gpt-5.6
* fix(llama-cpp): refresh server patch contexts
The new llama.cpp pin changed the slot reset and prompt batch code. GNU patch accepted stale hunks with fuzz, which left the L4T build with invalid source.
Refresh both server patches against the pinned source so each hunk applies at its intended location.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): adapt metrics result fields
The updated llama.cpp groups cumulative counters under server_metrics. Probe the result layout so the shared adapter also compiles against older forks.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): refresh TTS patch offsets
GNU patch rejects the stale pre-decode hunk after the score patch changes the same file. Anchor the TTS hunks to the pinned llama.cpp source so the full series applies without fuzz.
Assisted-by: Codex:gpt-5.4
* fix(llama-cpp): normalize batch threads
The updated llama.cpp creates its batch threadpool during model initialization, before the context-level fallback can replace the -1 sentinel. Resolve that sentinel from the inference thread count so model loading does not overflow the threadpool allocation.\n\nAssisted-by: Codex:gpt-5.4
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): refresh the TTS patch
The llama.cpp update moved and changed the generated-audio pipeline. Refresh the carried patch so backend builds can apply it to the new revision.
Assisted-by: Codex:gpt-5.4
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(config): do not read a TTS speaker-encoder mmproj as vision support
Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and
code predictor. VisionSupported() treated any non-empty MMProj as proof
of image input, so every such model would be advertised as vision-capable.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): add TTS request option parsing helper
Validates text and speaker reference presence and strictly parses the
top_k / top_p per-request params, in a header with no llama.cpp or gRPC
dependencies so the standalone C++ unit test gate picks it up.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): range-check the TTS top_k and top_p request params
Format validation alone let NaN, infinity and out-of-range values through.
The consumer copies both values into the audio generation input
unconditionally and only guards its separate sampler assignment with
"> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard
never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0
inclusive, with the bound written as a negated in-range test so NaN is
rejected rather than silently accepted.
Also cover the two checks the suite could not previously kill: the
whole-string check in the float parser and the int32 range check.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task
Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the
short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch,
the server-side half of the still-draft #26603, so TTS runs through the
slot scheduler instead of racing it. Remove that patch when #26603 merges.
The patch is rebased on top of the score patch: its tokenize-switch hunk
collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call
passes no variadic argument, which the macro cannot expand. The score
patch itself needed no refresh.
Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the
per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only
the schema branch loses it, since forks predating the server-schema split
still expect the old argument list.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): implement the TTS and TTSStream RPCs
Both were declared in backend.proto but unimplemented. They now submit a
SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape
PredictStream uses.
The streaming path emits a leading sample_rate message and then raw PCM,
because ModelTTSStream builds the WAV header itself; the non-streaming
path emits a complete WAV to the requested dst.
The streamed samples are converted from the pipeline's float32 to signed
16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while
the header ModelTTSStream writes announces 16-bit samples, so shipping
the floats verbatim would decode as noise.
prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside
the other grpc-server helpers, and register its standalone test with
ctest the way passthrough_options_test is registered.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation
The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a
codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp:
inp.code0 = sampled - codec_0;
For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens,
<|codec_0|> is 151936 and the codec codes end at 153983. The model's own
tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007,
every special above the codec range except <|codec_eos_token|> (154086)
which stays reachable as the stop token. Nothing masks the text range
0..151935, so the backbone can sample a text token at any step, the
subtraction goes negative, and ggml_compute_forward_get_rows aborts the
whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01).
Complete the mask upstream started: bias every token below <|codec_0|>
to -INFINITY for TTS tasks so only codec codes and the codec EOS remain
reachable. The biases are appended to task.params.sampling.logit_bias,
which common_sampler_init already merges with the model's suppress
tokens into one llama_sampler_init_logit_bias, so no sampler is added to
the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set
against a forward pass in the multi-millisecond range.
It lands in launch_slot_with_task rather than in a route handler so that
llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both
covered, and <|codec_0|> is resolved from the vocab rather than
hardcoded so a model without it is left alone.
This is reproducible with upstream's own llama-tts and no LocalAI code
loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is
neither a quantization artifact nor an artifact of the gRPC adapter.
Two further defects in the same draft pipeline still prevent end-to-end
audio; they are independent of this one and are recorded in the task
report for an upstream bug report.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask
Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81
("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour
after the previous pin. ggml_build_forward_expand marks a tensor and all
its ancestors for compute, so using it as a pure ordering hint defeated
ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE
branch against a stale inp_code0, hitting the get_rows bound assert in
ggml_compute_forward_get_rows.
That single defect accounts for every abort seen on this model, so
0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased.
The mask changed the observed behavior, but it was perturbing a graph
ordering bug rather than fixing a sampling one: at the new pin the whole
path works without it. Keeping it would have meant carrying a 152k-entry
logit bias, and rebasing it on every pin bump, for no benefit.
Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply
clean with no fuzz and needed no rebase:
non-streaming HTTP 200, 410924 bytes, 8.56 s
RIFF (little-endian) data, WAVE audio, Microsoft PCM,
16 bit, mono 24000 Hz
streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at
byte 0, same format, which also exercises the
float32-to-s16 conversion at runtime for the first time
Pristine unpatched llama-tts at the same pin now also completes, 130
frames to a valid WAV, where it aborted at frame 55 before.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): clear the TTS slot sequence between requests
Only the first TTS request in a backend process succeeded. Every later
one failed instantly, in about 0.13 s, with "TTS prompt processing
failed" from step_prompt, regardless of streaming or non-streaming and
regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the
process is kept alive between requests, so a deployment would have
served exactly one utterance per backend start.
The cause is missing KV hygiene, not anything in the gRPC adapter. TTS
slots never enter the shared batch: pre_decode() returns early for them
and process_tts_slots() drives them instead, so they skip the
prompt-cache bookkeeping that clears a slot's sequence between requests.
Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset
only clears host-side buffers, and the pipeline always decodes from
position 0 into the sequence identified by slot.id. So the second task
on a slot writes positions 0..N over the first task's tokens and
llama_decode fails.
Fix is one call to slot.prompt_clear(), the same helper the normal path
uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before
set_input. It goes into 0002 rather than a new patch file because it is
a defect in the code that patch introduces, and the header now records
it as ours so we know whether it still needs carrying if #26603 merges
without it.
Verified in one backend process, different text on every request:
three consecutive non-streaming requests, three consecutive streaming
requests, and an interleaved non-streaming, streaming, non-streaming,
streaming run. All ten returned HTTP 200 with
RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed
ones carrying exactly one RIFF header at byte 0, and every output
measured as real speech rather than silence or a truncated fragment.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): expose max_frames for TTS requests
The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and
when it does not, generation runs to upstream's 512-frame n_predict
default. At the model's 12.5 Hz frame rate that is 40.96 s of audio,
which a short input can trigger: one request in this session produced
40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to
-1, so callers had no way to bound it.
Add a max_frames key alongside top_k and top_p, parsed with the same
strict whole-string parsing so a typo is an error rather than a silently
truncated value, and rejected with a field-naming message when negative.
0 keeps the existing sentinel convention and means unset, so a request
that omits it behaves exactly as before.
Named max_frames rather than n_predict because frames are what the
parameter means at a TTS endpoint: one frame is 0.08 s of audio.
The 512-frame default is deliberately unchanged. Lowering it would
truncate legitimately long inputs, which is a worse failure than an
occasionally overlong one.
Verified end to end on one text of thirty words:
max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames
max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames
no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own
codec EOS after 149 frames, unchanged behavior
max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\""
max_frames=many InvalidArgument "max_frames must be an integer, got \"many\""
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): send the TTS sample rate up front, and tidy three review items
Four items from the Task 4 review.
Streaming first-byte latency. TTSStream sent the sample-rate reply only
once the first audio result arrived, and a chunk needs a whole 72-frame
window, roughly 5.8 s of audio and far longer in wall time on CPU. The
Go side blocks on that reply before it can emit the WAV header, so a
streaming client sat at zero bytes for the whole stretch. The rate is a
property of the loaded model and is available synchronously from
mtmd_gen_audio_get_info, so it now goes out immediately after post_task
and the rate_sent bookkeeping is gone. Measured on a warm model, first
byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV
with exactly one RIFF header at byte 0.
Unchecked close. The non-streaming path ignored ofstream::close(), so a
failure that only surfaces on flush was reported as success while
leaving a truncated file at dst. It now returns INTERNAL like the other
write failures.
Wrong comment on set_lang. gen_audio::inp::get() already maps a stored
blank to nullptr, so our guard is behavior-preserving, not
behavior-fixing. The comment claimed otherwise; the code was right.
Repetition penalty. penalty_last_n = -1 is inert at this pin, because
llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0)
and then builds a disabled sampler, so the 1.05 penalty never applies.
Upstream's README attributes looping to a missing repeat_penalty, so it
was worth testing as a root-cause fix for the model running to the frame
cap. Dropping the line lets the sampling default of 64 apply, which was
confirmed in the sampler chain trace as penalty_last_n = 64 with
repeat_penalty = 1.050. Over 15 uncapped short requests each way it did
not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it
active. Both lines are therefore kept for parity with upstream's draft,
and a comment now records that the pair is inert and why, so the next
reader does not believe a penalty is applied. max_frames remains the way
to bound output.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(llama-cpp): let unpatched forks opt out of the TTS task
turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do
not carry our patches. disable-tts-task.sh injects the same kind of
preprocessor switch disable-score-task.sh already uses, so those builds
answer UNIMPLEMENTED rather than failing to compile.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): keep a TTS speaker-encoder projector out of vision detection
Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the
first real gallery entry with an mmproj still came back vision-capable through
two paths the earlier fix did not close.
GuessUsecases has no FLAG_VISION branch, so it falls through to true for any
chat-ish model. That is not just a wrong answer at the call site:
syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and
the loader calls it more than once per config file, so the guessed FLAG_VISION
is written out and parsed back into KnownUsecases as if the operator had
declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same
explicit signals VisionSupported uses.
Second, llama.cpp builds an mtmd context for the speaker-encoder projector and
reports its media marker on the first chat probe, which resurrected vision
after the model had been used once. Apply the same declared-TTS exemption to
MediaMarker that the mmproj check already had.
Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability
and no image input modality, before load, after a TTS request, and after a chat
probe.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen3-TTS entries for the llama-cpp backend
Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each
pairing a backbone with the Q8_0 projector. Named to sit alongside the
existing qwen3-tts-cpp entries rather than replace them.
Also tags the llama-cpp backend text-to-speech / TTS so the backend browser
surfaces the capability.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: cover Qwen3-TTS on the llama-cpp backend
Adds the gallery variants, the two-file mmproj configuration, the
required voice reference, and the language and sampling knobs. Also
corrects the streaming-support list, which named only voxcpm.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): register llama-cpp as a TTS and voice-cloning backend
The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two
gallery entries for it, but never told the capability table. llama-cpp still
declared only the text RPCs and usecases, so:
- VoiceCloningForModel returned nil at the capability check, before it ever
reached the model's own tts.voice_cloning override, and /tts answered 400
"selected model does not support reference-audio voice cloning" for any
localai://voice-profiles/... voice. No model YAML could opt back in.
- GET /api/backends/usecases did not list tts for llama-cpp, so the gallery
greyed out the TTS filter for the entries this branch adds.
- The React TTS page saw voice_cloning: null and kept both models out of the
Voice Library.
Add the TTS RPCs and usecase, and the reference-audio contract.
The contract needs narrowing, because the per-backend switch in
VoiceCloningForModel ends in a permissive default: an unnarrowed entry would
have advertised reference-audio cloning on every GGUF chat model in the
gallery. Narrow on the declared TTS usecase rather than the model name. The
TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts];
name matching would have to guess at third-party repacks, and "base", the
substring the neighbouring Qwen and vLLM cases key on, is a routine word in
text-model names. The check reads the declared bit directly instead of going
through HasUsecases, which falls through to GuessUsecases and would hand the
decision to a heuristic that never had a llama.cpp TTS model in mind.
DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model,
and both the gallery filter and the importer read that field.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): declare what nemotron-3-nano-omni actually accepts
The entry is backend: vllm-omni with known_usecases: [chat, completion], no
mmproj and no media marker, so it used to report vision only through the
blanket GuessUsecases fallthrough that the vision branch in this branch
removed. Nemotron 3 Nano Omni is a multimodal understanding model: image,
video and audio in, text out. Declaring that is what the sibling
vllm-omni-qwen3-omni-30b already does.
known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output
modality, and this model generates none; video and audio input belong in
known_input_modalities, which is where AudioInputSupported and
VideoInputSupported read them from.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat
The llama-cpp importer hardcodes known_usecases: [chat] and assigns any
mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B-
Base-GGUF imported as a chat model with vision. Both fields were wrong, and
the model was unreachable from /tts and from the Voice Library.
Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision
repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header
is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the
projectors it can drive as a speech pipeline and refuses to build one without
it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP
detection already does, and declare tts when it is set. The mmproj assignment
then stops reading as vision on its own, since a declared-TTS model already
exempts its projector from vision detection.
The probe is best-effort like the MTP one: a network blip leaves the chat
default in place rather than failing the import.
Verified against the real artifacts on disk: the Qwen3-TTS projector reports
gen-audio, its backbone does not.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): stop non-TTS models crashing on the new pin
Two regressions, both hit every ordinary llama-cpp model and neither was
caught locally because every test on this branch loaded a TTS model.
The first is a null dereference. server_slot::tts_ctx::reset() called
mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline
is only allocated for models carrying a gen-audio mmproj, and upstream's
implementation reads ctx->pipeline before null-checking anything. Since
server_slot::reset() runs during slot initialization for every model, any
non-TTS model segfaulted the backend the moment it loaded. Guard the call
on the is_supported() predicate already defined beside it, and keep the
plain field resets unconditional.
The second is unrelated to TTS and came in with the pin bump.
PredictOptions.Penalty is a bare proto float, so a caller that names no
repetition penalty sends 0 rather than omitting the field. Since
9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat
outright because it would divide logits by zero, turning every such
request into "Failed to initialize samplers". Treat 0 as unset and leave
llama.cpp's own neutral default in place.
Verified with the same suite CI runs, which is what caught both:
tests/e2e-backends passes 6 of 6 including the load and predict specs
that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono
16-bit WAV with exactly one RIFF header on the streamed output.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): drop merged MiniMax-M3 patch
The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Bump LLAMA_VERSION to 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3 (73 commits
touching common/, src/ and tools/server/). Two upstream changes break the
backend:
* ggml-org/llama.cpp#20834 folded common_params::use_mmap / use_mlock /
use_direct_io into a single `load_mode` enum. LocalAI still exposes the three
as independent settings (`mmap`, `mmlock`, and the `direct_io` option), so
params_parse folds them once all three have been read, keeping the precedence
the separate booleans had: direct I/O bypasses the page cache, mlock implies
mmap, everything off is a plain buffered read. turboquant and bonsai compile
this same grpc-server.cpp against forks that predate the refactor, so
prepare.sh probes the checkout for LLAMA_LOAD_MODE_MMAP and generates
llama_compat.h with LOCALAI_LEGACY_LOAD_MODE set accordingly. Probing beats a
per-fork build flag here because the fork flavor targets disagree on whether
they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and it heals itself once a fork
rebases past the refactor.
* The MiniMax M3 patch no longer applies. Upstream merged the model half of
llama.cpp#24523 (LLM_ARCH_MINIMAX_M3, src/models/minimax-m3.cpp, the gguf-py
constants and conversion/minimax.py) but not the chat half, so the patch is
re-cut to carry only the common/chat.cpp template detection and PEG parser,
rebased onto the new pin and onto the thinking_end_tag -> thinking_end_tags
rename. Dropping it wholesale (as #11008 did, reverted in #11136) would have
silently regressed MiniMax M3 tool calling and thinking.
Verified with a CPU docker build of the backend plus LoadModel and Predict
against a real GGUF over gRPC in all four load modes.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link server-stream.cpp TU into grpc-server for upstream 0ed235ea (#10536)
Upstream llama.cpp 0ed235ea added an SSE stream-resumption layer in a new
translation unit tools/server/server-stream.cpp, which defines
stream_session, stream_pipe_producer and the g_stream_sessions manager.
server-context.cpp (already #included into grpc-server.cpp) now calls into
it via spipe->cleanup(), stream_aware_should_stop() and
stream_session_attach_pipe(), so without the new TU the grpc-server link
fails on every arch with:
undefined reference to `stream_pipe_producer::cleanup()'
prepare.sh already copies every tools/server/* file into tools/grpc-server/,
so the source is present; the only missing piece was including its
definitions. Add an __has_include-guarded #include "server-stream.cpp"
before server-context.cpp, mirroring the existing server-chat.cpp and
server-schema.cpp guards, keeping the source compatible with older
pins/forks that predate the split. The file is self-contained (its only
external symbols come from server-common, already in the TU) so it adds no
new undefined references; the http route-handler factories it also defines
are unused in the grpc path but harmless.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp): build renamed ggml-rpc-server target for upstream 0ed235ea (#10536)
Upstream renamed the RPC server CMake target and binary from `rpc-server`
to `ggml-rpc-server` (tools/rpc/CMakeLists.txt: `set(TARGET ggml-rpc-server)`),
so the RPC-enabled grpc build failed with "No rule to make target 'rpc-server'".
The grpc-server itself links fine after the server-stream.cpp fix; this only
updates the RPC target name and the binary path copied to llama-cpp-rpc-server.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): single x86 CPU build via ggml CPU_ALL_VARIANTS
Replace the per-microarch avx/avx2/avx512/fallback multi-binary build on
x86 with a single grpc-server plus the dlopen-able libggml-cpu-*.so set
that ggml's backend registry selects at runtime by probing host CPU
features. One build instead of four, broader microarch coverage (adds
alderlake AVX-VNNI, zen4 AVX512-BF16, sapphirerapids AMX), and the
shell-side /proc/cpuinfo probing in run.sh goes away.
Build/link notes:
- CPU_ALL_VARIANTS requires GGML_BACKEND_DL + BUILD_SHARED_LIBS=ON, so
ggml/llama become shared objects. SHARED_LIBS is now a make variable
(default OFF) so the override survives the recursive sub-make into the
VARIANT build dir instead of being re-clobbered by the base flags.
- The cpu-all target also builds "--target ggml": the per-microarch
backends are runtime-dlopened, not link deps, so they only compile via
ggml's add_dependencies().
- hw_grpc_proto is pinned STATIC. Under BUILD_SHARED_LIBS=ON it would
otherwise become a DSO referencing hidden-visibility symbols in the
static libprotobuf.a, which fails to link ("hidden symbol ... is
referenced by DSO"). Keeping it static links gRPC/protobuf into the
executable while only ggml/llama stay shared, so no PIC or base-image
change is required.
- package.sh bundles the libggml-*.so set into package/lib; ggml finds
them by scanning the bundled ld.so directory (/proc/self/exe), which
run.sh launches from.
Scope: x86 only. arm64/darwin keep the single fallback build. The
ik-llama-cpp / turboquant forks and the other ggml C++ backends are
unchanged; the same recipe applies but is out of scope here.
Validated with a full docker build plus a live inference smoke test:
the model loads, ggml selects the AVX512_BF16 variant on a Zen-class
host, and tokens generate correctly.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(llama-cpp,turboquant): extend CPU_ALL_VARIANTS to arm64 + turboquant
- llama-cpp: x86 AND arm64 now use the single llama-cpp-cpu-all build
(only hipblas keeps the fallback build). ggml's arm64 variant table
(armv8.x / armv9.x, plus apple_m* on darwin) is selected at runtime.
- turboquant: same recipe via a turboquant-cpu-all target. turboquant
copies backend/cpp/llama-cpp's CMakeLists.txt + Makefile per flavor, so
the hw_grpc_proto STATIC fix and the SHARED_LIBS / EXTRA_CMAKE_ARGS
make-vars are inherited; the target just passes SHARED_LIBS=ON, the DL
flags and --target ggml through, then collects the .so set. run.sh and
package.sh updated to ship/select turboquant-cpu-all.
- Makefile lib-collection find now also matches *.dylib (for the darwin
build, which emits dylibs rather than .so).
ik-llama-cpp is intentionally left unchanged: its pinned ggml has no
CPU_ALL_VARIANTS support and its IQK kernels require AVX2, so the
per-microarch dynamic backend set does not apply.
Scope still excludes the darwin packaging wiring (separate change).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(llama-cpp,turboquant): arm64 gcc-14 for SME variants + darwin cpu-all packaging
- arm64: ggml CPU_ALL_VARIANTS builds armv9.2 SME variants whose -march=...+sme
is rejected by the Ubuntu 24.04 default gcc-13. Build the arm64 variants with
gcc-14 (installed in the compile step). The host only selects a variant it
actually supports at runtime, but every variant must still compile.
- darwin: scripts/build/llama-cpp-darwin.sh builds llama-cpp-cpu-all instead of
the fallback binary, keeps Metal (GGML_METAL stays ON; --target ggml also builds
ggml-metal). The per-microarch libggml-cpu-*.dylib are placed in the package
root next to the binary (darwin has no bundled ld.so, so ggml's executable-dir
scan looks there), while the other shared dylibs go in lib/ for DYLD_LIBRARY_PATH.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp-darwin): distribute ggml backends by suffix (.so root, .dylib lib)
ggml emits its loadable backends (per-microarch CPU variants, metal, blas) with a
.so suffix even on darwin, while the core libraries (ggml-base/ggml/llama/
llama-common/mtmd) use .dylib. Split the distribution by suffix: .so DL backends
go in the package root for ggml's executable-directory scan, .dylib core libs go
in lib/ for DYLD_LIBRARY_PATH. The previous .dylib name-pattern matched none of the
variants.
Verified on an M4: ggml loads the apple_m4 CPU variant (SME=1) and Metal, model
loads and generates correct tokens.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp,turboquant): only CPU_ALL_VARIANTS for pure-CPU builds, GPU uses fallback
The previous gate sent every non-hipblas build through llama-cpp-cpu-all, so the
GPU image builds (cublas, sycl_f16/f32, vulkan, nvidia l4t) compiled the whole CPU
microarch variant matrix on top of their already-huge GPU backend - blowing the
build time (the sycl job was only 59% done after 2h11m) - and the arm64 l4t build
failed at `apt-get install gcc-14` (exit 100) on the Jetson base.
Gate on an empty BUILD_TYPE instead: only the pure CPU image (build-type: '' in
.github/backend-matrix.yml) builds the CPU_ALL_VARIANTS set; every GPU build gets a
single fallback CPU grpc-server, since the accelerator does the compute. This also
confines the arm64 gcc-14 step (needed for the armv9.2 SME variants) to the CPU
build, away from the GPU base images.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(llama-cpp): correct run.sh comment for arm64/darwin cpu-all
arm64 and darwin CPU images now also ship llama-cpp-cpu-all (not fallback-only);
only GPU images ship fallback-only. Fix the stale comment to match.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt grpc-server to upstream server-schema split
Upstream llama.cpp (e475fa2) extracted the JSON request-schema evaluation
out of the static server_task::params_from_json_cmpl into the new
server_schema::eval_llama_cmpl_schema (tools/server/server-schema.cpp).
The grpc-server unity build still called the old static member, breaking
every llama-cpp backend build with "no member named 'params_from_json_cmpl'
in 'server_task'".
Pull server-schema.cpp into the translation unit and call the new function,
keeping both guarded by __has_include so forks that predate the split (e.g.
llama-cpp-turboquant, which still exposes params_from_json_cmpl) keep
compiling against the old static member.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump to 8f83d6c for mtmd video input support
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): forward video input to mtmd (template + non-template paths)
Wire request->videos() into grpc-server.cpp mirroring the existing image
and audio handling: a video_data build + non-template files extraction, and
input_video chat chunks on the tokenizer-template path. allow_video is
auto-set at model load by the vendored upstream chat_params.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add video attachment support to the chat UI
Mirror the image/audio attachment path for video: emit video_url content
parts, accept video/* in the picker, keep video files as base64, show a
film icon badge, and render attached video inline with a <video> player.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): patch mtmd video stdin double-close (heap crash)
Upstream mtmd video input (ggml-org/llama.cpp#24269) double-fcloses the
ffmpeg/ffprobe stdin FILE: feed_stdin() fclose()s the FILE returned by
subprocess_stdin() (which is sp->stdin_file), then subprocess_destroy()
fclose()s the same pointer again -> heap corruption that aborts the
backend on any base64 input_video request (the CLI --video file path is
unaffected). Vendor a one-line fix (null sp->stdin_file after fclose)
via prepare.sh's patches/ until upstream merges it.
Verified e2e with gemma-4-e2b-it-qat-q4_0: video frames decode via
ffmpeg and the model answers correctly (red clip -> 'Red', blue -> 'Blue').
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): re-pin to upstream #24316, drop vendored stdin patch
Upstream replaced the ad-hoc video stdin handling with a proper RAII
refactor (ggml-org/llama.cpp#24316, "mtmd: refactor video subproc
handling"), which includes the same `sp->stdin_file = nullptr` guard our
patch added (plus join-before-destroy ordering). Re-pin LLAMA_VERSION to
that branch head and drop patches/0001 - it's now redundant.
Verified e2e with gemma-4-e2b-it-qat-q4_0: no crash, video frames decode
and the model answers correctly (red clip -> "Red", blue -> "Blue").
NOTE: #24316 is not yet merged, so this pins to its branch-head commit
(28ca1e60). Re-pin to the squash-merge commit on master once it lands,
otherwise `git fetch` may lose the commit after the branch is deleted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
