mirror/LocalAI
1
0 Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-11-01 18:19:50 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,818 Commits 373 Branches 181 Tags
ce85d2905421dd2ec7d35ce8b526e8fb507bd366
Commit Graph
53 Commits
Author SHA1 Message Date
mudler-agentandEttore Di Giacinto 2cadf319a0 feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958) 
* chore(backends): remove the liquid-audio Python backend

The backend targets liquid-audio, which loads LFM2 checkpoints only.
Remove its directory, the gallery file and the LFM2.5-Audio model
entries, the library and generated backend gallery entries, the
backend matrix and CI job, the Makefile targets and the golden test
fixtures that listed it.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(config): drop liquid-audio from capabilities, importers and UI

With the liquid-audio backend gone, remove its capability entry, its
importer, its artifact-gate entry and the realtime_audio heuristic
that only matched that backend. realtime_audio is now set only
through known_usecases. Update the Talk and fine-tuning pages, the
docs and the swagger text that named the backend.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump llama.cpp to e486a4b5

Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d,
the master head of 2026-10-28. It contains the LFM3-Audio support
(ggml-org/llama.cpp#31257) and the realtime server split
(ggml-org/llama.cpp#31269).

The score task patch is regenerated against the new pin. It applies
with git apply and with patch -p1, with no offset and no fuzz. Its
hunks in server-task.h and server-context.cpp were rebased by hand
next to SERVER_TASK_TYPE_REALTIME and the control dispatch.

The C++ backend was built on CPU for this change: the llama, mtmd and
server-context libraries and the grpc-server binary link.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): compile server-realtime.cpp into grpc-server

server-context.cpp includes server-realtime.h, and its realtime slot
type is defined in server-realtime.cpp. Upstream compiles that file
into the server library. The gRPC server is one translation unit, so
it must include the file too, or the link fails.

The include is guarded by __has_include. Forks that predate the
realtime split do not ship the file, so they keep building as before.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): probe realtime and TTS voice support in prepare.sh

prepare.sh writes llama_compat.h from probes on the checkout. Add two:
LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and
LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the
gRPC server choose the realtime and voice code paths per checkout, so
forks without them keep compiling.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): advertise speech recognition and speech-to-speech

llama-cpp now registers the AudioTranscription and AudioToAudioStream
RPCs, with the transcript and realtime_audio usecases. The default
usecase stays chat, so a plain GGUF is still a chat model. The
backend implements the RPCs in a follow-up change; until then the
calls return UNIMPLEMENTED.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(config): add pipeline.any_to_any for speech-to-speech models

A realtime_audio model that serves the whole speech loop itself sets
pipeline.any_to_any. The flag is off by default, so chained pipelines
and existing realtime_audio configs are unchanged. The realtime_audio
heuristic stays explicit: a llama-cpp model is never a Talk-page model
unless known_usecases says so.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(config): read the audio encoder and speech-to-speech projector flags

Add HasAudioEncoderProjector for clip.has_audio_encoder and
HasSpeechToSpeechProjector, which requires the LFM3 audio encoder
family together with the gen-audio decoder. Qwen3-TTS sets both encoder
flags too, so the flags alone cannot tell it apart from LFM3.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(importer): import LFM3-Audio projectors as speech-to-speech

An mmproj that carries the LFM3 audio encoder and the gen-audio decoder
now imports with known_usecases realtime_audio, chat, transcript and
tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer
template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS
included. The probe runs against a local HTTP server in the tests.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt

Models without a speaker reference (LFM3-Audio) take a named voice
through their "tts" chat template instead of a reference file. The
request is validated with needs_reference=false, the prompt is rendered
with common_chat_get_tts_prompt(), and the voice is set on the gen-audio
input. Qwen3-TTS Base keeps the reference-file rule. The branch is
guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the
old behaviour.

Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3
gets a system turn that names the language and an empty user turn.
Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a
caller prompt is still appended after a newline. The preset call is
guarded by a new LOCALAI_HAS_ASR_PRESET probe.

The pure parts sit in tts_request_options.h and transcription_prompt.h,
with standalone tests run by backend/cpp/run-unit-tests.sh.

Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts,
ttsstream and transcription specs against LFM3-Audio GGUF weights. It is
a manual run and is not wired into CI.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): mock AudioToAudioStream and pending realtime S2S specs

The mock backend serves AudioToAudioStream deterministically: session
created, commit with RMS, a tool call on the first turn with tools, then
transcript pieces and paced 24 kHz tone frames, with response.cancel
ending a reply as cancelled.

The realtime S2S specs cover the mock and real-weights variants. They
skip until the /v1/realtime S2S path lands, so the mock suite stays
green in CI.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot

Implement AudioToAudioStream on upstream's realtime slot, so a model
that answers after the audio (LFM3-Audio) can hold a conversation over
the existing duplex RPC. The proto schema does not change: tools travel
in the config, tool calls leave as function_call_arguments.done and tool
results arrive as a function_call_output item.

realtime_session.h keeps what needs no model: the turn text derived
from the chat template (with a tool turn), the conversation state, a
filter that keeps <tool_call> blocks out of the transcript and drops
audio produced inside them, and the translation of slot results into
events. grpc-server.cpp wires it to the slot with control tasks and
parses calls with the parser the chat template selects.

The unit test uses a fake renderer. A ctest-only check renders the real
LFM3 template and compares the slot text with the full render.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add the LFM3-Audio-2B entries

Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M
default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both
with the Q8_0 audio projector. Each declares pipeline.any_to_any and the
realtime_audio, chat, transcript and tts usecases the importer writes
for an LFM3 projector, with parallel:2 and the tokenizer template. The
Q4 entry lists the Q8 one as a variant.

File sizes and sha256 come from the Hugging Face API for the pinned
repo revision. Speaker reference files and the VAD model are not part
of the entries: VAD stays optional.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: describe LFM3-Audio speech-to-speech and its gallery entries

Adds a speech-to-speech section to the Realtime API page (any_to_any
configuration, VAD versus manual commit, tools, transcripts, barge-in,
limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in
the compatibility table, and LFM3-Audio mentions on the TTS and ASR
pages. The feature is marked experimental: the behaviour follows the
backend session contract and has not been run with real weights.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): run any-to-any models on AudioToAudioStream

A realtime_audio model with pipeline.any_to_any now opens one duplex
stream to the backend and keeps it for the session, replacing the
"any-to-any models not implemented" branch. The chained pipeline is
unchanged.

The endpoint forwards input audio (with a pre-roll from speech start
when a VAD model is set, otherwise every chunk and a client commit),
sends response.create, and turns the backend's events into the usual
realtime events: audio and transcript deltas, tool calls through the
existing function_call items, and a single response.done. A function
call output is forwarded before the next response. Barge-in sends
response.cancel ahead of the new speech and stops forwarding audio at
once. The user's speech is transcribed on the side and reported on the
user item; a failure there does not touch the response.

Server VAD is on only when pipeline.vad is set. Classifier mode,
compaction, streaming.llm, voice gating, sound detection and
semantic_vad are rejected for these models.

The mock backend now matches the llama.cpp contract (session.updated,
the cancel error when idle) and its VAD reports the loud stretches of
the audio, so the S2S e2e specs run a VAD turn. The specs are enabled.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec

The realtime S2S specs now check reply audio on real models: duration,
loudness, clipping, and keywords found in the model transcript or in a
separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp
and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR
and TTS models, then runs the real variant. It is manual and never part of
CI: `make test-realtime` still filters out real-models.

The backend suite gains an s2s capability (spoken turn, tool call, tool
result, spoken answer over AudioToAudioStream) and optional TTS duration
and RMS bounds plus a transcript keyword check, all enabled for
test-extra-backend-llama-cpp-lfm3.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): expect the idle-cancel error from the mock S2S

The mock now answers a response.cancel with no reply running with
response_cancel_not_active, as the llama.cpp backend does. The cancel
spec still treated that cancel as a no-op and read the error as extra
data after the stream ended.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(talk): show any-to-any models as one speech-to-speech model

/api/pipeline-models lists a pipeline.any_to_any model with any_to_any
set and its optional VAD. A realtime_audio model that is not any-to-any
is listed only when it has the four pipeline slots, since the session
takes the chained path for it.

The Talk page shows a speech-to-speech model as one model: no four-slot
view, the VAD slot only when a VAD model is set, and the female and male
voices offered. Without a VAD model the server does not detect turns, so
the page offers Done speaking, which sends input_audio_buffer.commit and
then response.create. Text-only replies stream through
response.output_text.delta and .done.

self_contained stays in the response for clients built before any_to_any.

Assisted-by: Claude:claude-haiku-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any

llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty
pipeline slots with the model name failed later with UNIMPLEMENTED.
Fail at session start and name the missing setting instead.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): hold manual-commit S2S audio until the commit

In manual-commit mode the endpoint streamed every appended chunk to the
backend, so input_audio_buffer.clear only emptied the Go buffer while the
model kept the audio, and idle audio filled its context between turns.
Keep the audio in the input buffer and send it, behind pending text and
tool-result items, when the client commits or asks for a response.

Frames and controls now go through one ordered write queue, so no gRPC
send happens under the audio buffer lock. A tool result created while a
VAD turn is open is refused with a clear error. When the stream ends
outside a response the session reports it and closes, and a bare EOF
from Send reports the status that Recv holds.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): drop the S2S readiness gate and stale comments

The gate was always true. Some comments referred to work items that
readers of the repository cannot see.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): add the liquid-audio migration note and the clear semantics

Say that the Python liquid-audio backend is gone, which gallery entries
replace it and that LFM2.5-Audio has no replacement. Describe what
input_audio_buffer.clear drops for speech-to-speech sessions now that
manual-commit audio is held back until the commit.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): do not promise turn detection when no VAD model is set

The idle note said the server notices when you stop talking. A
speech-to-speech model without a VAD model needs Done speaking, so the
note now says that.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): make the mock S2S backend keep the llama-cpp session limits

The mock accepted what the real backend refuses and hid two endpoint bugs.
It now treats output_modalities from the config as a fixed session mode and
picks the default per response from the tools set at that time. A fixed text
plus audio mode with tools reads the tool call out loud. After the first
commit or message, session.update of instructions, voice or tools is an
error that does not belong to a response, committing an empty buffer is an
error, and response.create while a reply runs is refused. Errors of a
refused response.create carry response_scoped in their meta.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): leave the S2S output mode to the backend unless the client sets one

The stream config sent text plus audio when the session had no tools yet.
The llama-cpp backend keeps a mode from the config as a fixed session mode,
and tools added later by session.update never cleared it. The model then
read tool calls out loud. Send output_modalities only when the client chose
them. The backend picks the default of each response from the tools it has
when the response starts.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama-cpp): mark the realtime errors that answer a response.create

The error event now carries response_scoped: true when it ends the request
for a response (a refused response.create, a failed commit made by it, or
a stream failure). Other errors, such as a refused session.update or a
commit of an empty buffer, are not about a response and one that runs
keeps generating. The client needs the difference to know whether a
response.done is still coming.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end an S2S response only on the errors that belong to it

Every backend error ended the response in flight, although the backend
also reports errors about other things (a refused session.update, a commit
of an empty buffer) and goes on generating. The rest of the reply was lost
and the next response.create met either a refusal or the leftover events of
the old reply. End the response only on an error marked response_scoped,
forward the others to the client and keep waiting for response.done. Drop
the events buffered by an earlier response when a new one starts, and skip
the commit of a turn that forwarded no audio.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): run the S2S input transcription detached from the commit slot

The commit waited for the side transcription of the user audio while it
held the commit slot, and response.create waits for that slot. The ASR
latency was added before every reply, and on a backend with few slots a
busy ASR task could stall the reply. The model already has the audio, so
run the transcription detached under the session context and release the
slot at once. The transcript is still reported on the user item.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): do not pin the place of the input transcript in the S2S order

The transcript of the user audio is reported on the side and may land
before or after the first response events. Check the order of the response
events without it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue

The reader waited on a channel it closes itself when the events buffer was
full, so it outlived the session after a turn ended on an error or timed
out. It now also stops when the session closes.

Stopping a response sent the cancel through the write queue and waited for
it before the grace timer started. With backpressure on the stream the wait
never ended and nothing read the events meanwhile. The timer now starts
first and the cancel is only queued.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): let a chained pipeline run without a VAD model

A pipeline that names no pipeline.vad failed to build, because the
VAD slot was always loaded, and a new session still started with
server_vad. The slot is now optional for chained pipelines as it
already is for any-to-any models: the session starts with turn
detection off and the client commits each turn. Turning server_vad on
for such a session is refused, and a switch to a pipeline without a
VAD drops turn detection.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): add LFM3-Audio VAD and chained-pipeline entries

Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as
the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the
-q8 forms (the model chained through its own transcription, chat and
speech slots, with and without the VAD). The gallery has no
dependencies between models, so the VAD entries state that
silero-vad-ggml must be installed first.

Drop the variants list from lfm3-audio-2b: it let the installer pick
the larger Q8 build on any host with enough memory, so a plain install
of the Q4 entry did not get Q4. All entries are now independent. Tests
cover the entry shapes, the single-build install and their acceptance
by the realtime endpoint.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): run the realtime specs for any-to-any and chained shapes

The realtime specs covered one any-to-any model with a VAD. They now run
for any-to-any and for a chained pipeline, each with and without a VAD,
on the mock backend in CI. VAD-less shapes also check that turn
detection starts off and that server_vad is refused. The manual
real-weights script writes one config per shape that mirrors the gallery
entries and runs the four shapes in turn (LFM3_MATRIX picks a subset).

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries

List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml
must be installed before the -vad entries, describe the chained pipeline
on one model and the behaviour of a pipeline without a VAD, and
describe the shape matrix of the e2e specs.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): wait for the detached input transcription in the S2S specs

The input transcription runs in its own goroutine, so the transcription
completed and failed events can arrive after the response ends. Poll for
them instead of reading the event list right after the response.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): release the S2S stream before running server-side tool calls

A response held the stream slot until it returned, and it ran the tool
calls from inside that scope. A server-side tool starts a follow-up
response on the same stream, which then waited for the slot its parent
still held, so the client never got response.done. Run the tool calls
once the slot is free; the event order is unchanged.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end an S2S session whose stream dies, and stop queuing after close

A stream that died during a response left the session open: later
requests failed one by one and VAD frames kept being forwarded. The
response now ends the session through the transport, and enqueue drops
requests once the session is closed so the write queue cannot grow.

The writer is now the only caller of Send and of CloseSend. close()
used to call CloseSend while the writer could be inside Send, which
gRPC does not allow.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): close the open S2S turn when the VAD stops

Switching turn detection to manual in the middle of a VAD turn left the
turn forwarding, and the next client commit sent the whole buffer a
second time. Stopping the VAD now commits the forwarded audio on the
backend and drops it from the input buffer.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): keep a tool result pending while an S2S user turn is open

The check that refuses a tool result during a user turn ran when the
item arrived, but the item was forwarded later, and the result was
marked as sent before the backend took it. If the user spoke in
between, the backend refused it and it was never retried. The result is
now claimed and queued under the audio lock only when no turn is open,
and otherwise stays pending for the next commit or response.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): empty the input buffer for models without a VAD

Without voice detection the microphone fills the server's input buffer
for the whole session, so Done speaking committed idle silence and the
echo of the previous reply. Clear the buffer when the session starts and
when a reply ends.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(importer): keep an existing parallel option on LFM3-Audio imports

The speech-to-speech import added parallel:2 even when the options
already carried another parallel value. Use the same check as the
hardware defaults, which also covers n_parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(talk): list chained pipelines that have no VAD

/api/pipeline-models required a VAD for a chained pipeline, so the
VAD-less LFM3-Audio pipeline entries never showed up on the Talk page.
Any pipeline with transcription, LLM and speech slots now qualifies.
The Talk page treats every model without a VAD as manual-turn, and the
session sheet says so for chained pipelines too.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(llama-cpp): bump llama.cpp to 0ac1a7fd

Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b,
the master head of 2026-10-29. It contains tool calls in the realtime
server (ggml-org/llama.cpp#31287).

The score task patch is regenerated so that it applies with no offset
and no fuzz.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): keep a null tool argument null

A tagged parameter whose text was "null" came out as an empty string:
unmarshalling JSON null into a string succeeds and leaves it empty, so
the value was taken for a string. Check the first byte of the JSON
value instead. A parameter with the text "null" now becomes JSON null,
as in llama.cpp.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(functions): add an lfm3 XML format preset

The LFM3 chat template writes a call as <tool_call>, <function=NAME>,
one <parameter=KEY> block per argument, then the closing tags. The
qwen3-coder format has the same shape and auto-detection already found
it, but xml_format_preset: lfm3 was not a known name.

Add the preset, with tests for one call, a call without arguments, two
calls, string and non-string values (numbers, booleans, null, arrays,
objects), a call between text, and unfinished input.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): parse tool calls with the chat template, as upstream

Since ggml-org/llama.cpp#31287 the realtime server handles tools with
the parser and grammar of the chat template. Do the same in the
speech-to-speech stream instead of a hard-coded tag filter and a parse
of the whole text at the end.

- Each response gets a common_chat_session with the tools of the
  session and parallel calls. Its grammar and preserved tokens go into
  the sampling of the response, so the slot keeps the call tokens in
  the text and constrains the call.
- The text of the response is fed to the session piece by piece
  (task_result_state::update_chat_msg). Content deltas become the
  transcript; tool-call deltas are collected into one
  function_call_arguments.done per call at the end. Audio after the
  start of a call is dropped.
- A tool_choice of auto, none or required is accepted in the params of
  the config, in session.update and in response.create, with the
  response overriding the session.
- The tool turn is derived from a conversation in which the assistant
  called a tool, like upstream, not from a plain reply.
- response.done carries raw_text, everything the model wrote with the
  call markup, for a caller that parses it another way. If the parser
  throws, the rest of the response is content and the raw text is the
  way out.

ToolCallFilter is no longer used and goes in the next commit.

The unit tests cover the splitter contract (content, call deltas, the
final piece, whole characters only). realtime_chat_check feeds
LFM3-format text through the session in pieces of 1, 7 and all bytes
and checks the call, the transcript and the grammar.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(realtime): drop the tool-call text filter

The parser of the chat template now splits content from tool calls, so
ToolCallFilter, which looked for the literal <tool_call> tags, has no
caller. Remove it with its tests. Every case it covered (a call
between text, split tags, several calls, an unfinished call, text after
a call) is handled by the parser, and realtime_chat_check runs the
LFM3 format through it in pieces of different sizes.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): fall back to the raw text for S2S tool calls

For an any-to-any model, the calls the backend parsed still win. When
there are none and the request had tools, parse the raw text that
response.done now carries the way the chained pipeline parses the text
of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so
a call format the backend parser does not know can still be served
with function.xml_format_preset or the other function settings of the
model. The content around the calls replaces the streamed transcript
only if the config captures it, as in the chained path.

Also pass tool_choice to the backend: the session mode with the stream
config and with session.update, and the mode of a response.create
override with that event. A named function cannot be forced, so it is
not sent.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): carry raw_text in the mock S2S response.done

Match the llama-cpp session: with tools set, response.done has raw_text,
the call markup of the LFM3 template for a tool turn and the transcript
for any other reply.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): parse tool blocks that have text between them

The iterative XML parser stopped at the first scope block that was
preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave
one call. Skip text between blocks once a call was found; text before
the first block still ends the parse as before. Blocks that follow each
other, or only whitespace, already worked.

Tests cover three calls back to back and with text between them, with
the lfm3 preset and with auto-detection. They also pin the current
reading of argument values: the parser has no tool schema, so a value
such as 42 or true is read as JSON even when the schema types it as a
string.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(gallery): read LFM3-Audio tool calls with the lfm3 preset

Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries
and in the config the importer writes for a speech-to-speech projector,
so the server parses the call format of the LFM3 chat template when the
backend returns no call (and in the chained pipeline entries, which use
the regular chat path). gallery/index.yaml is regenerated from the
library.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): describe how S2S tool calls are read

Replace the description of the hard-coded tag filter with the parser
and grammar of the chat template, tool_choice, and the raw-text
fallback in the server. Document function.xml_format_preset, and set it
in the real-weights e2e model configs like the gallery entries do.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(functions): keep earlier calls when a later block does not parse

After a valid call, text that holds the opening tag again (a prose
mention, a markdown example, a malformed second block) made the parser
treat it as a new block, fail, and return no calls at all. Remember the
state after each complete block and fall back to it when a later one
fails: the calls found so far are kept, parsing stops there and the
remaining text stays in the content.

Text skipped between two valid blocks now goes to the content instead of
being dropped, so StripToolCallMarkup keeps it.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): do not drop audio after a tool call started

In mixed mode the speech for text written before a call can still be
pending when the call starts, so dropping every audio frame after the
first call delta cut the speech while its transcript was sent. The
llama.cpp server never drops audio when it reads a result; do the same.
If a model speaks the call markup aloud, that is model behaviour to
observe on real weights, not something to filter here.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): flush held-back text when a response is cancelled

The parser got is_partial=true on a cancelled response, so text it held
back (a possible start of a tag) never reached the transcript. The llama.cpp
server parses a cancelled response with is_partial=false; do the same. A
cancelled response still makes no tool call. The path where the parser
throws already sends the text as content and needs no change.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): stop the fake S2S stream from sending after Close

The barge-in spec failed now and then with a data race and "send on
closed channel": its script pushes audio from a goroutine, and the
session closes the stream while that goroutine still runs. A push now
does nothing once the stream is closed, and Close waits for a push in
flight. The production close order is right: the writer closes the
stream only after it stops sending.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): align the S2S tool-call fallback with the backend

Four gaps in how the Go side reads the tool calls of an any-to-any reply:

- With tool_choice none (the response value wins over the session value)
  the backend builds no tool parser, yet the raw-text fallback still
  turned a call the model wrote into a function_call item. Skip it.
- Without a capture regex the fallback never cleaned the transcript, so the
  stored item and transcript.done kept the raw call markup. When the call
  is XML, use the text left after the markup is cut out; a reply that is
  only a call now has an empty transcript.
- A named function as tool_choice mapped to nothing, so the backend kept
  the previous mode. Refuse it with an error in session.update and
  response.create.
- A response-level tools list changed what the fallback looked for, but
  the backend only has the tools of the session. Use those in both.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(realtime): say how S2S tool_choice and response tools behave

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(e2e): use the real llama-cpp backend in the real realtime run

The suite always registered the mock backend under the name llama-cpp, so
a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend
found in REALTIME_BACKENDS_PATH. Register the mock only when that run is
off.

The real run converts the input audio with ffmpeg. Check for it in the
suite and in run-realtime-lfm3.sh before any download or backend start,
with a message that says what to install, and list it as a prerequisite
in the script header and the Makefile target.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(realtime): cover the tools path of a client session.update to the S2S backend

A tool call that never came back from an any-to-any model raised the
question whether the tools of the session reach the backend at all. Add a
spec that sends the session.update of the real-weights e2e (flat
Responses API tool, no session type) and checks that the backend gets
the tool in the chat-completions form.

Log the number of tools sent when the stream opens and when a
session.update carries them, so a run log shows it without reading the
wire. The "No tools received" line of the llama-cpp backend does not
show it: the server prints it for the input transcription request, which
has no tools by design.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): interrupt the chained pipeline reply while it is in flight

With unary stages, the default of the lfm3 pipeline gallery entries, the
chained pipeline synthesizes the whole reply and sends it as one audio
delta right before response.done. The barge-in spec waited for that
delta and then rendered the interruption with the TTS, so the first
response was already done when the speech started and ended completed
instead of cancelled. Real runs on a GPU showed it with both
quantizations.

For a pipeline, start the interruption at response.created, which comes
before the LLM and TTS stages. Render the interruption audio before the
first utterance is sent, so nothing sits between the reply and the
interruption. The any-to-any shape still waits for the first audio
delta, which that backend streams while it generates.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): accept a number word and its homophone in the spoken-answer check

The check that the reply audio says what the transcript says needs a
shared word of 4+ letters. A short answer failed it: the model wrote
"Four" and the ASR heard "for". Compare numbers in one spoken form, so
digits, number words and their homophones match each other. A homophone
only counts when the other text has the number itself, so "for" on both
sides still shares nothing and the check does not pass on a stop word.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools

The default system prompt of a realtime session was "Respond with
interleaved text and audio." whatever the tools. Under that prompt
LFM3-Audio answers in words ("I'll check the weather ...") and never
writes the call. The model card, and the llama.cpp realtime example
client, use "Respond with function calls, and interleaved text and
audio." when the session has tools, and the model then calls the tool.

Pick the default by the tools of the session, and treat the plain default
that the LocalAI server sends back as no choice: it follows the tools too,
also when they arrive with a session.update before the first audio. A
prompt the client wrote is kept as it is.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): transcribe the input of an any-to-any turn between responses

The input transcript of a speech-to-speech turn ran at the same time as
the reply, on the same model. llama.cpp serves both from one context: the
realtime slot wants embeddings on, the chat slot of the transcript wants
them off, and each flip makes it reserve its compute buffers again. In a
GB10 run that was two reserves of about 175 ms per step, the reply fell to
2.6 tokens per second and the transcript to 380 ms per token, until the
transcript was done.

When pipeline.transcription is the model of the stream, run the transcript
after the response, or before one that is not started yet, and keep
responses out while it runs. A separate transcription model still runs in
parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): send the input transcript of an any-to-any turn before its reply

The previous change moved the same-model transcript after the response.
A client that reads until response.done then never saw
conversation.item.input_audio_transcription.completed: the transcript
was made (the log shows it) but sent after the end of the window.

Run it in line with the commit instead. A VAD turn transcribes and then
responds. A client response.create already waits for the commit slot, so
it starts once the transcript is out. The transcript event now comes
before any output of the response of its turn, for a manual commit and for
a VAD turn, and never overlaps the response on the llama.cpp context. A
transcription model of its own still runs in parallel.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(realtime): transcribe the input of an any-to-any model only when asked

The stream answers from the audio, so the transcript of the input is for
the client only. On the model of the stream it is costly: llama.cpp
serves the transcript request and the realtime stream from one context
and rebuilds its compute buffers on every step while both are active.
On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0
to 1 on the llama.cpp server, and the reply ran at a few tokens per
second.

An any-to-any model no longer fills pipeline.transcription with its own
name. Without it the session does not transcribe the input; with another
model it does, in parallel with the reply; with the model itself named it
does, before the reply. The reply transcript always comes from the stream.
The gallery entries set none, so they get the fast path.

The mock configs and run-realtime-lfm3.sh name a transcription model, so
the real-weights spec still covers the input transcript.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(realtime): end a response as cancelled when a barge-in lands during the LLM call

The chained pipeline cancels the response context on a barge-in. When that
happens while the model is generating, the backend call fails with the
context error, and the response ended as failed: an error event and no
response.done. A client that waits for the terminal of the interrupted
response then waits forever. On a GB10 the q4 model answered before the
barge-in arrived, so only the slower q8 model showed it: the spec read
events for 30 s and timed out.

When the context is cancelled, a failed Predict call or a failed result is
a cancel, as the TTS step and the streamed LLM path already treat it. The
response ends with response.done{cancelled} and no error.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): wait for the input transcript that comes from a model of its own

An any-to-any model now gets its input transcript from a separate ASR
that runs next to the reply. The reply is often over first, so a spec that
stops reading at response.done saw no transcript in the VAD shape. Keep
reading, up to 30 s, until the transcript arrives. The contract is that it
comes, not when.

Add a guard spec for what the input transcription is handed: the whole
turn, in a WAV that says the rate it was recorded at (the session rate for
a client commit, 16 kHz for a VAD turn).

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): check the errors of Close in the realtime specs

golangci-lint (errcheck) flagged the unchecked Close of the websocket
connections, the response body and the fake stream. Use the
`defer func() { _ = x.Close() }()` form the other e2e specs use.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(e2e): write the mock S2S specs with Ginkgo and drop the env read

forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the
sanctioned places. Convert the mock S2S tests to Describe/It with a suite
bootstrap (the package had none), with the same 11 cases. The mock no
longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length
is a package variable they change per spec and restore afterwards.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-10-29 22:06:41 +01:00
Tai Anandlocalai-org-maint-bot eec532704c fix(functions): honor function_arguments_key when building the tool grammar (#11677) 
* fix(functions): honor function_arguments_key when building the tool grammar

All four call sites of `Functions.ToJSONStructure(name, args string)` pass
`FunctionsConfig.FunctionNameKey` as *both* arguments, so
`FunctionArgumentsKey` never reaches the grammar generator.

`ToJSONStructure` writes the two properties into the same map:

    property[nameKey] = FunctionName{Const: function.Name}
    property[argsKey] = Argument{...}

When `nameKey == argsKey` the second assignment overwrites the first, so a
model configured with `function_name_key` gets a grammar carrying only the
arguments object -- the `{"const": "<function name>"}` constraint is gone and
the grammar can no longer express which function was called.

With `function_name_key: function`, the generated property set collapses from

    {"function": {"const": "get_weather"}, "arguments": {...}}

to

    {"function": {"type": "object", "properties": {...}}}

Setting only `function_arguments_key` is equally broken in the other
direction: the grammar keeps emitting `arguments` while `ParseFunctionCall`
(pkg/functions/parse.go) looks up the configured key, so the parsed call comes
back with its arguments empty.

The default configuration is unaffected -- with both keys empty
`ToJSONStructure` falls back to `name`/`arguments` for both parameters, which
is why this went unnoticed.

The existing `ToJSONStructure()` unit test already calls the helper with two
distinct keys, so only the call sites were wrong. Extend that test with a case
that keeps both custom keys distinct and asserts the two properties survive.

Signed-off-by: Anai-Guo <antai12232931@outlook.com>

* test(functions): cover configured grammar keys

Route grammar construction through FunctionsConfig so the regression test
covers the key wiring used by every endpoint.

Assisted-by: Codex:gpt-5

* chore: empty commit to trigger workflow approval

Signed-off-by: Tai An <antai12232931@outlook.com>

---------

Signed-off-by: Anai-Guo <antai12232931@outlook.com>
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
 2026-09-28 00:52:51 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto 82487e68f3 fix(grammars): restore backslash escaping in llama31 grammar fixture (#11242) 
PR #11041 rewrote the testllama31inputResult1 fixture and un-escaped the
backslashes inside its Go raw-string literal, turning `[^"\\]` into
`[^"\]` and `["\\/bfnrt]` into `["\/bfnrt]`. The fixture is compared
line-by-line against the grammar built from PRIMITIVE_RULES in
bnf_rules.go, which is unchanged and still emits the doubled form, so
"generates a valid grammar from JSON schema" fails on every platform.

Restore the four fixture lines to their pre-#11041 form. The cyclic $ref
and depth specs added by that PR are untouched.

The regression reached master because only the DCO check ever reported
on #11041; its test runs were cancelled during the CI purge.


Assisted-by: Claude Code:claude-opus-5[1m] [Bash] [Edit]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-07-30 23:53:23 +02:00
Tai Anandlocalai-org-maint-bot 37f2087f97 fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash (#11020) (#11041) 
* fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash

JSONSchemaConverter.visit resolved $ref entries by recursively calling
itself with no cycle detection. A client-supplied grammar_json_functions
schema whose $defs contains a self- or mutually-referential $ref (e.g.
{"A": {"$ref": "#/$defs/A"}}) made visit recurse until the goroutine
stack was exhausted, producing a fatal "stack overflow" that kills the
whole process rather than failing the single request. The schema is
converted synchronously in the /v1/chat/completions handler before any
backend call, so this is an unauthenticated remote crash. Fixes #11020.

Track the $ref targets currently on the recursion stack and error out
when one is re-entered, while popping after each descent so sibling
(non-cyclic) reuse of the same $ref is still allowed.

Signed-off-by: Tai An <antai12232931@outlook.com>

* fix(grammars): add a bounded recursion depth and cover llama31 $ref cycles

Addresses the review on #11041. The stack-set approach catches cyclic
$ref chains, but a deeply nested yet acyclic client schema (thousands of
nested arrays/objects) can still recurse through visit until the
goroutine stack is exhausted, which is the same unauthenticated remote
crash surface as #11020.

- Add a bounded depth counter to JSONSchemaConverter.visit (incremented
  with a defer-based cleanup, capped at maxSchemaDepth = 256, far above
  any realistic schema) so an over-deep schema fails the request with an
  ordinary error instead of crashing the process.
- Apply the same cyclic-$ref guard and depth bound to
  LLama31SchemaConverter.visit, the other production grammar entry point
  named in #11020, which previously had no cycle detection at all.
- Regression tests: a deeply nested acyclic schema is rejected while a
  moderately nested one still builds, plus direct/indirect $ref cycle
  and depth tests for the llama31 converter.

Signed-off-by: Tai An <antai12232931@outlook.com>

* test(grammars): make llama31 cycle fixtures valid function-call shapes

The two new llama31 $ref-cycle specs asserted on "cyclic $ref" but the
converter requires each top-level oneOf alternative to carry its
function-name property before descending, so both fixtures failed
earlier with "no function name found in the schema" and never reached
the cycle guard.

Give each fixture a valid llama31 shape: construct the converter with
NewLLama31SchemaConverter("function"), put "function": {"const": "test"}
on the top-level alternative, and hang the cyclic $ref under an
arguments property, so all 29 grammar specs pass and the assertions
genuinely observe the cyclic $ref error.

Signed-off-by: Tai An <antai12232931@outlook.com>

---------

Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
 2026-07-30 12:06:17 +02:00
alaningtrump 80ec22945a refactor: use the built-in max/min to simplify the code (#10657) 
Signed-off-by: alaningtrump <alaningtrump@outlook.com>
 2026-07-03 17:59:26 +02:00
pos-ei-donandPoseidon 2cee318fad fix(functions): avoid quadratic-time debug logging in CleanupLLMResult / ParseFunctionCall (#10592) 
fix(functions): avoid quadratic-time debug logging in CleanupLLMResult/ParseFunctionCall

The streaming chat path (core/http/endpoints/openai/chat_stream_workers.go)
calls CleanupLLMResult / ParseFunctionCall once per delta chunk with the
*full accumulated* LLM result so far. Both functions xlog.Debug the entire
argument on entry and exit, so a single N-chunk stream emits roughly
chunk_size * N^2 bytes of debug output.

Under LOG_LEVEL=debug this was observed in a recent SGLang-via-LocalAI
session on a DGX Spark host (about 50K tokens, long streaming generation)
to drive container logs to ~96 GiB, which interacted with the streaming
hot loop on the same filesystem and contributed to a host-wide hard hang
once disk pressure built up. Workaround was setting LOG_LEVEL=info, but
the quadratic shape remains a foot-gun for anyone intentionally enabling
debug.

Replace the four result-content debug arguments with len(...) plus a
fixed-size head (200 bytes via a new truncForLog helper), bounding per-
call output to a constant. The debug signal stays useful: the first 200
chars are enough to identify which generation is in flight, and the
length lets you observe growth without paying for the payload itself.

No API change. No behaviour change for LOG_LEVEL != debug.

Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
Co-authored-by: Poseidon <philipp.wacker@ibf-solutions.com>
 2026-06-30 09:16:03 +02:00
LocalAI [bot]andEttore Di Giacinto 3e220373b0 fix(functions): validate auto-detected XML tool-call names — robust glm-4.5/Hermes guard (#9722, supersedes #9940) (#10059) 
fix(functions): validate auto-detected XML tool-call names (#9722)

The XML tool-call auto-detector tries every preset, including glm-4.5 whose
tool block is <tool_call>name...</tool_call>. When a Hermes/NousResearch model
emits <tool_call>{"name":"bash","arguments":{...}}</tool_call>, glm-4.5
mis-claims the block and returns the entire JSON object (or leading prose, or a
JSON array) as the function NAME. The misparse then wins over the JSON parser,
so streaming clients receive a tool call whose name is a JSON blob.

Guard the auto-detect paths in ParseXMLIterative: a returned tool name must look
like a real function name ([A-Za-z0-9_.-]+). Results that don't are dropped so
auto-detection falls through to the next format and ultimately to JSON parsing,
which handles Hermes correctly. An explicitly forced format (format != nil) is
left untouched and trusted verbatim.

This supersedes PR #9940, which dropped only names with a leading "{". That
narrower check misses leading prose ("Sure: {...}"), JSON arrays ("[{...}]")
and brace-less garbage ("name: bash, ..."); the name-shape check rejects all of
them while still accepting legitimate glm-4.5 calls. The fix applies to both the
streaming worker and the non-streaming ParseFunctionCall path, which both call
ParseXMLIterative with auto-detection.


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-05-29 12:03:33 +02:00
LocalAI [bot]andEttore Di Giacinto 73cfedc023 fix: tool-call JSON leaks into content with stream+tools on tokenizer-template models (#10052) (#10057) 
* fix(grammars): honor properties_order entry at index 0

The JSON-schema-to-GBNF property sort used `aOrder != 0 && bOrder != 0` as
its "is this key ordered?" guard. That treats index 0 — the first key listed
in properties_order — as unset, so `properties_order: name,arguments` fell
back to alphabetical ordering and still emitted "arguments" before "name".

Use presence in the order map instead: listed keys sort by their index and
ahead of unlisted keys, which keep a stable alphabetical order. This makes
the documented `properties_order: name,arguments` actually produce
name-first tool-call JSON. Relates to #10052.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

* fix(functions): defer tool grammar to the backend when the tokenizer template owns templating (#10052)

When use_tokenizer_template delegates templating to the backend (llama.cpp),
the backend also owns tool-call grammar generation and parsing. LocalAI was
still generating its own GBNF grammar and sending it down. With a grammar
present, llama.cpp does not hand the tools to its template, so its native
peg/json tool parser never engages: it streams the grammar-constrained
tool-call JSON back as plain content instead of emitting tool_calls. In
streaming mode the JSON object leaked into the content field, and the
Go-side incremental detector never gated content because the
LocalAI-generated grammar emitted "arguments" before "name".

The GGUF auto-import path already couples use_tokenizer_template with
grammar.disable, but that block is skipped when a template is already
configured, so gallery and hand-written configs (e.g. qwen3) that set the
tokenizer template directly never got the paired grammar.disable.

- SetDefaults now enforces the coupling for every config: when
  use_tokenizer_template is set, grammar generation is disabled and tools
  flow to the backend's native (name-first) pipeline. This also fixes
  already-installed models without editing each config.
- Set function.grammar.disable in the shared gallery/qwen3.yaml, which is
  the base config referenced by every qwen3 gallery entry.

Verified end to end against qwen3-4b with stream:true + tools: content no
longer carries the tool-call JSON, reasoning is classified separately, and
tool calls stream as proper name-first tool_calls deltas.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-05-29 10:12:53 +02:00
LocalAI [bot]andEttore Di Giacinto f17d99f6e5 fix(streaming/tools): stop healing-marker stubs from gating off content (#9999) 
* fix(streaming/tools): stop healing-marker stubs from gating off content

When the C++ autoparser is in pure-content fallback mode (e.g. qwen3
without --jinja) and the model emits a tool call as JSON, the streaming
worker calls ParseJSONIterative on each new chunk. parseJSONWithStack
heals partial input like `{` into `{"<marker>":1}` where <marker> is a
random integer. removeHealingMarkerFromJSON only stripped the marker
from values, so the synthetic key survived and downstream callers saw
a stub object with a random-looking key.

chat_stream_workers.go's JSON tool-call detector then bumped
lastEmittedCount past the stub even though no real tool call was
emitted, gating off ALL subsequent content chunks. The qwen3 + tools +
streaming case ended up dribbling only the first `{"` to clients and
then nothing, even when the model went on to call the noAction
`answer({"message": "…"})` pseudo-tool.

Three changes, each with its own regression test:

* removeHealingMarkerFromJSON now strips the marker suffix from keys
  too, dropping the entry when the truncated key is empty. Inputs like
  `{` no longer leak `{"<marker>":1}` to callers; partial keys like
  `{ "code` still preserve the model-typed prefix `code`.

* ParseJSONIterative skips empty-after-healing maps so a healed `{`
  doesn't surface as a stub result.

* The streaming JSON detector now breaks (not continues) on entries
  without a usable `name`, and only bumps lastEmittedCount past
  successfully-emitted entries. Defense-in-depth against any future
  partial-parse shape.

The parser tests cover eight partial-JSON-prefix shapes and verify no
marker characters leak into keys, plus the two early shapes (`{`,
`{"`) that should not surface a stub at all.

Fixes #9988

Assisted-by: Claude:opus-4-7 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(streaming/tools): cover the autoparser-correctly-working path

Extract the JSON tool-call streaming emit loop into emitJSONToolCallDeltas
and unit-test it against every shape that can hit the streaming worker:

* the bug case — a healing-marker stub at index 0 must NOT bump
  lastEmittedCount, so subsequent content chunks keep flowing;
* the autoparser-correctly-working case — empty jsonResults (because
  the C++ autoparser cleared the raw text and delivers tool calls via
  TokenUsage.ChatDeltas) is a no-op, leaving the deferred end-of-stream
  emitter to ship the autoparser's tool calls;
* a single complete tool call — emit one chunk, advance to 1;
* arguments arriving as a JSON-string vs as a nested object — both
  serialize to the wire as JSON-string arguments;
* multiple parallel tool calls — one chunk each;
* a real tool call followed by a partial stub — emit the real one,
  stop at the stub, resume on a later chunk once the stub completes.

Locks down the no-regression guarantee the user asked for: this PR's
fix is scoped to the pure-content fallback path; when the autoparser
actually classifies tool calls (jinja-recognized chat format with tool
support), the helper is a no-op and nothing changes.

Assisted-by: Claude:opus-4-7 [Read] [Edit] [Bash] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
 2026-05-25 23:55:35 +02:00
Richard Palethorpe 0245b33eab feat(realtime): Add Liquid Audio s2s model and assistant mode on talk page (#9801) 
* feat(liquid-audio): add LFM2.5-Audio any-to-any backend + realtime_audio usecase

Wires LiquidAI's LFM2.5-Audio-1.5B as a self-contained Realtime API model:
single engine handles VAD, transcription, LLM, and TTS in one bidirectional
stream — drop-in alternative to a VAD+STT+LLM+TTS pipeline.

Backend
- backend/python/liquid-audio/ — new Python gRPC backend wrapping the
  `liquid-audio` package. Modes: chat / asr / tts / s2s, voice presets,
  Load/Predict/PredictStream/AudioTranscription/TTS/VAD/AudioToAudioStream/
  Free and StartFineTune/FineTuneProgress/StopFineTune. Runtime monkey-patch
  on `liquid_audio.utils.snapshot_download` so absolute local paths from
  LocalAI's gallery resolve without a HF round-trip. soundfile in place of
  torchaudio.load/save (torchcodec drags NVIDIA NPP we don't bundle).
- backend/backend.proto + pkg/grpc/{backend,client,server,base,embed,
  interface}.go — new AudioToAudioStream RPC mirroring AudioTransformStream
  (config/frame/control oneof in; typed event+pcm+meta out).
- core/services/nodes/{health_mock,inflight}_test.go — add stubs for the
  new RPC to the test fakes.

Config + capabilities
- core/config/backend_capabilities.go — UsecaseRealtimeAudio, MethodAudio
  ToAudioStream, UsecaseInfoMap entry, liquid-audio BackendCapability row.
- core/config/model_config.go — FLAG_REALTIME_AUDIO bitmask, ModalityGroups
  membership in both speech-input and audio-output groups so a lone flag
  still reads as multimodal, GetAllModelConfigUsecases entry, GuessUsecases
  branch.

Realtime endpoint
- core/http/endpoints/openai/realtime.go — extract prepareRealtimeConfig()
  so the gate is unit-testable; accept realtime_audio models and self-fill
  empty pipeline slots with the model's own name (user-pinned slots win).
- core/http/endpoints/openai/realtime_gate_test.go — six specs covering nil
  cfg, empty pipeline, legacy pipeline, self-contained realtime_audio,
  user-pinned VAD slot, and partial legacy pipeline.

UI + endpoints
- core/http/routes/ui.go — /api/pipeline-models accepts either a legacy
  VAD+STT+LLM+TTS pipeline or a realtime_audio model; surfaces a
  self_contained flag so the Talk page can collapse the four cards.
- core/http/routes/ui_api.go — realtime_audio in usecaseFilters.
- core/http/routes/ui_pipeline_models_test.go — covers both code paths.
- core/http/react-ui/src/pages/Talk.jsx — self-contained badge instead of
  the four-slot grid; rename Edit Pipeline → Edit Model Config; less
  pipeline-specific wording.
- core/http/react-ui/src/pages/Models.jsx + locales/en/models.json — new
  realtime_audio filter button + i18n.
- core/http/react-ui/src/utils/capabilities.js — CAP_REALTIME_AUDIO.
- core/http/react-ui/src/pages/FineTune.jsx — voice + validation-dataset
  fields, surfaced when backend === liquid-audio, plumbed via
  extra_options on submit/export/import.

Gallery + importer
- gallery/liquid-audio.yaml — config template with known_usecases:
  [realtime_audio, chat, tts, transcript, vad].
- gallery/index.yaml — four model entries (realtime/chat/asr/tts) keyed by
  mode option. Fixed pre-existing `transcribe` typo on the asr entry
  (loader silently dropped the unknown string → entry never surfaced as a
  transcript model).
- gallery/lfm.yaml — function block for the LFM2 Pythonic tool-call format
  `<|tool_call_start|>[name(k="v")]<|tool_call_end|>` matching
  common_chat_params_init_lfm2 in vendored llama.cpp.
- core/gallery/importers/{liquid-audio,liquid-audio_test}.go — detector
  matches LFM2-Audio HF repos (excludes -gguf mirrors); mode/voice
  preferences plumbed through to options.
- core/gallery/importers/importers.go — register LiquidAudioImporter
  before LlamaCPPImporter.
- pkg/functions/parse_lfm2_test.go — seven specs for the response/argument
  regex pair on the LFM2 pythonic format.

Build matrix
- .github/backend-matrix.yml — seven liquid-audio targets (cuda12, cuda13,
  l4t-cuda-13, hipblas, intel, cpu amd64, cpu arm64). Jetpack r36 cuda-12
  is skipped (Ubuntu 22.04 / Python 3.10 incompatible with liquid-audio's
  3.12 floor).
- backend/index.yaml — anchor + 13 image entries.
- Makefile — .NOTPARALLEL, prepare-test-extra, test-extra,
  docker-build-liquid-audio.

Docs
- .agents/plans/liquid-audio-integration.md — phased plan; PR-D (real
  any-to-any wiring via AudioToAudioStream), PR-E (mid-audio tool-call
  detector), PR-G (GGUF entries once upstream llama.cpp PR #18641 lands)
  remain.
- .agents/api-endpoints-and-auth.md — expand the capability-surface
  checklist with every place a new FLAG_* needs to be registered.

Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): function calling + history cap for any-to-any models

Three pieces, all on the realtime_audio path that just landed:

1. liquid-audio backend (backend/python/liquid-audio/backend.py):
   - _build_chat_state grows a `tools_prelude` arg.
   - new _render_tools_prelude parses request.Tools (the OpenAI Chat
     Completions function array realtime.go already serialises) and
     emits an LFM2 `<|tool_list_start|>…<|tool_list_end|>` system turn
     ahead of the user history. Mirrors gallery/lfm.yaml's `function:`
     template so the model sees the same prompt shape whether served
     via llama-cpp or here. Without this the backend silently dropped
     tools — function calling was wired end-to-end on the Go side but
     the model never saw a tool list.

2. Realtime history cap (core/http/endpoints/openai/realtime.go):
   - Session grows MaxHistoryItems int; default picked by new
     defaultMaxHistoryItems(cfg) — 6 for realtime_audio models (LFM2.5
     1.5B degrades quickly past a handful of turns), 0/unlimited for
     legacy pipelines composing larger LLMs.
   - triggerResponse runs conv.Items through trimRealtimeItems before
     building conversationHistory. Helper walks the cut left if it
     would orphan a function_call_output, so tool result + call pairs
     stay intact.
   - realtime_gate_test.go: specs for defaultMaxHistoryItems and
     trimRealtimeItems (zero cap, under cap, over cap, tool-call pair
     preservation).

3. Talk page (core/http/react-ui/src/pages/Talk.jsx):
   - Reuses the chat page's MCP plumbing — useMCPClient hook,
     ClientMCPDropdown component, same auto-connect/disconnect effect
     pattern. No bespoke tool registry, no new REST endpoints; tools
     come from whichever MCP servers the user toggles on, exactly as
     on the chat page.
   - sendSessionUpdate now passes session.tools=getToolsForLLM(); the
     update re-fires when the active server set changes mid-session.
   - New response.function_call_arguments.done handler executes via
     the hook's executeTool (which round-trips through the MCP client
     SDK), then replies with conversation.item.create
     {type:function_call_output} + response.create so the model
     completes its turn with the tool output. Mirrors chat's
     client-side agentic loop, translated to the realtime wire shape.

UI changes require a LocalAI image rebuild (Dockerfile:308-313 bakes
react-ui/dist into the runtime image). Backend.py changes can be
swapped live in /backends/<id>/backend.py + /backend/shutdown.

Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): LocalAI Assistant ("Manage Mode") for the Talk page

Mirrors the chat-page metadata.localai_assistant flow so users can ask the
realtime model what's loaded / installed / configured. Tools are run
server-side via the same in-process MCP holder that powers the chat
modality — no transport switch, no proxy, no new wire protocol.

Wire:
- core/http/endpoints/openai/realtime.go:
  - RealtimeSessionOptions{LocalAIAssistant,IsAdmin}; isCurrentUserAdmin
    helper mirrors chat.go's requireAssistantAccess (no-op when auth
    disabled, else requires auth.RoleAdmin).
  - Session grows AssistantExecutor mcpTools.ToolExecutor.
  - runRealtimeSession, when opts.LocalAIAssistant is set: gate on admin,
    fail closed if DisableLocalAIAssistant or the holder has no tools,
    DiscoverTools and inject into session.Tools, prepend
    holder.SystemPrompt() to instructions.
  - Tool-call dispatch loop: when AssistantExecutor.IsTool(name), run
    ExecuteTool inproc, append a FunctionCallOutput to conv.Items, skip
    the function_call_arguments client emit (the client can't execute
    these — it doesn't know about them). After the loop, if any
    assistant tool ran, trigger another response so the model speaks the
    result. Mirrors chat's agentic loop, driven server-side rather than
    via client round-trip.

- core/http/endpoints/openai/realtime_webrtc.go: RealtimeCallRequest
  gains `localai_assistant` (JSON omitempty). Handshake calls
  isCurrentUserAdmin and builds RealtimeSessionOptions.

- core/http/react-ui/src/pages/Talk.jsx: admin-only "Manage Mode"
  checkbox under the Tools dropdown; passes localai_assistant: true to
  realtimeApi.call's body, captured in the connect callback's deps.

Mirroring chat's pattern means the in-process MCP tools surface "just
works" for the Talk page without exposing a Streamable-HTTP MCP endpoint
(which was the alternative). Clients with their own MCP servers can
still use the existing ClientMCPDropdown path in parallel; the realtime
handler distinguishes them by AssistantExecutor.IsTool() at dispatch
time.

Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): render Manage Mode tool calls in the Talk transcript

Previously the realtime endpoint only emitted response.output_item.added
for the FunctionCall item, and Talk.jsx's switch ignored the event — so
server-side tool runs were invisible in the UI. The model would speak
the result but the user had no way to see what tool was actually
called.

realtime.go: after executing an assistant tool inproc, emit a second
output_item.added/.done pair for the FunctionCallOutput item. Mirrors
the way the chat page displays tool_call + tool_result blocks.

Talk.jsx: handle both response.output_item.added and .done. Render
FunctionCall (with arguments) and FunctionCallOutput (pretty-printed
JSON when possible) as two transcript entries — `tool_call` with the
wrench icon, `tool_result` with the clipboard icon, both in mono-space
secondary-colour. Resets streamingRef after the result so the next
assistant text delta starts a fresh transcript entry instead of
appending to the previous turn.

Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor(realtime): bound the Manage Mode tool-loop + preserve assistant tools

Fallout from a review pass on the Manage Mode patches:

- Bound the server-side agentic loop. triggerResponse used to recurse on
  executedAssistantTool with no cap — a model that kept calling tools
  would blow the goroutine stack. New maxAssistantToolTurns = 10 (mirrors
  useChat.js's maxToolTurns). Public triggerResponse is now a thin shim
  over triggerResponseAtTurn(toolTurn int); recursion increments the
  counter and stops at the cap with an xlog.Warn.

- Preserve Manage Mode tools across client session.update. The handler
  used to blindly overwrite session.Tools, so toggling a client MCP
  server mid-session silently wiped the in-process admin tools. Session
  now caches the original AssistantTools slice at session creation and
  the session.update handler merges them back in (client names win on
  collision — the client is explicit).

- strconv.ParseBool for the localai_assistant query param instead of
  hand-rolled "1" || "true". Mirrors LocalAIAssistantFromMetadata.

- Talk.jsx: render both tool_call and tool_result on
  response.output_item.done instead of splitting them across .added and
  .done. The server's event pairing (added → done) stays correct; the
  UI just doesn't need to inspect both phases of the same item. One
  switch case instead of two, no behavioural change.

Out of scope (noted for follow-ups): extract a shared assistant-tools
helper between chat.go and realtime.go (duplication is small enough
that two parallel implementations stay readable for now), and an i18n
key for the Manage Mode helper text (Talk.jsx doesn't use i18n
anywhere else yet).

Assisted-by: claude-code:claude-opus-4-7-1m [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci(test-extra): wire liquid-audio backend smoke test

The backend ships test.py + a `make test` target and is listed in
backend-matrix.yml, so scripts/changed-backends.js already writes a
`liquid-audio=true|false` output when files under backend/python/liquid-audio/
change. The workflow just wasn't reading it.

- Expose the `liquid-audio` output on the detect-changes job
- Add a tests-liquid-audio job that runs `make` + `make test` in
  backend/python/liquid-audio, gated on the per-backend detect flag

The smoke covers Health() and LoadModel(mode:finetune); fine-tune mode
short-circuits before any HuggingFace download (backend.py:192), so the
job needs neither weights nor a GPU. The full-inference path remains
gated on LIQUID_AUDIO_MODEL_ID, which CI doesn't set.

The four new Go test files (core/gallery/importers/liquid-audio_test.go,
core/http/endpoints/openai/realtime_gate_test.go,
core/http/routes/ui_pipeline_models_test.go, pkg/functions/parse_lfm2_test.go)
are already picked up by the existing test.yml workflow via `make test` →
`ginkgo -r ./pkg/... ./core/...`; their packages all carry RunSpecs entries.

Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
 2026-05-13 21:57:27 +02:00
Ettore Di Giacinto 59108fbe32 feat: add distributed mode (#9124) 
* feat: add distributed mode (experimental)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix data races, mutexes, transactions

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactorings

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix events and tool stream in agent chat

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* use ginkgo

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(cron): compute correctly time boundaries avoiding re-triggering

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* enhancements, refactorings

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* do not flood of healthy checks

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* do not list obvious backends as text backends

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* tests fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactoring and consolidation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Drop redundant healthcheck

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* enhancements, refactorings

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-30 00:47:27 +02:00
Ettore Di Giacinto 031a36c995 feat: inferencing default, automatic tool parsing fallback and wire min_p (#9092) 
* feat: wire min_p

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat: inferencing defaults

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(refactor): re-use iterative parser

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: generate automatically inference defaults from unsloth

Instead of trying to re-invent the wheel and maintain here the inference
defaults, prefer to consume unsloth ones, and contribute there as
necessary.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: apply defaults also to models installed via gallery

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: be consistent and apply fallback to all endpoint

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-22 00:57:15 +01:00
Ettore Di Giacinto c3174f9543 chore(deps): bump llama-cpp to 'a0bbcdd9b6b83eeeda6f1216088f42c33d464e38' (#9079) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-20 08:12:21 +01:00
Ettore Di Giacinto b2f81bfa2e feat(functions): add peg-based parsing and allow backends to return tool calls directly (#8838) 
* feat(functions): add peg-based parsing

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat: support returning toolcalls directly from backends

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore: do run PEG only if backend didn't send deltas

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-08 22:21:57 +01:00
Ettore Di Giacinto 09ddaf94b2 feat(ui): move to React for frontend (#8772) 
* feat(ui): move to React

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add import model

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* syntax highlight

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Minor fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-05 21:47:12 +01:00
Ettore Di Giacinto 1fc8ad854f fix(toolcall): consider also literal \n between tags 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-03-01 11:20:46 +01:00
Ettore Di Giacinto 00abf1be1f fix(qwen3.5): add qwen3.5 preset and mimick llama.cpp's PEG (#8668) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-02-27 12:15:00 +01:00
Ettore Di Giacinto 34e054f607 fix(reasoning): support models with reasoning without starting thinking tag (#8132) 
* chore: extract reasoning to its own package

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* make sure we detect thinking tokens from template

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Allow to override via config, add tests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-01-20 21:07:59 +01:00
Ettore Di Giacinto d20a113aef fix(functions): do not duplicate function when valid JSON is inside XML tags (#8043) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-01-14 23:42:00 +01:00
Ettore Di Giacinto c88074a19e feat(api): support 'reasoning' api field (#7959) 
This PR adds support to support the 'reasoning' API field of the OpenAI
spec.

LocalAI now will extract automatically thinking tags in both SSE and
non-SSE mode. The changes are adapted as well to the Chat UI now that
will use the reasoning field to extract the thinking process and display
it in the chat.

This fixes https://github.com/mudler/LocalAI/issues/7944

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-01-10 19:06:12 +01:00
Ettore Di Giacinto 21c84f432f feat(function): Add tool streaming, XML Tool Call Parsing Support (#7865) 
* feat(function): Add XML Tool Call Parsing Support

Extend the function parsing system in LocalAI to support XML-style tool calls, similar to how JSON tool calls are currently parsed. This will allow models that return XML format (like <tool_call><function=name><parameter=key>value</parameter></function></tool_call>) to be properly parsed alongside text content.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* thinking before tool calls, more strict support for corner cases with no tools

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Support streaming tools

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Iterative JSON

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Iterative parsing

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Consume JSON marker

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fixup

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* add tests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fix pending TODOs

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Don't run other parsing with ParseRegex

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2026-01-05 18:25:40 +01:00
Ettore Di Giacinto c37785b78c chore(refactor): move logging to common package based on slog (#7668) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-12-21 19:33:13 +01:00
Ettore Di Giacinto 3728552e94 feat: import models via URI (#7245) 
* feat: initial hook to install elements directly

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* WIP: ui changes

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Move HF api client to pkg

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add simple importer for gguf files

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add opcache

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* wire importers to CLI

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add omitempty to config fields

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fix tests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add MLX importer

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Small refactors to star to use HF for discovery

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add tests

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Common preferences

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Add support to bare HF repos

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(importer/llama.cpp): add support for mmproj files

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* add mmproj quants to common preferences

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fix vlm usage in tokenizer mode with llama.cpp

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-11-12 20:48:56 +01:00
Ettore Di Giacinto 02cc8cbcaa feat(llama.cpp): consolidate options and respect tokenizer template when enabled (#7120) 
* feat(llama.cpp): expose env vars as options for consistency

This allows to configure everything in the YAML file of the model rather
than have global configurations

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(llama.cpp): respect usetokenizertemplate and use llama.cpp templating system to process messages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* WIP

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Detect template exists if use tokenizer template is enabled

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Better recognization of chat

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fixes to support tool calls while using templates from tokenizer

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Drop template guessing, fix passing tools to tokenizer

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Extract grammar and other options from chat template, add schema struct

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* WIP

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* WIP

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Automatically set use_jinja

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Cleanups, identify by default gguf models for chat

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Update docs

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-11-07 21:23:50 +01:00
Robbie Cronin 639ecb59b3 fix: handle multi-type arrays in JSON schema to prevent panic (#6495) 
Signed-off-by: robert-cronin <robert.owen.cronin@gmail.com>
 2025-10-17 11:07:37 +02:00
Ettore Di Giacinto df46a438b8 fix(grammars): handle empty parameters on object types (#6409) 
fix: handle empty parameters on object types

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-10-08 19:13:40 +02:00
leopardracer ce9a9a30e0 Improve Comments and Documentation for MixedMode and ParseJSON Functions (#5626) 
Update parse.go

Signed-off-by: leopardracer <136604165+leopardracer@users.noreply.github.com>
 2025-06-11 09:46:53 +02:00
Ettore Di Giacinto 67f7bffd18 chore(deps): update llama.cpp and sync with upstream changes (#4950) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-03-06 00:40:58 +01:00
Ettore Di Giacinto 1d6afbd65d feat(llama.cpp): Add support to grammar triggers (#4733) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2025-02-02 13:25:03 +01:00
Maximilian Kenfenheuer b4b67e00bd refactor: function argument parsing using named regex (#4708) 
Signed-off-by: Maximilian Kenfenheuer <maximilian.kenfenheuer@ksol.it>
 2025-01-28 22:58:02 +01:00
Maximilian Kenfenheuer 539e94db73 feat: function argument parsing using named regex (#4700) 
Signed-off-by: Maximilian Kenfenheuer <maximilian.kenfenheuer@ksol.it>
 2025-01-27 15:53:05 +00:00
Peter Cover a05737c7e4 chore: fix some function names in comment (#4665) 
Signed-off-by: petercover <raowanxiang@outlook.com>
 2025-01-22 19:35:53 +01:00
Ettore Di GiacintoandDave e198347886 feat(openai): add json_schema format type and strict mode (#3193) 
* feat(openai): add json_schema and strict mode

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* handle err vs _

security scanners prefer if we put these branches in, and I tend to agree.

Signed-off-by: Dave <dave@gray101.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Dave <dave@gray101.com>
Co-authored-by: Dave <dave@gray101.com>
 2024-08-07 15:27:02 -04:00
Ettore Di Giacinto 2169c3497d feat(grammar): add llama3.1 schema (#3015) 
* wip

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* get rid of panics

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* expose it properly from the config

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Simplify

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* forgot to commit

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Remove focus on test

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Small fixups

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-07-26 20:11:29 +02:00
Ettore Di GiacintoandDave 5eda7f578d refactor: break down json grammar parser in different files (#3004) 
* refactor: break down json grammar parser in different files

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix: patch to `refactor_grammars` - propagate errors (#3006)

propagate errors around

Signed-off-by: Dave Lee <dave@gray101.com>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Signed-off-by: Dave Lee <dave@gray101.com>
Co-authored-by: Dave <dave@gray101.com>
 2024-07-25 08:41:00 +02:00
Ettore Di Giacinto bf9dd1de7f feat(functions): parse broken JSON when we parse the raw results, use dynamic rules for grammar keys (#2912) 
* feat(functions): enhance parsing with broken JSON when we parse the raw results

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* breaking: make function name by default

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(grammar): dynamically generate grammars with mutating keys

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor: simplify condition

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Update docs

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-07-18 17:52:22 +02:00
Ettore Di Giacinto 3eaf59021c feat(grammar): expose properties_order (#2662) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-06-26 14:59:02 +02:00
Sertaç Özercan 5866fc8ded chore: fix go.mod module (#2635) 
Signed-off-by: Sertac Ozercan <sozercan@gmail.com>
 2024-06-23 08:24:36 +00:00
Ettore Di Giacinto 5d31e5269d feat(functions): allow response_regex to be a list (#2447) 
feat(functions): allow regex match to be a list

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-31 22:52:02 +02:00
Ettore Di Giacinto 3f7212c660 feat(functions): better free string matching, allow to expect strings after JSON (#2445) 
Allow now any non-character, both as suffix and prefix when mixed grammars are enabled

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-31 09:36:27 +02:00
Ettore Di Giacinto 669cd06dd9 feat(functions): allow parallel calls with mixed/no grammars (#2432) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-28 21:06:09 +02:00
Ettore Di Giacinto ea330d452d models(gallery): add mistral-0.3 and command-r, update functions (#2388) 
* models(gallery): add mistral-0.3 and command-r, update functions

Add also disable_parallel_new_lines to disable newlines in the JSON
output when forcing parallel tools. Some models (like mistral) might be
very sensible to that when being used for function calling.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* models(gallery): add aya-23-8b

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-23 19:16:08 +02:00
Ettore Di Giacinto 491e1d752b feat(functions): relax mixedgrammars (#2365) 
* feat(functions): relax mixedgrammars

Extend even more the functionalities and when mixed mode is enabled,
tolerate also both strings and JSON in the result - in this case we make
sure that the JSON can be correctly parsed.

This also updates the examples and the gallery model to configure the
grammar.

The changeset also breaks current function/grammar configuration as it
reserves now a stanza in the YAML config.

For example:

```yaml
function:
  grammar:
    # This allows the grammar to also return messages
    mixed_mode: true
    # Suffix to add to the grammar
    # prefix: '<tool_call>\n'
    # Force parallel calls in the grammar
    # parallel_calls: true
```

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor, add a way to disable mixed json and freestring

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* Fix linting issues

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-22 00:14:16 +02:00
Ettore Di Giacinto 5a6d120a56 feat(functions): don't use yaml.MapSlice (#2354) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-20 08:31:06 +02:00
Ettore Di Giacinto 73566a2bb2 feat(functions): allow to use JSONRegexMatch unconditionally (#2349) 
* feat(functions): allow to use JSONRegexMatch unconditionally

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(functions): make json_regex_match a list

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-19 18:24:49 +02:00
lenaxiaLenaxiamudler
 		6b6c8cdd5f feat(functions): Enable true regex replacement for the regexReplacement option (#2341) 
* Adding regex capabilities to ParseFunctionCall replacement

Signed-off-by: Lenaxia <github@47north.lat>

* Adding tests for the regex replace in ParseFunctionCall

Signed-off-by: Lenaxia <github@47north.lat>

* Fixing tests and adding a test case to validate double quote replacement works

Signed-off-by: Lenaxia <github@47north.lat>

* Make Regex replacement stable, drop lookaheads

Signed-off-by: mudler <mudler@localai.io>

---------

Signed-off-by: Lenaxia <github@47north.lat>
Signed-off-by: mudler <mudler@localai.io>
Co-authored-by: Lenaxia <github@47north.lat>
Co-authored-by: mudler <mudler@localai.io>
 2024-05-19 01:29:10 +02:00
Ettore Di Giacinto 02f1b477df feat(functions): simplify parsing, read functions as list (#2340) 
Signed-off-by: mudler <mudler@localai.io>
 2024-05-18 09:35:28 +02:00
Ettore Di Giacinto beb598e4f9 feat(functions): mixed JSON BNF grammars (#2328) 
feat(functions): support mixed JSON BNF grammar

This PR provides new options to control how functions are extracted from
the LLM, and also provides more control on how JSON grammars can be used
(also in conjunction).

New YAML settings introduced:

- `grammar_message`: when enabled, the generated grammar can also decide
  to push strings and not only JSON objects. This allows the LLM to pick
to either respond freely or using JSON.
- `grammar_prefix`: Allows to prefix a string to the JSON grammar
  definition.
- `replace_results`: Is a map that allows to replace strings in the LLM
  result.

As an example, consider the following settings for Hermes-2-Pro-Mistral,
which allow extracting both JSON results coming from the model, and the
ones coming from the grammar:

```yaml
function:
  # disable injecting the "answer" tool
  disable_no_action: true
  # This allows the grammar to also return messages
  grammar_message: true
  # Suffix to add to the grammar
  grammar_prefix: '<tool_call>\n'
  return_name_in_function_response: true
  # Without grammar uncomment the lines below
  # Warning: this is relying only on the capability of the
  # LLM model to generate the correct function call.
  # no_grammar: true
  # json_regex_match: "(?s)<tool_call>(.*?)</tool_call>"
  replace_results:
    "<tool_call>": ""
    "\'": "\""
```

Note: To disable entirely grammars usage in the example above, uncomment the
`no_grammar` and `json_regex_match`.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-15 20:03:18 +02:00
Ettore Di Giacinto 84e2407afa feat(functions): allow to set JSON matcher (#2319) 
Signed-off-by: mudler <mudler@localai.io>
 2024-05-14 09:39:20 +02:00
Ettore Di Giacinto c4186f13c3 feat(functions): support models with no grammar and no regex (#2315) 
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
 2024-05-14 00:32:32 +02:00