mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-05 12:03:27 -05:00
cfafec79cabc7da7e8a85908dffcd3e8747df70f
48
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler-agentandEttore Di Giacinto
|
2cadf319a0 |
feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958)
* chore(backends): remove the liquid-audio Python backend The backend targets liquid-audio, which loads LFM2 checkpoints only. Remove its directory, the gallery file and the LFM2.5-Audio model entries, the library and generated backend gallery entries, the backend matrix and CI job, the Makefile targets and the golden test fixtures that listed it. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): drop liquid-audio from capabilities, importers and UI With the liquid-audio backend gone, remove its capability entry, its importer, its artifact-gate entry and the realtime_audio heuristic that only matched that backend. realtime_audio is now set only through known_usecases. Update the Talk and fine-tuning pages, the docs and the swagger text that named the backend. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to e486a4b5 Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d, the master head of 2026-10-28. It contains the LFM3-Audio support (ggml-org/llama.cpp#31257) and the realtime server split (ggml-org/llama.cpp#31269). The score task patch is regenerated against the new pin. It applies with git apply and with patch -p1, with no offset and no fuzz. Its hunks in server-task.h and server-context.cpp were rebased by hand next to SERVER_TASK_TYPE_REALTIME and the control dispatch. The C++ backend was built on CPU for this change: the llama, mtmd and server-context libraries and the grpc-server binary link. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): compile server-realtime.cpp into grpc-server server-context.cpp includes server-realtime.h, and its realtime slot type is defined in server-realtime.cpp. Upstream compiles that file into the server library. The gRPC server is one translation unit, so it must include the file too, or the link fails. The include is guarded by __has_include. Forks that predate the realtime split do not ship the file, so they keep building as before. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): probe realtime and TTS voice support in prepare.sh prepare.sh writes llama_compat.h from probes on the checkout. Add two: LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the gRPC server choose the realtime and voice code paths per checkout, so forks without them keep compiling. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): advertise speech recognition and speech-to-speech llama-cpp now registers the AudioTranscription and AudioToAudioStream RPCs, with the transcript and realtime_audio usecases. The default usecase stays chat, so a plain GGUF is still a chat model. The backend implements the RPCs in a follow-up change; until then the calls return UNIMPLEMENTED. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): add pipeline.any_to_any for speech-to-speech models A realtime_audio model that serves the whole speech loop itself sets pipeline.any_to_any. The flag is off by default, so chained pipelines and existing realtime_audio configs are unchanged. The realtime_audio heuristic stays explicit: a llama-cpp model is never a Talk-page model unless known_usecases says so. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): read the audio encoder and speech-to-speech projector flags Add HasAudioEncoderProjector for clip.has_audio_encoder and HasSpeechToSpeechProjector, which requires the LFM3 audio encoder family together with the gen-audio decoder. Qwen3-TTS sets both encoder flags too, so the flags alone cannot tell it apart from LFM3. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(importer): import LFM3-Audio projectors as speech-to-speech An mmproj that carries the LFM3 audio encoder and the gen-audio decoder now imports with known_usecases realtime_audio, chat, transcript and tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS included. The probe runs against a local HTTP server in the tests. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt Models without a speaker reference (LFM3-Audio) take a named voice through their "tts" chat template instead of a reference file. The request is validated with needs_reference=false, the prompt is rendered with common_chat_get_tts_prompt(), and the voice is set on the gen-audio input. Qwen3-TTS Base keeps the reference-file rule. The branch is guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the old behaviour. Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3 gets a system turn that names the language and an empty user turn. Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a caller prompt is still appended after a newline. The preset call is guarded by a new LOCALAI_HAS_ASR_PRESET probe. The pure parts sit in tts_request_options.h and transcription_prompt.h, with standalone tests run by backend/cpp/run-unit-tests.sh. Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts, ttsstream and transcription specs against LFM3-Audio GGUF weights. It is a manual run and is not wired into CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): mock AudioToAudioStream and pending realtime S2S specs The mock backend serves AudioToAudioStream deterministically: session created, commit with RMS, a tool call on the first turn with tools, then transcript pieces and paced 24 kHz tone frames, with response.cancel ending a reply as cancelled. The realtime S2S specs cover the mock and real-weights variants. They skip until the /v1/realtime S2S path lands, so the mock suite stays green in CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot Implement AudioToAudioStream on upstream's realtime slot, so a model that answers after the audio (LFM3-Audio) can hold a conversation over the existing duplex RPC. The proto schema does not change: tools travel in the config, tool calls leave as function_call_arguments.done and tool results arrive as a function_call_output item. realtime_session.h keeps what needs no model: the turn text derived from the chat template (with a tool turn), the conversation state, a filter that keeps <tool_call> blocks out of the transcript and drops audio produced inside them, and the translation of slot results into events. grpc-server.cpp wires it to the slot with control tasks and parses calls with the parser the chat template selects. The unit test uses a fake renderer. A ctest-only check renders the real LFM3 template and compares the slot text with the full render. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the LFM3-Audio-2B entries Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both with the Q8_0 audio projector. Each declares pipeline.any_to_any and the realtime_audio, chat, transcript and tts usecases the importer writes for an LFM3 projector, with parallel:2 and the tokenizer template. The Q4 entry lists the Q8 one as a variant. File sizes and sha256 come from the Hugging Face API for the pinned repo revision. Speaker reference files and the VAD model are not part of the entries: VAD stays optional. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe LFM3-Audio speech-to-speech and its gallery entries Adds a speech-to-speech section to the Realtime API page (any_to_any configuration, VAD versus manual commit, tools, transcripts, barge-in, limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in the compatibility table, and LFM3-Audio mentions on the TTS and ASR pages. The feature is marked experimental: the behaviour follows the backend session contract and has not been run with real weights. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): run any-to-any models on AudioToAudioStream A realtime_audio model with pipeline.any_to_any now opens one duplex stream to the backend and keeps it for the session, replacing the "any-to-any models not implemented" branch. The chained pipeline is unchanged. The endpoint forwards input audio (with a pre-roll from speech start when a VAD model is set, otherwise every chunk and a client commit), sends response.create, and turns the backend's events into the usual realtime events: audio and transcript deltas, tool calls through the existing function_call items, and a single response.done. A function call output is forwarded before the next response. Barge-in sends response.cancel ahead of the new speech and stops forwarding audio at once. The user's speech is transcribed on the side and reported on the user item; a failure there does not touch the response. Server VAD is on only when pipeline.vad is set. Classifier mode, compaction, streaming.llm, voice gating, sound detection and semantic_vad are rejected for these models. The mock backend now matches the llama.cpp contract (session.updated, the cancel error when idle) and its VAD reports the loud stretches of the audio, so the S2S e2e specs run a VAD turn. The specs are enabled. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec The realtime S2S specs now check reply audio on real models: duration, loudness, clipping, and keywords found in the model transcript or in a separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR and TTS models, then runs the real variant. It is manual and never part of CI: `make test-realtime` still filters out real-models. The backend suite gains an s2s capability (spoken turn, tool call, tool result, spoken answer over AudioToAudioStream) and optional TTS duration and RMS bounds plus a transcript keyword check, all enabled for test-extra-backend-llama-cpp-lfm3. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): expect the idle-cancel error from the mock S2S The mock now answers a response.cancel with no reply running with response_cancel_not_active, as the llama.cpp backend does. The cancel spec still treated that cancel as a no-op and read the error as extra data after the stream ended. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(talk): show any-to-any models as one speech-to-speech model /api/pipeline-models lists a pipeline.any_to_any model with any_to_any set and its optional VAD. A realtime_audio model that is not any-to-any is listed only when it has the four pipeline slots, since the session takes the chained path for it. The Talk page shows a speech-to-speech model as one model: no four-slot view, the VAD slot only when a VAD model is set, and the female and male voices offered. Without a VAD model the server does not detect turns, so the page offers Done speaking, which sends input_audio_buffer.commit and then response.create. Text-only replies stream through response.output_text.delta and .done. self_contained stays in the response for clients built before any_to_any. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty pipeline slots with the model name failed later with UNIMPLEMENTED. Fail at session start and name the missing setting instead. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): hold manual-commit S2S audio until the commit In manual-commit mode the endpoint streamed every appended chunk to the backend, so input_audio_buffer.clear only emptied the Go buffer while the model kept the audio, and idle audio filled its context between turns. Keep the audio in the input buffer and send it, behind pending text and tool-result items, when the client commits or asks for a response. Frames and controls now go through one ordered write queue, so no gRPC send happens under the audio buffer lock. A tool result created while a VAD turn is open is refused with a clear error. When the stream ends outside a response the session reports it and closes, and a bare EOF from Send reports the status that Recv holds. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): drop the S2S readiness gate and stale comments The gate was always true. Some comments referred to work items that readers of the repository cannot see. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): add the liquid-audio migration note and the clear semantics Say that the Python liquid-audio backend is gone, which gallery entries replace it and that LFM2.5-Audio has no replacement. Describe what input_audio_buffer.clear drops for speech-to-speech sessions now that manual-commit audio is held back until the commit. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): do not promise turn detection when no VAD model is set The idle note said the server notices when you stop talking. A speech-to-speech model without a VAD model needs Done speaking, so the note now says that. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): make the mock S2S backend keep the llama-cpp session limits The mock accepted what the real backend refuses and hid two endpoint bugs. It now treats output_modalities from the config as a fixed session mode and picks the default per response from the tools set at that time. A fixed text plus audio mode with tools reads the tool call out loud. After the first commit or message, session.update of instructions, voice or tools is an error that does not belong to a response, committing an empty buffer is an error, and response.create while a reply runs is refused. Errors of a refused response.create carry response_scoped in their meta. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): leave the S2S output mode to the backend unless the client sets one The stream config sent text plus audio when the session had no tools yet. The llama-cpp backend keeps a mode from the config as a fixed session mode, and tools added later by session.update never cleared it. The model then read tool calls out loud. Send output_modalities only when the client chose them. The backend picks the default of each response from the tools it has when the response starts. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): mark the realtime errors that answer a response.create The error event now carries response_scoped: true when it ends the request for a response (a refused response.create, a failed commit made by it, or a stream failure). Other errors, such as a refused session.update or a commit of an empty buffer, are not about a response and one that runs keeps generating. The client needs the difference to know whether a response.done is still coming. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S response only on the errors that belong to it Every backend error ended the response in flight, although the backend also reports errors about other things (a refused session.update, a commit of an empty buffer) and goes on generating. The rest of the reply was lost and the next response.create met either a refusal or the leftover events of the old reply. End the response only on an error marked response_scoped, forward the others to the client and keep waiting for response.done. Drop the events buffered by an earlier response when a new one starts, and skip the commit of a turn that forwarded no audio. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): run the S2S input transcription detached from the commit slot The commit waited for the side transcription of the user audio while it held the commit slot, and response.create waits for that slot. The ASR latency was added before every reply, and on a backend with few slots a busy ASR task could stall the reply. The model already has the audio, so run the transcription detached under the session context and release the slot at once. The transcript is still reported on the user item. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): do not pin the place of the input transcript in the S2S order The transcript of the user audio is reported on the side and may land before or after the first response events. Check the order of the response events without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue The reader waited on a channel it closes itself when the events buffer was full, so it outlived the session after a turn ended on an error or timed out. It now also stops when the session closes. Stopping a response sent the cancel through the write queue and waited for it before the grace timer started. With backpressure on the stream the wait never ended and nothing read the events meanwhile. The timer now starts first and the cancel is only queued. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): let a chained pipeline run without a VAD model A pipeline that names no pipeline.vad failed to build, because the VAD slot was always loaded, and a new session still started with server_vad. The slot is now optional for chained pipelines as it already is for any-to-any models: the session starts with turn detection off and the client commits each turn. Turning server_vad on for such a session is refused, and a switch to a pipeline without a VAD drops turn detection. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add LFM3-Audio VAD and chained-pipeline entries Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the -q8 forms (the model chained through its own transcription, chat and speech slots, with and without the VAD). The gallery has no dependencies between models, so the VAD entries state that silero-vad-ggml must be installed first. Drop the variants list from lfm3-audio-2b: it let the installer pick the larger Q8 build on any host with enough memory, so a plain install of the Q4 entry did not get Q4. All entries are now independent. Tests cover the entry shapes, the single-build install and their acceptance by the realtime endpoint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): run the realtime specs for any-to-any and chained shapes The realtime specs covered one any-to-any model with a VAD. They now run for any-to-any and for a chained pipeline, each with and without a VAD, on the mock backend in CI. VAD-less shapes also check that turn detection starts off and that server_vad is refused. The manual real-weights script writes one config per shape that mirrors the gallery entries and runs the four shapes in turn (LFM3_MATRIX picks a subset). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml must be installed before the -vad entries, describe the chained pipeline on one model and the behaviour of a pipeline without a VAD, and describe the shape matrix of the e2e specs. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): wait for the detached input transcription in the S2S specs The input transcription runs in its own goroutine, so the transcription completed and failed events can arrive after the response ends. Poll for them instead of reading the event list right after the response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): release the S2S stream before running server-side tool calls A response held the stream slot until it returned, and it ran the tool calls from inside that scope. A server-side tool starts a follow-up response on the same stream, which then waited for the slot its parent still held, so the client never got response.done. Run the tool calls once the slot is free; the event order is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S session whose stream dies, and stop queuing after close A stream that died during a response left the session open: later requests failed one by one and VAD frames kept being forwarded. The response now ends the session through the transport, and enqueue drops requests once the session is closed so the write queue cannot grow. The writer is now the only caller of Send and of CloseSend. close() used to call CloseSend while the writer could be inside Send, which gRPC does not allow. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): close the open S2S turn when the VAD stops Switching turn detection to manual in the middle of a VAD turn left the turn forwarding, and the next client commit sent the whole buffer a second time. Stopping the VAD now commits the forwarded audio on the backend and drops it from the input buffer. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): keep a tool result pending while an S2S user turn is open The check that refuses a tool result during a user turn ran when the item arrived, but the item was forwarded later, and the result was marked as sent before the backend took it. If the user spoke in between, the backend refused it and it was never retried. The result is now claimed and queued under the audio lock only when no turn is open, and otherwise stays pending for the next commit or response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): empty the input buffer for models without a VAD Without voice detection the microphone fills the server's input buffer for the whole session, so Done speaking committed idle silence and the echo of the previous reply. Clear the buffer when the session starts and when a reply ends. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importer): keep an existing parallel option on LFM3-Audio imports The speech-to-speech import added parallel:2 even when the options already carried another parallel value. Use the same check as the hardware defaults, which also covers n_parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): list chained pipelines that have no VAD /api/pipeline-models required a VAD for a chained pipeline, so the VAD-less LFM3-Audio pipeline entries never showed up on the Talk page. Any pipeline with transcription, LLM and speech slots now qualifies. The Talk page treats every model without a VAD as manual-turn, and the session sheet says so for chained pipelines too. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to 0ac1a7fd Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b, the master head of 2026-10-29. It contains tool calls in the realtime server (ggml-org/llama.cpp#31287). The score task patch is regenerated so that it applies with no offset and no fuzz. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep a null tool argument null A tagged parameter whose text was "null" came out as an empty string: unmarshalling JSON null into a string succeeds and leaves it empty, so the value was taken for a string. Check the first byte of the JSON value instead. A parameter with the text "null" now becomes JSON null, as in llama.cpp. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(functions): add an lfm3 XML format preset The LFM3 chat template writes a call as <tool_call>, <function=NAME>, one <parameter=KEY> block per argument, then the closing tags. The qwen3-coder format has the same shape and auto-detection already found it, but xml_format_preset: lfm3 was not a known name. Add the preset, with tests for one call, a call without arguments, two calls, string and non-string values (numbers, booleans, null, arrays, objects), a call between text, and unfinished input. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): parse tool calls with the chat template, as upstream Since ggml-org/llama.cpp#31287 the realtime server handles tools with the parser and grammar of the chat template. Do the same in the speech-to-speech stream instead of a hard-coded tag filter and a parse of the whole text at the end. - Each response gets a common_chat_session with the tools of the session and parallel calls. Its grammar and preserved tokens go into the sampling of the response, so the slot keeps the call tokens in the text and constrains the call. - The text of the response is fed to the session piece by piece (task_result_state::update_chat_msg). Content deltas become the transcript; tool-call deltas are collected into one function_call_arguments.done per call at the end. Audio after the start of a call is dropped. - A tool_choice of auto, none or required is accepted in the params of the config, in session.update and in response.create, with the response overriding the session. - The tool turn is derived from a conversation in which the assistant called a tool, like upstream, not from a plain reply. - response.done carries raw_text, everything the model wrote with the call markup, for a caller that parses it another way. If the parser throws, the rest of the response is content and the raw text is the way out. ToolCallFilter is no longer used and goes in the next commit. The unit tests cover the splitter contract (content, call deltas, the final piece, whole characters only). realtime_chat_check feeds LFM3-format text through the session in pieces of 1, 7 and all bytes and checks the call, the transcript and the grammar. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): drop the tool-call text filter The parser of the chat template now splits content from tool calls, so ToolCallFilter, which looked for the literal <tool_call> tags, has no caller. Remove it with its tests. Every case it covered (a call between text, split tags, several calls, an unfinished call, text after a call) is handled by the parser, and realtime_chat_check runs the LFM3 format through it in pieces of different sizes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): fall back to the raw text for S2S tool calls For an any-to-any model, the calls the backend parsed still win. When there are none and the request had tools, parse the raw text that response.done now carries the way the chained pipeline parses the text of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so a call format the backend parser does not know can still be served with function.xml_format_preset or the other function settings of the model. The content around the calls replaces the streamed transcript only if the config captures it, as in the chained path. Also pass tool_choice to the backend: the session mode with the stream config and with session.update, and the mode of a response.create override with that event. A named function cannot be forced, so it is not sent. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): carry raw_text in the mock S2S response.done Match the llama-cpp session: with tools set, response.done has raw_text, the call markup of the LFM3 template for a tool turn and the transcript for any other reply. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): parse tool blocks that have text between them The iterative XML parser stopped at the first scope block that was preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave one call. Skip text between blocks once a call was found; text before the first block still ends the parse as before. Blocks that follow each other, or only whitespace, already worked. Tests cover three calls back to back and with text between them, with the lfm3 preset and with auto-detection. They also pin the current reading of argument values: the parser has no tool schema, so a value such as 42 or true is read as JSON even when the schema types it as a string. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): read LFM3-Audio tool calls with the lfm3 preset Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries and in the config the importer writes for a speech-to-speech projector, so the server parses the call format of the LFM3 chat template when the backend returns no call (and in the chained pipeline entries, which use the regular chat path). gallery/index.yaml is regenerated from the library. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): describe how S2S tool calls are read Replace the description of the hard-coded tag filter with the parser and grammar of the chat template, tool_choice, and the raw-text fallback in the server. Document function.xml_format_preset, and set it in the real-weights e2e model configs like the gallery entries do. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep earlier calls when a later block does not parse After a valid call, text that holds the opening tag again (a prose mention, a markdown example, a malformed second block) made the parser treat it as a new block, fail, and return no calls at all. Remember the state after each complete block and fall back to it when a later one fails: the calls found so far are kept, parsing stops there and the remaining text stays in the content. Text skipped between two valid blocks now goes to the content instead of being dropped, so StripToolCallMarkup keeps it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): do not drop audio after a tool call started In mixed mode the speech for text written before a call can still be pending when the call starts, so dropping every audio frame after the first call delta cut the speech while its transcript was sent. The llama.cpp server never drops audio when it reads a result; do the same. If a model speaks the call markup aloud, that is model behaviour to observe on real weights, not something to filter here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): flush held-back text when a response is cancelled The parser got is_partial=true on a cancelled response, so text it held back (a possible start of a tag) never reached the transcript. The llama.cpp server parses a cancelled response with is_partial=false; do the same. A cancelled response still makes no tool call. The path where the parser throws already sends the text as content and needs no change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): stop the fake S2S stream from sending after Close The barge-in spec failed now and then with a data race and "send on closed channel": its script pushes audio from a goroutine, and the session closes the stream while that goroutine still runs. A push now does nothing once the stream is closed, and Close waits for a push in flight. The production close order is right: the writer closes the stream only after it stops sending. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): align the S2S tool-call fallback with the backend Four gaps in how the Go side reads the tool calls of an any-to-any reply: - With tool_choice none (the response value wins over the session value) the backend builds no tool parser, yet the raw-text fallback still turned a call the model wrote into a function_call item. Skip it. - Without a capture regex the fallback never cleaned the transcript, so the stored item and transcript.done kept the raw call markup. When the call is XML, use the text left after the markup is cut out; a reply that is only a call now has an empty transcript. - A named function as tool_choice mapped to nothing, so the backend kept the previous mode. Refuse it with an error in session.update and response.create. - A response-level tools list changed what the fallback looked for, but the backend only has the tools of the session. Use those in both. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): say how S2S tool_choice and response tools behave Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(e2e): use the real llama-cpp backend in the real realtime run The suite always registered the mock backend under the name llama-cpp, so a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend found in REALTIME_BACKENDS_PATH. Register the mock only when that run is off. The real run converts the input audio with ffmpeg. Check for it in the suite and in run-realtime-lfm3.sh before any download or backend start, with a message that says what to install, and list it as a prerequisite in the script header and the Makefile target. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover the tools path of a client session.update to the S2S backend A tool call that never came back from an any-to-any model raised the question whether the tools of the session reach the backend at all. Add a spec that sends the session.update of the real-weights e2e (flat Responses API tool, no session type) and checks that the backend gets the tool in the chat-completions form. Log the number of tools sent when the stream opens and when a session.update carries them, so a run log shows it without reading the wire. The "No tools received" line of the llama-cpp backend does not show it: the server prints it for the input transcription request, which has no tools by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): interrupt the chained pipeline reply while it is in flight With unary stages, the default of the lfm3 pipeline gallery entries, the chained pipeline synthesizes the whole reply and sends it as one audio delta right before response.done. The barge-in spec waited for that delta and then rendered the interruption with the TTS, so the first response was already done when the speech started and ended completed instead of cancelled. Real runs on a GPU showed it with both quantizations. For a pipeline, start the interruption at response.created, which comes before the LLM and TTS stages. Render the interruption audio before the first utterance is sent, so nothing sits between the reply and the interruption. The any-to-any shape still waits for the first audio delta, which that backend streams while it generates. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): accept a number word and its homophone in the spoken-answer check The check that the reply audio says what the transcript says needs a shared word of 4+ letters. A short answer failed it: the model wrote "Four" and the ASR heard "for". Compare numbers in one spoken form, so digits, number words and their homophones match each other. A homophone only counts when the other text has the number itself, so "for" on both sides still shares nothing and the check does not pass on a stop word. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools The default system prompt of a realtime session was "Respond with interleaved text and audio." whatever the tools. Under that prompt LFM3-Audio answers in words ("I'll check the weather ...") and never writes the call. The model card, and the llama.cpp realtime example client, use "Respond with function calls, and interleaved text and audio." when the session has tools, and the model then calls the tool. Pick the default by the tools of the session, and treat the plain default that the LocalAI server sends back as no choice: it follows the tools too, also when they arrive with a session.update before the first audio. A prompt the client wrote is kept as it is. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): transcribe the input of an any-to-any turn between responses The input transcript of a speech-to-speech turn ran at the same time as the reply, on the same model. llama.cpp serves both from one context: the realtime slot wants embeddings on, the chat slot of the transcript wants them off, and each flip makes it reserve its compute buffers again. In a GB10 run that was two reserves of about 175 ms per step, the reply fell to 2.6 tokens per second and the transcript to 380 ms per token, until the transcript was done. When pipeline.transcription is the model of the stream, run the transcript after the response, or before one that is not started yet, and keep responses out while it runs. A separate transcription model still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): send the input transcript of an any-to-any turn before its reply The previous change moved the same-model transcript after the response. A client that reads until response.done then never saw conversation.item.input_audio_transcription.completed: the transcript was made (the log shows it) but sent after the end of the window. Run it in line with the commit instead. A VAD turn transcribes and then responds. A client response.create already waits for the commit slot, so it starts once the transcript is out. The transcript event now comes before any output of the response of its turn, for a manual commit and for a VAD turn, and never overlaps the response on the llama.cpp context. A transcription model of its own still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): transcribe the input of an any-to-any model only when asked The stream answers from the audio, so the transcript of the input is for the client only. On the model of the stream it is costly: llama.cpp serves the transcript request and the realtime stream from one context and rebuilds its compute buffers on every step while both are active. On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0 to 1 on the llama.cpp server, and the reply ran at a few tokens per second. An any-to-any model no longer fills pipeline.transcription with its own name. Without it the session does not transcribe the input; with another model it does, in parallel with the reply; with the model itself named it does, before the reply. The reply transcript always comes from the stream. The gallery entries set none, so they get the fast path. The mock configs and run-realtime-lfm3.sh name a transcription model, so the real-weights spec still covers the input transcript. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end a response as cancelled when a barge-in lands during the LLM call The chained pipeline cancels the response context on a barge-in. When that happens while the model is generating, the backend call fails with the context error, and the response ended as failed: an error event and no response.done. A client that waits for the terminal of the interrupted response then waits forever. On a GB10 the q4 model answered before the barge-in arrived, so only the slower q8 model showed it: the spec read events for 30 s and timed out. When the context is cancelled, a failed Predict call or a failed result is a cancel, as the TTS step and the streamed LLM path already treat it. The response ends with response.done{cancelled} and no error. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): wait for the input transcript that comes from a model of its own An any-to-any model now gets its input transcript from a separate ASR that runs next to the reply. The reply is often over first, so a spec that stops reading at response.done saw no transcript in the VAD shape. Keep reading, up to 30 s, until the transcript arrives. The contract is that it comes, not when. Add a guard spec for what the input transcription is handed: the whole turn, in a WAV that says the rate it was recorded at (the session rate for a client commit, 16 kHz for a VAD turn). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): check the errors of Close in the realtime specs golangci-lint (errcheck) flagged the unchecked Close of the websocket connections, the response body and the fake stream. Use the `defer func() { _ = x.Close() }()` form the other e2e specs use. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): write the mock S2S specs with Ginkgo and drop the env read forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the sanctioned places. Convert the mock S2S tests to Describe/It with a suite bootstrap (the package had none), with the same 11 cases. The mock no longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length is a package variable they change per spec and restore afterwards. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
6d7e8171be |
feat(agents): per-model output cap for agent job completions
Agent jobs (/api/agent/jobs, local and on distributed agent workers) run
cogito against a LocalAI client that sends no max_tokens of its own, so
every completion gets the client's fixed default of 16384. max_iterations
bounds the number of turns and loop_detection repeated tool calls, but
neither stops one completion that keeps generating, and 16384 is the same
for a small tool-picking model and a reasoning model that thinks at length.
Add agent.max_generation_tokens to the model config. When set, both job
paths apply it to the client with SetMaxTokens; requests that carry their
own cap still win, so cogito's length retry can raise it. Unset keeps
today's behaviour.
The specs run each job path against an MCP fixture and a recording LLM
endpoint and check the max_tokens the completions carry; they fail when
either call site drops the cap.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
pos-ei-don
|
073adc50c9 |
feat(sglang): map reasoning_effort to per-level thinking budgets
The Go side already forwards the request's effort as
Metadata["reasoning_effort"], but the sglang backend only read
enable_thinking: every level got the same model-level thinking_budget,
so low/medium/high produced identical reasoning.
- New model option `effort_budgets:low=…;medium=…;high=…` picks the
budget of the request's level; unlisted levels and requests without an
effort keep `thinking_budget`. Parsing reuses _parse_thinking_budget.
- A level listed in effort_budgets turns reasoning on for that request,
so levels the Go side does not know cannot silently disable it.
- ApplyReasoningEffort now treats `xhigh` and `max` as thinking levels.
Before, they left DisableReasoning unset, so with a model default of
"off" asking for more reasoning produced none.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit
|
||
|
pos-ei-don
|
a1e0faa9f0 |
docs(vllm): explain sampling validation
Describe stop strings, greedy decoding, and seed handling after sampling
parameters are validated during construction.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit
|
||
|
|
e7cc993b0f |
feat(llama-cpp): add lazy_mode model option (#12743)
* feat(llama-cpp): add lazy_mode model option Expose upstream --lazy-mode (on, auto, off) as a model option, for example `lazy_mode:on`. It controls on-demand reading of large tensors such as per-layer embeddings; `on` requires mmap. The option is also accepted as `lazy-mode` or `lzm`. Invalid values log a warning and keep the upstream default (auto). The pinned llama.cpp already provides the lazy_mode field in common_params. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to c0fd78626405 Update the default llama.cpp pin to the current upstream master. The changes since the old pin that the backend touches are additive: a new common_decision_type value, new mtmd audio-stream and download helper declarations. The two server patches still apply in order, and the common_params fields and server headers the backend uses are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
mudler-agentandEttore Di Giacinto
|
849c177834 |
feat(llama-cpp): bump llama.cpp and add moe_cache_mib option (#12729)
Bump LLAMA_VERSION to b76b13379d3b (upstream 0.7.0 plus later commits). This is the same one-line change as the bot PR #12725. Add the moe_cache_mib model option. It sets upstream's moe_cache_size, a GPU cache for MoE experts kept in host memory, from a size in MiB. It is off by default. Invalid, negative or non-numeric values are ignored with a log line. Forks that predate the field ignore the option, using the same prepare.sh probe as the other fork-skew switches. Document the option, and cover the MiB to bytes conversion with a standalone unit test. Assisted-by: Claude:claude-sonnet-5-5 g++ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
c7ce0c602c |
fix: stop listing the models directory on request paths and cache GGUF header reads (#12683)
* fix(http): stop listing the models directory on embeddings and bearer requests Every embeddings request read the whole models directory before the body was parsed, even when the body named its model. On a network-backed models directory this cost seconds per request. Embeddings now binds the body first and looks up a default model only when the body has none, like chat. The pre-bind default lookup and the gpt-4o constant default are gone from that route. The first-available default lookup skips the directory when the filter cannot accept an unconfigured file. Every usecase filter rejects files without a config, so the scan never added results for them. Match-all, nil and name filters keep the scan. The bearer token check is now an exact test: a configured name, or one stat of that name in the models directory. The value is a literal, not a regular expression, and a name that is not a local relative path never matches a file. Assisted-by: Claude Code:claude-haiku-5-5 * perf(config): cache GGUF header reads for model defaults Model defaults re-parsed the GGUF header on every SetDefaults call, which runs per request and at startup reload. Put a bounded LRU in front of the parser, keyed by absolute path and validated by size and modification time. The cached value is a snapshot of the fields the defaults use, not the parser output. A changed size or mtime is a miss. Missing files and parse errors are not cached, so the existing fallback is unchanged. LOCALAI_DISABLE_GUESSING, the empty-path guard, the alias early return, the panic recovery and the context-size default are all kept in front of the cache. Assisted-by: Claude:claude-haiku-5-5 * test(http): cover listing-free default lookup on every usecase route Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): persist the GGUF header cache per instance The header cache kept parsed GGUF results only in memory, so every restart parsed each model file again before its defaults were known. Write the cache to a per-instance file in the data path and load it at startup. A restarted frontend with a fixed instance ID starts warm. The file is versioned and records the estimator inputs (gguf-parser version, candidate contexts, auto context size, offload count). A mismatch on any of them makes the file ignored. Each entry keeps path, size and mtime, so a changed model file is a miss on load. Writes run on one background saver, debounced, through a temp file and rename. The entry bound applies to the file as well. Without a data path the cache stays in memory only. Documented as a per-instance exception in the model configuration docs. Assisted-by: Claude:claude-haiku-5-5 * docs(embeddings): document model selection and the no-model error Assisted-by: Claude Code:claude-haiku-5-5 * refactor(config): split the header cache into a generic cache The GGUF header cache mixed two things: the mechanics (bounded LRU keyed by path and identity, per-instance file, debounced atomic save, tolerant load) and the GGUF fields. Move the mechanics into headerCache[S], driven by a headerCacheSpec[S] that names the file prefix, schema, fingerprint, parser and the snapshot encode/decode pair. The GGUF cache becomes one instance of it. File names, the identity check, LRU size, debounce, atomic write and the corruption handling are unchanged. The on-disk JSON keeps the "estimator" key so existing files still load. Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): share one GGUF header cache file between frontends Every frontend used to write its own header cache file, named after its instance ID, so a header read by one frontend was never reused by another. The cache now uses one file, header-cache.json, in the data path, for all frontends and for standalone mode. The instance ID no longer changes the file name. Frontends share the file in three ways. A save first merges the file's entries in, so one frontend's write does not drop another's entries. A miss reloads the file when its modification time has changed, and a hit never reads it. Writes go to a temp file that is renamed into place, so a reader never sees a partial file. Two frontends saving at the same moment can still drop each other's last entries. That costs one re-parse, never a wrong value, because every entry is checked against its model file before use. The shared file is valid only when every frontend mounts the same data volume. The distributed-mode and model-configuration docs state this mode and its volume limit. Assisted-by: Claude:claude-haiku-5-5 * feat(config): give an instance ID its own header cache file Assisted-by: Claude Code:claude-haiku-5-5 * style(config): gofmt the distributed config struct Assisted-by: Claude Code:claude-haiku-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Stefan Walcz
|
e3341d21a0 |
feat(pii): extend a detection over the previous word (extend_to_previous_word)
The mirror of extend_to_next_word. A name model often tags the surname with
high confidence and scores the first name in front of it below min_score,
so "Gast Greta Waldmeister" became "Gast Greta [REDACTED:ner:LASTNAME]".
extend_to_previous_word lists entity groups whose accepted detections also
cover the word directly before them on the same line. The preceding word
must consist of letters (plus '-' and the apostrophe): a word with digits -
an amount, a date, a room number - is never taken, nor is a protected term,
and the search never crosses a line break.
Docs: operations/middleware.md and advanced/model-configuration.md.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Stefan Walcz
|
e16b9d04cc |
feat(pii): protected terms that pass the NER filter unchanged
A name model reads many business names as surnames. Measured with
privacy-filter-multilingual on 24 synthetic German receipts and
invoices: the supplier name ("Hotel Seeblick", "Café Lindenhof", ...)
was masked as LASTNAME (0.78-0.98) or FIRSTNAME in 9 of 12 receipts
that carried one. The supplier is exactly what the upstream model needs
to categorise a receipt, and no score floor helps at those confidences.
pii_detection.protected_terms (inline) and
pii_detection.protected_terms_files (files inside the models path, one
term per line, '#' comments, re-read when size or mtime change) list
values that must pass unchanged. Each whole-word, case-insensitive
occurrence is replaced by a neutral placeholder before the detector
runs; every detection is mapped back onto the original text with the
placeholder parts cut out, so a protected term is never masked or
blocked, the detector never sees it as context, and event offsets keep
addressing the original text. A word extension (extend_to_next_word)
is dropped rather than run into a protected term.
Terms shorter than three characters are ignored. A missing or
unreadable file is logged and skipped: fewer protected terms means more
masking, never less. Paths are verified to stay inside the models path.
Opt-in, empty by default. Covered by Ginkgo specs in
core/services/routing/pii/protected_test.go (detector-visible text,
exact restoration, straddling detections, whole-word/case/line-wrap
matching, the segment path, file loading and reload), a YAML round-trip
spec, registry entries and the middleware/model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
Stefan Walcz
|
9b13408c94 |
feat(pii): extend a detection over the next word (extend_to_next_word)
Address models tend to tag a postal code with high confidence but miss
or under-score the town right after it. Measured with
privacy-filter-multilingual on 24 synthetic German invoices: the postal
code was masked every time, while the town after it stayed in the clear
in 6 of 18 cases (4 not detected at all, 2 tagged CITY at 0.43/0.47,
below the configured floor). Masking the code alone leaves the pair half
redacted: "[REDACTED:ner:ZIPCODE] Ludwigshafen".
pii_detection.extend_to_next_word lists entity groups whose accepted
detections also cover the word that directly follows them on the same
line, with the same action and group. Horizontal whitespace is skipped,
a line break ends the search, and nothing is extended when no word
follows, so a hit never grows over bare whitespace or onto the next
line (or across the segment separator on the multi-message path).
The option is opt-in and empty by default; existing detector configs
behave exactly as before. Covered by Ginkgo specs in
core/services/routing/pii/extend_test.go (fail without the change), a
YAML round-trip spec, a registry entry for the model editor, and the
middleware and model-configuration docs.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
(cherry picked from commit
|
||
|
|
4d7bdc6ff2 |
feat(ui): redesign the web UI around a shared kit and a calm palette (#12526)
* build(ui): vendor the shared UI kit snapshot at 0.2.0 The restyle needs the kit's tokens, motion layer and component classes. Take a pinned snapshot instead of depending on the kit at build time, and keep a lock file with the version and per-file checksums so a later update shows exactly what changed. The product theme stays outside the vendored directory. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the ink and teal theme and bridge the old variables Define the product colours as the shared UI kit's roles, for light and dark, in theme-localai.css. The kit's contrast check passes on every pair. theme.css keeps the existing --color-* and --shadow-* names but now points each at a role, so App.css and the pages get the new palette without edits. Radii move to the kit scale. index.html now sets data-theme before first paint with the same rule as ThemeContext (stored choice, otherwise dark), because the contract layout of the theme file no longer defaults to dark by itself. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): restyle the shared chrome with the UI kit grammar Adjust the shared classes so every page picks up the same interaction language without per-page edits: - Sidebar sits on the canvas and the current row lifts onto a card. Section labels are tracked uppercase, the badge is a soft pill, and the phone drawer leaves the tab order when closed. - Buttons are flat: hover swaps the surface, press scales to .97, focus is a 2px ring with a 2px offset, danger is a tinted wash. - Inputs use the card surface and the control edge; switches, tabs, filter chips, badges and cards follow the same rules. Cards no longer lift on hover; only linked or button cards react. - Menus and popovers scale in from the trigger corner with 40px items. Dialogs get a veil fade and a spring settle. Toasts become pills at the bottom centre. - The page transition is a 250 ms fade with a 6px rise. It fills backwards so a finished animation no longer leaves a transform that confined dialog veils to the main column. The focus-ring test now checks the outline instead of a box shadow, and new specs cover the theme roles, the first-paint theme and the sidebar lift. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): move leftover hard-coded colours onto the theme roles The YAML editor restated the old blue palette in JavaScript, and a few pages kept literal blues, indigo and violet tints, or fallbacks that only applied because a variable was never defined. Point them at the theme variables so they follow light and dark and the new palette. The status badges in the account pages built their tint by appending "22" to a variable, which is not valid once the variable is defined, so they had no background. Use the wash roles instead. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): raise the type scale and control size toward the kit Body and list text moves to 15px and the rest of the scale follows the kit's 12/13/15/17/21/32/44 steps. Page titles, section headings and stat values are bold with tighter tracking; titles are 32px. Buttons, inputs, selects, tabs and nav rows are 40px high with the 12px radius, compact controls 32px. Tabs become a segmented control. The sidebar widens to 240px (64px collapsed) and nav rows get more room. Identifiers and counts in the split views use the mono face, and the stat grid becomes separate inset tiles. The Geist stack stays: it is bundled, and the thin look came from the size, weight and negative tracking, not the face. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): separate cards, panes and floating surfaces from the canvas Cards, the Models and Installed split panes, the composers and the confirm dialog use a stronger card edge, the rest shadow and the 20px radius, so they read as layers in dark as well as light. Menus and popovers move to a float surface (the hover tone in dark) with the float shadow. The selected rail row gets an accent wash and a 3px accent edge. The send buttons are a clear accent when there is something to send and a quiet inset when not; the Home button carries data-empty for that, since submitting an empty box does nothing. The assistant card becomes an accent wash with a square icon. New surfaces spec checks the pane edge, the selected row, both send buttons and the popover in both themes. The voice library empty-state spec now waits for the layout to settle before comparing two boxes. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): tidy the sidebar header and mark the current row with a dot The header gives the configured horizontal logo a fixed width and centres it in a 72px band, lined up with the nav icons. The collapsed rail shows the configured icon logo centred, and its nav rows become 40px tiles centred in the 64px rail. The current row gets the kit's accent dot, hidden in the rail. The theme, language and account controls stay in the sidebar footer: the app has no global search or command palette to put in a top bar, so a bar would only hold controls that already have a place. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): centre the avatar in the collapsed sidebar rail The collapsed avatar link was set to "flex: 0", which gives it a zero flex basis; with min-width: 0 the link shrank to its padding and the icon overflowed from the link's left edge, about 14px right of the icon column. Use "flex: 0 0 auto" in the collapsed and tablet rail. The footer controls now share the nav icon column in the expanded sidebar too (6px footer padding, 40px control boxes), and the tablet rail gets the same footer padding and hidden language code as the collapsed one. New spec measures the centre x of the nav icons, mark, avatar, language, theme and collapse icons in the collapsed, expanded and tablet states, in both themes, and asserts they agree within 1px. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): size the console and settings rails and stack settings on phones The Operate console rail, the Settings section rail and the account tab bar still used 13px text and the old underline tabs. They now use the 15px nav size, 40px rows and the segmented tab control. Form row labels are 15px with 13px hints. On a phone the Settings section rail sat beside the form and squeezed every row into a few characters. Below 720px the rail stacks above the content as a scrolling row and form rows wrap their control below the label. The save button no longer carries the icon font class, which drew a missing glyph before its label. The language menu is wide enough to keep Bahasa Indonesia on one line. Assisted-by: Claude Code:claude-sonnet-5-5 * build(ui): update the vendored UI kit snapshot to 0.3.0 Take the 0.3.0 snapshot: the sprite now carries the full outline icon set, and the new icons/fa-map.json maps Font Awesome names to icon ids. The map lets the app move off Font Awesome in the following commits. The lock file is regenerated with the new checksums. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add an Icon component backed by the kit sprite Icon draws an inline svg that points into the kit's outline sprite. The sprite is inlined into the page once, so the references resolve under any base path and in the embedded build without a request. Icons size with the font (1em), take currentColor, hide from assistive tech unless given a title, and spin on request. An unknown id draws a neutral circle. FaIcon and iconFromFa resolve Font Awesome names through the kit's map, for names that arrive at run time. iconHtml does the same for markup built as a string. The GitHub and Apple marks are small local glyphs, as the kit ships no brand marks. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): draw shared components and helpers with Icon Replace the Font Awesome elements in the shared components and in the utility modules with the Icon component. Lookup tables now hold kit icon ids instead of class strings. Code-block copy buttons and artifact cards, which build HTML strings, use iconHtml and a sanitizer-safe slot. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): draw model, backend and account pages with Icon Replace the Font Awesome elements on the home, models, backends, import, settings, login, account and users pages with the Icon component. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): draw chat, studio and recognition pages with Icon Replace the Font Awesome elements on the chat, media generation, talk and face and voice pages with the Icon component. The talk status table keeps its spin and pulse states as Icon props. The connected and error states now use a dotted circle and an alert circle, so they differ from the idle ring by shape as well as by colour. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): draw agent, node and operate pages with Icon Replace the Font Awesome elements on the agents, skills, collections, jobs, fine-tune, quantize, nodes, swarm, usage, traces and activity pages with the Icon component. Two class strings on layout elements held leftover button and icon classes from an earlier merge; they are cleaned up so the elements keep only their own classes. Assisted-by: Claude Code:claude-sonnet-5-5 * style(ui): size and align icons for the svg component Icon rules that targeted the font element now target the svg: the descendant "i" selectors in App.css and auth.css become ".lai-icon". The svg is 1.2em with a 2 unit line so it matches the visual size of the old glyphs at the 12 to 16px sizes the app uses, sits on the text baseline, and follows the context font size. Large empty-state marks get a lighter line. Menu icons get a 16px box and the readiness badge icons keep their 20px circle with padding. Add the pulse used by the talk status. Assisted-by: Claude Code:claude-sonnet-5-5 * build(ui): remove Font Awesome No source file references the icon font any more. Drop the package and its stylesheet import. The build no longer ships the solid, regular and brand font files. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): keep focus traps off the svg use references The dialog and drawer focus traps collect focusable elements with a "[href]" selector. An icon's use element carries an href, so it became the first "focusable" element and Tab at the end of the dialog stopped there instead of wrapping to the first button. Match "a[href]" instead. Assisted-by: Claude Code:claude-sonnet-5-5 * style(ui): keep icon sizes overridable and set the line width per svg Give the icon base rule zero specificity so a rule that sizes one icon (nav column, menu box, avatar, language switcher) wins whatever its order in the file. The sprite symbols fix their own line width; the inlined copy drops it so the width set on each svg applies, as the --lai-stroke custom property, and large marks can use a lighter line. Pin the avatar and the language globe to the boxes the sidebar alignment spec expects. Import the map as JSON with an import attribute so Node can load it in the spec. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): select icons by the svg markup and cover the sprite Specs that found icons by their Font Awesome class now select the svg by its data-icon. The dead-icon audit checks that every svg resolves to a sprite symbol and has a size. The class hygiene spec fails on any remaining Font Awesome class. A new spec checks every mapped icon id has a symbol, that the sprite is inlined once, that an icon paints at the root and under a forwarded path prefix, and that Font Awesome names map as documented. Assisted-by: Claude Code:claude-sonnet-5-5 * build(ui): update the vendored UI kit snapshot to 0.4.0 Take the 0.4.0 snapshot: hub tabs with count and attention badges, the six chart series tokens and the grid colour in the theme contract, and sample themes on a calmer palette. The kit headers are renamed and the lock file is regenerated with the new checksums, as for the earlier snapshots. Assisted-by: Claude Code:claude-sonnet-5-5 * style(ui): switch the theme to the calm palette Rewrite the LocalAI theme on the calm palette: a muted teal accent on a near-neutral green-grey canvas, desaturated status colours, no glow and no coloured shadows. The theme fills every role of the shared UI kit's 0.4.0 theme contract for light and dark, including the six chart series and the grid line. The bridge in theme.css keeps the old --color-* names working, adds the dark surface ladder (card, raised, float) and a strong edge, and points the fixed data hues at the chart series. Two values differ from the first sketch. The dark text on the accent fill is #021512 instead of #04201d: it reads 5.58:1 on the fill at rest and 6.4:1 on the hover fill, against 5.08:1 at rest for the lighter value. The light control edge is #6b7d7a. The kit's contrast script passes for all text pairs (4.5:1), control and focus pairs (3:1) and series colours (3:1). Leftovers that no longer fit the palette are fixed: the usage chart takes the six series colours in order, the audio and animation canvases fall back to the new accent, the face box loses its glow, and two gradient fills are now flat. The theme tests expect the new canvas colours. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): replace the console rail with hub tab bars Build and Operate no longer open a second navigation rail beside the page. Each is a hub: one row of the kit's hub tabs above the page, with count and attention badges that scroll sideways on a phone. Every URL and route stays as it was, plus a new /app/build landing page that lists the Build tools with a line each. Build tabs: Overview, Agents, Skills, Memory, Jobs, Fine-Tune, Quantize, Import, Voices (recognition and library) and Faces. Operate tabs: Status, This machine, Swarm (distributed mode only), Runtime (backends, activity, failover), Traffic (usage, traces, middleware) and Settings (settings, users), plus the API link. A tab that holds several pages shows a second row of links, and a sub-page such as a node detail keeps its tab highlighted. The feature and admin gates decide which tabs are drawn, and badges show only values the Operate summary already has. The sidebar lists Build and Operate under a Workspace label next to the Create group. The voice library moves under Build and the model import page gains the Build tab bar. The old rail styles, the rail signals and the console config are removed, and the Operate overview docs describe the tab bar. The specs that drove the rail now drive the tabs, and a new spec covers the tab for each route, gating, badges and the phone layout. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Home as a calm console Home now opens on one command bar: the model chip shows which models are warm, the MCP chip and attach buttons sit beside it, and Send is a solid button with an Enter glyph. Typing "/" opens a grouped, keyboard-driven action list built on the kit command list; every action has a destination in the product. Memory use folds into a one-line strip that opens into the loaded models, with Stop per model and Stop all. It opens by itself while a model is being staged and after a failure, and shows nodes and aggregate memory in a cluster. The list of resident models carries no per-model size because the API reports none. "Jump back in" lists the conversations stored in the browser, one card per day, with j and k to move, Enter to resume and delete with an undo toast. First run keeps the install steps and the recommended models. The assistant prompt is a dismissible line, the library links are one quiet row and the API section is collapsed. Chat accepts an empty new-chat hand-off for /new. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Home console Update the Home specs for the new structure and add specs for the slash menu, the model chip, the memory strip (expand, stop, staging, failure, cluster), the resume list (grouping, j/k, Enter, delete with undo), first run, the send hand-off, a non-admin user and the phone layout. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the fit, disk and cleanup helpers for the models page Pure functions and hooks that the rebuilt Models page reads, with node tests for the rules. modelLedger turns an estimate and the memory budget into one of three verdicts (fits, spills to CPU, over) with the headroom in bytes, and reads the models disk from the resources reading. The disk counts as low under 10 percent or under 20 GB free, and is absent when the server reports none or runs as a cluster controller. cleanupPlan ranks installed models from what the API reports: loaded, pinned, or named by an agent, a task, a failover chain or an alias keeps a model protected; another installed build of the same gallery model is a duplicate; disabled models rank above idle ones. The API records no last use or use count, so none is used. When a lookup fails, nothing is called safe. useModelRemoval holds a removal in the browser for an undo window and sends the existing delete call only when the window ends. Leaving the page drops the batch without deleting anything. The undo toast takes optional labels so other pages can reuse it. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Models as a ledger with a disk strip and cleanup review Explore is one dense table. Each row carries the size, a solid memory bar and the headroom in words ("3.7 free", "+1.5 on CPU", "0.9 over"), worked out from the estimate at the chosen context length. Capability chips show the server's count for each facet, search keeps its meaning and "/" jumps to it, and a density switch (also "d") picks comfortable or compact rows. Selection is a surface step and a check, never a rail. Arrow keys move, Enter installs and Esc closes the inspector, which keeps the fit summary, VRAM by context chart, variants, files, links, tags and licence. A failed install shows its error in the row with a Retry that dismisses the old failure first. A failed or empty listing says which it is, and a host with no GPU is measured against memory and says so. Installed uses the same table with state filters that carry counts, a state per row, Load or Stop on the row, the row menu and the sort by size. Sizes come from the files the gallery lists, so a model it does not know shows a dash. A strip in the header shows the free space on the models disk. It turns amber under 10 percent or under 20 GB free, hides when the server reports no disk or runs as a cluster controller, and opens the cleanup review. Explore says how much an install leaves free. The review ranks installed models as Safe to remove, Probably safe and Your call from real facts only, lists protected models with the reason, and says plainly that usage history is not recorded. A sticky bar shows what a choice frees. Confirming runs a dry run that checks again and lists what will go. Removal waits 30 seconds with an undo; nothing is deleted before that, and leaving the page deletes nothing. The old rail, filter band and popover styles are removed. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Models ledger, Installed table and cleanup review Update the Models, lifecycle, cluster fit, height, search focus and surfaces specs for the table and inspector, keeping what each one checks. New specs, on a shared 41-model gallery stub with three machine profiles: the fit bar and headroom words for a 24 GB card, an 8 GB laptop and a host with no GPU; facet counts, search, "/" and Escape; selection, arrow keys, Enter to install, density; the disk strip when normal, low and hidden; and the states (loading, empty, offline, install failed, phone). Installed covers filters with counts, row actions, the row menu, sizes and sort. The cleanup specs cover grouping, protected models, the honest-data note, the effect bar, the dry run, the undo window, a failed delete, leaving the page, and the phone sheet. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the placement helpers and the estimate hooks The Placement section and the model page need the same few rules, so they sit in plain functions that can be read and tested alone. placement.js holds what gpu_layers, tensor_split and main_gpu mean (unset asks for every layer and the llama.cpp engine trims it, zero is CPU only, 99999999 is the value LocalAI itself writes for all layers), the device list taken from the resources reading, the split by free memory, the part of an estimate that grows with context (read from two lengths, since that term is linear), the fit states with their limit (95 percent of free memory, and the leftover has to fit in system memory too), and a bisection for the largest layer count whose estimate fits. The estimate returns one total and no layer count, so the search runs over 1 to 256 and stops at the first count that no longer changes it. modelWalk.js keeps the order of the list a model page was opened from, in memory and in session storage, for the previous and next buttons. usePlacementEstimate reads /api/models/vram-estimate for a choice, again at twice the context, and with every layer, and keeps readings for the session. useModelPage reads a gallery entry by name, an estimate by context size (from the model's own files when the gallery does not list it), the builds and the loaded models. usePlacementConfig edits the four placement keys of an installed model and saves only what changed. useModelActions is the Load, Stop, disable, pin and remove logic of the Installed table, shared with the model page. MemoryBar is one solid bar with a tick at the capacity of its pool; over capacity it grows past the tick and the tick turns red. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the Placement section to the model editor Run this model on: CPU only (gpu_layers: 0), Auto (the key stays unset) or Custom. Custom takes a number, has an All layers button that writes 99999999, and shows a slider only when the estimate reports the model's layer count, which it does not today. Context size has presets and a number field because the KV cache follows it. With two or more GPUs there is a split (written as percentages, with a button that takes them from the free memory of each card) and a main GPU. A bar per GPU and one for system memory show what other programs use, the model's weights and working memory, and the part that grows with context, with the room left or how far over it is. Under them a verdict in plain words: Fits in GPU, Spills to CPU, Too many layers for the GPU, Runs on CPU only, No GPU found, Not enough memory. It says "slower" and never a multiplier, because the estimate has none. Fit it for me asks the estimate for the largest layer count that fits the free GPU memory and says what it set, with Undo; it is hidden when the estimate is unavailable or the host has no GPU. Loading shows skeletons, an unavailable estimate shows a note with Retry, and a server that schedules onto other machines shows no bars, because its device list is the controller's. The editor shows the section for an installed model, with a link in its section rail. Auto sends null for the key, since a patch only merges, and a null read back opens as Auto. The docs describe the section and what each mode writes. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): open a model on its own page A model has an address, /app/models/<name>, for an installed model and a gallery entry alike. Open it from the arrow at the end of a row, a double click, "o" on the selected row, the inspector's Open details button, or a tap on a phone. The title block holds the main action: Install with a chevron that chooses the build, or Load and Stop with a menu (disable, pin, edit configuration, logs, delete with a confirm). A strip answers whether it fits, what it does and what installing leaves free. Tabs: Overview (about, a memory bar, state, the pages it opens in, and the agents, tasks, chains and aliases that name it); Fit and memory (verdict, context sizes, the bar split into weights and context, and memory by context against the limit, with a data table); Variants and files (builds with size and fit, install any, the files of the chosen build). For an installed model also Usage and history, which says what the API does not record instead of drawing an empty chart, Configuration, which is the Placement section with the file it writes and a link to the full editor, and Logs, the backend log viewer without its page. Keys 1 to 6 switch tabs, [ ] and j k walk the list the page was opened from, Esc or Backspace go back. The list stays mounted behind the page, so Back finds its view, search, filters, selection and scroll as they were, and focus returns to the row's arrow. The page covers loading, an unknown name with the closest matches, the gallery being out of reach, an install in progress with Cancel, and a failed install with Retry. The docs describe the page and its keys. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the model page and the Placement section New specs for the model page: reaching it from Explore, Installed, a double click, "o", a pasted link and a phone tap; the walker and Back with the search, a filter, the selection, the Installed view and the scroll kept, and no second read of the gallery; the title block, the answer strip, tabs by click, keys and arrows; Fit and memory, builds and files with the install call each one makes; an installed model's actions, used-by, the honest usage tab, configuration and logs; loading, an unknown name, offline, an install in flight and a failed one; and the phone. New specs for Placement: every mode and the keys it writes, the slider only when a layer count exists, the context presets, the bars and every verdict, two GPUs, no GPU, a cluster, an unread machine, a loading and an unavailable estimate, Fit it for me and Undo, and the section in the model editor with its save. The phone tap on a row now opens the page, so the two phone specs that expected the inspector as the page check the page and keep the inspector check for a window between a phone and a desk. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): link Studio results and open workspaces from a prompt Each workspace now records the result it was made from (parentId and an edge kind such as take, animate or to-3d) and reads a prompt, model, size, count and source from the query string, so one page can hand work to another. A source result is fetched from the server's own output file and becomes the start image, the picture for 3D, or the audio file. A note on the page says when the source loaded or could not be loaded. Diarization had no history; it now keeps the file name, the model and a speaker count, never the recording. Prompts are cut at 2000 characters when stored. The pure helpers (type suggestion, grouping, lineage layout, favourites, clearing) have node tests. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Studio front page as a composer with your work The front page is a prompt box with a chip per type, a type suggestion from the words, starters, and the options each workspace accepts. Generate opens the workspace with those filled in. A type with no model is a dashed chip that shows a gallery model, its size, memory need and an Install button only when picked; the typed words stay while it installs. Under it, Your work lists results from every workspace as a masonry with filters, counts, favourites and a Clear history action. Results made from each other stack into a project tile and open as a lineage board with a dock for running a new take or branching to the next step; steps the destination cannot start from yet are disabled with the reason. The docs describe the page, what is stored in the browser, and the query parameters a workspace accepts. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Studio composer, your work and the lineage view Specs for the type suggestion, the keys, hand-off to each workspace, the install path for a missing model, the masonry filters, favourites and clearing, stacking, the lineage board, new take and branch, steps that are disabled with a reason, and the phone layout. Existing Studio specs move from lanes to chips with the same intent. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the shared Studio workspace frame and move Images onto it The seven Studio workspaces get one layout: a row of type tabs, a compose card (optional sources as chips, a prompt with starters, a model chip, the essential options as chips, an Advanced fold that names what is inside, the memory the model needs, and one action with the reason when it cannot run), a run area, and a strip of recent results of the type. The run area shows a job card with the time that has passed and an indeterminate bar, because these endpoints report no phase or percentage; a failure with what the server said and one action; or the result with a toolbar: Favourite (the list the front page keeps), Download, Use in (the hand-off targets, disabled with the reason when a destination cannot start from the result), Re-run with edits (the take's values go back in the form, changed fields are outlined and listed) and Lineage. A type with no model shows the install note from the front page. Images is the first workspace on the frame. It keeps its size, count, steps, seed, negative prompt, source image and reference images, and its history writes, including the parent link and edge of a hand-off run. useMediaHistory.addEntry now returns the id of the entry it stored. The docs describe the workspace page. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move Video onto the workspace frame Video keeps its size list, duration, frame rate, steps, seed, CFG scale, frame count, negative prompt, start and end image and avatar audio. The start and end image are source chips, the avatar audio opens the recording and paste input from a chip, and the rest sit in the Advanced fold. A start image from a hand-off shows as a chip with its picture. Results play in the video player with the shared toolbar. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move TTS onto the workspace frame TTS keeps the saved-voice picker for cloning models, the typed voice for the others, the voice library deep link, and the delivery instructions, which now sit in the Advanced fold. The result is the waveform player with the words under it. The stored entry also keeps the voice id so Re-run with edits can select the same saved voice. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move Sound onto the workspace frame Sound keeps its Simple and Advanced modes and every field of both: the description, instrumental, vocal language, caption, lyrics, BPM, duration, key, language, time signature and think mode. The mode switch, instrumental and duration are in the compose card, the rest in a More options fold. The stored entry keeps all of the fields, so Re-run with edits restores the form as it was. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move Transform onto the workspace frame Transform keeps its audio and reference inputs with upload and record, the echo test, the key=value parameters (now in the Advanced fold), the input and output spectra and the three waveform players. The audio that was chosen shows before the run, waiting to be transformed. Re-run with edits puts back the model and parameters and fetches the audio and reference the server kept for that run. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move 3D onto the workspace frame 3D keeps the picture input with paste and webcam, the animation operations a model declares, quality and background, the shape and material steps, guidance and seed, the GLB and animation viewers, the remesh control and the download. A 3D result now has a title from the motion prompt when it has no label, so the strip and the front page name animation results by what was asked. Re-run with edits is shown disabled with the reason, because only a small thumbnail of the picture is kept. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move Diarization onto the workspace frame Diarization keeps its model and recording inputs, the option to prepare speakers to remember, the clean-speech previews, naming and remembering a speaker, and the history entry with only the file name, model and counts. The result now shows a timeline with one lane per speaker, the talk time of each speaker, and the segments with their start time and text. RTTM, SRT (only when the run has text) and JSON are built in the browser from the result. The helpers for talk time, axis ticks and the two text formats have node tests. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): remove the styles and lists the old workspace layout used Nothing renders the two-column workbench, the control column, the old history lists, the generation progress tiles, the TTS voice picker or the result echo any more. The inline-style baseline drops with them. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the workspace frame and one run per type Specs for the type tabs, the compose card and its reason when Generate cannot run, starters, the Advanced fold, the job card with no invented progress, a failed run and its one action, the install note, the strip with its favourites filter, Use in with its disabled steps, Lineage, the parent link, Re-run with edits and its list of changes, deleting and clearing, and the hand-off note. One run through each of Video, TTS, Sound, Transform, 3D and Diarization, the phone layout of all seven, and reduced motion. Existing Studio specs move from the old control column to the compose card with the same intent. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the chat thread with raised user turns, prose replies and one-line activity Your messages are raised blocks on the right at a 760 px measure and the model's replies are plain prose under its name and a warm or not loaded dot. Reasoning, tool calls and their results fold into one quiet line that opens inline into steps. Code blocks carry a Copy button and a Canvas button that opens that block in the canvas, image attachments are thumbnails that open in the lightbox, and files are chips. Per-message actions show on hover, on focus and on the last turn, and a turn takes focus so the arrow keys and C, E, R and B work. A failed reply keeps the text written so far and shows the reason with one Retry action. The Agent chat page keeps the older rules: the new styles are scoped to the chat page and use their own class names. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): use the Home command bar as the Chat composer Chat now ends in the same object as Home: the model chip, the MCP chip, a Canvas chip, the message box with attach buttons, a solid Send and the hint line, with the slash menu on the kit command list. The slash menu lists what Chat can do today (switch model, new chat, conversations, manage mode, canvas, find, settings, export, clear). While a reply is streaming Send becomes Stop, which Esc also presses, and Up in an empty box edits your last message. Attached images show as thumbnails and a line under the bar carries the speed and the token count. HomeComposer takes optional props for this (extra chips, its own slash list, Stop, paste, a stricter Enter); Home passes none of them. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): open conversations from a Ctrl K menu with day groups, undo and a slim header The conversations list opens as a centred menu on Ctrl or Cmd K. It groups chats by day like the Home resume list, shows the model that answered and the time, searches names and message text, and moves with the arrow keys. Enter opens a chat, F2 renames it and Delete removes it. Removing a chat hides the row and shows the kit undo toast; the chat is deleted for good only when the undo time ends. Rename, duplicate, copy and export are on each row, as before. The header is one slim bar: the Chats button, the chat name (click to rename), a context meter when the context size is known, settings and a More menu with rename, duplicate, copy, export, model info, keyboard shortcuts and clear. A dialog lists the shortcuts the page answers to. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): show loaded state, capabilities and fit in the Chat model switcher The model chip in Chat opens the same list as Home, grouped as Loaded now and Installed. Each row says warm or not loaded and marks models that understand images. When the list opens, the page reads the host memory once and asks the server to estimate each listed model at the chat's context size (up to twelve, three at a time), then shows what the model needs and whether it fits: free memory, how much would run on the CPU, or how far over the machine it is. A model with no estimate shows no fit text, and no load time is shown because the API does not report one. A memory bar closes the list. The picker takes the model list from the page when it has one, and useModels can skip its own request. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): move chat settings into a sheet and add find in chat, jump to latest and a wider canvas Settings open as a kit sheet: the system prompt, temperature, top P and top K (each says "model default" until it is changed and has a Reset), the context size with quick sizes and a note that it only drives the meter, Manage mode and Focus mode, the model info for admins with its Edit config button, and Clear conversation behind a confirmation. The old slide-out drawer and the model info panel are gone. Ctrl or Cmd Shift F (or the search button, or /find) opens a search bar over the thread. It marks matches in the messages already on the page, shows "n of m" and steps with Enter and Shift+Enter. Nothing is sent to the server. Jump to latest is a pill above the composer. Esc stops a reply, then closes the search, then closes the canvas. The canvas panel gets the kit look: tabs, a Code and Preview switch, Copy and Download, a full-page layout on narrow windows, and translated labels. The Agent chat page shares it and gets the same look. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add the empty, no-model, loading and phone states to Chat An empty chat opens with the composer under one line, starters to try, whether the model is loaded, and the Jump back in list: the same rows as Home, read from the chats the page holds. With no chat model installed, an install card offers the starter models for this hardware, the gallery and import, and the composer stays so the text is not lost. While a reply waits for a model, a load card shows what the page knows: the phase the server names, the node, the bytes and the time left when the server reports them, and a progress bar. A model that is just not loaded yet gets a plain note, with no invented phases or estimates. The foot warns when the context is nearly full. On a phone the header drops its labels, the model list and the settings open as sheets from the bottom, per-message actions stay in view and the canvas takes the whole page. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Talk as a calm voice page over what the connection really does Talk is one stage and one transcript. The stage has the pipeline chip, the voice and language chips, an outline orb that follows the real microphone and playback levels, a heading and a sentence for the current state, and the controls. The transcript lists You, Reply, Tool and Result lines and can be copied. Session settings (instructions, voice, language, tools, Manage mode and the pipeline's parts) open in a sheet. The states are the ones the code reaches: no pipeline model, idle, connecting, listening, thinking (also while a tool runs), speaking, an interrupted reply (the server cancelled it; a note marks the cut), a blocked microphone, a link that failed during a session, and any other error with its reason and a link to the traces. Push to talk and hands-free are not on the page, so they are not shown. Diagnostics keep their waveform, spectrum and stats, drawn in theme colours. The page text moves into the talk namespace, and the old Talk and visualizer styles and the inline-style count go down with the rebuild. Assisted-by: Claude Code:claude-sonnet-5-5 * refactor(ui): remove the chat styles and strings the rebuilt page replaced The settings drawer, the model info panel, the bubble avatars, the conversation menu popover, the context bar, the recent strip, the staging bar, the file badges and the focus-mode rules have no user now. Their rules, the Chat page's focus class and seven unused empty-state strings are removed. The Agent chat page keeps the shared message, sidebar and input rules it still renders with. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the rebuilt Chat and Talk pages Add a Chat page under features (thread, message actions and keys, the message box and its slash actions, the model list with loaded state and fit, conversations on Ctrl K, settings, find, canvas and the empty, no-model and loading states) and a Talk section to the realtime API page with the states the page shows. Manage mode now turns on from the chat settings or /assistant, and the client MCP steps point at the MCP chip. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): settle the rough edges of the new Chat and Talk pages The undo toast sat under the conversations menu, so the Undo button could not be pressed while the menu was open; the menu, the sheets and the fullscreen canvas now stay below the toast layer. Esc in a rename box saved the text through the blur that follows it; it now cancels. The image viewer closed on Esc only when the page did not re-render on the same key, so its key listener is registered once and reads the latest handlers. Keys on a focused message no longer type their letter into the editor they open, "/" from outside a text field starts a command as it does on Home, and Esc leaves the page's own dialogs alone. Code in the canvas is highlighted for languages that have no preview. The conversations menu drops its key hints on a phone so Clear all stays in view. Talk hides Test tone while connecting and calls a server error "Something went wrong", since the call can still be open. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the rebuilt Chat and Talk pages Specs for the thread layout and the activity fold, code blocks, image thumbnails and the viewer, per-message actions and their keys, a failed reply with its one Retry, Stop and Esc while streaming, the composer and every slash action, the conversations menu (groups, search, resume, rename, delete with undo that ends by itself, one chat left), the model switcher with loaded state, vision and fit text from stubbed estimates, the settings sheet, the canvas panel, find in chat, Jump to latest, the empty, no-model and loading states, the phone layout and reduced motion. Talk is driven over a fake WebRTC link through idle, connecting, listening, thinking, speaking, interrupted, blocked, lost, error and no pipeline, its settings sheet and its phone layout. Node tests cover the message text helpers and the conversation grouping. The existing chat specs move to the new structure with the same intent: the transcript spec now describes the raised turn and the prose reply, and the render smoke accepts Talk's own header. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): keep a bounded run log for agents in the browser The server keeps no run history for an agent, so a run is one task and the events until the agent answers, written to browser storage while the page watches the stream: up to 50 runs per agent, task, step and answer text only. Stored chats from the earlier agent chat page read as runs with stable ids. A run still marked running five minutes after its last event reads as stopped. Helpers read an agent's config into chips, build the list of changed fields against the saved config, hide secret values and offer starting points. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Agents area around runs The Agents page shows what needs a look (work in flight, a run that failed in the last day), then each agent with its model, attached memory and skills, and a strip of its last 14 runs. An agent has its own page: model, tools, memory, skills, instructions, a task box and its runs. A run has an address, shows the thread while it works (steps folded into one line, the tool in use, the answer as it arrives) and settles into a report about a second and a half after the agent answers: task, outcome, follow-ups, evidence and steps, with wide tables opening wider on demand. A failure says in plain words what happened and offers Run again. Create and edit fold into sections with a ready mark and a one-line summary, start from a template or an optional model-written draft, and open a preview sheet with the config as saved and the changes against the saved agent. Status becomes a quiet panel in the same language, and the old chat link opens the agent page. There is no Stop, approval, steer, version or dry-run control, because the agent API has no call behind them. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Agents launcher, agent page, runs and editor Specs for the Now strip and run strip, search and the empty state, the agent page, starting a run, the live thread, settling into the report, the run address across a reload and for a run from another browser, follow-ups with their history, failures, the folding editor with ready marks, templates, the preview sheet with hidden secrets and changes, the status page, and the phone, 1440 and 2560 layouts. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe runs and the new agent create flow Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add helpers for tasks, schedules and job outcomes Reads a cron expression the way the server does (five fields or an @ shortcut), checks it, and puts the common shapes in words. The next run is left out on purpose, because the schedule follows the server clock, which the browser cannot read. Also groups jobs by day, sums the last seven days, and gives each job one outcome line from its result or error. A rerun call starts a new job with the same parameters and media. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Jobs area around runs The Jobs page opens with one sentence about the last seven days, then the tasks (model, schedule in words, last 14 jobs, enabled switch, Run now) and a run history grouped by day. Each row has an outcome sentence and opens to the error or the start of the result with one next action. Deleting a task waits 30 seconds with an undo button. A task opens as a page with its recent runs, its prompt with the gaps marked and its schedule. The task form folds into sections, takes a schedule as a preset or a checked cron expression, warns about prompt gaps the schedule does not fill, and has a preview sheet. A job opens as a document: task, outcome, delivery and the recorded steps; a failed job says what happened and offers Run again. Run now now sends attached media through the job call, which is the only one that takes it. "Clear History" only ever cancelled running jobs, so it is now called Stop running jobs. Webhook headers of a saved task show as JSON instead of [object Object]. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Jobs page, task pages and job pages Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the Jobs page and the task form Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add helpers that say who uses a skill or a collection An agent loads a skill when skills are on and the skill is in its selection (an empty selection means every skill). It reads the one collection that carries its own name, when its knowledge base is on. The helpers derive that from the saved agent configs, build the config that adds or removes a skill or a collection, and estimate tokens as characters divided by four. Removing the last selected skill switches skills off, because an empty selection would mean every skill. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Skills and Memory as one library Skills and collections sit in a list with an open item beside it. Each row says who uses it, read from the saved agent configs, or says it is not used yet. Chat reads neither, so it is never named. An item opens in a pane with a Used by strip (names link to the agent, a small x removes it, with undo) and an Add to menu that shows what the addition costs. A collection can be added only to the agent that carries its name. The Memory pane searches the collection alone and shows ranked passages with scores, lists web sources with their refresh interval and the files, shows the server message when an upload fails, and names the endpoints and where files stay. The Simulate a message sheet runs a collection search and shows an agent's skills with a token estimate. It runs no model. The collection details route now opens the same page. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): show use and cost in the agent form pickers Each skill in the agent form says which other agents use it and what it adds to every message, with a total for the selection. The memory section names the collection the agent reads. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Skills and Memory libraries Specs for the used-by lines (including an agent that uses every skill), the filters, search, add to agent, remove with undo, the last-skill case, an unreadable agent list, the empty states, git repositories, the Memory question box, sources, uploads that fail, the Simulate sheet with the parts the API can run, the agent form hints and the phone layout. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the Skills and Memory libraries Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add helpers for the Operate status page and backend rows Pure functions for the parts that need rules. They work out the memory pool the page measures (GPU memory, system memory, or the workers of a cluster that are answering), which pools are too full, the headline and the four ledger rows, the geometry of the capacity chart, and what removing a backend would leave without a runtime (models name their backend, and a meta backend names the concrete one it points at). A second set says what a backend row states: installing, queued, removing, failed, update available, current or absent. LocalAI keeps no memory history, so the chart reads a bounded buffer of readings the page took itself and says so. A reading with no total is dropped rather than drawn as zero. Two hooks are shared by the pages that need them. One retries a failed operation after moving the failure into the record. The other holds a cancel for an undo window, because the server cannot take a cancel back. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Operate Status, This machine, Backends, Activity and Logs Status opens with one sentence ("2 things need you", or "Everything is running") and four rows: Needs you, Capacity, Running now and Recent failures. A row with a problem opens by itself and holds the button that deals with it: Update a backend, Retry or Dismiss a failed operation, Unload a model. A quiet row stays one line. A new installation gets a first-run screen, a cluster sums the memory of the workers that are answering, and a page still waiting for an answer says so. The chart under the rows is drawn from readings the page took while it was open and is labelled that way, because LocalAI keeps no memory history. This machine leads with GPU memory as one bar, then host memory split by running model, then VRAM, RAM, CPU and disk with a bar each. The running models become a kit table with the same menu and stop dialog. Backends is one list with Installed and Catalog views. A row says what the backend is doing (a progress bar with Cancel, Queued, Failed with Retry, Update 1.2.0, Current), carries the one button that matters, and opens in place. Removing a backend names the models and the meta backends that would stop working. Check for updates, Update all, From URL and a first-run recommendation for llama-cpp are in the header. Activity keeps its three sections as quiet rows. Cancel waits eight seconds with an undo toast, because the server cannot take a cancel back; a cancelled install can be started again from the record. Logs gets a process list, a picker, stream and text filters, Follow and Times switches, and a Clear with an undo window. Not shown, because the API has no data for them: GPU temperature and power, a size per backend, an earlier version to roll back to, a dependency lookup beyond the models and meta backends that name a backend, and models that failed to load. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover Operate Status, This machine, Backends, Activity and Logs New specs for the Status headline and ledger (healthy, needs attention, one thing, a full memory pool alone, loading, first run, cluster), its actions (Update, Retry, Dismiss, Unload with its dialog), the capacity chart built from readings taken while the page is open and bounded, the phone layout, no coloured edge on a row, and reduced motion. The Backends specs cover the two views, install progress with Cancel and its undo window, Retry on a failed install, Update, Update all, Check for updates, removal with the models and meta backends it would break, Install from URL, the first-run recommendation, a cluster, and a phone. Activity gains cancel with undo, Cancel now, a second cancel, leaving the page, progress, and starting a cancelled install again. Logs covers the stream and text filters, Follow, Times, Export, Clear with undo, the process picker and list. This machine covers the GPU strip, several GPUs, no GPU and Add a machine. Existing specs keep their intent and follow the new structure: rows open in place instead of in a pane, Update replaces Upgrade, the notice spec now pins that an update is a row state and not a banner or a rail, and a cancel waits for its undo window. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the rebuilt Operate Status, Backends and Activity pages Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add helpers for the Swarm pages Pure functions for what the pages work out from the cluster API: a node's state in words, which nodes a placement rule may use, what a rule would ask for, what a drain or a lost node would leave without service, the nodes a bulk backend update reaches, and the join commands for a worker, a peer instance and a memory shard. Hooks read the roster, the loaded replicas and the rules. Everything runs in the browser from data the page already holds, and says when it cannot see free memory or disk. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Swarm hub: nodes, node page, placement rules, failover Nodes is a sortable table with comfortable and compact rows, a Needs attention filter by reason, a map of the cluster that is not drawn on a phone, the running models, and a bulk backend update for the nodes that drifted. A node is a page: state, vitals, a drain preview computed from the loaded replicas and the rules, tabs for models, backends, logs and capacity and labels, and Remove that asks for the node's name. Placement rules are written as sentences, show where each model is loaded now, and edit in a side sheet with a preview of the nodes a draft could use. Deleting a rule waits a few seconds so it can be taken back. Failover keeps its chains, adds what the router does when a worker stops answering and a per-node preview of what would stop. Add a node covers a registered worker, a peer instance and a memory shard, with a command to copy and a live line that says when the machine arrived. P2P keeps its page in the same vocabulary, and the node logs page follows the local logs page. Previews are labelled as worked out in the browser. Per-GPU readings and node events are not drawn because the API does not return them. Failover moves to Swarm when distributed mode is on. Legacy fleet components and their styles are removed. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Swarm hub New specs for adding a node (each join method, the command, copy, waiting and found, approve, a single install, P2P, a phone), placement rules (sentences, where models are loaded, the preview matrix, the sheet and its preview, delete with undo) and failover on a cluster. Node detail covers its tabs, the drain preview and its dialog, resume, remove with the typed name, a node that stopped answering, and unload. The nodes specs follow the new structure and keep their intent: the table, filters, grouping, pagination, bulk actions, the map, and running models with stop, logs and the loading, error and empty states. The scheduling, failover, P2P, hub and smoke specs follow the renames. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the rebuilt Swarm pages Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): add helpers for the Traffic pages Pure functions for what the pages work out from the usage ledger, the trace summary, the trace buffers and the resources reading: the shared time window, grouping, sorting and filtering of usage rows, chart series and axes that start at zero, the overview figures, per-model statistics, the state of a trace and the words for a failure, the backend operations that ran during a request, CSV export, the Prometheus metric list and scrape config, and a bounded buffer of host readings. A figure whose source cannot say is null, never zero. The trace summary call takes the window in hours, and a helper reads /metrics with its status. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Traffic hub: overview, usage, models, host, traces, middleware Traffic opens on an overview: five figures (requests, failed, p95, tokens in and out) and three charts, each naming its source. A second row of links reaches Usage, Models, GPU and host, Traces, Middleware and Prometheus, and one time window is shared by the first three. Usage groups by model, user or API key, filters, sorts, opens a row on its own chart, exports the rows it holds as CSV or JSON in the browser, and keeps the opt-in cost estimate and the quota forecast. A user who is not an admin sees only their own numbers. Models joins the ledger, the backend-operation buffer and the loaded models. GPU and host shows the current reading and two charts of readings taken since the page opened. Traces gets filters, a settings strip and an explained off state. An API request is a page: the error LocalAI recorded, a timeline with the backend operations that ran meanwhile, and bodies that stay closed until revealed. Middleware draws the pipeline as five steps and shows the rules of the selected step. Prometheus documents /metrics, checks it against the server and gives a scrape config to copy. Alerts is not built: LocalAI has no alert rules. Per-model latency percentiles, GPU utilisation and compare with the previous period are not drawn because the API does not return them. Legacy usage, trace and middleware styles and the usage source components are removed. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Traffic hub New specs for the overview (figures, charts with a data table and arrow key readout, failed and first-run and tracing-off states, the shared window, a phone), usage (group by, filters, sort, export, cost, quotas, a non-admin, empty and loading), models, GPU and host (snapshot, the since-opened labelling, a cluster), the traces list, a trace page (the real error, the timeline, reveal, no headers, a trace that left the buffer), Prometheus and the Middleware pipeline, with shared fixtures. The usage, traces, middleware, hub and smoke specs follow the new structure and keep their intent. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the Traffic hub Add an operations page for the Traffic tab: which record each page reads, what it leaves out and why, the trace page and its reveal, the GPU and host readings kept since the page opened, and the Prometheus endpoint. Link it from the operations index, the tracing page and the middleware page. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): let metric names wrap in the Prometheus table on a phone The long metric names pushed the type and "on this server" columns out of view. Names now wrap inside the table. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Settings with groups by intent, search, a pending bar and history The fifteen sections become eight groups by intent: memory and models, speed and defaults, backends and galleries, access and security, debugging and traces, agents and responses, swarm and sharing, look and feel. Search covers names, descriptions, keys and the old section name, and says where a result used to be. Edits wait in a bar with Discard, Show diff and Apply. The diff lists old and new values and the checks the browser can make: durations parse the way Go parses them, a GPU memory budget is one the server accepts, a gallery box holds JSON, and warnings repeat what the handler and the field text say. Apply sends only the changed keys. Undo saves the previous values again; it is a new save, not a rollback. History lists the changes applied from this browser, since LocalAI keeps no settings log, and Revert stages the old value. A value is marked as changed only where the built-in default is known from the CLI defaults. A row says "Applies now" or "Needs restart" only where the handler or the docs say so. Three things were wrong before and are fixed with the rebuild: the gallery boxes and the shared API keys box were sent under names the server ignores, the "Enable CSRF Protection" switch showed the disable flag the wrong way round, and every save restarted peer-to-peer networking because every field was sent. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Users and keys, Account, sign-in, invite and the 404 page Users and keys is a tabbed page under the Settings tab: people, invites and API keys. The people table filters by state and role, sorts, approves or disables (disabling offers an undo that sets the status back), and opens a side sheet for one person's features, model allow-list and limits. Role, password reset and delete sit in the row menu; delete asks for the name. Invites choose a lifetime of 1, 7 or 30 days and show the link once. API keys can be created with a lifetime, are shown once in full, can be paused, and are revoked after a ten second undo window in which nothing is sent. LocalAI lists keys only to their owner, so the tab shows the signed-in person's own keys and says so. Account has Profile, Security, API keys and Usage. Usage shows the last 30 days, tokens by model and the limits an admin set. The Security tab now shows for a GitHub or SSO account and says the password is not theirs to change. Sign-in asks for one field per step and draws a provider button only for a provider /api/auth/status lists. It has the notice for a sign-up that waits for approval, the first-admin screen, the key-only screen and the invite page. An address outside the app now gets the 404 page too, which names the address and lists the places the sidebar lists, with the same gates. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover Settings, Users and keys, Account, sign-in and the 404 page Settings: groups, search by name, key and old section, the changed marker only where a default is known, apply hints, the pending bar and diff, the checks, apply sending only changed keys, undo as a second save, discard, history, the CSRF inversion and the gallery and API key wire forms, and the phone layout. Users and keys: the table, filters, sort, approve, disable with undo, the row menu, the access sheet, invites, key creation with a one-time reveal, the ten second revoke with undo and with a page leave, and the non-admin redirect. Account, each sign-in variant (error, pending, first admin, key-only, invite, provider buttons) and the 404 page have specs too. Fixtures are shared with the screenshot scripts. Existing specs follow the new structure. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the rebuilt Settings, Users and keys, Account and sign-in pages Runtime settings: the eight groups and where each old section went, search, the pending bar, the diff and its checks, apply, undo, the history, and which settings show a default or an apply note and why. Authentication: the sign-in screen variants, the Account tabs, key lifetimes, the one-time key reveal, the revoke undo window, and the fact that keys are listed only to their owner. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): let the Settings undo toast stand alone and read back a generated P2P token The saved message and the undo toast sat on the same spot at the bottom of the page. The undo toast now carries the saved message. A new P2P token is made by the server when the page sends 0. The page reads it back after the save so the field shows the token and not the placeholder. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): fit the users table, API keys and Account figures on a phone On a phone the users table dropped its Role and Status columns off the screen edge with the row actions. The role and state now sit under the name, so the actions stay in view. API key rows no longer put the key icon on a line of its own, and the three Account figures keep one row. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the phone users table, reduced motion and the empty Account state Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): drop the apply note from three settings the save handler does not mention Size-aware eviction, automatic backend upgrades and development backends said Applies now, but nothing in the handler or the docs says when they take effect. A row now carries a note only where the code or the docs say so. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild Voices and Faces as one identity family Voices is one page with three tabs: Speakers (voiceprints for recognising who is speaking), Speech voices (the text-to-speech reference library, kept apart because it is a different store) and From a recording (a link into the diarization workspace). Faces uses the same layout. Who is this and Same person? give the answer in a sentence with the real distance and cut-off, a word for how far inside the cut-off it sits, and a distance scale with the cut-off drawn on it. The cut-off slider re-reads the answer in the browser; the identify call sends the cut-off, and verify uses the threshold the model returns. The old confidence percentage is gone because it is not a probability. The server has no list call, so the people list stays in the browser and the page says so. After a search that asked for more people than it got back, a saved person the server did not return is marked, and people the server returned that the browser does not know are listed. Nothing is claimed from a short or cut-off search. Enrolling is a sheet: sample, name, labels, permission. A copy of the sample in the browser is opt-in, and an administrator can also keep the recording as a speech voice in the same step. Removing a person waits ten seconds behind an Undo toast and sends nothing before then. Errors say what happened (no face found, model missing, call failed), a blocked or missing microphone is explained, and a missing model or a missing permission renders a page that says what turns the feature on instead of a redirect. Analyze, detect and raw embedding move under More tools, with attribute guesses off by default. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Voices and Faces pages Specs for who is this (match, no match, working, failed, missing model), the cut-off slider, same person, a blocked, allowed and insecure microphone, the registry notes and the not-on-the-server marks, the enrol sheet and its opt-in copy, delete with undo on a fake clock, the disabled and no-permission states, the phone layout, reduced motion and Faces. Existing library and diarization specs follow the new structure and keep their intent. Node tests cover the distance words, scale layout, stored list and error mapping. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the Voices and Faces pages Add a WebUI section to the voice and face recognition pages: the two tools, the cut-off, what the people list is and why it can be stale, the undo window, and what is stored where. Point the Voice Library and Fish Audio notes at Build, Voices, Speech voices. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): rebuild the Build landing, Fine-tune, Quantize, Import and Explorer The Build landing says what each tool is for and what it needs from the machine: the installed backend, the GPU memory, RAM and disk the server reports, and a job that is running or the newest one when it failed. A tool that cannot run says why and what enables it. Fine-tune and Quantize share one page: set up, a check list that is redrawn as the form changes, a run view with progress, stages and a log, and a result with real next steps (export, import, chat, Models). The checks state only what the server reports. A job needs no estimate the server cannot make, so none is invented. Stop on a fine-tuning job asks whether to keep a checkpoint, a failed job shows the server's message, and a memory failure offers two changes that are applied to a copy of the setup. Import is a guided flow: source, review, import, done. The server returns no preview before an import starts, so the review reads the spelling of the source, prints the request the form will send and runs the checks that can be made early. The estimate that arrives when the import starts is set against free memory and disk. The ambiguity picker and the Write YAML tab stay. Explorer shows what GET /networks returns and lists a swarm with POST /network/add, with a join sheet that carries the token and commands. Build tools the account may not use say so instead of redirecting. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the Build landing, the tool pages, Import and Explorer Specs for the landing (a tool ready, missing a backend, with no GPU, a running or failed job, a feature switched off, a member without admin, phone, reduced motion), the shared tool pattern for Fine-tune and Quantize (set up, live checks, start request, running with progress, chart and log, the stop choice, failure with the server message, finish with next steps, earlier jobs, the account-disabled page, phone), Import (source detection, review, checks, ambiguity, running with the estimate against free memory, done, Write YAML, phone) and Explorer (list, join, list a swarm, empty, not an explorer, retry, phone). Existing specs follow the new structure and keep their intent. Node tests cover the machine facts, tool status, checks, log lines, source detection, the import request and the join commands. Assisted-by: Claude Code:claude-sonnet-5-5 * docs: describe the Build tool pages, the import flow and the Explorer Fine-tuning and quantization now describe the set up, check, run and result steps and what the check list can and cannot say. The import section explains the review step and why the size and memory appear only after the import starts. The distributed page describes the Explorer list, the join sheet and what listing a swarm publishes. Assisted-by: Claude Code:claude-sonnet-5-5 * feat(ui): turn the hardware recommendations into a "Best for this machine" shelf The shelf in the Models inspector put five columns into a 400 px pane, so long model ids wrapped letter by letter underneath the size and the memory figures. Each row now stacks the tag, the id and the size and memory facts beside one Install button, and the id wraps inside its own column. Once a model is installed the shelf narrows to the best fit and keeps the others behind a "N more that fit" toggle. Specs cover the ranking, the layout, the narrowing and the install request against a gallery fixture that carries the 4K estimate the shelf sizes against. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): quiet the Studio tab markers and say their state in words The type tabs drew a saturated green dot for every modality that has a model. The dot now uses a text colour, filled when a model is installed and hollow when none is, and each tab carries "(model installed)" or "(no model installed)" as hidden text so the state is not only a colour. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): stack the model editor empty state and drop its section hues "No fields configured" sat in a flex row, so the icon, the title and the text ran together. It now uses the stacked empty-state layout. The section icons took a different status colour each (amber, red, green); they now share one quiet colour, with the accent on the current section. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): keep the Home memory sentence whole on a phone and drop side rails On a 390 px screen the memory strip clipped "2 models loaded" to make room for the figure. The sentence now takes the first line and the figure and device wrap under it. The sweep also removed coloured left rails from the editor section rail, the skill editor list, the install strip and the audio transform notice (now an outlined note), plus unused chat rules that carried rails and two glow animations that nothing referenced. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): line up hub pages, list the model templates, and stop clipped text Medium-width pages inside Build and Operate were centred while the tab bar above them was flush left, so the title started 60 px right of the first tab. They now start at the bar's edge. Add Model offered nine templates as a grid of identical cards with chip clouds and inline styles. It is now one list of rows, each with the field names it fills in on a single muted line. Two clipped strings are fixed: the Studio voice field cut its placeholder mid-word, and the phone job list ended the schedule line in an ellipsis. The recommendation shelf also separates size and memory with a dot, and the docs describe the shelf. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): keep the hidden Studio tab state inside its tab The hidden state text added to each type tab was absolutely positioned against the page, so on a phone it sat outside the scrolling tab row and widened the page by hundreds of pixels. The tab is now the containing block. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): cover the on-disk sizes on the Installed table, model page and cleanup sheet The fixtures stub GET /api/models/storage. The default report is empty, so existing specs keep the gallery estimates. makeStorage() builds a report from files and the models that use them, the way the server does, and storageSpec() is a models directory with shared and missing files. New specs cover the Size column and its shared line, the fallback when the call fails or the user is not an admin, the files list on the model page, a missing file, the bytes a removal frees with shared files, and the cleanup findings. Node tests cover the storage helpers and the batch arithmetic. Assisted-by: Claude Code:claude-sonnet-5-5 * test(ui): wait for the page before pressing keys and ticking the clock Two specs failed in loaded full runs and passed alone. The Alt+1 to Alt+7 spec pressed a key before the composer had armed its key handler. The capacity chart spec advanced the fake clock before the poller had mounted, so it counted fewer readings than it expected. Both now wait for the page to mount. The key spec retries a press that lands during a re-render, and the clock spec advances in small steps and polls for the row count. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): hide the Installed footer when the storage report is empty An empty report from the storage call made the footer read "0.0 GB on disk" next to sizes taken from the gallery estimate. An empty report says nothing about the disk, so the footer now shows only the model count. A spec covers it. Assisted-by: Claude Code:claude-sonnet-5-5 * fix(ui): keep Explore pane actions inside the pane The inspector actions sat in a non-wrapping flex row beside the title, so the buttons ran past the pane edge once it got narrow. The row now takes its own line and wraps. The primary action (Install, Retry, Open) comes first. Manage installation becomes a ghost button, and Open details moves to the end of the row, so one action stands out and the others are quiet. No action or test id is removed. Add a spec that checks, in light and dark at several widths and with a pane forced to 320 px, that every action stays inside the pane box and that the pane keeps its inner padding. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * refactor(ui): drop the type chips from the Studio composer The Studio tabs and the composer's type chips listed the same seven modes, so the page said the same thing twice. Keep the tabs as the one place to switch modes. The composer now shows the type it will open as a small label in its header. The type suggestion from the typed words stays as the quiet hint line under the prompt, and Alt+1 to Alt+7 still pick a type. The composer root carries data-type, data-types and data-missing so tests can read the state. Specs pick a type through a shared Alt+digit helper and read a missing model from the tab dot instead of a chip. Remove the unused chip locale strings and CSS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * refactor(ui): remove the section crumb above page titles Page headers drew a small uppercase crumb with a short rule before it above the title. On the hub pages it repeated the hub name, so Build sat above a heading that also said Build. PageHeader now renders only the title, the supporting line and the actions. Drop the eyebrow prop, the route-derived section name, its CSS and the unused section helper, and remove the explicit eyebrow props from the pages that passed one. Pages stay reachable through the sidebar and the hub tab bar. Add a spec that checks several pages show their title with nothing ahead of it in the header. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * refactor(ui): remove left accent rails from tiles, rows and quotes Several surfaces marked state with a coloured strip on the left edge. Replace each one with a cue that is not a rail: - Stat cards lose the strip; the icon and value still carry the colour. - The highlighted card is a raised surface with a firmer edge. - The selected rail row is an accent wash with a hairline outline. - The status stripe on rail items is a small status dot. - The active failover row is a tinted row. - Quotes in markdown and chat prose are italic instead of barred. - The variant detail panel has a full hairline border. Add a spec that walks the main routes in light and dark and fails on a left border thicker than 1px, a sideways inset shadow, a narrow absolute strip in ::before or ::after, or a narrow tall child pinned to a left edge. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Ettore Di Giacinto
|
70ce62901f |
refactor: name the capability decisions instead of systemone
The usecase describes what a model can do, and the category is the Decisions API. SystemOne stays as the wire contract: the /v1/systemone routes, the Score RPC question_type and the swagger tag are unchanged. The usecase, flag, auth feature, UI label, gallery tags and docs page are now decisions. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c7f278dd0d |
docs: document the systemone usecase and decisions API
Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
50c284fdcc |
fix: make the remaining VerifyPath checks effective (#12326)
utils.VerifyPath joins its argument onto the base path, so a path that the caller already joined always passes. Several callers gave it joined paths, and their checks could not fail: - modeladmin (config view, patch, edit, pin and state): the config file path from the loader. A config loaded from outside the models directory (--models-config-file) could be pinned, and the pin wrote the outside file. The patch and state paths stopped later, in the mutation snapshot, with a different error. - core/backend/tts.go: the model path joined onto the models path. - The trellis2cpp and stablediffusion-ggml backends: option paths (*_path) joined onto the model path. A "../" value outside the model directory was accepted. Add utils.VerifyResolvedPath for a full path. modeladmin and tts use it. The backends now check the relative option value before they join it. A rename in modeladmin checks the new relative name. For models from a config file outside the models directory, the admin API and web UI now return ErrPathNotTrusted for view, edit, pin, and enable or disable. The docs describe this. Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
956cf4b9b6 |
docs: explain mixed CPU/GPU inference (#12143)
Show partial layer offload and CPU expert placement for llama-cpp. Correct the documented gpu_layers default to match the backend config. Refs #10557 Assisted-by: Codex:gpt-6 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
3e4a44be9d |
fix(diffusers): forward original config for single files
Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
155804e144 |
docs: clarify model configuration precedence
Assisted-by: Codex:gpt-5 [Codex] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
9bd7d17ff6 |
[model-config] feat: add environment variables support for backends (#10721)
* feat: add environment variables support for backends in model configurations - Add field to model configuration to pass environment variables to backend processes - Update backend options and model configuration handling - Add documentation for environment variables configuration with examples including CUDA_VISIBLE_DEVICES Assisted-by: qwen-agentworld-35b-a3b Signed-off-by: nold <nold42@pm.me> * fix(test): Test environment variables configuration parsing from YAML Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: nold <Nold360@users.noreply.github.com> --------- Signed-off-by: nold <nold42@pm.me> Signed-off-by: nold <Nold360@users.noreply.github.com> Co-authored-by: nold <nold42@pm.me> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Stefan Walcz
|
109244a76a |
[chat] feat: template.system_messages_after_first — merge or forward late system turns (#11906)
* feat(chat): template.system_messages_after_first — merge or forward late system turns Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise 'System message must be at the beginning' for system-role messages that appear after the leading system block, while agent frameworks (cogito tool selection and adjustment prompts) legitimately append system instructions mid-conversation. Every such request failed with a 500 (48 errors in one 10-task agent run). New per-model option template.system_messages_after_first: merge fold late system turns into the leading system message user forward them as user-role turns at their original position Default (unset) keeps the current pass-through behaviour. Fixes #11876 Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * docs(model-config): document template.system_messages_after_first Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> * fix(config/meta): register template.system_messages_after_first in the field registry TestAllFieldsHaveRegistryEntries requires every model-config field to have a registry entry. Adds the entry (templates section, select component) and the option list for the new field so the coverage gate passes. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> --------- Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
Plamen K. Kosseff
|
ba88fb13ce |
fix(diffusers): auto-detect CUDA instead of defaulting to CPU (#11891)
The device fell back to CPU unless the model config set cuda: true, while MPS right below was auto-detected — GPU hosts silently rendered on CPU for any gallery entry missing the flag. Use CUDA whenever torch reports it available (ROCm builds included), keep cuda: true as an explicit force, and allow pinning with the device: model option (e.g. options: ["device:cpu"]). Gallery entries stay untouched. Assisted-by: Claude:claude-fable-5 Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com> |
||
|
Abdullah Mansour | عبد الله منصور
|
a98501d6ce |
docs(llama-cpp): clarify multimodal speculative decoding (#11700)
* docs(llama-cpp): clarify multimodal speculative decoding Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example. Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker] Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> * docs(llama-cpp): clarify multimodal MTP references Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility. Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker] Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> --------- Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> |
||
|
Claudio Maradonna
|
30e53f8d9f |
fix(ds4): enforce generation boundaries (#11821)
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits. Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit. Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Claudio Maradonna <git@codeshifter.xyz> |
||
|
Richard Palethorpe
|
49ef40a187 |
feat(classifier/VAD): support voice control on low power devices (#10804)
* feat(llama-cpp): route Score through the slot loop Score previously bypassed the slot loop with a direct llama_decode: a conflict guard aborted the whole process if scoring raced generation, the config validator had to reject score alongside chat/completion/embeddings, and every candidate re-decoded the full shared prompt. Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks are scheduled like any other slot work: generation and scoring serialize naturally, the shared prompt is decoded once per call, and the slot's prompt cache carries the conversation prefix across calls. Context checkpoints at the score boundary and at the cache-divergence point keep SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s to ~0.5s on a desktop CPU. The conflict guard and the validation split are removed; declaring score with generation usecases on one config is now supported and shares the slot cache. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier wire types and pipeline config Wire types and YAML config for realtime classifier mode: sessions carry a localai_classifier extension (options with canned replies/tool calls, softmax threshold, normalization, history trimming, fallback modes, and a deterministic wake-word address gate), mirrored by pipeline.classifier in the model YAML and surfaced in the config-meta registry. The localai.classifier.result server event reports the full score distribution per turn. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier response flow Classifier-mode responses: instead of autoregressive generation, each user turn is prefill-scored against the option list (router.ScoreClassifier prompt/candidate shapes over the Score primitive) and the winning option's canned reply and tool call are emitted through the existing response machinery. Below-threshold turns take the configured fallback (none / canned reply / generate); empty transcripts and unaddressed turns (wake word not mentioned) skip scoring entirely. The scoring probe defaults to the latest user message only — small scorers echo canned replies from prior turns back as the top option otherwise. Built for hardware that can afford prompt processing but not decode: with slot-based Score the option list stays KV-cached across turns, so a turn costs roughly one forward pass over the new words. session_update_error events now carry the validation cause instead of a generic message. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): bound the VAD tick's scan window and buffer retention The VAD tick loop re-scanned the entire input buffer every 300ms and only trimmed it on zero-segment ticks or commits. Audio that keeps producing segments without a committing pause (steady noise a mic pipeline lets through, music, continuous speech) grew the buffer toward the 100MB cap with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero per buffered second: past ~90s retained, ticks run back to back and pin ~4 cores until the stream stops. Silero's recurrent state only carries a few hundred ms of context, so rescanning old audio buys nothing. Clip the slice handed to the VAD to the largest silence the commit test can need to measure (server_vad silence window or the semantic eagerness fallback) plus a warm-up margin, and rebase the returned segment times so every downstream consumer keeps whole-buffer coordinates. An open turn whose clipped window is all silence now commits (the silence outran the window) instead of being discarded as no-speech. Independently, retain at most 90s of raw buffer, rebasing the live-feed and EOU cursors on trim — this also bounds the previously unbounded VAD-error path. Turn boundaries are otherwise unchanged: no forced commits, no new coordinator states. pipeline.turn_detection.vad_window_sec can widen the scan window; values below the automatic floor are ignored. The tick body is extracted into vadTick so specs can drive turn detection synchronously (same shape as classifySoundWindow); the babble reproduction that pinned 4 cores now plateaus under 10% of one core. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(backend): let per-model threads override the global default ModelOptions overrode a set per-model threads value with the app-level --threads whenever the latter was non-zero — and WithThreads defaults it to the physical core count, so it always was. The YAML threads: knob has been dead config: a tiny VAD model could never opt down from the global pool size. SetDefaults already fills an unset per-model value from the app config, which is the intended precedence; resolve threads through a helper that honors it (explicit threads: 0 still means unset). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(gallery): single-thread the silero VAD Silero is a ~2MB recurrent model with no exploitable graph parallelism: measured per-call latency is identical at 1 and 10 ORT threads, while every extra pool thread just spin-waits between the realtime loop's frequent tiny inferences. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(realtime): classifier mode, VAD scan window, threads precedence Document the realtime classifier mode (options, threshold guidance, wake-word address gate, empty-transcript handling), the VAD scan window and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the per-model threads precedence, and the M3 classifier note in the realtime state-machine design doc. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): score all candidates in one batched decode One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot decodes the shared prefix (prompt + longest common candidate token prefix) once, then forks one sequence per candidate off it (metadata-only for the unified KV cache, copy-on-write for recurrent state) and decodes every candidate's unique tail in one llama_decode. Previously each candidate was its own task that restored the boundary checkpoint and re-decoded its full tail sequentially, paying per-candidate task and decode overhead. The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and recurrent-state cells) beyond the parallel slots via the new common_params::n_seq_score_forks. Forking requires the unified KV cache (already this backend's default) since per-sequence streams would shrink n_ctx_seq; an explicit kv_unified:false disables forking and Score calls that need it fail cleanly. Candidates beyond the fork/output budget decode in successive chunks. Wire contract and scores are unchanged: per-token logprobs are stitched from the shared region and the forked tails. Verified bitwise deterministic call-to-call and independent of candidate order (no cross-fork leakage via equal-length candidate swap); ranking matches the per-candidate implementation on the drone battery (winner softmax 0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and empty candidates all pass. Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm realtime classifier turns 196-303ms. The 9-candidate drone turn decodes ~17 unique tail tokens in one batch instead of nine sequential ~220ms checkpoint-restore tasks. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): gate scoring capacity by model usecase Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity. Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step The builder-prebuilt path installs gcc-14 with apt directly and ignored the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already honors, so an ubuntu mirror outage broke every arm64 backend build. Pass the args into the stage and run apt-mirror.sh (already in the build context via COPY . /LocalAI) before the apt step. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier argument slots via constrained completion Hybrid classify-then-complete: a classifier option's canned tool call can declare typed argument slots (number | enum | string, with defaults and prompt hints) referenced as "{{name}}" in the arguments template. When the option wins, the slots are filled by a short grammar-constrained completion that continues the exact scoring prompt — rendered by the same cached ScoreClassifier, so the llama.cpp prompt cache is already warm — with the chosen route JSON re-opened at the first slot field. A GBNF grammar pins the field skeleton and frees only the values; temperature 0, a couple dozen tokens at most (~300ms on a desktop CPU for two slots). Slot declarations and hints ride the option descriptions in the shared system prompt, informing scoring and the fill alike at no per-turn token cost. The localai.classifier.result event carries the final arguments and a fill_latency_ms. On inference failure the slots' defaults apply; a slot without a default fails the response (or falls through with fallback.mode: generate). Slot filling requires completion alongside score in the scoring model's known_usecases. Verified end-to-end on the Pi drone demo: "fly forward three meters" in distance mode classifies forward and infers {"distance": 3, "units": "meters"} in ~310ms, and the drone flies exactly 3 units. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): splice filled slot values into classifier replies A classifier option's spoken reply can now reference its tool's argument slots ("Going forward {{distance}} {{units}}."): the values inferred by the slot-fill completion — or the recovery defaults — are spliced into the reply as plain text before it is emitted, so what the assistant says confirms what it actually inferred. Placeholders without a value stay literal, and options without slots are untouched. FillToolArguments now returns the raw slot values alongside the spliced arguments JSON to make the reply templating possible. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): harden classifier slot completion Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): prewarm the classifier scoring prompt on registration Swapping a session's classifier option list (a voice-switched command mode, for instance) made the next turns pay a full re-prefill of the new option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and worse: on hybrid-memory models like LFM2.5, whose state cannot be partially rewound (llama.cpp can only restore checkpoints), *every* probe change re-prefilled from scratch whenever the last checkpoint missed the probe boundary, so even same-list turns intermittently cost full prefills. Registering an option list (pipeline seed or session.update) now fires a best-effort background prewarm: two throwaway scores with distinct probes. The first prefills the new option-list prompt; the second, diverging exactly where per-turn probe text starts, plants the backend's rewind point (KV checkpoint) at the stable-prefix boundary that every real turn reuses. The prewarm hides behind the canned mode-switch reply — by the time it finishes speaking, the cache is warm. Idempotent per option set, detached from the registering request's lifetime. Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after a mode switch 2374ms -> 340ms; intermittent same-list full prefills (1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently, options: [parallel:2] on the scoring model additionally keeps one slot per list via prefix-similarity routing (+26MB RSS, unified KV). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new small models are headed) cannot rewind their state, so any prompt-cache reuse that needs a rewind falls back to a full re-prefill. For classifier scoring that meant every probe change re-processed the whole option-list prompt: the server's checkpoints were placed reactively (at wherever the previous task happened to diverge), so a checkpoint past the next divergence was erased rather than restored — measured as intermittent 2-10s turns on prompts with a 95%+ common prefix. The classifier now computes the probe-invariant prompt prefix once (the byte-wise common prefix of two synthetic probe renders) and declares its length with every Score request; the server maps it to a token boundary and forces a KV checkpoint exactly there on each score prefill. That checkpoint sits at or before every future divergence under the same option list, so it always survives and always restores — repeat scoring costs probe+candidates regardless of how the probe changes. Also: - prewarm reruns on every option-list registration instead of memoizing per list: with boundary checkpoints a redundant rewarm costs two probe-sized decodes, while skipping one after a slot eviction (three lists sharing fewer slots evict in LRU cascades) silently moves a full re-prefill onto the user's next turn - new llama.cpp backend option rs_seq:N exposes bounded recurrent-state rollback outside speculative decoding; measured impractical for deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap insurance for small-state models - docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot similarity threshold funnels distinct lists onto one slot) Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady state: every turn 285-421ms including mode switches, vs 2.4s post-switch and intermittent 1.3-2.9s re-prefills before. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): align classifier cache guidance Document the single-score prewarm behavior and clean the vendored score patch formatting. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(llama-cpp): guard score task for fork backends TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields. Assisted-by: Codex:gpt-5 [gh] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(dev): generate gRPC code before commit lint The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
40d35c0385 |
docs: onboarding overhaul, dedup, and error docs (#7711) (#10895)
* docs: fix CPU image tag (latest, not latest-cpu) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: use canonical localai/localai registry in models guide Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: replace dead llama-stable backend with llama-cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct mitm-proxy intercept config and redaction tier Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix text-to-audio endpoint and broken notice block Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix VAD example, stale FAQ, broken link, CLI list, whats-new dump Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: render advanced/reference section indexes (consolidate _index) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove duplicate getting-started build/kubernetes pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold container image reference into installation/containers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove stale advanced fine-tuning page (superseded by features/fine-tuning) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold distribution/longcat/sound pages into their parents Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: make getting-started index accurate and complete Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: carry one concrete model through the getting-started path Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add end-to-end 'build your first agent' walkthrough Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add runtime errors reference; consolidate troubleshooting from FAQ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add agent actions catalog Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: agent-scoped MCP, skills walkthrough, agentic disambiguation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add concrete gallery install lines to media feature pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: merge installation into getting-started (URLs preserved via aliases) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add Operations section; move operator pages and P2P API reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: journey-ordered top nav and grouped feature sections Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add docs-with-code process gate (PR template + agent instructions) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove em/en dashes from documentation prose Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
pos-ei-don
|
06b4a29387 |
docs(config): document grpc.attempts timing + tuning guidance (#10868)
The gRPC configuration table only listed the two fields with a one-line description each, without defaults, without explaining what the total load window looks like, and without hinting when a user should adjust them. In practice the default 20 attempts x 2 s = 40 s window is way too tight for large NVFP4 / FP8 models on slow storage or first-run CUDA-graph capture, and the resulting kill (exitCode=120, 'context canceled') looks like a backend crash even though the backend is still making legitimate forward progress. Extend the section with: - Defaults column (20 and 2) added to the table - Prose explaining that these govern the readiness handshake between LocalAI and a freshly spawned backend (Health polling loop) - Total-load-window formula - Concrete failure signature so users can recognize a timeout-kill vs. a real backend crash - Example configuration for a ~10 min cold-load window (grpc.attempts 140, attempts_sleep_time 5), with a note that inference-timeouts and the watchdog are unaffected. |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bcc41219f7 |
feat: materialize Hugging Face model artifacts (#10825)
* feat(config): add model artifact source contract Assisted-by: Codex:GPT-5 [Codex] * feat(downloader): add authenticated raw-byte progress Assisted-by: Codex:GPT-5 [Codex] * feat(huggingface): resolve immutable snapshot manifests Assisted-by: Codex:GPT-5 [Codex] * feat(models): add artifact storage primitives Assisted-by: Codex:GPT-5 [Codex] * feat(models): materialize pinned Hugging Face snapshots Assisted-by: Codex:GPT-5 [Codex] * feat(models): bind managed snapshots at runtime Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): materialize model artifacts during install Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): declare managed Hugging Face artifacts Assisted-by: Codex:GPT-5 [Codex] * feat(models): preload managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * fix(gallery): retain shared artifact caches on delete Assisted-by: Codex:GPT-5 [Codex] * feat(models): report artifact acquisition progress Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load managed models from ModelFile Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load staged speech model snapshots Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): use staged snapshots in engine backends Assisted-by: Codex:GPT-5 [Codex] * test(distributed): cover staged artifact snapshots Assisted-by: Codex:GPT-5 [Codex] * docs: explain managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * docs: add product design context Assisted-by: Codex:GPT-5 [Codex] * feat(ui): show model artifact download progress Assisted-by: Codex:GPT-5 [Codex] * Eagerly materialize Hugging Face artifacts Materialize HF-backed model references as managed GGUF artifacts during load, with lazy download retained only as fallback. Assisted-by: Codex:GPT-5 [shell] * Refactor HF downloads through a shared executor Assisted-by: Codex:GPT-5 [shell] * drop Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
4056283aa4 |
[voice] feat: add managed voice cloning profiles (#10799)
* feat(ui): add voice library workflow Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech. Assisted-by: Codex:gpt-5 * feat(voice): add managed voice cloning profiles Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends. Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
b00422e45f |
feat(backends): add LongCat video and avatar generation (#10792)
* feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
5569b2de56 |
feat(config): context_size: -1 to auto-use model's full trained context (#10752)
* feat(config): clamp negative context_size to default in EffectiveContextSize Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(config): resolve context_size=-1 to model trained max with VRAM warn Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(config): treat negative context_size as unset when GGUF is unparseable Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(config): document context_size=-1 auto-max Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(backend): drop em dashes from EffectiveContextSize comment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
40dae953f4 |
feat: interleaved thinking with tool calls (reasoning_content alias + Anthropic thinking blocks) (#10744)
* feat(schema): accept reasoning_content as inbound alias for reasoning Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content on assistant turns. Accept it as an inbound alias so reasoning survives the tool-result loop; canonical reasoning wins when both are present. Emission is unchanged (still reasoning). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(schema): pin interleaved reasoning+tool_calls round-trip Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(openai): pin reachedTokenBudget truncation detection Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): add thinking and signature fields to content blocks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): parse inbound thinking blocks into reasoning Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): emit thinking blocks with synthetic signature on tool turns Extract buildAnthropicContentBlocks so non-streaming content assembly is unit-testable, and prepend a thinking block (with an opaque synthetic signature) before text/tool_use blocks when the request opts into thinking. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): stream thinking_delta and signature_delta before tool_use Extract anthropicStreamSequence so the streaming block order is unit-testable, and emit content_block_start(thinking) -> thinking_delta -> signature_delta -> content_block_stop before the tool_use block sequence when thinking is enabled. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add interleaved thinking with tool calls guide Add a features guide describing interleaved thinking: an assistant turn carrying reasoning and tool_calls together, the reasoning-round-trip contract (including the reasoning_content inbound alias and Anthropic thinking blocks with a synthetic signature), per-backend enablement (reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for vLLM/SGLang plus the vLLM auto-config hook), a worked request/response example, and known limitations. Cross-link from model-configuration, text-generation, and openai-functions. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Roman Mazurenkoandrvmzes
|
a3fdfbc0d1 |
feat(llama-cpp): add device selection option (#10724)
Allow llama.cpp model configs to select the backend devices used for offload, matching upstream --device behavior so users can exclude a display or debug GPU. Signed-off-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local> Co-authored-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
066abf82c0 |
feat(llama-cpp): cpu_moe/n_cpu_moe options + generic upstream-flag passthrough (#10490)
* feat(llama-cpp): add main-model cpu_moe/n_cpu_moe options Mirror the existing draft_cpu_moe/draft_n_cpu_moe siblings for the main model, matching upstream --cpu-moe / --n-cpu-moe (common/arg.cpp). Lets users keep MoE expert weights on CPU to manage VRAM on large MoE models. Closes part of #10483 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): forward unknown '-' options to upstream arg parser Any options: entry starting with '-' is collected and passed verbatim to llama.cpp's own common_params_parse (LLAMA_EXAMPLE_SERVER) at the end of params_parse, so every upstream llama-server flag works without a new hand-wired branch. Passthrough runs last and wins on overlap; n_parallel is snapshotted to survive parser_init's SERVER reset, and help/usage/completion flags are skipped to avoid exiting the backend. Closes #10483 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(llama-cpp): document cpu_moe/n_cpu_moe and option passthrough Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): terminate tensor/kv override vectors after passthrough The tensor_buft_overrides padding and the kv/draft override terminators ran before the generic option passthrough, so a passthrough flag (--cpu-moe, --override-tensor, --override-kv, ...) appended a real entry after the null sentinel - tripping the model loader's back().pattern == nullptr assertion (crash) or being silently dropped. Move all three termination/padding blocks to the end of params_parse, after both the named-option loop and common_params_parse have pushed their real entries. Also widen the exit()-flag skip list so --version, --license, --list-devices and --cache-list cannot terminate the backend. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3fa7b2955c |
feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360)
Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
1ab61a0875 |
feat: generic chat_template_kwargs (model config + per-request metadata) (#10359)
* feat(config): add chat_template_kwargs model field + resolver Adds the ChatTemplateKwargs model-config map and RequestMetadata carrier, plus ResolveChatTemplateKwargs which layers the config map under coerced request metadata. Foundation for generic jinja chat-template kwargs (issue #10329). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): forward resolved chat_template_kwargs blob to backends gRPCPredictOpts now merges per-request client metadata over the server-derived enable_thinking/reasoning_effort (reaching all backends via the standalone keys) and serialises the resolved chat_template_kwargs map into a JSON blob for llama.cpp, written last so a client cannot clobber it. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(http): wire request metadata to config.RequestMetadata The OpenAI request metadata field was parsed but unused; stamp it onto the per-request ModelConfig so gRPCPredictOpts forwards it as chat_template_kwargs overrides. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): generic chat_template_kwargs merge (drop per-key blocks) Replace the per-key enable_thinking/reasoning_effort handling in both the streaming and non-streaming chat paths with a single block that parses the chat_template_kwargs JSON blob resolved by the Go layer and merges every key into body_json. New jinja template levers (e.g. preserve_thinking) now need no C++ change. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document custom chat_template_kwargs (model + per-request) Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(backend): pin reasoning_effort as a string in the chat_template_kwargs blob Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(http): e2e guard pinning chat_template_kwargs forwarded to gRPC Adds an ECHO_PREDICT_METADATA marker to the mock-backend that echoes the received PredictOptions.Metadata, and an app_test.go spec that drives a real /v1/chat/completions request (model chat_template_kwargs + per-request metadata override) and asserts the exact metadata + chat_template_kwargs blob the REST layer forwards to gRPC. Locks the REST->gRPC contract against regressions. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(config): grandfather chat_template_kwargs in registry coverage chat_template_kwargs is a free-form map[string]any (like engine_args, already on the list), not a scalar the config UI registry can surface, so it is exempt from the registry-entry requirement. Fixes the TestAllFieldsHaveRegistryEntries failure introduced by the new field. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
f618636c71 |
docs: fix broken relref to realtime page (#10255)
Hugo fails the gh-pages build with REF_NOT_FOUND because the relref in model-configuration.md uses the 'docs/' prefix; refs are resolved relative to content/, so the page lives at 'features/openai-realtime' (as the other ref in the same file already uses). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
e837921c2c |
feat: forward reasoning_effort to the backend so jinja models honor it (#10184)
* feat: forward reasoning_effort to the backend so jinja models honor it reasoning_effort was only mapped to the binary enable_thinking toggle and otherwise reached Go-side templates — it was never sent to the backend. So jinja-templated models whose chat template keys on reasoning_effort (gpt-oss Harmony, LFM2.5) could not be driven by it: LFM2.5 ignores enable_thinking and kept emitting <think>. Forward the effective reasoning_effort to the backend as a chat_template_kwarg (mirroring enable_thinking) in grpc-server.cpp, and put it in PredictOptions metadata (gRPCPredictOpts). Add a config-level default: ModelConfig.reasoning_effort and Pipeline.reasoning_effort, resolved by ModelConfig.ApplyReasoningEffort (request value overrides config default, none->disable / level->enable, an operator's reasoning.disable wins). request.go now uses that helper. Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): set the pipeline LLM's reasoning_effort Apply Pipeline.ReasoningEffort to the pipeline's LLM config when the realtime model is built (per-session copy, overrides the LLM's own reasoning_effort), and surface the resolved effort on the template input so Go-templated models get it too. jinja models receive it via the backend metadata. This lets a realtime pipeline disable thinking on models that only honor reasoning_effort (e.g. LFM2.5), which enable_thinking can't. Assisted-by: Claude:claude-opus-4-8 go test, golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
4a2cc64d07 |
feat(reasoning): honor per-request reasoning_effort on chat completions (#10082)
The OpenAI `reasoning_effort` field only reached the prompt template; it never toggled the backend's thinking. Map it onto ReasoningConfig.DisableReasoning (which becomes the enable_thinking gRPC metadata) in the request merge, so reasoning_effort="none" disables reasoning per request: the use case from #10072 (run a single Qwen3-style model and turn reasoning off for low-latency tasks while keeping it on for others). Effort levels (minimal/low/medium/high) enable thinking unless the model config explicitly disabled it (reasoning.disable: true wins and is never re-enabled by a request); "none" always disables. Closes #10072 Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
959de86761 |
feat(llama-cpp): make server-side prompt cache work by default (#9925)
Aligns LocalAI's llama-cpp gRPC backend with upstream's auto-on prompt cache path so repeated system prompts (agents, OpenAI/Anthropic-compatible CLIs, coding assistants) skip prefill on subsequent calls without any YAML changes. Reported in #9921. Upstream's server enables `kv_unified=true` (and bumps `n_parallel` to 4) when slot count is auto, which unlocks `cache_idle_slots`. LocalAI hardcodes `n_parallel=1` and so far also hardcoded `kv_unified=false`, which silently force-disables idle-slot saving at server init. The host prompt cache was allocated but never written across requests. Changes in backend/cpp/llama-cpp/grpc-server.cpp: - params.kv_unified: false -> true (single-slot path now benefits from the prompt cache; users can opt out with `kv_unified:false`) - params.n_ctx_checkpoints: 8 -> 32 (match upstream default) - params.cache_idle_slots = true initialized explicitly (upstream default) - params.checkpoint_every_nt = 8192 initialized explicitly (upstream default) - New option parsers: cache_idle_slots / idle_slots_cache, checkpoint_every_nt / checkpoint_every_n_tokens Docs: - features/text-generation.md: fix misleading `cache_ram` description (it's the host-side prompt cache, not the KV cache), document the kv_unified + cache_ram + cache_idle_slots interaction, add rows for the two newly-exposed options, and add a worked example for the agent/CLI workload from the issue. - advanced/model-configuration.md: mark the legacy `prompt_cache_path` / `prompt_cache_all` / `prompt_cache_ro` YAML fields as unused by the llama-cpp gRPC backend (they target upstream's CLI completion tool and are not consumed by grpc-server.cpp) and point readers at the new prompt-cache explainer. Closes #9921 Assisted-by: claude:opus-4.7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
d77a9137d8 |
feat(llama-cpp): bump to MTP-merge SHA and automatically set MTP defaults (#9852)
* feat(llama-cpp): bump to MTP-merge SHA and document draft-mtp spec type Update LLAMA_VERSION to 0253fb21 (post ggml-org/llama.cpp#22673 merge, 2026-05-16) to pick up Multi-Token Prediction support. No grpc-server.cpp changes are required: the existing `spec_type` option delegates to upstream's `common_speculative_types_from_names()`, which already accepts the new `draft-mtp` name. The `n_rs_seq` cparam needed by MTP is auto-derived inside `common_context_params_to_llama` from `params.speculative.need_n_rs_seq()`, and when no `draft_model` is set the upstream server builds the MTP context off the target model itself. Docs: extend the speculative-decoding section of the model-configuration guide with the new type, both load paths (MTP head embedded in the main GGUF vs. separate `mtp-*.gguf` sibling), the PR's recommended `spec_n_max:2-3`, and the chained `draft-mtp,ngram-mod` recipe. Also notes that the upstream `-hf` auto-discovery of `mtp-*.gguf` siblings is not wired through LocalAI's gRPC layer. Agent guide: short note explaining that new upstream spec types are picked up automatically and that MTP needs no gRPC plumbing. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): auto-detect MTP heads and enable draft-mtp on import + load Detect upstream's `<arch>.nextn_predict_layers` GGUF metadata key (set by `convert_hf_to_gguf.py` for Qwen3.5/3.6 family models and similar) and, when present and the user has not configured a `spec_type` explicitly, auto-append the upstream-recommended speculative-decoding tuple: - spec_type:draft-mtp - spec_n_max:6 - spec_p_min:0.75 The 0.75 p_min is pinned defensively because upstream marks the current default with a "change to 0.0f" TODO; locking it here keeps acceptance thresholds stable across future llama.cpp bumps. Detection runs in two places: - The model importer (`POST /models/import-uri`, the `/import-model` UI) range-fetches the GGUF header for HuggingFace / direct-URL imports via `gguf.ParseGGUFFileRemote`, with a 30s timeout and non-fatal error handling. OCI/Ollama URIs are skipped because the artifact is not directly streamable; the load-time hook covers them once the file is on disk. - The llama-cpp load-time hook (`guessGGUFFromFile`) reads the local header on every model start and appends the same options if `spec_type` is not already set. Both paths share `ApplyMTPDefaults` and respect an explicit user-set `spec_type:` / `speculative_type:` so YAML overrides win. Ginkgo specs cover the append, preserve-user-choice, legacy alias, and nil safety paths. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importer): resolve huggingface:// URIs before MTP header probe `gguf.ParseGGUFFileRemote` only speaks HTTP(S), but the importer was handing it the raw `huggingface://...` URI directly (and similarly for any other custom downloader scheme). Live-test against `huggingface://ggml-org/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-MTP-Q8_0.gguf` exposed this: the probe failed with `unsupported protocol scheme "huggingface"`, was caught by the non-fatal error path, and the MTP options were silently never applied to the generated YAML. Route every candidate URI through `downloader.URI.ResolveURL()` and require the resolved form to be HTTP(S). After the fix the probe successfully reads `<arch>.nextn_predict_layers=1` from the real HF GGUF and the emitted ConfigFile carries spec_type:draft-mtp, spec_n_max:6, spec_p_min:0.75 as intended. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
6e1dbae256 |
feat(llama-cpp): expose 12 missing common_params via options[] (#9814)
The llama.cpp backend already accepts a free-form options: array in the
model config that maps to common_params fields, but a coverage audit
against upstream pin 7f3f843c flagged 12 user-visible knobs that were
neither set via the typed proto fields nor reachable via options:.
Wire them up under the existing if/else chain in params_parse, before
the speculative section. Each new option follows the file's prevailing
patterns (try/catch around numeric parses, the same true/1/yes/on bool
form used elsewhere, hardware_concurrency() fallback for thread counts,
mirror of draft_override_tensor for override_tensor).
Top-level / batching / IO:
- n_ubatch (alias ubatch) -- physical batch size; was previously
force-aliased to n_batch at line 482, blocking embedding/rerank
workloads that need independent control
- threads_batch (alias n_threads_batch) -- main-model batch threads;
mirrors the existing draft_threads_batch
- direct_io (alias use_direct_io) -- O_DIRECT model loads
- verbosity -- llama.cpp log threshold (line 479 had this commented
out)
- override_tensor (alias tensor_buft_overrides) -- per-tensor buffer
overrides for the main model; mirrors draft_override_tensor
Embedding / multimodal:
- pooling_type (alias pooling) -- mean/cls/last/rank/none; previously
only auto-flipped to RANK for rerankers
- embd_normalize (alias embedding_normalize) -- and the embedding
handler now reads params_base.embd_normalize instead of a hardcoded
2 at the previous embd_normalize literal in Embedding()
- mmproj_use_gpu (alias mmproj_offload) -- mmproj on CPU vs GPU
- image_min_tokens / image_max_tokens -- per-image vision token budget
Reasoning surface (the audit-focus three; LocalAI's existing
ReasoningConfig.DisableReasoning only feeds the per-request
chat_template_kwargs.enable_thinking and does not touch any of these):
- reasoning_format -- none/auto/deepseek/deepseek-legacy parser
- enable_reasoning (alias reasoning_budget) -- -1/0/>0 thinking budget
- prefill_assistant -- trailing-assistant-message prefill toggle
All 14 referenced fields exist on both the upstream pin and the
turboquant fork's common.h, so no LOCALAI_LEGACY_LLAMA_CPP_SPEC guard
is needed.
Docs: extend model-configuration.md with new "Reasoning Models",
"Multimodal Backend Options", "Embedding & Reranking Backend Options",
and "Other Backend Tuning Options" subsections; also refresh the
Speculative Type Values table to show the new dash-separated canonical
names alongside the underscore aliases LocalAI still accepts.
Assisted-by: claude-code:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
bc4cd3dd85 |
feat(llama-cpp): bump to 1ec7ba0c, adapt grpc-server, expose new spec-decoding options (#9765)
* chore(llama.cpp): bump to 1ec7ba0c14f33f17e980daeeda5f35b225d41994
Picks up the upstream `spec : parallel drafting support` change
(ggml-org/llama.cpp#22838) which reshapes the speculative-decoding API
and `server_context_impl`.
Adapt the grpc-server wrapper accordingly:
* `common_params_speculative::type` (single enum) became `types`
(`std::vector<common_speculative_type>`). Update both the
"default to draft when a draft model is set" branch and the
`spec_type`/`speculative_type` option parser. The parser now also
tolerates comma-separated lists, mirroring the upstream
`common_speculative_types_from_names` semantics.
* `common_params_speculative_draft::n_ctx` is gone (draft now shares
the target context size). Keep the `draft_ctx_size` option name for
backward compatibility and ignore the value rather than failing.
* `server_context_impl::model` was renamed to `model_tgt`; update the
two reranker / model-metadata call sites.
Replaces #9763. Builds cleanly under the linux/amd64 cpu-llama-cpp
target locally.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): expose new speculative-decoding option keys
Upstream `spec : parallel drafting support` (ggml-org/llama.cpp#22838)
adds the `ngram_mod`, `ngram_map_k`, and `ngram_map_k4v` speculative
families and beefs up the draft-model knobs. The previous bump only
adapted the API; this exposes the new fields through the grpc-server
options dictionary so model configs can drive them.
New `options:` keys (all under `backend: llama-cpp`):
ngram_mod (`ngram_mod` type):
spec_ngram_mod_n_min / spec_ngram_mod_n_max / spec_ngram_mod_n_match
ngram_map_k (`ngram_map_k` type):
spec_ngram_map_k_size_n / spec_ngram_map_k_size_m / spec_ngram_map_k_min_hits
ngram_map_k4v (`ngram_map_k4v` type):
spec_ngram_map_k4v_size_n / spec_ngram_map_k4v_size_m /
spec_ngram_map_k4v_min_hits
ngram lookup caches (`ngram_cache` type):
spec_lookup_cache_static / lookup_cache_static
spec_lookup_cache_dynamic / lookup_cache_dynamic
Draft-model tuning (active when `spec_type` is `draft`):
draft_cache_type_k / spec_draft_cache_type_k
draft_cache_type_v / spec_draft_cache_type_v
draft_threads / spec_draft_threads
draft_threads_batch / spec_draft_threads_batch
draft_cpu_moe / spec_draft_cpu_moe (bool flag)
draft_n_cpu_moe / spec_draft_n_cpu_moe (first N MoE layers on CPU)
draft_override_tensor / spec_draft_override_tensor
(comma-separated <tensor regex>=<buffer type>; re-implements upstream's
static parse_tensor_buffer_overrides since it isn't exported)
`spec_type` already accepted comma-separated lists after the previous
commit, matching upstream's `common_speculative_types_from_names`.
Docs: refresh `docs/content/advanced/model-configuration.md` with
per-family tables and a note about multi-type chaining.
Builds locally with `make docker-build-llama-cpp` (linux/amd64
cpu-llama-cpp AVX variant).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(turboquant): bridge new llama.cpp spec API to the legacy fork layout
The previous commits in this series adapted backend/cpp/llama-cpp/grpc-server.cpp
to the post-#22838 (parallel drafting) llama.cpp API. The turboquant build
reuses the same grpc-server.cpp through backend/cpp/turboquant/Makefile,
which copies it into turboquant-<flavor>-build/ and runs patch-grpc-server.sh
on the copy. The fork branched before the API refactor, so it errors out on:
* `ctx_server.impl->model_tgt` (fork still has `model`)
* `params.speculative.{ngram_mod,ngram_map_k,ngram_map_k4v,ngram_cache}.*`
(none of these sub-structs exist in the fork)
* `params.speculative.draft.{cache_type_k/v, cpuparams[, _batch].n_threads,
tensor_buft_overrides}` (fork uses the pre-#22397 flat layout)
* `params.speculative.types` vector / `common_speculative_types_from_names`
(fork has a scalar `type` and only the singular helper)
Approach:
1. backend/cpp/llama-cpp/grpc-server.cpp: introduce a single feature switch
`LOCALAI_LEGACY_LLAMA_CPP_SPEC`. When defined, the two `speculative.type[s]`
discriminations (the "default to draft when a draft model is set" branch
and the `spec_type` / `speculative_type` option parser) fall back to the
singular scalar form, and the entire new-option block (ngram_mod / map_k
/ map_k4v / ngram_cache / draft.{cache_type_*, cpuparams*,
tensor_buft_overrides}) is preprocessed out. The macro is *not* defined
in the source tree — stock llama-cpp builds get the full new API.
2. backend/cpp/turboquant/patch-grpc-server.sh: two new patch steps applied
to the per-flavor build copy at turboquant-<flavor>-build/grpc-server.cpp:
- substitute `ctx_server.impl->model_tgt` -> `ctx_server.impl->model`
- inject `#define LOCALAI_LEGACY_LLAMA_CPP_SPEC 1` before the first
`#include`, so the guarded blocks above drop out for the fork build.
Both patches are idempotent and follow the existing sed/awk pattern in
this script (KV cache types, `get_media_marker`, flat speculative
renames). Stock llama-cpp's `grpc-server.cpp` is never touched.
Drop both legacy patches once the turboquant fork rebases past
ggml-org/llama.cpp#22397 / #22838.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(turboquant): close draft_ctx_size brace inside legacy guard
The previous turboquant fix wrapped the new option-handler blocks in
`#ifndef LOCALAI_LEGACY_LLAMA_CPP_SPEC ... #endif` but placed the guard
in the middle of an `else if` chain — the `} else if` openings of the
new blocks were responsible for closing the previous block's brace.
With the macro defined the new blocks vanish, draft_ctx_size's `{`
loses its closer, the for-loop's `}` is consumed instead, and the
file ends with a stray opening brace — clang reports it as
`function-definition is not allowed here before '{'` on the next
top-level `int main(...)` and `expected '}' at end of input`.
Move the chain split inside the draft_ctx_size branch:
} else if (... "draft_ctx_size") {
// ...
#ifdef LOCALAI_LEGACY_LLAMA_CPP_SPEC
} // legacy: chain ends here
#else
} else if (... "spec_ngram_mod_n_min") { // modern: chain continues
...
} else if (... "draft_override_tensor") {
...
} // closes last branch
#endif
} // closes for-loop
Brace count is now balanced under both preprocessor branches (verified
with `tr -cd '{' | wc -c` against the patched and unpatched outputs).
Local `make docker-build-turboquant` builds the linux/amd64 cpu-llama-cpp
`turboquant-avx` variant cleanly.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): forward AMDGPU_TARGETS into Dockerfile.turboquant builder-prebuilt
Dockerfile.turboquant's `builder-prebuilt` stage was missing the
`ARG AMDGPU_TARGETS` / `ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}` pair that
`builder-fromsource` already has (and that `Dockerfile.llama-cpp`
mirrors across both stages). When CI uses the prebuilt base image
(quay.io/go-skynet/ci-cache:base-grpc-*, the common path) the build-arg
passed by the workflow never reaches the env inside the compile stage.
backend/cpp/llama-cpp/Makefile:38 (introduced by #9626) errors out on
hipblas builds when AMDGPU_TARGETS is empty, and the turboquant
Makefile reuses backend/cpp/llama-cpp via a sibling build dir, so the
same check fires from turboquant-fallback under BUILD_TYPE=hipblas:
Makefile:38: *** AMDGPU_TARGETS is empty — set it to a comma-separated
list of gfx targets e.g. gfx1100,gfx1101. Stop.
make: *** [Makefile:66: turboquant-fallback] Error 2
The bug is latent on master because the docker layer cache stays warm
across builds — the compile step rarely re-runs from scratch. The
llama.cpp bump in this PR invalidates the cache, so the missing env var
becomes load-bearing and the hipblas turboquant CI job fails.
Mirror the existing pattern from Dockerfile.llama-cpp.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
95efb8a562 |
feat(backend): add turboquant llama.cpp-fork backend (#9355)
* feat(backend): add turboquant llama.cpp-fork backend
turboquant is a llama.cpp fork (TheTom/llama-cpp-turboquant, branch
feature/turboquant-kv-cache) that adds a TurboQuant KV-cache scheme.
It ships as a first-class backend reusing backend/cpp/llama-cpp sources
via a thin wrapper Makefile: each variant target copies ../llama-cpp
into a sibling build dir and invokes llama-cpp's build-llama-cpp-grpc-server
with LLAMA_REPO/LLAMA_VERSION overridden to point at the fork. No
duplication of grpc-server.cpp — upstream fixes flow through automatically.
Wires up the full matrix (CPU, CUDA 12/13, L4T, L4T-CUDA13, ROCm, SYCL
f32/f16, Vulkan) in backend.yml and the gallery entries in index.yaml,
adds a tests-turboquant-grpc e2e job driven by BACKEND_TEST_CACHE_TYPE_K/V=q8_0
to exercise the KV-cache config path (backend_test.go gains dedicated env
vars wired into ModelOptions.CacheTypeKey/Value — a generic improvement
usable by any llama.cpp-family backend), and registers a nightly auto-bump
PR in bump_deps.yaml tracking feature/turboquant-kv-cache.
scripts/changed-backends.js gets a special-case so edits to
backend/cpp/llama-cpp/ also retrigger the turboquant CI pipeline, since
the wrapper reuses those sources.
* feat(turboquant): carry upstream patches against fork API drift
turboquant branched from llama.cpp before upstream commit 66060008
("server: respect the ignore eos flag", #21203) which added the
`logit_bias_eog` field to `server_context_meta` and a matching
parameter to `server_task::params_from_json_cmpl`. The shared
backend/cpp/llama-cpp/grpc-server.cpp depends on that field, so
building it against the fork unmodified fails.
Cherry-pick that commit as a patch file under
backend/cpp/turboquant/patches/ and apply it to the cloned fork
sources via a new apply-patches.sh hook called from the wrapper
Makefile. Simplifies the build flow too: instead of hopping through
llama-cpp's build-llama-cpp-grpc-server indirection, the wrapper now
drives the copied Makefile directly (clone -> patch -> build).
Drop the corresponding patch whenever the fork catches up with
upstream — the build fails fast if a patch stops applying, which
is the signal to retire it.
* docs: add turboquant backend section + clarify cache_type_k/v
Document the new turboquant (llama.cpp fork with TurboQuant KV-cache)
backend alongside the existing llama-cpp / ik-llama-cpp sections in
features/text-generation.md: when to pick it, how to install it from
the gallery, and a YAML example showing backend: turboquant together
with cache_type_k / cache_type_v.
Also expand the cache_type_k / cache_type_v table rows in
advanced/model-configuration.md to spell out the accepted llama.cpp
quantization values and note that these fields apply to all
llama.cpp-family backends, not just vLLM.
* feat(turboquant): patch ggml-rpc GGML_OP_COUNT assertion
The fork adds new GGML ops bringing GGML_OP_COUNT to 97, but
ggml/include/ggml-rpc.h static-asserts it equals 96, breaking
the GGML_RPC=ON build paths (turboquant-grpc / turboquant-rpc-server).
Carry a one-line patch that updates the expected count so the
assertion holds. Drop this patch whenever the fork fixes it upstream.
* feat(turboquant): allow turbo* KV-cache types and exercise them in e2e
The shared backend/cpp/llama-cpp/grpc-server.cpp carries its own
allow-list of accepted KV-cache types (kv_cache_types[]) and rejects
anything outside it before the value reaches llama.cpp's parser. That
list only contains the standard llama.cpp types — turbo2/turbo3/turbo4
would throw "Unsupported cache type" at LoadModel time, meaning
nothing the LocalAI gRPC layer accepted was actually fork-specific.
Add a build-time augmentation step (patch-grpc-server.sh, called from
the turboquant wrapper Makefile) that inserts GGML_TYPE_TURBO2_0/3_0/4_0
into the allow-list of the *copied* grpc-server.cpp under
turboquant-<flavor>-build/. The original file under backend/cpp/llama-cpp/
is never touched, so the stock llama-cpp build keeps compiling against
vanilla upstream which has no notion of those enum values.
Switch test-extra-backend-turboquant to set
BACKEND_TEST_CACHE_TYPE_K=turbo3 / _V=turbo3 so the e2e gRPC suite
actually runs the fork's TurboQuant KV-cache code paths (turbo3 also
auto-enables flash_attention in the fork). Picking q8_0 here would
only re-test the standard llama.cpp path that the upstream llama-cpp
backend already covers.
Refresh the docs (text-generation.md + model-configuration.md) to
list turbo2/turbo3/turbo4 explicitly and call out that you only get
the TurboQuant code path with this backend + a turbo* cache type.
* fix(turboquant): rewrite patch-grpc-server.sh in awk, not python3
The builder image (ubuntu:24.04 stage-2 in Dockerfile.turboquant)
does not install python3, so the python-based augmentation step
errored with `python3: command not found` at make time. Switch to
awk, which ships in coreutils and is already available everywhere
the rest of the wrapper Makefile runs.
* Apply suggestion from @mudler
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
---------
Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
|
||
|
Ettore Di Giacinto
|
06fbe48b3f |
feat(llama.cpp): wire speculative decoding settings (#9238)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
f73a158153 |
docs: Document GPU auto-fit mode limitations and trade-offs (closes #8562) (#8954)
* docs: Add documentation about GPU auto-fit mode limitations (closes #8562) - Document the default gpu_layers behavior (9999999) that disables auto-fit - Explain the trade-off between auto-fit and VRAM threshold unloading - Add recommendations for users who want to enable gpu_layers: -1 - Note known issues with tensor_buft_override buffer errors - Link to issue #8562 for future improvements Signed-off-by: team-coding-agent-1 <team-coding-agent-1@localai.dev> * Apply suggestion from @mudler Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> --------- Signed-off-by: team-coding-agent-1 <team-coding-agent-1@localai.dev> Signed-off-by: Ettore Di Giacinto <mudler@users.noreply.github.com> Co-authored-by: team-coding-agent-1 <team-coding-agent-1@localai.dev> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
580517f9db |
feat: pass-by metadata to predict options (#8795)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
dd8e74a486 |
feat(realtime): Add audio conversations (#6245)
* feat(realtime): Add audio conversations Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(realtime): Vendor the updated API and modify for server side Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): Update to the GA realtime API Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore: Document realtime API and add docs to AGENTS.md Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat: Filter reasoning from spoken output Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): Send delta and done events for tool calls and audio transcripts Ensure that content is sent in both deltas and done events for function call arguments and audio transcripts. This fixes compatibility with clients that rely on delta events for parsing. 💘 Generated with Crush Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): Improve tool call handling and error reporting - Refactor Model interface to accept []types.ToolUnion and *types.ToolChoiceUnion instead of JSON strings, eliminating unnecessary marshal/unmarshal cycles - Fix Parameters field handling: support both map[string]any and JSON string formats - Add PredictConfig() method to Model interface for accessing model configuration - Add comprehensive debug logging for tool call parsing and function config - Add missing return statement after prediction error (critical bug fix) - Add warning logs for NoAction function argument parsing failures - Improve error visibility throughout generateResponse function 💘 Generated with Crush Assisted-by: Claude Sonnet 4.5 via Crush <crush@charm.land> Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
c491c6ca90 |
feat(openresponses): Support reasoning blocks (#8133)
* feat(openresponses): support reasoning blocks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * allow to disable reasoning, refactor common logic Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add option to only strip reasoning Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add configurations for custom reasoning tokens Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2cc4809b0d |
feat: docs revamp (#7313)
* docs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Small enhancements Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Enhancements * Default to zen-dark Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fixups Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |