Merge branch 'master' into feat/audio-cpp-backend-v2

* feat(llama-cpp): route Score through the slot loop

Score previously bypassed the slot loop with a direct llama_decode: a
conflict guard aborted the whole process if scoring raced generation, the
config validator had to reject score alongside chat/completion/embeddings,
and every candidate re-decoded the full shared prompt.

Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks
are scheduled like any other slot work: generation and scoring serialize
naturally, the shared prompt is decoded once per call, and the slot's
prompt cache carries the conversation prefix across calls. Context
checkpoints at the score boundary and at the cache-divergence point keep
SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole
prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s
to ~0.5s on a desktop CPU.

The conflict guard and the validation split are removed; declaring score
with generation usecases on one config is now supported and shares the
slot cache.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier wire types and pipeline config

Wire types and YAML config for realtime classifier mode: sessions carry a
localai_classifier extension (options with canned replies/tool calls,
softmax threshold, normalization, history trimming, fallback modes, and a
deterministic wake-word address gate), mirrored by pipeline.classifier in
the model YAML and surfaced in the config-meta registry. The
localai.classifier.result server event reports the full score distribution
per turn.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier response flow

Classifier-mode responses: instead of autoregressive generation, each user
turn is prefill-scored against the option list (router.ScoreClassifier
prompt/candidate shapes over the Score primitive) and the winning option's
canned reply and tool call are emitted through the existing response
machinery. Below-threshold turns take the configured fallback (none /
canned reply / generate); empty transcripts and unaddressed turns (wake
word not mentioned) skip scoring entirely. The scoring probe defaults to
the latest user message only — small scorers echo canned replies from
prior turns back as the top option otherwise.

Built for hardware that can afford prompt processing but not decode: with
slot-based Score the option list stays KV-cached across turns, so a turn
costs roughly one forward pass over the new words.

session_update_error events now carry the validation cause instead of a
generic message.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): bound the VAD tick's scan window and buffer retention

The VAD tick loop re-scanned the entire input buffer every 300ms and only
trimmed it on zero-segment ticks or commits. Audio that keeps producing
segments without a committing pause (steady noise a mic pipeline lets
through, music, continuous speech) grew the buffer toward the 100MB cap
with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero
per buffered second: past ~90s retained, ticks run back to back and pin
~4 cores until the stream stops.

Silero's recurrent state only carries a few hundred ms of context, so
rescanning old audio buys nothing. Clip the slice handed to the VAD to the
largest silence the commit test can need to measure (server_vad silence
window or the semantic eagerness fallback) plus a warm-up margin, and
rebase the returned segment times so every downstream consumer keeps
whole-buffer coordinates. An open turn whose clipped window is all silence
now commits (the silence outran the window) instead of being discarded as
no-speech. Independently, retain at most 90s of raw buffer, rebasing the
live-feed and EOU cursors on trim — this also bounds the previously
unbounded VAD-error path. Turn boundaries are otherwise unchanged: no
forced commits, no new coordinator states.

pipeline.turn_detection.vad_window_sec can widen the scan window; values
below the automatic floor are ignored. The tick body is extracted into
vadTick so specs can drive turn detection synchronously (same shape as
classifySoundWindow); the babble reproduction that pinned 4 cores now
plateaus under 10% of one core.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(backend): let per-model threads override the global default

ModelOptions overrode a set per-model threads value with the app-level
--threads whenever the latter was non-zero — and WithThreads defaults it
to the physical core count, so it always was. The YAML threads: knob has
been dead config: a tiny VAD model could never opt down from the global
pool size.

SetDefaults already fills an unset per-model value from the app config,
which is the intended precedence; resolve threads through a helper that
honors it (explicit threads: 0 still means unset).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(gallery): single-thread the silero VAD

Silero is a ~2MB recurrent model with no exploitable graph parallelism:
measured per-call latency is identical at 1 and 10 ORT threads, while
every extra pool thread just spin-waits between the realtime loop's
frequent tiny inferences.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(realtime): classifier mode, VAD scan window, threads precedence

Document the realtime classifier mode (options, threshold guidance,
wake-word address gate, empty-transcript handling), the VAD scan window
and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the
per-model threads precedence, and the M3 classifier note in the realtime
state-machine design doc.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): score all candidates in one batched decode

One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot
decodes the shared prefix (prompt + longest common candidate token
prefix) once, then forks one sequence per candidate off it
(metadata-only for the unified KV cache, copy-on-write for recurrent
state) and decodes every candidate's unique tail in one llama_decode.
Previously each candidate was its own task that restored the boundary
checkpoint and re-decoded its full tail sequentially, paying
per-candidate task and decode overhead.

The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and
recurrent-state cells) beyond the parallel slots via the new
common_params::n_seq_score_forks. Forking requires the unified KV cache
(already this backend's default) since per-sequence streams would shrink
n_ctx_seq; an explicit kv_unified:false disables forking and Score calls
that need it fail cleanly. Candidates beyond the fork/output budget
decode in successive chunks.

Wire contract and scores are unchanged: per-token logprobs are stitched
from the shared region and the forked tails. Verified bitwise
deterministic call-to-call and independent of candidate order (no
cross-fork leakage via equal-length candidate swap); ranking matches the
per-candidate implementation on the drone battery (winner softmax
0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and
empty candidates all pass.

Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm
realtime classifier turns 196-303ms. The 9-candidate drone turn decodes
~17 unique tail tokens in one batch instead of nine sequential ~220ms
checkpoint-restore tasks.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): gate scoring capacity by model usecase

Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity.

Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step

The builder-prebuilt path installs gcc-14 with apt directly and ignored
the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already
honors, so an ubuntu mirror outage broke every arm64 backend build. Pass
the args into the stage and run apt-mirror.sh (already in the build
context via COPY . /LocalAI) before the apt step.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier argument slots via constrained completion

Hybrid classify-then-complete: a classifier option's canned tool call can
declare typed argument slots (number | enum | string, with defaults and
prompt hints) referenced as "{{name}}" in the arguments template. When
the option wins, the slots are filled by a short grammar-constrained
completion that continues the exact scoring prompt — rendered by the same
cached ScoreClassifier, so the llama.cpp prompt cache is already warm —
with the chosen route JSON re-opened at the first slot field. A GBNF
grammar pins the field skeleton and frees only the values; temperature 0,
a couple dozen tokens at most (~300ms on a desktop CPU for two slots).

Slot declarations and hints ride the option descriptions in the shared
system prompt, informing scoring and the fill alike at no per-turn token
cost. The localai.classifier.result event carries the final arguments and
a fill_latency_ms. On inference failure the slots' defaults apply; a slot
without a default fails the response (or falls through with
fallback.mode: generate). Slot filling requires completion alongside
score in the scoring model's known_usecases.

Verified end-to-end on the Pi drone demo: "fly forward three meters" in
distance mode classifies forward and infers {"distance": 3, "units":
"meters"} in ~310ms, and the drone flies exactly 3 units.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): splice filled slot values into classifier replies

A classifier option's spoken reply can now reference its tool's argument
slots ("Going forward {{distance}} {{units}}."): the values inferred by
the slot-fill completion — or the recovery defaults — are spliced into
the reply as plain text before it is emitted, so what the assistant says
confirms what it actually inferred. Placeholders without a value stay
literal, and options without slots are untouched.

FillToolArguments now returns the raw slot values alongside the spliced
arguments JSON to make the reply templating possible.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): harden classifier slot completion

Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): prewarm the classifier scoring prompt on registration

Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.

Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.

Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix

Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new
small models are headed) cannot rewind their state, so any prompt-cache
reuse that needs a rewind falls back to a full re-prefill. For classifier
scoring that meant every probe change re-processed the whole option-list
prompt: the server's checkpoints were placed reactively (at wherever the
previous task happened to diverge), so a checkpoint past the next
divergence was erased rather than restored — measured as intermittent
2-10s turns on prompts with a 95%+ common prefix.

The classifier now computes the probe-invariant prompt prefix once (the
byte-wise common prefix of two synthetic probe renders) and declares its
length with every Score request; the server maps it to a token boundary
and forces a KV checkpoint exactly there on each score prefill. That
checkpoint sits at or before every future divergence under the same
option list, so it always survives and always restores — repeat scoring
costs probe+candidates regardless of how the probe changes.

Also:
- prewarm reruns on every option-list registration instead of memoizing
  per list: with boundary checkpoints a redundant rewarm costs two
  probe-sized decodes, while skipping one after a slot eviction (three
  lists sharing fewer slots evict in LRU cascades) silently moves a full
  re-prefill onto the user's next turn
- new llama.cpp backend option rs_seq:N exposes bounded recurrent-state
  rollback outside speculative decoding; measured impractical for
  deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap
  insurance for small-state models
- docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot
  similarity threshold funnels distinct lists onto one slot)

Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady
state: every turn 285-421ms including mode switches, vs 2.4s post-switch
and intermittent 1.3-2.9s re-prefills before.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): align classifier cache guidance

Document the single-score prewarm behavior and clean the vendored score patch formatting.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(llama-cpp): guard score task for fork backends

TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields.

Assisted-by: Codex:gpt-5 [gh]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(dev): generate gRPC code before commit lint

The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>

.docker/llama-cpp-compile.sh

@@ -28,6 +28,10 @@ if [ -z "${BUILD_TYPE:-}" ]; then
   # variants with it (the host never *selects* SME unless it has it, but every variant must
   # still compile).
   if [ "${TARGETARCH}" = "arm64" ]; then
+    # The prebuilt base inherits default ports.ubuntu.com sources; honor the
+    # APT_*_MIRROR build args here like the from-source path does, so this
+    # apt step survives a mirror outage.
+    sh /LocalAI/.docker/apt-mirror.sh || true
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi

.github/backend-matrix.yml

@@ -66,6 +66,34 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-kokoro'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'true'
+    backend: "kokoro"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""

backend/Dockerfile.llama-cpp

@@ -111,6 +111,10 @@ RUN make -BC /LocalAI/backend/cpp/llama-cpp package
 # ============================================================================
 FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
 
+ARG APT_MIRROR
+ENV APT_MIRROR=${APT_MIRROR}
+ARG APT_PORTS_MIRROR
+ENV APT_PORTS_MIRROR=${APT_PORTS_MIRROR}
 ARG BUILD_TYPE
 ENV BUILD_TYPE=${BUILD_TYPE}
 ARG CUDA_DOCKER_ARCH

backend/backend.proto

@@ -181,6 +181,13 @@ message ScoreRequest {
   // PredictOptions.ModelIdentity for the full rationale. Empty means "no
   // identity supplied" and backends MUST skip the check.
   string ModelIdentity = 5;
+  // Byte length of the prompt prefix that stays identical across
+  // repeated scoring calls (e.g. a classifier's option-list system
+  // prompt — everything before the per-turn probe text). Backends that
+  // snapshot state (hybrid/recurrent models cannot rewind otherwise)
+  // use it to place a reuse point exactly at the boundary, so the next
+  // call re-processes only the tokens after it. 0 means unknown.
+  int32 stable_prefix_len = 6;
 }
 
 // CandidateScore is one row in the ScoreResponse, matching by index
@@ -493,6 +500,11 @@ message ModelOptions {
   // Proxy carries the cloud-proxy backend's per-model configuration.
   // Empty for non-proxy backends.
   ProxyOptions Proxy = 74;
+
+  // EnableScore reserves backend resources for the Score RPC. It is derived
+  // from the model's explicit `known_usecases: [score]` declaration so models
+  // that never score retain their ordinary serving footprint.
+  bool EnableScore = 75;
 }
 
 // ProxyOptions configures the cloud-proxy backend. UpstreamURL and
@@ -508,6 +520,12 @@ message ProxyOptions {
   string api_key_file = 5;
   string upstream_model = 6;
   int32 request_timeout_seconds = 7;
+  // cache_prompt enables automatic Anthropic prompt-cache breakpoints
+  // (cache_control: ephemeral) on the stable prefix — system, tools, and
+  // the last message block — when translating to the Anthropic provider.
+  // Cuts input cost on repeated/agentic calls (cache read = 0.1x). Only
+  // meaningful for mode=translate + provider=anthropic; ignored otherwise.
+  bool cache_prompt = 8;
 }
 
 message Result {

backend/cpp/audio-cpp/CMakeLists.txt

@@ -0,0 +1,95 @@
+# SPDX-License-Identifier: MIT
+
+cmake_minimum_required(VERSION 3.20)
+project(audio-cpp-grpc-server LANGUAGES CXX)
+
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_EXTENSIONS OFF)
+
+set(AUDIO_CPP_DIR "${CMAKE_CURRENT_SOURCE_DIR}/audio.cpp"
+    CACHE PATH "Path to the audio.cpp source tree")
+set(LOCALAI_BACKEND_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/../../backend.proto"
+    CACHE FILEPATH "Path to the LocalAI backend protocol")
+
+option(ENGINE_ENABLE_CUDA "Build audio.cpp with CUDA support" OFF)
+option(ENGINE_ENABLE_VULKAN "Build audio.cpp with Vulkan support" OFF)
+option(ENGINE_ENABLE_METAL "Build audio.cpp with Metal support" OFF)
+option(AUDIO_CPP_BUILD_TESTS "Build LocalAI audio.cpp unit tests" OFF)
+option(AUDIO_CPP_BUILD_GRPC "Build the LocalAI gRPC server" ON)
+
+find_package(Threads REQUIRED)
+
+if(NOT EXISTS "${AUDIO_CPP_DIR}/CMakeLists.txt")
+  message(FATAL_ERROR
+      "AUDIO_CPP_DIR does not point to an audio.cpp source tree: ${AUDIO_CPP_DIR}")
+endif()
+add_subdirectory("${AUDIO_CPP_DIR}" "${CMAKE_CURRENT_BINARY_DIR}/audio.cpp")
+
+add_library(localai_audio_cpp_runtime STATIC
+    audio_cpp_runtime.cpp
+    model_config.cpp)
+target_include_directories(localai_audio_cpp_runtime
+    PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}")
+target_link_libraries(localai_audio_cpp_runtime PUBLIC engine_runtime)
+
+if(AUDIO_CPP_BUILD_GRPC)
+  find_package(Protobuf CONFIG REQUIRED)
+  find_package(gRPC CONFIG REQUIRED)
+  find_program(PROTOC_EXECUTABLE NAMES protoc REQUIRED)
+  find_program(GRPC_CPP_PLUGIN_EXECUTABLE NAMES grpc_cpp_plugin REQUIRED)
+
+  get_filename_component(LOCALAI_BACKEND_PROTO_DIR
+      "${LOCALAI_BACKEND_PROTO}" DIRECTORY)
+  set(LOCALAI_PROTO_SOURCES
+      "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.cc"
+      "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.cc")
+  set(LOCALAI_PROTO_HEADERS
+      "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.h"
+      "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.h")
+
+  add_custom_command(
+      OUTPUT ${LOCALAI_PROTO_SOURCES} ${LOCALAI_PROTO_HEADERS}
+      COMMAND "${PROTOC_EXECUTABLE}"
+      ARGS
+        --cpp_out "${CMAKE_CURRENT_BINARY_DIR}"
+        --grpc_out "${CMAKE_CURRENT_BINARY_DIR}"
+        -I "${LOCALAI_BACKEND_PROTO_DIR}"
+        --plugin=protoc-gen-grpc="${GRPC_CPP_PLUGIN_EXECUTABLE}"
+        "${LOCALAI_BACKEND_PROTO}"
+      DEPENDS "${LOCALAI_BACKEND_PROTO}"
+      VERBATIM)
+
+  add_library(localai_backend_proto STATIC
+      ${LOCALAI_PROTO_SOURCES}
+      ${LOCALAI_PROTO_HEADERS})
+  target_include_directories(localai_backend_proto
+      PUBLIC "${CMAKE_CURRENT_BINARY_DIR}")
+  target_link_libraries(localai_backend_proto
+      PUBLIC protobuf::libprotobuf gRPC::grpc++)
+
+  # Task 2 replaces this generated entry point with the LocalAI service.
+  set(AUDIO_CPP_SERVER_PLACEHOLDER
+      "${CMAKE_CURRENT_BINARY_DIR}/audio-cpp-grpc-server-placeholder.cpp")
+  file(GENERATE OUTPUT "${AUDIO_CPP_SERVER_PLACEHOLDER}"
+      CONTENT "int main() { return 0; }\n")
+
+  add_executable(audio-cpp-grpc-server "${AUDIO_CPP_SERVER_PLACEHOLDER}")
+  target_link_libraries(audio-cpp-grpc-server PRIVATE
+      localai_audio_cpp_runtime
+      engine_runtime
+      localai_backend_proto
+      gRPC::grpc++
+      gRPC::grpc++_reflection)
+endif()
+
+if(AUDIO_CPP_BUILD_TESTS)
+  enable_testing()
+  add_executable(audio-cpp-runtime-test tests/runtime_tests.cpp)
+  target_link_libraries(audio-cpp-runtime-test PRIVATE
+      localai_audio_cpp_runtime
+      Threads::Threads)
+  add_test(
+      NAME audio-cpp-runtime-test
+      COMMAND audio-cpp-runtime-test "${AUDIO_CPP_DIR}")
+endif()

backend/cpp/audio-cpp/Makefile

@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: MIT
+
+AUDIO_CPP_VERSION?=f8fb0c19739193adfad0d9e58da99f25eda65256
+AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
+AUDIO_CPP_SRC?=
+
+CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
+BUILD_DIR := build
+
+BUILD_TYPE ?=
+JOBS ?= $(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+UNAME_S := $(shell uname -s)
+
+CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
+CMAKE_ARGS += -DENGINE_ENABLE_CUDA=OFF
+CMAKE_ARGS += -DENGINE_ENABLE_VULKAN=OFF
+CMAKE_ARGS += -DENGINE_ENABLE_METAL=OFF
+
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS += -DENGINE_ENABLE_CUDA=ON
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS += -DENGINE_ENABLE_VULKAN=ON
+else ifeq ($(UNAME_S),Darwin)
+	CMAKE_ARGS += -DENGINE_ENABLE_METAL=ON
+endif
+
+.PHONY: all grpc-server test test-unit clean purge
+
+all: grpc-server
+
+audio.cpp:
+ifneq ($(AUDIO_CPP_SRC),)
+	ln -sfn $(abspath $(AUDIO_CPP_SRC)) audio.cpp
+else
+	mkdir -p audio.cpp
+	cd audio.cpp && \
+	git init -q && \
+	git remote add origin $(AUDIO_CPP_REPO) && \
+	git fetch --depth 1 origin $(AUDIO_CPP_VERSION) && \
+	git checkout --detach FETCH_HEAD && \
+	git submodule update --init --recursive --depth 1
+endif
+
+grpc-server: audio.cpp
+	mkdir -p $(BUILD_DIR)
+	cd $(BUILD_DIR) && cmake $(CMAKE_ARGS) $(CURRENT_MAKEFILE_DIR)
+	cmake --build $(BUILD_DIR) --config Release \
+		--target audio-cpp-grpc-server -j $(JOBS)
+	cp $(BUILD_DIR)/audio-cpp-grpc-server grpc-server
+
+test:
+	bash tests/build_contract_test.sh
+
+test-unit: audio.cpp
+	mkdir -p $(BUILD_DIR)-unit
+	cd $(BUILD_DIR)-unit && cmake $(CMAKE_ARGS) \
+		-DAUDIO_CPP_BUILD_TESTS=ON -DAUDIO_CPP_BUILD_GRPC=OFF \
+		$(CURRENT_MAKEFILE_DIR)
+	cmake --build $(BUILD_DIR)-unit --config Release \
+		--target audio-cpp-runtime-test -j $(JOBS)
+	ctest --test-dir $(BUILD_DIR)-unit --output-on-failure
+
+clean:
+	rm -rf $(BUILD_DIR) $(BUILD_DIR)-unit grpc-server
+
+purge: clean
+	rm -rf audio.cpp

backend/cpp/audio-cpp/audio_cpp_runtime.cpp

@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: MIT
+
+#include "audio_cpp_runtime.h"
+
+#include <algorithm>
+#include <stdexcept>
+#include <string>
+#include <utility>
+
+namespace audio_cpp {
+namespace {
+
+using engine::runtime::CapabilitySet;
+using engine::runtime::RunMode;
+using engine::runtime::TaskSpec;
+
+const engine::runtime::TaskCapability * find_task_capability(
+    const CapabilitySet & capabilities,
+    const TaskSpec & task) {
+    const auto it = std::find_if(
+        capabilities.supported_tasks.begin(),
+        capabilities.supported_tasks.end(),
+        [&](const engine::runtime::TaskCapability & capability) {
+            return capability.task == task.task;
+        });
+    return it == capabilities.supported_tasks.end() ? nullptr : &*it;
+}
+
+void validate_capability(
+    const engine::runtime::ILoadedVoiceModel & model,
+    const AudioCppModelConfig & config) {
+    if (model.metadata().family != config.family) {
+        throw std::runtime_error(
+            "loaded audio.cpp model family '" + model.metadata().family +
+            "' does not match requested family '" + config.family + "'");
+    }
+
+    const auto * capability = find_task_capability(
+        model.capabilities(),
+        config.task);
+    if (capability == nullptr) {
+        throw std::runtime_error(
+            "loaded audio.cpp model does not support requested task '" +
+            std::string(engine::runtime::to_string(config.task.task)) + "'");
+    }
+    if (std::find(
+            capability->modes.begin(),
+            capability->modes.end(),
+            config.task.mode) == capability->modes.end()) {
+        throw std::runtime_error(
+            "loaded audio.cpp model does not support requested mode '" +
+            std::string(engine::runtime::to_string(config.task.mode)) +
+            "' for task '" +
+            std::string(engine::runtime::to_string(config.task.task)) + "'");
+    }
+}
+
+void validate_session(
+    const engine::runtime::IVoiceTaskSession & session,
+    const AudioCppModelConfig & config) {
+    if (session.family() != config.family) {
+        throw std::runtime_error("audio.cpp session returned the wrong family");
+    }
+    if (session.task_kind() != config.task.task) {
+        throw std::runtime_error("audio.cpp session returned the wrong task");
+    }
+    if (session.run_mode() != config.task.mode) {
+        throw std::runtime_error("audio.cpp session returned the wrong mode");
+    }
+    if (config.task.mode == RunMode::Offline &&
+        dynamic_cast<const engine::runtime::IOfflineVoiceTaskSession *>(&session) == nullptr) {
+        throw std::runtime_error("audio.cpp session does not implement offline execution");
+    }
+    if (config.task.mode == RunMode::Streaming &&
+        dynamic_cast<const engine::runtime::IStreamingVoiceTaskSession *>(&session) == nullptr) {
+        throw std::runtime_error("audio.cpp session does not implement streaming execution");
+    }
+}
+
+}  // namespace
+
+AudioCppRuntime::AudioCppRuntime()
+    : registry_(engine::runtime::make_default_registry()) {}
+
+AudioCppRuntime::AudioCppRuntime(engine::runtime::ModelRegistry registry)
+    : registry_(std::move(registry)) {}
+
+AudioCppRuntime::~AudioCppRuntime() {
+    free();
+}
+
+void AudioCppRuntime::load(const AudioCppModelConfig & config) {
+    std::lock_guard<std::mutex> lock(mutex_);
+
+    auto candidate_model = registry_.load(config.load);
+    if (candidate_model == nullptr) {
+        throw std::runtime_error("audio.cpp registry returned a null model");
+    }
+    validate_capability(*candidate_model, config);
+
+    auto candidate_session = candidate_model->create_task_session(
+        config.task,
+        config.session);
+    if (candidate_session == nullptr) {
+        throw std::runtime_error("audio.cpp model returned a null session");
+    }
+    validate_session(*candidate_session, config);
+
+    free_locked();
+    model_ = std::move(candidate_model);
+    session_ = std::move(candidate_session);
+}
+
+void AudioCppRuntime::free() {
+    std::lock_guard<std::mutex> lock(mutex_);
+    free_locked();
+}
+
+engine::runtime::TaskResult AudioCppRuntime::run(
+    const engine::runtime::TaskRequest & request) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    auto & session = require_session_locked();
+    session.prepare(engine::runtime::build_preparation_request(request));
+    return require_offline_locked().run(request);
+}
+
+void AudioCppRuntime::start_stream(
+    const engine::runtime::TaskRequest & request) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    auto & session = require_session_locked();
+    session.prepare(engine::runtime::build_preparation_request(request));
+    require_streaming_locked().start_stream(request);
+}
+
+engine::runtime::StreamEvent AudioCppRuntime::process_audio_chunk(
+    const engine::runtime::AudioChunk & chunk) {
+    std::lock_guard<std::mutex> lock(mutex_);
+    return require_streaming_locked().process_audio_chunk(chunk);
+}
+
+engine::runtime::TaskResult AudioCppRuntime::finish_stream() {
+    std::lock_guard<std::mutex> lock(mutex_);
+    return require_streaming_locked().finish_stream();
+}
+
+engine::runtime::IVoiceTaskSession & AudioCppRuntime::require_session_locked() {
+    if (session_ == nullptr) {
+        throw std::runtime_error("audio.cpp runtime has no loaded session");
+    }
+    return *session_;
+}
+
+engine::runtime::IOfflineVoiceTaskSession &
+AudioCppRuntime::require_offline_locked() {
+    auto * offline = dynamic_cast<engine::runtime::IOfflineVoiceTaskSession *>(
+        &require_session_locked());
+    if (offline == nullptr) {
+        throw std::runtime_error("loaded audio.cpp session is not offline");
+    }
+    return *offline;
+}
+
+engine::runtime::IStreamingVoiceTaskSession &
+AudioCppRuntime::require_streaming_locked() {
+    auto * streaming = dynamic_cast<engine::runtime::IStreamingVoiceTaskSession *>(
+        &require_session_locked());
+    if (streaming == nullptr) {
+        throw std::runtime_error("loaded audio.cpp session is not streaming");
+    }
+    return *streaming;
+}
+
+void AudioCppRuntime::free_locked() {
+    session_.reset();
+    model_.reset();
+}
+
+}  // namespace audio_cpp

backend/cpp/audio-cpp/audio_cpp_runtime.h

@@ -0,0 +1,46 @@
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include "model_config.h"
+
+#include "engine/framework/runtime/registry.h"
+#include "engine/framework/runtime/session.h"
+
+#include <memory>
+#include <mutex>
+
+namespace audio_cpp {
+
+class AudioCppRuntime {
+public:
+    AudioCppRuntime();
+    explicit AudioCppRuntime(engine::runtime::ModelRegistry registry);
+    ~AudioCppRuntime();
+
+    AudioCppRuntime(const AudioCppRuntime &) = delete;
+    AudioCppRuntime & operator=(const AudioCppRuntime &) = delete;
+
+    void load(const AudioCppModelConfig & config);
+    void free();
+
+    engine::runtime::TaskResult run(
+        const engine::runtime::TaskRequest & request);
+    void start_stream(const engine::runtime::TaskRequest & request);
+    engine::runtime::StreamEvent process_audio_chunk(
+        const engine::runtime::AudioChunk & chunk);
+    engine::runtime::TaskResult finish_stream();
+
+private:
+    engine::runtime::IVoiceTaskSession & require_session_locked();
+    engine::runtime::IOfflineVoiceTaskSession & require_offline_locked();
+    engine::runtime::IStreamingVoiceTaskSession & require_streaming_locked();
+    void free_locked();
+
+    std::mutex mutex_;
+    engine::runtime::ModelRegistry registry_;
+    std::unique_ptr<engine::runtime::ILoadedVoiceModel> model_;
+    std::unique_ptr<engine::runtime::IVoiceTaskSession> session_;
+};
+
+}  // namespace audio_cpp

backend/cpp/audio-cpp/model_config.cpp

@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: MIT
+
+#include "model_config.h"
+
+#include <limits>
+#include <stdexcept>
+#include <string>
+
+namespace audio_cpp {
+namespace {
+
+using engine::core::BackendType;
+using engine::runtime::RunMode;
+using engine::runtime::VoiceTaskKind;
+
+std::string require_option(
+    const std::unordered_map<std::string, std::string> & options,
+    const std::string & name) {
+    const auto it = options.find(name);
+    if (it == options.end() || it->second.empty()) {
+        throw std::invalid_argument("audio.cpp model config requires " + name);
+    }
+    return it->second;
+}
+
+VoiceTaskKind parse_task(const std::string & value) {
+    static const std::unordered_map<std::string, VoiceTaskKind> tasks = {
+        {"vad", VoiceTaskKind::Vad},
+        {"asr", VoiceTaskKind::Asr},
+        {"diarization", VoiceTaskKind::Diarization},
+        {"source-separation", VoiceTaskKind::SourceSeparation},
+        {"audio-generation", VoiceTaskKind::AudioGeneration},
+        {"tts", VoiceTaskKind::Tts},
+        {"voice-cloning", VoiceTaskKind::VoiceCloning},
+        {"voice-conversion", VoiceTaskKind::VoiceConversion},
+        {"speech-to-speech", VoiceTaskKind::SpeechToSpeech},
+        {"alignment", VoiceTaskKind::Alignment},
+        {"voice-design", VoiceTaskKind::VoiceDesign},
+        {"speaker-recognition", VoiceTaskKind::SpeakerRecognition},
+        {"svc", VoiceTaskKind::Svc},
+    };
+    const auto it = tasks.find(value);
+    if (it == tasks.end()) {
+        throw std::invalid_argument("unsupported audio.cpp task: " + value);
+    }
+    return it->second;
+}
+
+RunMode parse_mode(const std::string & value) {
+    if (value == "offline") {
+        return RunMode::Offline;
+    }
+    if (value == "streaming") {
+        return RunMode::Streaming;
+    }
+    throw std::invalid_argument("unsupported audio.cpp mode: " + value);
+}
+
+BackendType parse_backend(const std::string & value) {
+    if (value == "cpu") {
+        return BackendType::Cpu;
+    }
+    if (value == "cuda") {
+        return BackendType::Cuda;
+    }
+    if (value == "vulkan") {
+        return BackendType::Vulkan;
+    }
+    if (value == "metal") {
+        return BackendType::Metal;
+    }
+    if (value == "best") {
+        return BackendType::BestAvailable;
+    }
+    throw std::invalid_argument("unsupported audio.cpp backend: " + value);
+}
+
+int parse_integer(
+    const std::string & name,
+    const std::string & value,
+    int minimum) {
+    size_t parsed = 0;
+    long result = 0;
+    try {
+        result = std::stol(value, &parsed);
+    } catch (const std::exception &) {
+        throw std::invalid_argument("invalid audio.cpp " + name + ": " + value);
+    }
+    if (parsed != value.size() ||
+        result < minimum ||
+        result > std::numeric_limits<int>::max()) {
+        throw std::invalid_argument("invalid audio.cpp " + name + ": " + value);
+    }
+    return static_cast<int>(result);
+}
+
+void copy_namespaced_option(
+    const std::string & key,
+    const std::string & prefix,
+    const std::string & value,
+    std::unordered_map<std::string, std::string> & destination) {
+    const std::string name = key.substr(prefix.size());
+    if (name.empty()) {
+        throw std::invalid_argument("audio.cpp option namespace requires a name: " + key);
+    }
+    destination[name] = value;
+}
+
+}  // namespace
+
+AudioCppModelConfig parse_model_config(
+    const std::filesystem::path & model_path,
+    const std::unordered_map<std::string, std::string> & options) {
+    AudioCppModelConfig config;
+    config.model_path = model_path;
+    config.family = require_option(options, "family");
+    config.task.task = parse_task(require_option(options, "task"));
+    config.task.mode = RunMode::Offline;
+    config.load.model_path = model_path;
+    config.load.family_hint = config.family;
+    config.session.backend.type = BackendType::Cpu;
+
+    if (const auto it = options.find("mode"); it != options.end()) {
+        config.task.mode = parse_mode(it->second);
+    }
+    if (const auto it = options.find("backend"); it != options.end()) {
+        config.session.backend.type = parse_backend(it->second);
+    }
+    if (const auto it = options.find("device"); it != options.end()) {
+        config.session.backend.device = parse_integer("device", it->second, 0);
+    }
+    if (const auto it = options.find("threads"); it != options.end()) {
+        config.session.backend.threads = parse_integer("threads", it->second, 1);
+    }
+    if (const auto it = options.find("model_spec"); it != options.end()) {
+        config.load.model_spec_override = std::filesystem::path(it->second);
+    }
+    if (const auto it = options.find("config_id"); it != options.end()) {
+        config.load.config_id = it->second;
+    }
+    if (const auto it = options.find("weight_id"); it != options.end()) {
+        config.load.weight_id = it->second;
+    }
+
+    for (const auto & [key, value] : options) {
+        if (key.rfind("load.", 0) == 0) {
+            copy_namespaced_option(key, "load.", value, config.load.options);
+        } else if (key.rfind("session.", 0) == 0) {
+            copy_namespaced_option(key, "session.", value, config.session.options);
+        }
+    }
+
+    return config;
+}
+
+}  // namespace audio_cpp

backend/cpp/audio-cpp/model_config.h

@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+#include "engine/framework/runtime/model.h"
+#include "engine/framework/runtime/session.h"
+
+#include <filesystem>
+#include <string>
+#include <unordered_map>
+
+namespace audio_cpp {
+
+struct AudioCppModelConfig {
+    std::filesystem::path model_path;
+    std::string family;
+    engine::runtime::TaskSpec task;
+    engine::runtime::ModelLoadRequest load;
+    engine::runtime::SessionOptions session;
+};
+
+AudioCppModelConfig parse_model_config(
+    const std::filesystem::path & model_path,
+    const std::unordered_map<std::string, std::string> & options);
+
+}  // namespace audio_cpp

backend/cpp/audio-cpp/tests/build_contract_test.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: MIT
+
+set -euo pipefail
+
+backend_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+tmp_dir="$(mktemp -d)"
+trap 'rm -rf "${tmp_dir}"' EXIT
+
+fixture_dir="${tmp_dir}/audio.cpp"
+prefix_dir="${tmp_dir}/prefix"
+tools_dir="${tmp_dir}/tools"
+mkdir -p "${fixture_dir}" "${prefix_dir}/lib/cmake/Protobuf" \
+  "${prefix_dir}/lib/cmake/gRPC" "${tools_dir}"
+
+cat >"${fixture_dir}/engine_runtime.cpp" <<'EOF'
+void audio_cpp_build_contract_fixture() {}
+EOF
+
+cat >"${fixture_dir}/CMakeLists.txt" <<'EOF'
+cmake_minimum_required(VERSION 3.20)
+project(AudioCppBuildContractFixture LANGUAGES CXX)
+
+option(ENGINE_ENABLE_CUDA "Build with CUDA" OFF)
+option(ENGINE_ENABLE_VULKAN "Build with Vulkan" OFF)
+option(ENGINE_ENABLE_METAL "Build with Metal" OFF)
+
+foreach(wrong_option IN ITEMS
+    AUDIO_CPP_ENABLE_CUDA
+    AUDIO_CPP_ENABLE_VULKAN
+    AUDIO_CPP_ENABLE_METAL
+    AUDIOCPP_ENABLE_CUDA
+    AUDIOCPP_ENABLE_VULKAN
+    AUDIOCPP_ENABLE_METAL
+    ENGINE_CUDA
+    ENGINE_VULKAN
+    ENGINE_METAL
+    GGML_CUDA
+    GGML_VULKAN
+    GGML_METAL)
+  if(DEFINED ${wrong_option})
+    message(FATAL_ERROR "legacy or unsupported audio.cpp option: ${wrong_option}")
+  endif()
+endforeach()
+
+add_library(engine_runtime STATIC engine_runtime.cpp)
+EOF
+
+cat >"${prefix_dir}/lib/cmake/Protobuf/ProtobufConfig.cmake" <<'EOF'
+set(Protobuf_FOUND TRUE)
+set(Protobuf_VERSION 0.0.0)
+if(NOT TARGET protobuf::libprotobuf)
+  add_library(protobuf::libprotobuf INTERFACE IMPORTED)
+endif()
+EOF
+
+cat >"${prefix_dir}/lib/cmake/gRPC/gRPCConfig.cmake" <<'EOF'
+set(gRPC_FOUND TRUE)
+if(NOT TARGET gRPC::grpc++)
+  add_library(gRPC::grpc++ INTERFACE IMPORTED)
+endif()
+if(NOT TARGET gRPC::grpc++_reflection)
+  add_library(gRPC::grpc++_reflection INTERFACE IMPORTED)
+endif()
+EOF
+
+cat >"${tools_dir}/protoc" <<'EOF'
+#!/usr/bin/env sh
+exit 0
+EOF
+cat >"${tools_dir}/grpc_cpp_plugin" <<'EOF'
+#!/usr/bin/env sh
+exit 0
+EOF
+chmod +x "${tools_dir}/protoc" "${tools_dir}/grpc_cpp_plugin"
+
+assert_cache_bool() {
+  local cache_file="$1"
+  local name="$2"
+  local expected="$3"
+  grep -q "^${name}:BOOL=${expected}$" "${cache_file}" || {
+    echo "expected ${name}:BOOL=${expected} in ${cache_file}" >&2
+    return 1
+  }
+}
+
+configure_case() {
+  local name="$1"
+  local cuda="$2"
+  local vulkan="$3"
+  local metal="$4"
+  local build_dir="${tmp_dir}/build-${name}"
+
+  PATH="${tools_dir}:${PATH}" cmake \
+    -S "${backend_dir}" \
+    -B "${build_dir}" \
+    -DCMAKE_PREFIX_PATH="${prefix_dir}" \
+    -DAUDIO_CPP_DIR="${fixture_dir}" \
+    -DENGINE_ENABLE_CUDA="${cuda}" \
+    -DENGINE_ENABLE_VULKAN="${vulkan}" \
+    -DENGINE_ENABLE_METAL="${metal}" \
+    >/dev/null
+
+  assert_cache_bool "${build_dir}/CMakeCache.txt" ENGINE_ENABLE_CUDA "${cuda}"
+  assert_cache_bool "${build_dir}/CMakeCache.txt" ENGINE_ENABLE_VULKAN "${vulkan}"
+  assert_cache_bool "${build_dir}/CMakeCache.txt" ENGINE_ENABLE_METAL "${metal}"
+
+  grep -q 'engine_runtime' \
+    "${build_dir}/CMakeFiles/audio-cpp-grpc-server.dir/link.txt" || {
+    echo "audio-cpp-grpc-server does not link engine_runtime" >&2
+    return 1
+  }
+}
+
+configure_case cpu OFF OFF OFF
+configure_case cuda ON OFF OFF
+configure_case vulkan OFF ON OFF
+configure_case metal OFF OFF ON
+
+if PATH="${tools_dir}:${PATH}" cmake \
+  -S "${backend_dir}" \
+  -B "${tmp_dir}/build-wrong-option" \
+  -DCMAKE_PREFIX_PATH="${prefix_dir}" \
+  -DAUDIO_CPP_DIR="${fixture_dir}" \
+  -DGGML_CUDA=ON \
+  >/dev/null 2>&1; then
+  echo "strict audio.cpp fixture accepted legacy GGML_CUDA option" >&2
+  exit 1
+fi
+
+make_database="${tmp_dir}/make-database"
+make -C "${backend_dir}" -pn >"${make_database}"
+audio_cpp_version="$(
+  sed -n 's/^AUDIO_CPP_VERSION = //p' "${make_database}" | head -n 1
+)"
+[[ "${audio_cpp_version}" =~ ^[0-9a-f]{40}$ ]] || {
+  echo "AUDIO_CPP_VERSION must be a pinned 40-character commit" >&2
+  exit 1
+}
+
+fetch_plan="${tmp_dir}/fetch-plan"
+make -C "${backend_dir}" -Bn audio.cpp >"${fetch_plan}"
+grep -q 'github.com/0xShug0/audio.cpp' "${fetch_plan}"
+grep -q "${audio_cpp_version}" "${fetch_plan}"
+
+cat >"${tools_dir}/uname" <<'EOF'
+#!/usr/bin/env sh
+if [ "$#" -eq 1 ] && [ "$1" = "-s" ]; then
+  echo Darwin
+  exit 0
+fi
+echo "build contract requires uname -s" >&2
+exit 64
+EOF
+chmod +x "${tools_dir}/uname"
+
+darwin_plan="${tmp_dir}/darwin-plan"
+PATH="${tools_dir}:${PATH}" make -C "${backend_dir}" -n \
+  AUDIO_CPP_SRC="${fixture_dir}" grpc-server >"${darwin_plan}"
+grep -q -- '-DENGINE_ENABLE_CUDA=OFF' "${darwin_plan}"
+grep -q -- '-DENGINE_ENABLE_VULKAN=OFF' "${darwin_plan}"
+grep -q -- '-DENGINE_ENABLE_METAL=ON' "${darwin_plan}"
+
+echo "audio.cpp build contract: PASS"

backend/cpp/audio-cpp/tests/runtime_tests.cpp

@@ -0,0 +1,490 @@
+// SPDX-License-Identifier: MIT
+
+#include "audio_cpp_runtime.h"
+#include "model_config.h"
+
+#include "engine/framework/runtime/model.h"
+#include "engine/framework/runtime/registry.h"
+#include "engine/framework/runtime/session.h"
+
+#include <atomic>
+#include <chrono>
+#include <condition_variable>
+#include <exception>
+#include <filesystem>
+#include <future>
+#include <iostream>
+#include <memory>
+#include <mutex>
+#include <stdexcept>
+#include <string>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+namespace {
+
+using engine::runtime::AudioChunk;
+using engine::runtime::CapabilitySet;
+using engine::runtime::ILoadedVoiceModel;
+using engine::runtime::IOfflineVoiceTaskSession;
+using engine::runtime::IStreamingVoiceTaskSession;
+using engine::runtime::IVoiceModelLoader;
+using engine::runtime::IVoiceTaskSession;
+using engine::runtime::ModelInspection;
+using engine::runtime::ModelLoadRequest;
+using engine::runtime::ModelMetadata;
+using engine::runtime::RunMode;
+using engine::runtime::SessionOptions;
+using engine::runtime::SessionPreparationRequest;
+using engine::runtime::StreamEvent;
+using engine::runtime::TaskCapability;
+using engine::runtime::TaskRequest;
+using engine::runtime::TaskResult;
+using engine::runtime::TaskSpec;
+using engine::runtime::VoiceTaskKind;
+
+void require(bool condition, const std::string & message) {
+    if (!condition) {
+        throw std::runtime_error(message);
+    }
+}
+
+template <typename Function>
+void require_throws(Function && function, const std::string & expected) {
+    try {
+        function();
+    } catch (const std::exception & error) {
+        require(
+            std::string(error.what()).find(expected) != std::string::npos,
+            "expected error containing '" + expected + "', got '" + error.what() + "'");
+        return;
+    }
+    throw std::runtime_error("expected exception containing '" + expected + "'");
+}
+
+struct SessionGate {
+    std::mutex mutex;
+    std::condition_variable condition;
+    bool first_entered = false;
+    bool release_first = false;
+    std::atomic<int> entries{0};
+};
+
+struct FakeState {
+    std::mutex mutex;
+    std::vector<std::string> events;
+    CapabilitySet capabilities;
+    bool fail_load = false;
+    bool fail_session = false;
+    int generation = 0;
+    std::shared_ptr<SessionGate> gate;
+
+    void record(std::string event) {
+        std::lock_guard<std::mutex> lock(mutex);
+        events.push_back(std::move(event));
+    }
+};
+
+class FakeSession final
+    : public IOfflineVoiceTaskSession,
+      public IStreamingVoiceTaskSession {
+public:
+    FakeSession(
+        std::shared_ptr<FakeState> state,
+        int generation,
+        TaskSpec task,
+        SessionOptions options)
+        : state_(std::move(state)),
+          generation_(generation),
+          task_(task),
+          options_(std::move(options)) {}
+
+    ~FakeSession() override {
+        state_->record("session-" + std::to_string(generation_) + "-destroyed");
+    }
+
+    std::string family() const override { return "fake-family"; }
+    VoiceTaskKind task_kind() const override { return task_.task; }
+    RunMode run_mode() const override { return task_.mode; }
+
+    void prepare(const SessionPreparationRequest & request) override {
+        prepared_ = request;
+    }
+
+    TaskResult run(const TaskRequest &) override {
+        if (state_->gate != nullptr) {
+            const int entry = ++state_->gate->entries;
+            if (entry == 1) {
+                std::unique_lock<std::mutex> lock(state_->gate->mutex);
+                state_->gate->first_entered = true;
+                state_->gate->condition.notify_all();
+                state_->gate->condition.wait(
+                    lock,
+                    [&] { return state_->gate->release_first; });
+            }
+        }
+
+        TaskResult result;
+        result.text_output = engine::runtime::Transcript{
+            "generation-" + std::to_string(generation_),
+            "en",
+        };
+        return result;
+    }
+
+    engine::runtime::StreamingPolicy streaming_policy() const override {
+        engine::runtime::StreamingPolicy policy;
+        policy.input = engine::runtime::StreamingInputKind::AudioChunks;
+        policy.output = engine::runtime::StreamingOutputKind::PullEvents;
+        policy.preferred_audio_chunk_samples = 160;
+        return policy;
+    }
+
+    void start_stream(const TaskRequest &) override {
+        streaming_ = true;
+    }
+
+    std::optional<StreamEvent> next_stream_event() override {
+        return std::nullopt;
+    }
+
+    void set_stream_event_sink(engine::runtime::StreamEventCallback sink) override {
+        sink_ = std::move(sink);
+    }
+
+    TaskResult finish_stream() override {
+        streaming_ = false;
+        return run({});
+    }
+
+    void reset() override {
+        streaming_ = false;
+    }
+
+    StreamEvent process_audio_chunk(const AudioChunk & chunk) override {
+        require(streaming_, "stream was not started");
+        StreamEvent event;
+        event.audio_output = engine::runtime::AudioBuffer{
+            chunk.sample_rate,
+            chunk.channels,
+            chunk.samples,
+        };
+        if (sink_) {
+            sink_(event);
+        }
+        return event;
+    }
+
+    TaskResult finalize() override {
+        streaming_ = false;
+        return run({});
+    }
+
+private:
+    std::shared_ptr<FakeState> state_;
+    int generation_;
+    TaskSpec task_;
+    SessionOptions options_;
+    SessionPreparationRequest prepared_;
+    engine::runtime::StreamEventCallback sink_;
+    bool streaming_ = false;
+};
+
+class FakeLoadedModel final : public ILoadedVoiceModel {
+public:
+    FakeLoadedModel(std::shared_ptr<FakeState> state, int generation)
+        : state_(std::move(state)),
+          generation_(generation) {
+        metadata_.family = "fake-family";
+        metadata_.variant = "complete-fake";
+        metadata_.description = "complete test implementation";
+        metadata_.config_candidates = {"config.json"};
+        metadata_.weight_candidates = {"weights.gguf"};
+    }
+
+    ~FakeLoadedModel() override {
+        state_->record("model-" + std::to_string(generation_) + "-destroyed");
+    }
+
+    const ModelMetadata & metadata() const noexcept override {
+        return metadata_;
+    }
+
+    const CapabilitySet & capabilities() const noexcept override {
+        return state_->capabilities;
+    }
+
+    std::unique_ptr<IVoiceTaskSession> create_task_session(
+        const TaskSpec & task,
+        const SessionOptions & options) const override {
+        if (state_->fail_session) {
+            throw std::runtime_error("session creation failed");
+        }
+        return std::make_unique<FakeSession>(state_, generation_, task, options);
+    }
+
+private:
+    std::shared_ptr<FakeState> state_;
+    int generation_;
+    ModelMetadata metadata_;
+};
+
+class FakeLoader final : public IVoiceModelLoader {
+public:
+    explicit FakeLoader(std::shared_ptr<FakeState> state)
+        : state_(std::move(state)) {}
+
+    std::string family() const override { return "fake-family"; }
+
+    bool can_load(const ModelLoadRequest & request) const override {
+        return request.family_hint == family();
+    }
+
+    ModelInspection inspect(const ModelLoadRequest & request) const override {
+        ModelInspection inspection;
+        inspection.metadata.family = family();
+        inspection.metadata.variant = "complete-fake";
+        inspection.metadata.description = "complete test loader";
+        inspection.metadata.config_candidates = {"config.json"};
+        inspection.metadata.weight_candidates = {"weights.gguf"};
+        inspection.capabilities = state_->capabilities;
+        inspection.model_root = request.model_path;
+        return inspection;
+    }
+
+    std::unique_ptr<ILoadedVoiceModel> load(
+        const ModelLoadRequest &) const override {
+        if (state_->fail_load) {
+            throw std::runtime_error("model load failed");
+        }
+        const int generation = ++state_->generation;
+        return std::make_unique<FakeLoadedModel>(state_, generation);
+    }
+
+    CapabilitySet advertised_capabilities() const override {
+        return state_->capabilities;
+    }
+
+    std::string advertised_instructions_policy() const override {
+        return "explicit";
+    }
+
+    std::vector<std::string> advertised_api_endpoints() const override {
+        return {"/v1/audio/transcriptions"};
+    }
+
+private:
+    std::shared_ptr<FakeState> state_;
+};
+
+audio_cpp::AudioCppModelConfig offline_asr_config(
+    const std::filesystem::path & model_path) {
+    return audio_cpp::parse_model_config(
+        model_path,
+        {
+            {"family", "fake-family"},
+            {"task", "asr"},
+            {"mode", "offline"},
+            {"backend", "cpu"},
+            {"device", "2"},
+            {"threads", "3"},
+            {"load.cache", "memory"},
+            {"session.language", "en"},
+        });
+}
+
+std::unique_ptr<audio_cpp::AudioCppRuntime> make_runtime(
+    const std::shared_ptr<FakeState> & state) {
+    engine::runtime::ModelRegistry registry;
+    registry.register_loader(std::make_shared<FakeLoader>(state));
+    return std::make_unique<audio_cpp::AudioCppRuntime>(std::move(registry));
+}
+
+void test_model_config(const std::filesystem::path & model_path) {
+    const auto config = offline_asr_config(model_path);
+    require(config.model_path == model_path, "model path was not preserved");
+    require(config.family == "fake-family", "family was not parsed");
+    require(config.task.task == VoiceTaskKind::Asr, "task was not parsed");
+    require(config.task.mode == RunMode::Offline, "mode was not parsed");
+    require(
+        config.session.backend.type == engine::core::BackendType::Cpu,
+        "backend was not parsed");
+    require(config.session.backend.device == 2, "device was not parsed");
+    require(config.session.backend.threads == 3, "threads were not parsed");
+    require(config.load.options.at("cache") == "memory", "load option prefix was not stripped");
+    require(
+        config.session.options.at("language") == "en",
+        "session option prefix was not stripped");
+
+    require_throws(
+        [&] { audio_cpp::parse_model_config(model_path, {{"task", "asr"}}); },
+        "family");
+    require_throws(
+        [&] { audio_cpp::parse_model_config(model_path, {{"family", "fake-family"}}); },
+        "task");
+    require_throws(
+        [&] {
+            audio_cpp::parse_model_config(
+                model_path,
+                {{"family", "fake-family"}, {"task", "asr"}, {"mode", "batch"}});
+        },
+        "mode");
+    require_throws(
+        [&] {
+            audio_cpp::parse_model_config(
+                model_path,
+                {{"family", "fake-family"}, {"task", "asr"}, {"backend", "tpu"}});
+        },
+        "backend");
+}
+
+void test_capability_validation(const std::filesystem::path & model_path) {
+    auto state = std::make_shared<FakeState>();
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Tts, {RunMode::Offline}},
+    };
+    auto runtime = make_runtime(state);
+
+    require_throws(
+        [&] { runtime->load(offline_asr_config(model_path)); },
+        "task");
+
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Asr, {RunMode::Streaming}},
+    };
+    require_throws(
+        [&] { runtime->load(offline_asr_config(model_path)); },
+        "mode");
+}
+
+void test_atomic_replacement(const std::filesystem::path & model_path) {
+    auto state = std::make_shared<FakeState>();
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Asr, {RunMode::Offline}},
+    };
+    auto runtime = make_runtime(state);
+    runtime->load(offline_asr_config(model_path));
+
+    state->fail_load = true;
+    require_throws(
+        [&] { runtime->load(offline_asr_config(model_path)); },
+        "model load failed");
+    require(
+        runtime->run({}).text_output->text == "generation-1",
+        "old model was not retained after load failure");
+
+    state->fail_load = false;
+    state->fail_session = true;
+    require_throws(
+        [&] { runtime->load(offline_asr_config(model_path)); },
+        "session creation failed");
+    require(
+        runtime->run({}).text_output->text == "generation-1",
+        "old model was not retained after session creation failure");
+}
+
+void test_teardown_order(const std::filesystem::path & model_path) {
+    auto state = std::make_shared<FakeState>();
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Asr, {RunMode::Offline}},
+    };
+    auto runtime = make_runtime(state);
+    runtime->load(offline_asr_config(model_path));
+    runtime->free();
+
+    std::lock_guard<std::mutex> lock(state->mutex);
+    require(state->events.size() == 2, "expected one session and one model teardown");
+    require(
+        state->events[0] == "session-1-destroyed",
+        "session was not destroyed before model");
+    require(
+        state->events[1] == "model-1-destroyed",
+        "model teardown event was not second");
+}
+
+void test_runtime_serializes_calls(const std::filesystem::path & model_path) {
+    auto state = std::make_shared<FakeState>();
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Asr, {RunMode::Offline}},
+    };
+    state->gate = std::make_shared<SessionGate>();
+    auto runtime = make_runtime(state);
+    runtime->load(offline_asr_config(model_path));
+
+    auto first = std::async(std::launch::async, [&] { return runtime->run({}); });
+    {
+        std::unique_lock<std::mutex> lock(state->gate->mutex);
+        state->gate->condition.wait(
+            lock,
+            [&] { return state->gate->first_entered; });
+    }
+
+    std::promise<void> release_second;
+    std::shared_future<void> second_barrier = release_second.get_future().share();
+    std::promise<void> second_attempted_promise;
+    auto second_attempted = second_attempted_promise.get_future();
+    auto second = std::async(std::launch::async, [&] {
+        second_barrier.wait();
+        second_attempted_promise.set_value();
+        return runtime->run({});
+    });
+
+    release_second.set_value();
+    second_attempted.wait();
+    require(
+        second.wait_for(std::chrono::milliseconds(50)) == std::future_status::timeout,
+        "second call completed while first call held the runtime");
+    require(
+        state->gate->entries.load() == 1,
+        "second call entered the upstream session concurrently");
+
+    {
+        std::lock_guard<std::mutex> lock(state->gate->mutex);
+        state->gate->release_first = true;
+    }
+    state->gate->condition.notify_all();
+    first.get();
+    second.get();
+    require(state->gate->entries.load() == 2, "second call never reached the session");
+}
+
+void test_streaming_surface(const std::filesystem::path & model_path) {
+    auto state = std::make_shared<FakeState>();
+    state->capabilities.supported_tasks = {
+        {VoiceTaskKind::Asr, {RunMode::Streaming}},
+    };
+    auto runtime = make_runtime(state);
+    auto config = offline_asr_config(model_path);
+    config.task.mode = RunMode::Streaming;
+    runtime->load(config);
+    runtime->start_stream({});
+    const auto event = runtime->process_audio_chunk({16000, 1, 0, {0.25f}});
+    require(event.audio_output.has_value(), "streaming chunk result was lost");
+    require(
+        event.audio_output->samples == std::vector<float>{0.25f},
+        "streaming chunk samples changed");
+    require(
+        runtime->finish_stream().text_output->text == "generation-1",
+        "streaming final result was lost");
+}
+
+}  // namespace
+
+int main(int argc, char ** argv) {
+    try {
+        require(argc == 2, "runtime_test requires an existing model path argument");
+        const std::filesystem::path model_path(argv[1]);
+        test_model_config(model_path);
+        test_capability_validation(model_path);
+        test_atomic_replacement(model_path);
+        test_teardown_order(model_path);
+        test_runtime_serializes_calls(model_path);
+        test_streaming_surface(model_path);
+        std::cout << "audio.cpp runtime unit tests: PASS\n";
+        return 0;
+    } catch (const std::exception & error) {
+        std::cerr << "audio.cpp runtime unit tests: FAIL: " << error.what() << '\n';
+        return 1;
+    }
+}

backend/cpp/bonsai/Makefile

@@ -41,6 +41,7 @@ define bonsai-build
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-$(1)-build llama.cpp
@@ -77,6 +78,7 @@ bonsai-cpu-all:
 	# and are applied by apply-patches.sh below.
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build purge
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I bonsai build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BONSAI_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../bonsai-cpu-all-build llama.cpp

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=0a7ad776b9068348e6cb09df8cafa9cadd285298
+# Upstream pin lives below as DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=0a7ad776b9068348e6cb09df8cafa9cadd285298
+DS4_VERSION?=54b36ed9ba42da31b24f2d1a5feb075c2475dbb1
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=0a4e10c7fb65d2dd5a4afb78339c7d373a8cdfaa
+IK_LLAMA_VERSION?=b054a8b983827c01aec59d4dc273a27c492c51c4
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=0d47ea7427463093e69128bf2c2f9cd06b3ee5b3
+LLAMA_VERSION?=1cbfd1988311775425d36c0ce066590f7d3049cf
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/disable-score-task.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Mark a copied gRPC server as targeting a llama.cpp fork that does not carry
+# LocalAI's slot-based Score patches. The RPC remains present in the shared
+# protobuf service, but responds with UNIMPLEMENTED instead of referencing
+# server task types and common_params fields absent from those forks.
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q '^#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK' "$SRC"; then
+    echo "==> $SRC already disables the LocalAI score task, skipping"
+    exit 0
+fi
+
+awk '
+    !done && /^#include/ {
+        print "#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK 1"
+        print "// ^ injected by disable-score-task.sh for an unpatched llama.cpp fork"
+        print ""
+        done = 1
+    }
+    { print }
+    END {
+        if (!done) {
+            print "disable-score-task.sh: no #include anchor found" > "/dev/stderr"
+            exit 1
+        }
+    }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+echo "==> LocalAI score task disabled in $SRC"

backend/cpp/llama-cpp/grpc-server.cpp

@@ -152,40 +152,6 @@ static std::string base64_encode_bytes(const unsigned char* data, size_t len) {
 
 bool loaded_model; // TODO: add a mutex for this, but happens only once loading the model
 
-// Score bypasses the slot loop (see the comment on Score below) so it
-// must not run concurrently with any slot-loop RPC. These counters
-// are a defence-in-depth tripwire — ModelConfig.Validate already
-// rejects llama-cpp configs that mix score with chat/completion/
-// embeddings, so a healthy deployment never trips them. seq_cst is
-// load-bearing for the increment-then-check pattern below.
-static std::atomic<int> slot_loop_inflight{0};
-static std::atomic<int> score_inflight{0};
-
-// Increment-then-check, not check-then-increment: two simultaneous
-// racers both observe the other's increment and both abort cleanly.
-// Reversed, both could see zero and proceed.
-struct conflict_guard {
-    std::atomic<int>& self;
-    conflict_guard(const char* rpc, std::atomic<int>& self_, std::atomic<int>& other, const char* other_name)
-        : self(self_) {
-        self.fetch_add(1, std::memory_order_seq_cst);
-        int o = other.load(std::memory_order_seq_cst);
-        if (o > 0) {
-            fprintf(stderr,
-                "FATAL: %s called with %s=%d. The llama-cpp backend cannot "
-                "service Score and slot-loop RPCs concurrently — Score "
-                "bypasses the slot loop and races the llama_context. Bind "
-                "Score-using features to a model dedicated to scoring "
-                "(known_usecases: [score] with no chat/completion/embeddings).\n",
-                rpc, other_name, o);
-            std::abort();
-        }
-    }
-    ~conflict_guard() {
-        self.fetch_sub(1, std::memory_order_seq_cst);
-    }
-};
-
 static std::function<void(int)> shutdown_handler;
 static std::atomic_flag is_terminating = ATOMIC_FLAG_INIT;
 
@@ -732,6 +698,22 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
                     // If conversion fails, keep default value (0)
                 }
             }
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        } else if (!strcmp(optname, "n_rs_seq") || !strcmp(optname, "rs_seq")) {
+            // Recurrent-state rollback snapshots per sequence. Hybrid models
+            // (deltanet/conv layers) cannot rewind their state, so without
+            // snapshots any prompt-cache reuse that needs a rewind — e.g. a
+            // score task whose probe changed under a stable option-list
+            // prefix — falls back to a full re-prefill. Costs recurrent-state
+            // memory x (1 + N) per sequence; unsupported archs clamp to 0.
+            if (optval != NULL) {
+                try {
+                    params.n_rs_seq = std::stoi(optval_str);
+                } catch (const std::exception& e) {
+                    // If conversion fails, keep default value (0)
+                }
+            }
+#endif
         } else if (!strcmp(optname, "slot_prompt_similarity") || !strcmp(optname, "sps")) {
             if (optval != NULL) {
                 try {
@@ -1392,6 +1374,17 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
         }
     }
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+    // Score-task suffix forking: reserve seq ids (and recurrent-state cells)
+    // beyond the slots so one scoring call decodes all candidate tails in a
+    // single batch (SERVER_TASK_TYPE_SCORE, patches/). Requires the unified
+    // KV cache — with per-sequence streams the extra ids would shrink every
+    // sequence's context to n_ctx / n_seq_max. Decided after both option
+    // passes so an explicit kv_unified:false wins and disables forking.
+    params.score_enabled = request->enablescore();
+    params.n_seq_score_forks = params.score_enabled && params.kv_unified ? SERVER_SCORE_FORK_SEQS : 0;
+#endif
+
     // Terminate/pad the override vectors only after BOTH the named-option loop
     // and the generic passthrough (common_params_parse above) have pushed their
     // real entries, so back() is the null sentinel the model loader asserts on.
@@ -1478,6 +1471,16 @@ public:
         common_params params;
         params_parse(ctx_server, request, params);
 
+#ifndef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        if (params.score_enabled && !params.kv_unified) {
+            const std::string error_msg =
+                "Score requires the unified KV cache; remove kv_unified:false or remove score from known_usecases";
+            result->set_message(error_msg);
+            result->set_success(false);
+            return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, error_msg);
+        }
+#endif
+
         common_init();
         // Ensure debug logs are enabled after common_init() sets up logging
         common_log_set_verbosity_thold(params.verbosity);
@@ -1680,7 +1683,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("PredictStream", slot_loop_inflight, score_inflight, "score_inflight");
         json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
 
@@ -2249,7 +2251,6 @@ public:
          if (params_base.model.path.empty()) {
              return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
          }
-         conflict_guard guard("Predict", slot_loop_inflight, score_inflight, "score_inflight");
          json data = parse_options(true, request, params_base, ctx_server.get_llama_context());
 
         data["stream"] = false;
@@ -2783,7 +2784,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("Embedding", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
 
         body["stream"] = false;
@@ -2893,7 +2893,6 @@ public:
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "\"documents\" must be a non-empty string array");
         }
 
-        conflict_guard guard("Rerank", slot_loop_inflight, score_inflight, "score_inflight");
 
         // Create and queue the task
         auto rd = ctx_server.get_response_reader();
@@ -2970,37 +2969,16 @@ public:
     // Score returns the model's joint log-probability of each candidate
     // continuation given a shared prompt.
     //
-    // WHY bypass the slot/task queue: upstream server_context exposes
-    // get_llama_context as "main thread only" and the slot loop's
-    // update_slots() owns the context whenever a task is in flight.
-    // No public synchronization primitive is available — so Score is
-    // unsafe to call concurrently with active generation through this
-    // backend. In practice routing-classifier calls happen before the
-    // request is routed to a generation backend, so the model used
-    // for Score is typically idle. Concurrent Score calls are
-    // serialised by a local mutex; KV-cache state is isolated behind
-    // a dedicated sequence ID cleared between candidates.
-    //
-    // A patch to server-context.cpp that adds SERVER_TASK_TYPE_SCORE
-    // and routes scoring through the slot loop would be the correct
-    // long-term fix; tracked as a follow-up.
-    //
-    // Perf TODO (measured: ~450 ms warm for 3 candidates on Arch-
-    // Router-1.5B Q4_K_M + Intel SYCL): the current loop re-decodes
-    // `prompt + candidate` from scratch for every candidate, throwing
-    // away the prompt's KV cache between iterations. A smarter
-    // version would:
-    //   1. Decode just the prompt once into score_seq_id.
-    //   2. Snapshot/cp that sequence (llama_memory_seq_cp) into a
-    //      per-candidate sequence id.
-    //   3. For each candidate, decode only its tokens onto the copy
-    //      (continuing from the saved prompt state), read logits.
-    //   4. llama_memory_seq_rm the copy.
-    // Estimated speedup: 3-candidate calls 450 ms -> ~150-200 ms,
-    // 6-candidate calls 630 ms -> ~220 ms. Single source-file change,
-    // no proto / Go-side changes needed. Worth doing once routing is
-    // wired into the middleware and Score is on the hot path of every
-    // chat request.
+    // Scoring runs as a single SERVER_TASK_TYPE_SCORE task through the
+    // slot loop (added by patches/ on top of upstream server-context), so
+    // it is safe to interleave with generation on the same process and it
+    // reuses any KV prefix the slot already holds across turns. The task
+    // decodes the shared prefix (prompt + longest common candidate token
+    // prefix) once on the slot's sequence; every candidate's unique tail
+    // then rides its own forked sequence and all tails are decoded
+    // together in one batch, so a warm scoring call costs roughly one
+    // forward pass over the new prompt tokens plus one batched pass over
+    // the candidate tails.
     grpc::Status Score(ServerContext* context, const backend::ScoreRequest* request, backend::ScoreResponse* response) override {
         auto auth = checkAuth(context);
         if (!auth.ok()) return auth;
@@ -3009,40 +2987,21 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
+#ifdef LOCALAI_LLAMA_CPP_NO_SCORE_TASK
+        (void) request;
+        (void) response;
+        return grpc::Status(grpc::StatusCode::UNIMPLEMENTED,
+            "Score is unavailable in this llama.cpp fork backend");
+#else
+        if (!params_base.score_enabled) {
+            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION,
+                "Score was not enabled when the model was loaded; add score to known_usecases");
+        }
         if (request->candidates_size() == 0) {
             return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "candidates must be non-empty");
         }
 
-        // Tripwire against the slot loop. Acquired before score_mutex
-        // so it fires even when this Score is queued behind another.
-        conflict_guard guard("Score", score_inflight, slot_loop_inflight, "slot_loop_inflight");
-
-        // Serialise concurrent Score calls. The slot loop is still
-        // free to race with us — see the class comment above.
-        static std::mutex score_mutex;
-        std::lock_guard<std::mutex> score_lock(score_mutex);
-
-        llama_context * lctx = ctx_server.get_llama_context();
-        if (lctx == nullptr) {
-            return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "llama context unavailable (sleeping?)");
-        }
         const llama_vocab * vocab = ctx_server.impl->vocab;
-        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
-        const int32_t n_ctx = llama_n_ctx(lctx);
-        llama_memory_t mem = llama_get_memory(lctx);
-
-        // The KV-cache is sized to seq_to_stream.size() at load
-        // (typically equal to n_slots, often 1). Sequence IDs must
-        // be in [0, n_seq_max), so we can't pick a high-value
-        // "private" ID — we have to share with the slot. We clear
-        // the cache before AND after each candidate to keep
-        // scoring isolated from whatever state the slot held, and
-        // the static mutex above guarantees no other Score call is
-        // racing in the meantime. The slot loop is still free to
-        // race (see comment on this method) — Score must not run
-        // concurrently with generation through this backend.
-        const llama_seq_id score_seq_id = 0;
-        llama_memory_seq_rm(mem, score_seq_id, -1, -1);
 
         // Tokenize the shared prompt once with add_special=true so
         // BOS is prepended when the model requires it. parse_special
@@ -3051,6 +3010,15 @@ public:
         std::vector<llama_token> prompt_tokens = common_tokenize(vocab, prompt, /*add_special=*/true, /*parse_special=*/true);
         const int32_t prompt_len = (int32_t) prompt_tokens.size();
 
+        // Per candidate: full prompt+candidate token list and the
+        // divergence point, kept for piece rendering and empty-candidate
+        // handling after the task comes back.
+        std::vector<std::vector<llama_token>> cand_tokens(request->candidates_size());
+        std::vector<int32_t> cand_divergence(request->candidates_size(), 0);
+
+        // candidates that actually have tokens to score
+        std::vector<int32_t> included;
+
         for (int ci = 0; ci < request->candidates_size(); ci++) {
             const std::string & candidate_text = request->candidates(ci);
 
@@ -3067,9 +3035,135 @@ public:
                     break;
                 }
             }
+            divergence = std::min<int32_t>(divergence, (int32_t) full_tokens.size());
+
             const int32_t cand_len = (int32_t) full_tokens.size() - divergence;
+            if (cand_len > 0 && divergence < 1) {
+                // Need at least one prior token (typically BOS) to
+                // predict the first candidate token's logit. Tokeniser
+                // models without BOS + an empty prompt fall in here.
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+            }
+            if (cand_len > SERVER_SCORE_MAX_CAND_TOKENS) {
+                // The context reserves logits outputs for at most this many
+                // candidate tokens per slot (server_n_outputs_max).
+                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
+                    "Score: candidate " + std::to_string(ci) + " is " + std::to_string(cand_len) +
+                    " tokens; the maximum is " + std::to_string(SERVER_SCORE_MAX_CAND_TOKENS));
+            }
+
+            cand_divergence[ci] = divergence;
+            cand_tokens[ci] = std::move(full_tokens);
+
+            if (cand_len > 0) {
+                included.push_back(ci);
+            }
+        }
+
+        auto rd = ctx_server.get_response_reader();
+        bool posted_task = false;
+
+        // Shared prefix bounds, needed again when stitching the results:
+        // n_shared is the longest common token prefix of the scored
+        // candidates, n_score_prompt the earliest divergence from the
+        // bare prompt (scored logprobs start there).
+        int32_t n_shared       = 0;
+        int32_t n_score_prompt = 0;
+
+        if (!included.empty()) {
+            const auto & first = cand_tokens[included[0]];
+
+            // the common prefix of a set is the shortest common prefix
+            // against any fixed member
+            n_shared = (int32_t) first.size();
+            for (int32_t ci : included) {
+                const auto & ft = cand_tokens[ci];
+                const int32_t lim = std::min<int32_t>(n_shared, (int32_t) ft.size());
+                int32_t match = 0;
+                while (match < lim && ft[match] == first[match]) {
+                    match++;
+                }
+                n_shared = match;
+            }
+
+            // below its divergence every candidate equals the prompt
+            // tokens, so n_score_prompt <= n_shared always holds
+            n_score_prompt = cand_divergence[included[0]];
+            for (int32_t ci : included) {
+                n_score_prompt = std::min(n_score_prompt, cand_divergence[ci]);
+            }
+
+            // Map the caller's stable-prefix byte length onto a token
+            // index: the last prompt token that ends at or before the
+            // boundary. A checkpoint forced there survives every future
+            // probe under the same option list, which is what keeps
+            // repeat scoring cheap on models that cannot rewind state.
+            int32_t n_stable_prompt = 0;
+            if (request->stable_prefix_len() > 0) {
+                size_t consumed = 0;
+                for (int32_t ti = 0; ti < n_score_prompt; ti++) {
+                    const size_t piece_len = common_token_to_piece(vocab, prompt_tokens[ti]).size();
+                    // BOS and other zero-length specials consume no prompt bytes
+                    if (consumed + piece_len > (size_t) request->stable_prefix_len()) {
+                        break;
+                    }
+                    consumed += piece_len;
+                    n_stable_prompt = ti + 1;
+                }
+            }
+
+            server_task task(SERVER_TASK_TYPE_SCORE);
+            task.id             = rd.queue_tasks.get_new_id();
+            task.index          = 0;
+            task.tokens         = server_tokens(llama_tokens(first.begin(), first.begin() + n_shared), false);
+            task.n_score_prompt = n_score_prompt;
+            task.n_stable_prompt = n_stable_prompt;
+            task.score_suffixes.reserve(included.size());
+            for (int32_t ci : included) {
+                task.score_suffixes.emplace_back(cand_tokens[ci].begin() + n_shared, cand_tokens[ci].end());
+            }
+
+            std::vector<server_task> tasks;
+            tasks.push_back(std::move(task));
+            rd.post_tasks(std::move(tasks));
+            posted_task = true;
+        }
+
+        // Wait for the shared-prefix and per-candidate logprob vectors.
+        // Context overflow and decode failures surface here as task errors.
+        std::vector<float> shared_logprobs;
+        std::vector<std::vector<float>> cand_logprobs;
+        if (posted_task) {
+            auto all_results = rd.wait_for_all([&context]() { return context->IsCancelled(); });
+            if (all_results.is_terminated) {
+                return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
+            }
+            if (all_results.error) {
+                return grpc::Status(grpc::StatusCode::INTERNAL,
+                    all_results.error->to_json().value("message", "Error in receiving score results"));
+            }
+            if (all_results.results.size() != 1) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "expected a single score result");
+            }
+            auto * score_res = dynamic_cast<server_task_result_score*>(all_results.results[0].get());
+            if (score_res == nullptr) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "unexpected result type for score task");
+            }
+            shared_logprobs = std::move(score_res->shared_logprobs);
+            cand_logprobs   = std::move(score_res->cand_logprobs);
+            if (cand_logprobs.size() != included.size()) {
+                return grpc::Status(grpc::StatusCode::INTERNAL, "score result candidate count mismatch");
+            }
+        }
+
+        size_t inc = 0; // index into included / cand_logprobs
+        for (int ci = 0; ci < request->candidates_size(); ci++) {
+            const int32_t divergence = cand_divergence[ci];
+            const int32_t cand_len   = (int32_t) cand_tokens[ci].size() - divergence;
+
             backend::CandidateScore * cs = response->add_candidates();
-            cs->set_num_tokens(cand_len);
+            cs->set_num_tokens(cand_len > 0 ? cand_len : 0);
             if (cand_len <= 0) {
                 cs->set_log_prob(0.0);
                 if (request->length_normalize()) {
@@ -3077,101 +3171,57 @@ public:
                 }
                 continue;
             }
-            if (divergence < 1) {
-                // Need at least one prior token (typically BOS) to
-                // predict the first candidate token's logit. Tokeniser
-                // models without BOS + an empty prompt fall in here.
-                return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT,
-                    "Score: prompt produced no leading tokens; need at least one (e.g. BOS) to predict candidate");
+
+            // Stitch the candidate's scored logprobs back together: the
+            // stretch inside the shared prefix (identical for every
+            // candidate) followed by its forked suffix. Suffix entries
+            // before the candidate's own divergence are prompt tokens
+            // decoded only as context — not scored.
+            std::vector<float> lp;
+            lp.reserve(cand_len);
+            for (int32_t t = divergence; t < n_shared; t++) {
+                const int32_t idx = t - n_score_prompt;
+                if (idx < 0 || idx >= (int32_t) shared_logprobs.size()) {
+                    return grpc::Status(grpc::StatusCode::INTERNAL,
+                        "Score: shared logprob index out of range for candidate " + std::to_string(ci));
+                }
+                lp.push_back(shared_logprobs[idx]);
             }
-            if ((int32_t) full_tokens.size() > n_ctx) {
-                return grpc::Status(grpc::StatusCode::OUT_OF_RANGE,
-                    "Score: prompt+candidate exceeds context size (got " +
-                    std::to_string(full_tokens.size()) + ", n_ctx=" + std::to_string(n_ctx) + ")");
+            const auto & sfx_lp = cand_logprobs[inc++];
+            for (int32_t j = std::max(0, divergence - n_shared); j < (int32_t) sfx_lp.size(); j++) {
+                lp.push_back(sfx_lp[j]);
             }
 
-            // Build a batch covering the entire prompt+candidate. We
-            // need logits at (divergence-1) onward — those are the
-            // predictions for each candidate token.
-            llama_batch batch = llama_batch_init((int32_t) full_tokens.size(), 0, 1);
-            for (int32_t i = 0; i < (int32_t) full_tokens.size(); i++) {
-                batch.token[i]    = full_tokens[i];
-                batch.pos[i]      = i;
-                batch.n_seq_id[i] = 1;
-                batch.seq_id[i][0] = score_seq_id;
-                // logits[i] is "do we want the prediction *for the
-                // next token*, computed from this position?"
-                // We want predictions for candidate tokens at
-                // positions divergence .. full_tokens.size()-1, which
-                // come from logits at positions (divergence-1) ..
-                // (full_tokens.size()-2).
-                bool need_logit = (i >= divergence - 1) && (i < (int32_t) full_tokens.size() - 1);
-                batch.logits[i] = need_logit ? 1 : 0;
-            }
-            batch.n_tokens = (int32_t) full_tokens.size();
-
-            // Decode the batch. If decode fails (e.g. KV slot
-            // exhaustion), surface as INTERNAL — the caller will
-            // typically fall back to a sampling-based classifier.
-            int decode_err = llama_decode(lctx, batch);
-            if (decode_err != 0) {
-                llama_batch_free(batch);
-                llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+            if ((int32_t) lp.size() != cand_len) {
                 return grpc::Status(grpc::StatusCode::INTERNAL,
-                    "llama_decode failed during Score: " + std::to_string(decode_err));
+                    "Score: result for candidate " + std::to_string(ci) + " is missing token logprobs");
             }
 
-            // Sum log-probabilities of the actual candidate tokens.
             double total_log_prob = 0.0;
             for (int32_t k = 0; k < cand_len; k++) {
-                // The k-th candidate token sits at full_tokens index
-                // (divergence + k). Its predicting logit is at batch
-                // position (divergence + k - 1).
-                int32_t logit_pos = divergence + k - 1;
-                const float * logits = llama_get_logits_ith(lctx, logit_pos);
-                if (logits == nullptr) {
-                    llama_batch_free(batch);
-                    llama_memory_seq_rm(mem, score_seq_id, -1, -1);
+                const float token_log_prob = lp[k];
+                if (std::isnan(token_log_prob)) {
                     return grpc::Status(grpc::StatusCode::INTERNAL,
-                        "llama_get_logits_ith returned null at position " + std::to_string(logit_pos));
+                        "Score: incomplete result for candidate " + std::to_string(ci) +
+                        " at token " + std::to_string(k));
                 }
-                llama_token target_token = full_tokens[divergence + k];
-
-                // Compute log_softmax(logits)[target_token] with the
-                // max-subtraction stability trick.
-                float max_logit = logits[0];
-                for (int32_t v = 1; v < n_vocab; v++) {
-                    if (logits[v] > max_logit) max_logit = logits[v];
-                }
-                double sum_exp = 0.0;
-                for (int32_t v = 0; v < n_vocab; v++) {
-                    sum_exp += std::exp((double)(logits[v] - max_logit));
-                }
-                double token_log_prob = (double)(logits[target_token] - max_logit) - std::log(sum_exp);
-                total_log_prob += token_log_prob;
+                total_log_prob += (double) token_log_prob;
 
                 if (request->include_token_logprobs()) {
                     backend::TokenLogProb * tlp = cs->add_tokens();
-                    std::string piece = common_token_to_piece(lctx, target_token);
-                    tlp->set_token(piece);
+                    tlp->set_token(common_token_to_piece(vocab, cand_tokens[ci][divergence + k]));
                     tlp->set_log_prob(token_log_prob);
                 }
             }
 
             cs->set_log_prob(total_log_prob);
-            if (request->length_normalize() && cand_len > 0) {
+            if (request->length_normalize()) {
                 cs->set_length_normalized_log_prob(total_log_prob / (double) cand_len);
             }
-
-            llama_batch_free(batch);
-            // Drop this candidate's KV-cache contribution so the next
-            // candidate starts from a clean state. Without this, the
-            // next decode would conflict at positions 0..N-1 for our
-            // sequence ID.
-            llama_memory_seq_rm(mem, score_seq_id, -1, -1);
         }
 
         return grpc::Status::OK;
+#endif
     }
 
     grpc::Status TokenizeString(ServerContext* context, const backend::PredictOptions* request, backend::TokenizationResponse* response) override {
@@ -3182,7 +3232,6 @@ public:
         if (params_base.model.path.empty()) {
             return grpc::Status(grpc::StatusCode::FAILED_PRECONDITION, "Model not loaded");
         }
-        conflict_guard guard("TokenizeString", slot_loop_inflight, score_inflight, "score_inflight");
         json body = parse_options(false, request, params_base, ctx_server.get_llama_context());
         body["stream"] = false;
 
@@ -3204,7 +3253,6 @@ public:
 
     grpc::Status GetMetrics(ServerContext* /*context*/, const backend::MetricsRequest* /*request*/, backend::MetricsResponse* response) override {
 
-        conflict_guard guard("GetMetrics", slot_loop_inflight, score_inflight, "score_inflight");
 
 // request slots data using task queue
         auto rd = ctx_server.get_response_reader();

backend/cpp/llama-cpp/patches/0001-add-server-task-type-score.patch

@@ -0,0 +1,599 @@
+diff --git a/common/common.cpp b/common/common.cpp
+index 8f13217..fc584e1 100644
+--- a/common/common.cpp
++++ b/common/common.cpp
+@@ -1591,8 +1591,10 @@ struct llama_context_params common_context_params_to_llama(const common_params &
+     auto cparams = llama_context_default_params();
+
+     cparams.n_ctx             = params.n_ctx;
+-    cparams.n_seq_max         = params.n_parallel;
+-    cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
++    // score-task forks need seq ids (and recurrent-state cells) of their
++    // own beyond the parallel slots
++    cparams.n_seq_max         = params.n_parallel + params.n_seq_score_forks;
++    cparams.n_rs_seq          = std::max(params.speculative.need_n_rs_seq(), (uint32_t) std::max(0, params.n_rs_seq));
+     cparams.n_outputs_max     = std::max(params.n_outputs_max, 0);
+     cparams.n_batch           = params.n_batch;
+     cparams.n_ubatch          = params.n_ubatch;
+diff --git a/common/common.h b/common/common.h
+index bffc176..e313bd6 100644
+--- a/common/common.h
++++ b/common/common.h
+@@ -455,6 +455,9 @@ struct common_params {
+     int32_t n_keep                =     0; // number of tokens to keep from initial prompt
+     int32_t n_chunks              =    -1; // max number of chunks to process (-1 = unlimited)
+     int32_t n_parallel            =     1; // number of parallel sequences to decode
++    int32_t n_seq_score_forks     =     0; // extra seq ids beyond n_parallel, reserved for server score-task forks
++    int32_t n_rs_seq              =     0; // recurrent-state rollback snapshots per seq (hybrid models cannot rewind without them; lets score tasks reuse a cached prompt across probe changes)
++    bool    score_enabled          = false; // reserve server resources for the Score task type
+     int32_t n_sequences           =     1; // number of sequences to decode
+     int32_t n_outputs_max         =     0; // max outputs in a batch (0 = n_batch)
+     int32_t grp_attn_n            =     1; // group-attention factor
+diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
+index 780df32..1d2fe8f 100644
+--- a/tools/CMakeLists.txt
++++ b/tools/CMakeLists.txt
+@@ -41,3 +41,4 @@ else()
+     add_subdirectory(fit-params)
+     add_subdirectory(results)
+ endif()
++add_subdirectory(grpc-server)
+diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
+index 715477e..de5bed8 100644
+--- a/tools/server/server-context.cpp
++++ b/tools/server/server-context.cpp
+@@ -49,7 +49,16 @@ static uint32_t server_n_outputs_max(const common_params & params) {
+
+     const uint32_t n_outputs_per_seq = 1 + common_speculative_n_max(&params.speculative);
+
+-    const uint64_t n_outputs = (uint64_t) params.n_parallel * n_outputs_per_seq;
++    // score tasks (SERVER_TASK_TYPE_SCORE) output logits for every candidate
++    // token, so reserve room for a bounded candidate tail per parallel slot
++    if (!params.score_enabled) {
++        return std::max<uint32_t>(1, std::min<uint64_t>(n_batch,
++            (uint64_t) params.n_parallel * n_outputs_per_seq));
++    }
++
++    const uint32_t n_outputs_score_seq = 1 + SERVER_SCORE_MAX_CAND_TOKENS;
++
++    const uint64_t n_outputs = (uint64_t) params.n_parallel * std::max(n_outputs_per_seq, n_outputs_score_seq);
+
+     return std::max<uint32_t>(1, std::min<uint64_t>(n_batch, n_outputs));
+ }
+@@ -202,6 +211,26 @@ struct server_slot {
+
+     std::vector<completion_token_output> generated_token_probs;
+
++    // SERVER_TASK_TYPE_SCORE: shared-prefix token logprobs harvested
++    // incrementally across batch views (NaN = not yet produced)
++    std::vector<float> score_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: per-candidate suffix token logprobs; entry
++    // [c][0] comes from the last shared token's logits during prompt
++    // processing, the rest from the forked suffix decode
++    std::vector<std::vector<float>> score_cand_logprobs;
++
++    // SERVER_TASK_TYPE_SCORE: the prompt completed but some candidate has
++    // suffix tokens beyond the first, so a forked decode is still needed
++    bool score_suffix_pending = false;
++
++    // SERVER_TASK_TYPE_SCORE: where the current task's tokens diverged from
++    // the slot's previous cache. When the memory cannot rewind there and a
++    // re-prefill follows, a checkpoint at this position lets the next
++    // scoring call over the same stable prefix (e.g. a classifier's option
++    // list) resume from it instead of re-processing the whole prompt.
++    int32_t score_divergence = -1;
++
+     bool has_next_token = true;
+     bool has_new_line   = false;
+     bool truncated      = false;
+@@ -311,6 +340,10 @@ struct server_slot {
+         }
+         generated_tokens.clear();
+         generated_token_probs.clear();
++        score_logprobs.clear();
++        score_cand_logprobs.clear();
++        score_suffix_pending = false;
++        score_divergence = -1;
+         json_schema = json();
+
+         // clear speculative decoding stats
+@@ -2205,6 +2238,229 @@ private:
+         queue_results.send(std::move(res));
+     }
+
++    // log(sum(exp(logits))) with max-subtraction for stability — the
++    // log_softmax denominator shared by every token read from one output
++    static double score_log_denom(const float * logits, int32_t n_vocab) {
++        float max_logit = logits[0];
++        for (int32_t v = 1; v < n_vocab; ++v) {
++            max_logit = std::max(max_logit, logits[v]);
++        }
++        double sum_exp = 0.0;
++        for (int32_t v = 0; v < n_vocab; ++v) {
++            sum_exp += std::exp((double)(logits[v] - max_logit));
++        }
++        return (double) max_logit + std::log(sum_exp);
++    }
++
++    // Harvest logprobs for SCORE tasks from the current batch view: the
++    // shared-prefix scored tokens, and — from the last shared token's
++    // logits — the first suffix token of every candidate. The scored
++    // region can straddle ubatch boundaries for long prompts, so this
++    // accumulates view by view instead of reading everything when the
++    // prompt completes.
++    void collect_score_logprobs(server_slot & slot, const llama_batch & batch) {
++        const int32_t n_prompt = slot.task->n_score_prompt;
++        const int32_t n_total  = slot.task->n_tokens();
++        const auto &  suffixes = slot.task->score_suffixes;
++
++        const size_t n_shared_scored = (size_t) std::max(0, n_total - n_prompt);
++
++        if (slot.score_logprobs.size() != n_shared_scored) {
++            slot.score_logprobs.assign(n_shared_scored, NAN);
++        }
++        if (slot.score_cand_logprobs.size() != suffixes.size()) {
++            slot.score_cand_logprobs.resize(suffixes.size());
++            for (size_t c = 0; c < suffixes.size(); ++c) {
++                slot.score_cand_logprobs[c].assign(suffixes[c].size(), NAN);
++            }
++        }
++
++        const int32_t n_vocab = llama_vocab_n_tokens(vocab);
++
++        for (int32_t i = 0; i < batch.n_tokens; ++i) {
++            if (!batch.logits[i] || batch.seq_id[i][0] != slot.id) {
++                continue;
++            }
++
++            // the output at position p predicts the task token at index p + 1;
++            // score tasks are text-only, so positions equal token indices
++            const int32_t target = batch.pos[i] + 1;
++            if (target < n_prompt || target > n_total) {
++                continue;
++            }
++
++            const float * logits = llama_get_logits_ith(slot.ctx_tgt, i);
++            if (logits == nullptr) {
++                SLT_ERR(slot, "failed to get logits for score target %d\n", target);
++                continue;
++            }
++
++            const double log_denom = score_log_denom(logits, n_vocab);
++
++            if (target < n_total) {
++                const llama_token tok = slot.task->tokens[target];
++                slot.score_logprobs[target - n_prompt] = (float) ((double) logits[tok] - log_denom);
++            } else {
++                // the last shared token predicts the first suffix token of
++                // every candidate
++                for (size_t c = 0; c < suffixes.size(); ++c) {
++                    if (!suffixes[c].empty()) {
++                        slot.score_cand_logprobs[c][0] = (float) ((double) logits[suffixes[c][0]] - log_denom);
++                    }
++                }
++            }
++        }
++    }
++
++    void send_score(server_slot & slot) {
++        auto res = std::make_unique<server_task_result_score>();
++        res->id              = slot.task->id;
++        res->index           = slot.task->index;
++        res->shared_logprobs = std::move(slot.score_logprobs);
++        res->cand_logprobs   = std::move(slot.score_cand_logprobs);
++
++        slot.score_logprobs.clear();
++        slot.score_cand_logprobs.clear();
++
++        SLT_DBG(slot, "sending score result, n_shared = %zu, n_cand = %zu\n",
++                res->shared_logprobs.size(), res->cand_logprobs.size());
++
++        queue_results.send(std::move(res));
++    }
++
++    // Decode the candidate suffixes of a completed score prompt: fork one
++    // sequence per candidate off the slot's shared prefix (metadata-only
++    // for the unified KV cache, copy-on-write for recurrent state) and
++    // decode all unique suffix tokens in as few llama_decode calls as the
++    // fork/batch/output budgets allow, harvesting a logprob for every
++    // suffix token that predicts a following one.
++    bool decode_score_suffixes(server_slot & slot) {
++        const auto & suffixes = slot.task->score_suffixes;
++
++        auto * mem = llama_get_memory(ctx_tgt);
++
++        // seq ids beyond the slots are reserved for score forks at context
++        // creation (common_params::n_seq_score_forks)
++        const int32_t seq_base    = (int32_t) slots.size();
++        const int32_t n_forks_max = std::min<int32_t>(SERVER_SCORE_FORK_SEQS, (int32_t) llama_n_seq_max(ctx_tgt) - seq_base);
++
++        if (n_forks_max < 1) {
++            SLT_ERR(slot, "no fork sequences reserved for score suffixes (n_seq_max = %d, n_slots = %d)\n",
++                    (int32_t) llama_n_seq_max(ctx_tgt), seq_base);
++            return false;
++        }
++
++        const int32_t   n_batch_max = llama_n_batch(ctx_tgt);
++        const int32_t   n_vocab     = llama_vocab_n_tokens(vocab);
++        const llama_pos pos0        = slot.prompt.tokens.pos_next();
++
++        std::vector<size_t> pending;
++        for (size_t c = 0; c < suffixes.size(); ++c) {
++            // single-token suffixes were fully scored from the last shared
++            // token's logits during prompt processing
++            if (suffixes[c].size() > 1) {
++                if ((int32_t) suffixes[c].size() > n_batch_max) {
++                    SLT_ERR(slot, "score suffix of candidate %zu (%zu tokens) exceeds n_batch (%d)\n",
++                            c, suffixes[c].size(), n_batch_max);
++                    return false;
++                }
++                pending.push_back(c);
++            }
++        }
++
++        size_t next = 0;
++        while (next < pending.size()) {
++            // greedy-pack candidates into one decode within the fork,
++            // batch and reserved-output budgets
++            std::vector<size_t> chunk;
++            int32_t n_tok = 0;
++            int32_t n_out = 0;
++            while (next < pending.size() && (int32_t) chunk.size() < n_forks_max) {
++                const int32_t m = (int32_t) suffixes[pending[next]].size();
++                if (!chunk.empty() && (n_tok + m > n_batch_max || n_out + m - 1 > SERVER_SCORE_MAX_CAND_TOKENS)) {
++                    break;
++                }
++                chunk.push_back(pending[next]);
++                n_tok += m;
++                n_out += m - 1;
++                next++;
++            }
++
++            llama_batch fb = llama_batch_init(n_tok, 0, 1);
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                const llama_seq_id seq = seq_base + (llama_seq_id) k;
++                const auto & sfx = suffixes[chunk[k]];
++
++                llama_memory_seq_rm(mem, seq, -1, -1);
++                llama_memory_seq_cp(mem, slot.id, seq, -1, -1);
++
++                for (size_t j = 0; j < sfx.size(); ++j) {
++                    common_batch_add(fb, sfx[j], pos0 + (llama_pos) j, { seq }, j + 1 < sfx.size());
++                }
++            }
++
++            const int ret = llama_decode(ctx_tgt, fb);
++
++            if (ret == 0) {
++                int32_t i = 0;
++                for (size_t k = 0; k < chunk.size(); ++k) {
++                    const auto & sfx = suffixes[chunk[k]];
++                    auto & out = slot.score_cand_logprobs[chunk[k]];
++
++                    for (size_t j = 0; j < sfx.size(); ++j, ++i) {
++                        if (j + 1 >= sfx.size()) {
++                            continue; // last suffix token predicts nothing
++                        }
++                        const float * logits = llama_get_logits_ith(ctx_tgt, i);
++                        if (logits == nullptr) {
++                            SLT_ERR(slot, "failed to get logits for suffix token %zu of score candidate %zu\n", j, chunk[k]);
++                            continue;
++                        }
++                        const double log_denom = score_log_denom(logits, n_vocab);
++                        out[j + 1] = (float) ((double) logits[sfx[j + 1]] - log_denom);
++                    }
++                }
++            }
++
++            for (size_t k = 0; k < chunk.size(); ++k) {
++                llama_memory_seq_rm(mem, seq_base + (llama_seq_id) k, -1, -1);
++            }
++
++            llama_batch_free(fb);
++
++            if (ret != 0) {
++                SLT_ERR(slot, "score suffix decode failed, ret = %d\n", ret);
++                return false;
++            }
++        }
++
++        return true;
++    }
++
++    // score slots whose prompt completed this iteration decode their
++    // candidate suffixes here, after every batch view was consumed — a
++    // mid-view llama_decode would clobber logits other slots still read
++    void update_score_suffixes() {
++        for (auto & slot : slots) {
++            if (!slot.score_suffix_pending) {
++                continue;
++            }
++            slot.score_suffix_pending = false;
++
++            if (!slot.is_processing() || !slot.task || slot.task->type != SERVER_TASK_TYPE_SCORE) {
++                continue; // the task was aborted mid-iteration
++            }
++
++            if (decode_score_suffixes(slot)) {
++                send_score(slot);
++            } else {
++                send_error(slot, "failed to decode score candidate suffixes", ERROR_TYPE_SERVER);
++            }
++            slot.release();
++        }
++    }
++
+     //
+     // Functions to process the task
+     //
+@@ -2341,6 +2597,7 @@ private:
+             case SERVER_TASK_TYPE_INFILL:
+             case SERVER_TASK_TYPE_EMBEDDING:
+             case SERVER_TASK_TYPE_RERANK:
++            case SERVER_TASK_TYPE_SCORE:
+                 {
+                     // special case: if input is provided via CLI, tokenize it first
+                     // otherwise, no need to tokenize as it's already done inside the HTTP thread
+@@ -2832,6 +3089,13 @@ private:
+                 break; // stop any further processing
+             }
+         }
++
++        try {
++            update_score_suffixes();
++        } catch (const std::exception & e) {
++            SRV_ERR("update_score_suffixes() failed: %s\n", e.what());
++            abort_all_slots("update_score_suffixes() failed: " + std::string(e.what()));
++        }
+     }
+
+     void pre_decode() {
+@@ -3154,6 +3418,16 @@ private:
+                                     n_past = std::min(n_past, slot.alora_invocation_start - 1);
+                                 }
+
++                                // score tasks need the logits that predict the first candidate
++                                // token, so the last shared-prompt token must be (re-)decoded
++                                // even when the cache already covers it
++                                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                                    n_past = std::min(n_past, std::max(0, slot.task->n_score_prompt - 1));
++                                    // remember the divergence point before the checkpoint
++                                    // logic below possibly resets n_past to 0
++                                    slot.score_divergence = n_past;
++                                }
++
+                                 const auto n_cache_reuse = slot.task->params.n_cache_reuse;
+
+                                 const bool can_cache_reuse =
+@@ -3395,8 +3669,12 @@ private:
+
+                     bool do_checkpoint = params_base.n_ctx_checkpoints > 0;
+
+-                    // make checkpoints only for completion tasks
+-                    do_checkpoint = do_checkpoint && slot.task->type == SERVER_TASK_TYPE_COMPLETION;
++                    // make checkpoints for completion tasks, and for score tasks at the
++                    // shared-prompt boundary: models whose memory cannot be partially
++                    // rewound (SWA/hybrid/recurrent) would otherwise re-process the whole
++                    // prompt for every candidate of a scoring call
++                    do_checkpoint = do_checkpoint && (slot.task->type == SERVER_TASK_TYPE_COMPLETION ||
++                                                      slot.task->type == SERVER_TASK_TYPE_SCORE);
+
+                     // make a checkpoint of the parts of the memory that cannot be rolled back.
+                     // checkpoints are created only if:
+@@ -3463,10 +3741,17 @@ private:
+                         // embedding requires all tokens in the batch to be output;
+                         // MTP also wants logits at every prompt position so the
+                         // streaming hook can mirror t_h_nextn into ctx_dft.
++                        // score tasks need outputs at the positions that predict
++                        // each candidate token (the token at index i predicts the
++                        // task token at index i+1).
++                        const bool need_score_logit =
++                            slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            slot.prompt.n_tokens() + 1 >= slot.task->n_score_prompt &&
++                            slot.prompt.n_tokens() + 1 <  slot.task->n_tokens();
+                         add_ok &= batch.add(slot.id,
+                             cur_tok,
+                             slot.prompt.tokens.pos_next(),
+-                            slot.need_embd());
++                            slot.need_embd() || need_score_logit);
+                         slot.prompt.tokens.push_back(cur_tok);
+
+                         slot.n_prompt_tokens_processed++;
+@@ -3481,6 +3766,32 @@ private:
+                             }
+                         }
+
++                        // score tasks: break at the shared-prompt boundary so the checkpoint
++                        // below lands exactly there — the other candidates of the same
++                        // scoring call re-process only their own tokens. Also break at the
++                        // point where this task diverged from the previous cache: after a
++                        // forced re-prefill a checkpoint there serves the next scoring call
++                        // over the same stable prefix (e.g. a classifier's option list).
++                        // The caller-declared stable-prefix boundary is the strongest of
++                        // these: a checkpoint there is at or before every future task's
++                        // divergence within the same option list, so it always survives
++                        // and always restores.
++                        if (do_checkpoint && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                            (slot.prompt.n_tokens() == slot.task->n_score_prompt - 1 ||
++                             (slot.task->n_stable_prompt > 0 &&
++                              slot.prompt.n_tokens() == slot.task->n_stable_prompt &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1) ||
++                             (slot.prompt.n_tokens() == slot.score_divergence &&
++                              slot.prompt.n_tokens() <  slot.task->n_score_prompt - 1))) {
++                            bool have_ckpt = false;
++                            for (const auto & ckpt : slot.prompt.checkpoints) {
++                                have_ckpt |= ckpt.n_tokens == slot.prompt.n_tokens();
++                            }
++                            if (!have_ckpt) {
++                                break;
++                            }
++                        }
++
+                         // process the last few tokens of the prompt separately in order to allow for a checkpoint to be created.
+                         // create checkpoints that many tokens before the end of the prompt:
+                         //  - 4 + n_ubatch
+@@ -3513,6 +3824,15 @@ private:
+                     const bool is_user_start = spans.is_user_start(n_tokens_start);
+                     const bool is_last_user_message = n_tokens_start == last_user_pos;
+
++                    // a batch starting at the score boundary or divergence point must
++                    // always checkpoint — min-step spacing would otherwise suppress it
++                    // and every candidate / next scoring call would re-process the prompt
++                    const bool is_score_boundary = slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                                                   (n_tokens_start == slot.task->n_score_prompt - 1 ||
++                                                    (slot.task->n_stable_prompt > 0 &&
++                                                     n_tokens_start == slot.task->n_stable_prompt) ||
++                                                    n_tokens_start == slot.score_divergence);
++
+                     // entire prompt has been processed
+                     if (slot.prompt.n_tokens() == slot.task->n_tokens()) {
+                         slot.state = SLOT_STATE_DONE_PROMPT;
+@@ -3528,8 +3848,8 @@ private:
+                         slot.init_sampler();
+                     } else {
+                         // skip ordinary mid-prompt checkpoints, unless the batch starts a user
+-                        // message or we are near the end of the prompt
+-                        if (!is_user_start && !near_prompt_end) {
++                        // message, the score boundary, or we are near the end of the prompt
++                        if (!is_user_start && !is_score_boundary && !near_prompt_end) {
+                             do_checkpoint = false;
+                         }
+                     }
+@@ -3546,10 +3866,10 @@ private:
+                     // do not checkpoint after mtmd chunks
+                     do_checkpoint = do_checkpoint && !has_mtmd;
+
+-                    // no need to create checkpoints that are too close together, unless it's the last user message
++                    // no need to create checkpoints that are too close together, unless it's the last user message or the score boundary
+                     do_checkpoint = do_checkpoint && (
+                             slot.prompt.checkpoints.empty() ||
+-                            is_last_user_message || near_prompt_end ||
++                            is_last_user_message || near_prompt_end || is_score_boundary ||
+                             n_tokens_start > slot.prompt.checkpoints.back().n_tokens + params_base.checkpoint_min_step);
+                     SLT_DBG(slot, "main/do_checkpoint = %s, pos_min = %d, pos_max = %d\n", do_checkpoint ? "yes" : "no", pos_min, pos_max);
+
+@@ -3703,6 +4023,13 @@ private:
+                 }
+             }
+
++            // score slots harvest logprobs from every view that contains
++            // their outputs, not just the one holding the final token
++            if (slot.task && slot.task->type == SERVER_TASK_TYPE_SCORE &&
++                (slot.state == SLOT_STATE_PROCESSING_PROMPT || slot.state == SLOT_STATE_DONE_PROMPT)) {
++                collect_score_logprobs(slot, batch_view);
++            }
++
+             if (!is_inside_view(slot.i_batch)) {
+                 // the required token not in this sub-batch, skip
+                 return;
+@@ -3724,6 +4051,25 @@ private:
+                     return;
+                 }
+
++                if (slot.task->type == SERVER_TASK_TYPE_SCORE) {
++                    // shared-prefix logprobs (and every candidate's first
++                    // suffix logprob) were accumulated per view above;
++                    // candidates with more suffix tokens still need the
++                    // forked decode at the end of update_slots()
++                    for (const auto & sfx : slot.task->score_suffixes) {
++                        if (sfx.size() > 1) {
++                            slot.score_suffix_pending = true;
++                            break;
++                        }
++                    }
++                    if (!slot.score_suffix_pending) {
++                        send_score(slot);
++                        slot.release();
++                    }
++                    slot.i_batch = -1;
++                    return;
++                }
++
+                 GGML_ASSERT(slot.task->need_sampling());
+
+                 // prompt evaluated for next-token prediction
+diff --git a/tools/server/server-task.h b/tools/server/server-task.h
+index c3eea2e..fb3c178 100644
+--- a/tools/server/server-task.h
++++ b/tools/server/server-task.h
+@@ -13,10 +13,25 @@
+
+ using json = nlohmann::ordered_json;
+
++// SERVER_TASK_TYPE_SCORE emits one logits output per candidate token (plus
++// the forced last-token output), and the context's output budget
++// (n_outputs_max) is reserved up front — so candidate length must be
++// bounded. Raising this raises the worst-case compute-buffer reservation
++// by ~n_vocab * 4 bytes per extra output.
++constexpr int32_t SERVER_SCORE_MAX_CAND_TOKENS = 64;
++
++// Maximum sequences forked off the shared prefix in one score suffix
++// decode. The context is created with this many seq ids (and
++// recurrent-state cells) beyond the parallel slots — see
++// common_params::n_seq_score_forks; candidates in excess of the budget
++// are decoded in successive chunks.
++constexpr int32_t SERVER_SCORE_FORK_SEQS = 16;
++
+ enum server_task_type {
+     SERVER_TASK_TYPE_COMPLETION,
+     SERVER_TASK_TYPE_EMBEDDING,
+     SERVER_TASK_TYPE_RERANK,
++    SERVER_TASK_TYPE_SCORE,
+     SERVER_TASK_TYPE_INFILL,
+     SERVER_TASK_TYPE_CANCEL,
+     SERVER_TASK_TYPE_CONTROL,
+@@ -153,6 +168,18 @@ struct server_task {
+     task_params   params;
+     server_tokens tokens;
+
++    // used by SERVER_TASK_TYPE_SCORE: `tokens` holds the shared prefix
++    // (prompt + longest common candidate token prefix) and logprobs are
++    // returned for its tokens from n_score_prompt onward. Each candidate's
++    // tokens beyond the shared prefix ride a forked sequence.
++    int32_t n_score_prompt = 0;
++    std::vector<llama_tokens> score_suffixes;
++    // token index where the caller-declared stable prompt prefix ends
++    // (0 = no hint): the option-list system prompt that repeats across
++    // scoring calls. A context checkpoint is forced there so models that
++    // cannot rewind state re-process only the per-call tail next time.
++    int32_t n_stable_prompt = 0;
++
+     // only used by CLI, this allow tokenizing CLI inputs on server side
+     // we need this because mtmd_context and vocab are not accessible outside of server_context
+     bool                    cli = false;
+@@ -197,6 +224,7 @@ struct server_task {
+         switch (type) {
+             case SERVER_TASK_TYPE_COMPLETION:
+             case SERVER_TASK_TYPE_INFILL:
++            case SERVER_TASK_TYPE_SCORE:
+                 return true;
+             default:
+                 return false;
+@@ -494,6 +522,25 @@ struct server_task_result_rerank : server_task_result {
+     virtual json to_json() override;
+ };
+
++struct server_task_result_score : server_task_result {
++    // log P(token | prefix) for the shared-prefix tokens after
++    // n_score_prompt, in order; NaN marks positions the decode never
++    // produced an output for
++    std::vector<float> shared_logprobs;
++
++    // per candidate: logprobs of its suffix tokens, in task order (entry
++    // 0 is the token right after the shared prefix, predicted by the last
++    // shared token's logits)
++    std::vector<std::vector<float>> cand_logprobs;
++
++    virtual json to_json() override {
++        return json {
++            {"shared_logprobs", shared_logprobs},
++            {"cand_logprobs",   cand_logprobs},
++        };
++    }
++};
++
+ struct server_task_result_error : server_task_result {
+     error_type err_type = ERROR_TYPE_SERVER;
+     std::string err_msg;

backend/cpp/turboquant/Makefile

@@ -47,6 +47,7 @@ define turboquant-build
 	# original under backend/cpp/llama-cpp/, so the stock llama-cpp build
 	# stays compiling against vanilla upstream.
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:$(1)$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-$(1)-build llama.cpp
@@ -84,6 +85,7 @@ turboquant-cpu-all:
 	rm -rf $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/patches
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build purge
 	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build/grpc-server.cpp
 	$(info $(GREEN)I turboquant build info:cpu-all-variants$(RESET))
 	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(TURBOQUANT_VERSION) \
 	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../turboquant-cpu-all-build llama.cpp

backend/go/cloud-proxy/provider_anthropic.go

@@ -32,7 +32,9 @@ import (
 type anthropicRequest struct {
 	Model         string               `json:"model"`
 	MaxTokens     int32                `json:"max_tokens"`
-	System        string               `json:"system,omitempty"`
+	// System is `any`: a bare string normally, or []anthropicSystemBlock
+	// when cache_prompt is on (the block form carries cache_control).
+	System        any                  `json:"system,omitempty"`
 	Messages      []anthropicMessage   `json:"messages"`
 	Stream        bool                 `json:"stream,omitempty"`
 	Temperature   *float64             `json:"temperature,omitempty"`
@@ -52,9 +54,30 @@ type anthropicMessage struct {
 }
 
 type anthropicTool struct {
-	Name        string          `json:"name"`
-	Description string          `json:"description,omitempty"`
-	InputSchema json.RawMessage `json:"input_schema"`
+	Name         string                 `json:"name"`
+	Description  string                 `json:"description,omitempty"`
+	InputSchema  json.RawMessage        `json:"input_schema"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
+}
+
+// anthropicCacheControl marks a prompt-cache breakpoint. Anthropic caches
+// everything up to and including a block tagged {"type":"ephemeral"} (5-min
+// TTL) and serves that prefix at the cache-read rate (0.1x input) on later
+// calls that share it — the win on agentic/multi-turn workloads.
+type anthropicCacheControl struct {
+	Type string `json:"type"` // "ephemeral"
+}
+
+// ephemeralCacheControl is the single reused breakpoint marker.
+var ephemeralCacheControl = &anthropicCacheControl{Type: "ephemeral"}
+
+// anthropicSystemBlock is the block form of the top-level system field.
+// Anthropic accepts system as a bare string OR a list of text blocks; the
+// block form is required to attach cache_control to the system prompt.
+type anthropicSystemBlock struct {
+	Type         string                 `json:"type"` // "text"
+	Text         string                 `json:"text"`
+	CacheControl *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 // anthropicToolChoice mirrors the four shapes Anthropic accepts:
@@ -81,8 +104,9 @@ type anthropicContentBlock struct {
 	// Tool-result block fields. tool_result uses `content` (not
 	// `text`) and pairs with `tool_use_id`; modelling them as
 	// distinct fields avoids ambiguity at marshal time.
-	ToolUseID     string `json:"tool_use_id,omitempty"`
-	ResultContent string `json:"content,omitempty"`
+	ToolUseID     string                 `json:"tool_use_id,omitempty"`
+	ResultContent string                 `json:"content,omitempty"`
+	CacheControl  *anthropicCacheControl `json:"cache_control,omitempty"`
 }
 
 type anthropicResponse struct {
@@ -156,6 +180,11 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 	if req.ToolChoice != nil && req.ToolChoice.Type == anthropicToolChoiceNone {
 		req.Tools, req.ToolChoice = nil, nil
 	}
+	// Prompt-cache breakpoint on the last tool: Anthropic caches the entire
+	// tool block up to the marked tool — usually a large, fully stable prefix.
+	if cfg.cachePrompt && len(req.Tools) > 0 {
+		req.Tools[len(req.Tools)-1].CacheControl = ephemeralCacheControl
+	}
 
 	var systemParts []string
 	for _, m := range opts.GetMessages() {
@@ -189,15 +218,54 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 			})
 		}
 	}
-	req.System = strings.Join(systemParts, "\n\n")
+	// System: block form (with cache_control) when caching is on, else the
+	// bare string. Only set when non-empty so `omitempty` still drops it.
+	if len(systemParts) > 0 {
+		joined := strings.Join(systemParts, "\n\n")
+		if cfg.cachePrompt {
+			req.System = []anthropicSystemBlock{{Type: "text", Text: joined, CacheControl: ephemeralCacheControl}}
+		} else {
+			req.System = joined
+		}
+	}
 
 	if len(req.Messages) == 0 && opts.GetPrompt() != "" {
 		req.Messages = []anthropicMessage{{Role: "user", Content: opts.GetPrompt()}}
 	}
 
+	// Prompt-cache breakpoint on the final message block caches the whole
+	// conversation prefix up to the newest turn. With the system + tools
+	// breakpoints above, Anthropic serves the entire stable head at the
+	// cache-read rate on the next agentic iteration (max 4 breakpoints; we
+	// use at most 3, so we never exceed the limit).
+	if cfg.cachePrompt {
+		markLastMessageCacheable(req.Messages)
+	}
+
 	return json.Marshal(req)
 }
 
+// markLastMessageCacheable tags the final block of the last message with a
+// cache_control breakpoint. String content is promoted to a single text
+// block so the marker has somewhere to attach; block content gets the marker
+// on its last element.
+func markLastMessageCacheable(msgs []anthropicMessage) {
+	if len(msgs) == 0 {
+		return
+	}
+	last := &msgs[len(msgs)-1]
+	switch c := last.Content.(type) {
+	case string:
+		if c != "" {
+			last.Content = []anthropicContentBlock{{Type: "text", Text: c, CacheControl: ephemeralCacheControl}}
+		}
+	case []anthropicContentBlock:
+		if len(c) > 0 {
+			c[len(c)-1].CacheControl = ephemeralCacheControl
+		}
+	}
+}
+
 // appendToolResult appends a tool_result block as a user message,
 // merging into a preceding user message that already carries blocks.
 // Anthropic concatenates consecutive same-role messages on its end,

backend/go/cloud-proxy/provider_anthropic_test.go

@@ -328,3 +328,62 @@ func TestBuildAnthropic_RoundTripsAssistantToolCalls(t *testing.T) {
 	g.Expect(r0["tool_use_id"]).To(Equal("call_abc"))
 	g.Expect(r0["content"]).To(Equal(`{"models":["a","b"]}`))
 }
+
+// TestPredict_Anthropic_PromptCache verifies that cache_prompt injects
+// exactly the intended cache_control breakpoints (system, last tool, last
+// message) when on, and none when off — asserting on the raw upstream body
+// because System becomes a block list that the typed struct hides.
+func TestPredict_Anthropic_PromptCache(t *testing.T) {
+	g := NewWithT(t)
+
+	// run issues one translate Predict and returns the raw body the fake
+	// Anthropic upstream received.
+	run := func(cachePrompt bool) string {
+		var rawBody string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			b, _ := io.ReadAll(r.Body)
+			rawBody = string(b)
+			w.Header().Set("Content-Type", "application/json")
+			_, _ = io.WriteString(w, `{"id":"m","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"model":"claude-3-5-sonnet-20241022","usage":{"input_tokens":5,"output_tokens":2}}`)
+		}))
+		defer srv.Close()
+
+		t.Setenv("CLOUD_PROXY_ANTHROPIC_FAKE", "sk-ant-fake")
+		cp := NewCloudProxy()
+		err := cp.Load(&pb.ModelOptions{
+			Model: "claude-local",
+			Proxy: &pb.ProxyOptions{
+				UpstreamUrl:   srv.URL,
+				Mode:          modeTranslate,
+				Provider:      providerAnthropic,
+				ApiKeyEnv:     "CLOUD_PROXY_ANTHROPIC_FAKE",
+				UpstreamModel: "claude-3-5-sonnet-20241022",
+				CachePrompt:   cachePrompt,
+			},
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+
+		_, err = cp.Predict(&pb.PredictOptions{
+			Messages: []*pb.Message{
+				{Role: "system", Content: "be brief"},
+				{Role: "user", Content: "hello"},
+			},
+			Tools:  `[{"type":"function","function":{"name":"t","parameters":{"type":"object"}}}]`,
+			Tokens: 32,
+		})
+		g.Expect(err).NotTo(HaveOccurred())
+		return rawBody
+	}
+
+	// cache_prompt ON: three ephemeral breakpoints (system + last tool +
+	// last message), and system is emitted in block form.
+	on := run(true)
+	g.Expect(strings.Count(on, `"cache_control":{"type":"ephemeral"}`)).To(Equal(3),
+		"expected 3 breakpoints (system, tool, last message); body=%s", on)
+	g.Expect(on).To(ContainSubstring(`"system":[{"type":"text","text":"be brief"`))
+
+	// cache_prompt OFF: no breakpoints, system stays a bare string.
+	off := run(false)
+	g.Expect(off).NotTo(ContainSubstring("cache_control"))
+	g.Expect(off).To(ContainSubstring(`"system":"be brief"`))
+}

backend/go/cloud-proxy/proxy.go

@@ -48,6 +48,7 @@ type proxyConfig struct {
 	upstreamModel string
 	localModel    string // ModelOptions.Model — fallback when upstream_model is unset
 	apiKey        string // resolved at Load time
+	cachePrompt   bool   // inject Anthropic prompt-cache breakpoints (translate+anthropic)
 }
 
 func NewCloudProxy() *CloudProxy {
@@ -106,6 +107,7 @@ func (c *CloudProxy) Load(opts *pb.ModelOptions) error {
 		upstreamModel: po.GetUpstreamModel(),
 		localModel:    opts.GetModel(),
 		apiKey:        key,
+		cachePrompt:   po.GetCachePrompt(),
 	})
 	xlog.Info("cloud-proxy: ready",
 		"upstream", po.GetUpstreamUrl(),

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=306faee45fab641d54f9f941f075de1e9c0d3278
+CRISPASR_VERSION?=754b67289cf1137e3ed722885705f94132fc614f
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=f4e17dea695dd12ae76bea98ba58030996b98118
+DEPTHANYTHING_VERSION?=2028b47ac75a8659c6a9aa617baf09be193eb55f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+# Upstream pin lives below as PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=1da853421de9710cbe894a0110711de5a0516486
+PARAKEET_VERSION?=e747acdaee69b916cef62263ae5f718bda9ff3f3
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=2d0385ba85af358f7115dda608a63eafd9de7ffd
+STABLEDIFFUSION_GGML_VERSION?=22516991cbdf725e69b0b4a87e52ca16cce07c2d
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/README.md

@@ -29,29 +29,6 @@ ABI v2) through purego:
   LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
   choice constraints are also exposed by the ABI.
 
-## Hardware coverage
-
-The CUDA builds require the CUDA 13 toolchain and target Blackwell only:
-`sm_120a` + `sm_121a` on x86_64, `sm_121a` (GB10 / DGX Spark) on arm64. CUDA
-12.x nvcc cannot compile the Blackwell fp4 kernels, so no CUDA 12 variant is
-shipped and `backend/index.yaml` maps the `nvidia-cuda-12` /
-`nvidia-l4t-cuda-12` capabilities at the CPU build. Practically:
-
-| Host | Installed build |
-|---|---|
-| x86_64 + CUDA 13 | `cuda13-vllm-cpp` |
-| DGX Spark / GB10 (JetPack 7, CUDA 13) | `nvidia-l4t-arm64-vllm-cpp` |
-| Jetson AGX Orin (sm_87, JetPack 6, CUDA 12) | `cpu-vllm-cpp` |
-| Apple Silicon | `metal-vllm-cpp` |
-| Anything else | `vulkan-vllm-cpp` or `cpu-vllm-cpp` |
-
-The capability a host reports comes from `/run/localai/capability` inside the
-LocalAI container, which the image bakes in at build time (see `Dockerfile`).
-A DGX Spark running the CUDA 12 `-nvidia-l4t-arm64` image therefore reports
-`nvidia-l4t-cuda-12` and gets the CPU build; use the `-nvidia-l4t-arm64-cuda-13`
-image, or set `LOCALAI_FORCE_META_BACKEND_CAPABILITY=nvidia-l4t-cuda-13`, to
-get the GPU one.
-
 Model config example:
 
 ```yaml

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=080bbbe85230f624f0b52127f1ae1218247989f9
+WHISPER_CPP_VERSION?=97c56f1dc1d1100a9d859c865a20c82d22f823ed
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/index.yaml

@@ -167,10 +167,6 @@
     inference time. It loads Hugging Face safetensors and GGUF checkpoints, supports
     structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine,
     and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan.
-    The CUDA builds require the CUDA 13 toolchain and target Blackwell only: sm_120a
-    plus sm_121a on x86_64, and sm_121a (GB10 / DGX Spark) on arm64. Older NVIDIA
-    hardware and CUDA 12 hosts - including Jetson AGX Orin (sm_87, JetPack 6) - run
-    the CPU build instead.
   urls:
     - https://github.com/mudler/vllm.cpp
   tags:
@@ -188,12 +184,6 @@
     nvidia-cuda-13: "cuda13-vllm-cpp"
     nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp"
     nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp"
-    # No CUDA 12 variant exists: 12.x nvcc cannot compile the Blackwell fp4
-    # kernels, so those hosts run the CPU build. Mapped explicitly rather than
-    # left to the "default" catch-all so the fallback is visible here instead
-    # of looking like an oversight.
-    nvidia-cuda-12: "cpu-vllm-cpp"
-    nvidia-l4t-cuda-12: "cpu-vllm-cpp"
 - !!merge <<: *vllm-cpp
   name: "vllm-cpp-development"
   capabilities:
@@ -204,8 +194,6 @@
     nvidia-cuda-13: "cuda13-vllm-cpp-development"
     nvidia-l4t: "nvidia-l4t-arm64-vllm-cpp-development"
     nvidia-l4t-cuda-13: "nvidia-l4t-arm64-vllm-cpp-development"
-    nvidia-cuda-12: "cpu-vllm-cpp-development"
-    nvidia-l4t-cuda-12: "cpu-vllm-cpp-development"
 - &crispasr
   name: "crispasr"
   alias: "crispasr"
@@ -1467,6 +1455,7 @@
   alias: "kokoro"
   name: "kokoro"
   capabilities:
+    default: "cpu-kokoro"
     nvidia: "cuda12-kokoro"
     intel: "intel-kokoro"
     amd: "rocm-kokoro"
@@ -5198,11 +5187,22 @@
 - !!merge <<: *kokoro
   name: "kokoro-development"
   capabilities:
+    default: "cpu-kokoro-development"
     nvidia: "cuda12-kokoro-development"
     intel: "intel-kokoro-development"
     amd: "rocm-kokoro-development"
     nvidia-l4t: "nvidia-l4t-kokoro-development"
     metal: "metal-kokoro-development"
+- !!merge <<: *kokoro
+  name: "cpu-kokoro"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:latest-cpu-kokoro
+- !!merge <<: *kokoro
+  name: "cpu-kokoro-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-kokoro"
+  mirrors:
+    - localai/localai-backends:master-cpu-kokoro
 - !!merge <<: *kokoro
   name: "cuda12-kokoro-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-kokoro"

backend/python/mlx-vlm/requirements-mps.txt

@@ -1 +1,3 @@
-git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
+git+https://github.com/Blaizzy/mlx-vlm@v0.4.4
+torch
+torchvision

backend/python/nemo/install.sh

@@ -14,4 +14,28 @@ if [ "x${BUILD_PROFILE}" == "xintel" ]; then
     EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
 fi
 
+# Darwin needs a newer interpreter than libbackend's 3.10 default. nemo_toolkit
+# pulls in text2num, a Rust extension built with maturin, and its macOS arm64
+# wheels start at cp311 (3.0.2 publishes cp311/cp312/cp313/cp314 and no cp310).
+# On 3.10 pip therefore falls back to the sdist and dies in the PEP 517 hook
+# with "No module named 'maturin'", since EXTRA_PIP_INSTALL_FLAGS carries
+# --no-build-isolation and nothing installs the build backend. Moving to 3.12
+# takes the prebuilt wheel and needs no Rust toolchain on the runner at all.
+#
+# Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
+# for the same package and have no reason to move.
+if [ "x${BUILD_PROFILE}" == "xmps" ] || [ "x${BUILD_PROFILE}" == "xmetal" ]; then
+    PYTHON_VERSION="3.12"
+    # PYTHON_PATCH must move with it. libbackend builds the portable-Python URL
+    # as cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}-..., and
+    # the default patch is 18 for 3.10.18; leaving it alone asks for a 3.12.18
+    # that was never released and the download 404s.
+    #
+    # 11, not the 12 that sglang/install.sh uses for l4t13: at the 20250818 tag
+    # python-build-standalone published 3.12.12 for linux aarch64 but not for
+    # aarch64-apple-darwin, where 3.12.11 is the newest. Verified against the
+    # release assets rather than copied across.
+    PYTHON_PATCH="11"
+fi
+
 installRequirements

backend/python/sglang/requirements-cublas12-after.txt

@@ -2,3 +2,16 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
+
+# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
+# `diffusion` extra with no version bound of its own, and install.sh passes a
+# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
+# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
+# backend imports wheel_stub without declaring it as a build dependency; with
+# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
+# every cublas sglang image fails with "No module named 'wheel_stub'".
+#
+# Bounding this one package rather than dropping the global flag: the flag is
+# load-bearing for flash-attn-4, and this is the narrower change. Raise the
+# bound once 0.46.0 final ships.
+nvidia-modelopt<0.46

backend/python/sglang/requirements-cublas13-after.txt

@@ -2,3 +2,16 @@
 # (FunctionCallParser, ReasoningParser) move between releases.
 # 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
 sglang[all]>=0.5.11
+
+# Keep nvidia-modelopt on a stable release. sglang[all] pulls it in through its
+# `diffusion` extra with no version bound of its own, and install.sh passes a
+# GLOBAL --prerelease=allow (needed because flash-attn-4 only ships 4.0.0b*
+# wheels). Unbounded plus prereleases-allowed resolves to 0.46.0rc0, whose build
+# backend imports wheel_stub without declaring it as a build dependency; with
+# --no-build-isolation also in EXTRA_PIP_INSTALL_FLAGS nothing installs it, and
+# every cublas sglang image fails with "No module named 'wheel_stub'".
+#
+# Bounding this one package rather than dropping the global flag: the flag is
+# load-bearing for flash-attn-4, and this is the narrower change. Raise the
+# bound once 0.46.0 final ships.
+nvidia-modelopt<0.46

core/application/router_factories.go

@@ -46,12 +46,12 @@ type lazyScorer struct {
 	modelName string
 }
 
-func (l *lazyScorer) Score(ctx context.Context, prompt string, candidates []string) ([]backend.CandidateScore, error) {
+func (l *lazyScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]backend.CandidateScore, error) {
 	cfg := l.app.adapterConfig(l.modelName)
 	if cfg == nil {
 		return nil, fmt.Errorf("scorer: model %q no longer available", l.modelName)
 	}
-	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, candidates)
+	return backend.NewScorer(l.app.modelLoader, *cfg, l.app.applicationConfig).Score(ctx, prompt, stablePrefixLen, candidates)
 }
 
 // TokenCounter returns a func so the middleware's literal field type accepts

core/application/router_factories_test.go

@@ -109,7 +109,7 @@ var _ = Describe("router_factories lazy config resolution", func() {
 			Expect(lazy.modelName).To(Equal("score-test"))
 
 			removeCfg("score-test")
-			_, err := sc.Score(context.Background(), "prompt", []string{"a"})
+			_, err := sc.Score(context.Background(), "prompt", 0, []string{"a"})
 			Expect(err).To(HaveOccurred())
 			Expect(err.Error()).To(ContainSubstring("no longer available"))
 		})

core/backend/options.go

@@ -166,6 +166,21 @@ func estimateModelSizeBytes(c config.ModelConfig, modelsPath string) int64 {
 	return int64(result.SizeBytes)
 }
 
+// effectiveThreads resolves the thread count a backend is asked to use.
+// Per-model threads wins: SetDefaults already fills an unset per-model value
+// from the app-level --threads, so overriding a set value with the app value
+// here would make the YAML `threads:` knob dead config (it did, for years —
+// e.g. a tiny VAD model could never opt down from the global pool size).
+func effectiveThreads(c config.ModelConfig, appThreads int) int {
+	if c.Threads != nil && *c.Threads > 0 {
+		return *c.Threads
+	}
+	if appThreads > 0 {
+		return appThreads
+	}
+	return 1
+}
+
 func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...model.Option) []model.Option {
 	defOpts := []model.Option{
 		model.WithBackendString(c.Backend),
@@ -178,16 +193,7 @@ func ModelOptions(c config.ModelConfig, so *config.ApplicationConfig, opts ...mo
 		defOpts = append(defOpts, model.WithModelFile(c.ModelFileName()))
 	}
 
-	threads := 1
-
-	if c.Threads != nil {
-		threads = *c.Threads
-	}
-
-	if so.Threads != 0 {
-		threads = so.Threads
-	}
-
+	threads := effectiveThreads(c, so.Threads)
 	c.Threads = &threads
 
 	grpcOpts := grpcModelOpts(c, so.SystemState.Model.ModelsPath)
@@ -416,6 +422,7 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 		Options:              withCompanionArtifactOptions(c.Options, c.Artifacts),
 		Overrides:            c.Overrides,
 		EngineArgs:           engineArgsJSON,
+		EnableScore:          c.HasUsecases(config.FLAG_SCORE),
 		CLIPSkip:             int32(c.Diffusers.ClipSkip),
 		ControlNet:           c.Diffusers.ControlNet,
 		ContextSize:          int32(ctxSize),
@@ -475,6 +482,7 @@ func grpcModelOpts(c config.ModelConfig, modelPath string) *pb.ModelOptions {
 			ApiKeyFile:            c.Proxy.APIKeyFile,
 			UpstreamModel:         c.Proxy.UpstreamModel,
 			RequestTimeoutSeconds: int32(c.Proxy.RequestTimeoutSeconds),
+			CachePrompt:           c.Proxy.CachePrompt,
 		}
 	}
 

core/backend/options_internal_test.go

@@ -120,6 +120,7 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(512))
+		Expect(opts.EnableScore).To(BeFalse())
 	})
 
 	It("sizes the batch to the context window for score models", func() {
@@ -128,6 +129,14 @@ var _ = Describe("grpcModelOpts NBatch", func() {
 		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &scoreUsecase}
 		opts := grpcModelOpts(cfg, "/tmp/models")
 		Expect(opts.NBatch).To(BeEquivalentTo(4096))
+		Expect(opts.EnableScore).To(BeTrue())
+	})
+
+	It("enables score resources for a model with multiple usecases", func() {
+		usecases := config.FLAG_CHAT | config.FLAG_SCORE
+		cfg := config.ModelConfig{Threads: &threads, LLMConfig: config.LLMConfig{ContextSize: &ctx}, KnownUsecases: &usecases}
+		opts := grpcModelOpts(cfg, "/tmp/models")
+		Expect(opts.EnableScore).To(BeTrue())
 	})
 
 	It("keeps an explicit batch over the score default", func() {
@@ -355,3 +364,23 @@ var _ = Describe("gRPCPredictOpts model identity", func() {
 		Expect(opts.ModelIdentity).To(BeEmpty())
 	})
 })
+
+var _ = Describe("effectiveThreads", func() {
+	It("lets a per-model threads value override the app-level --threads", func() {
+		one := 1
+		cfg := config.ModelConfig{Threads: &one}
+		Expect(effectiveThreads(cfg, 10)).To(Equal(1),
+			"per-model threads is a real knob, not dead config under --threads")
+	})
+
+	It("falls back to the app-level threads when the model sets none", func() {
+		Expect(effectiveThreads(config.ModelConfig{}, 10)).To(Equal(10))
+		zero := 0
+		Expect(effectiveThreads(config.ModelConfig{Threads: &zero}, 10)).To(Equal(10),
+			"an explicit threads: 0 means unset, not zero threads")
+	})
+
+	It("never resolves to a non-positive thread count", func() {
+		Expect(effectiveThreads(config.ModelConfig{}, 0)).To(Equal(1))
+	})
+})

core/backend/preload.go

@@ -28,7 +28,7 @@ func PreloadModelByName(ctx context.Context, cl *config.ModelConfigLoader, ml *m
 		return nil, err
 	}
 
-	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath)
+	stages, err := pipelineStages(cl, &cfg.Pipeline, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 		return nil, err
 	}
@@ -59,7 +59,7 @@ var loadStage = PreloadModel
 // pipeline itself uses. A stage that fails to resolve is a misconfiguration,
 // so it fails fast rather than being deferred to load. A pipeline with no
 // stages set returns nil, which callers treat as "not a pipeline".
-func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string) ([]PreloadStage, error) {
+func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath string, opts ...config.ConfigLoaderOption) ([]PreloadStage, error) {
 	voiceRec := ""
 	if p.VoiceRecognition != nil {
 		voiceRec = p.VoiceRecognition.Model
@@ -76,7 +76,7 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 		if s.name == "" {
 			continue
 		}
-		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath)
+		cfg, err := cl.LoadResolvedModelConfig(s.name, modelPath, opts...)
 		if err != nil {
 			return nil, fmt.Errorf("%s (%s): %w", s.role, s.name, err)
 		}
@@ -87,9 +87,11 @@ func pipelineStages(cl *config.ModelConfigLoader, p *config.Pipeline, modelPath
 
 // PreloadStages loads every present stage at once and waits for all of them, so
 // a pipeline warms in the time of its slowest stage rather than the sum. Absent
-// (nil-config) stages are skipped. A failed stage does not cancel the others —
-// they all run to completion so the joined error names every broken stage at
-// once, alongside the names that did load.
+// stages are skipped. Some callers represent an unset optional stage with a
+// nil config, while others materialize a default config with an empty name. A
+// failed stage does not cancel the others — they all run to completion so the
+// joined error names every broken stage at once, alongside the names that did
+// load.
 func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config.ApplicationConfig, stages []PreloadStage) ([]string, error) {
 	var (
 		wg     sync.WaitGroup
@@ -98,7 +100,7 @@ func PreloadStages(ctx context.Context, ml *model.ModelLoader, appConfig *config
 		errs   []error
 	)
 	for _, s := range stages {
-		if s.Cfg == nil {
+		if s.Cfg == nil || s.Cfg.Name == "" {
 			continue
 		}
 		wg.Add(1)

core/backend/preload_internal_test.go

@@ -103,12 +103,13 @@ var _ = Describe("PreloadStages", func() {
 		return PreloadStage{Role: role, Cfg: &config.ModelConfig{Name: name}}
 	}
 
-	It("loads every present stage, skips absent (nil-config) ones, and returns the loaded names", func() {
+	It("loads every present stage, skips absent stages, and returns the loaded names", func() {
 		stubLoader(nil)
 
 		loaded, err := PreloadStages(context.Background(), nil, nil, []PreloadStage{
 			mkStage("vad", "vad-m"),
-			{Role: "transcription"}, // absent stage
+			{Role: "transcription"},
+			mkStage("tts", ""),
 			mkStage("llm", "llm-m"),
 		})
 

core/backend/score.go

@@ -23,6 +23,10 @@ type ScoreOptions struct {
 	// token count. Useful when comparing candidates of different
 	// lengths — without it, longer candidates score lower by default.
 	LengthNormalize bool
+	// StablePrefixLen is the byte length of the prompt prefix that stays
+	// identical across repeated scoring calls (0 = unknown); forwarded to
+	// the backend as a state-reuse boundary hint.
+	StablePrefixLen int
 }
 
 // CandidateScore is the per-candidate result. Mirrors pb.CandidateScore
@@ -42,9 +46,13 @@ type TokenLogProb struct {
 // Scorer evaluates a model's joint log-probability of each candidate
 // continuation given a shared prompt. Implemented by NewScorer over a
 // model-loaded backend; the router's score classifier consumes this
-// for multi-label policy selection.
+// for multi-label policy selection. stablePrefixLen is the byte length
+// of the prompt prefix that stays identical across calls (0 = unknown)
+// — backends use it to place a state-reuse point at the boundary, which
+// is what keeps repeat scoring fast on models that cannot rewind
+// (hybrid/recurrent architectures).
 type Scorer interface {
-	Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error)
+	Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error)
 }
 
 // NewScorer binds (loader, modelConfig, appConfig) into a Scorer. The
@@ -61,8 +69,8 @@ type modelScorer struct {
 	appConfig   *config.ApplicationConfig
 }
 
-func (m *modelScorer) Score(ctx context.Context, prompt string, candidates []string) ([]CandidateScore, error) {
-	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true}, m.loader, m.modelConfig, m.appConfig)
+func (m *modelScorer) Score(ctx context.Context, prompt string, stablePrefixLen int, candidates []string) ([]CandidateScore, error) {
+	fn, err := ModelScore(prompt, candidates, ScoreOptions{LengthNormalize: true, StablePrefixLen: stablePrefixLen}, m.loader, m.modelConfig, m.appConfig)
 	if err != nil {
 		return nil, err
 	}
@@ -103,6 +111,7 @@ func ModelScore(prompt string, candidates []string, opts ScoreOptions, loader *m
 			Candidates:           candidates,
 			IncludeTokenLogprobs: opts.IncludeTokenLogprobs,
 			LengthNormalize:      opts.LengthNormalize,
+			StablePrefixLen:      int32(opts.StablePrefixLen),
 		})
 		results := scoreResponseToCandidates(resp, opts.IncludeTokenLogprobs)
 		if appConfig.EnableTracing {

core/cli/run.go

@@ -243,6 +243,23 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return nil
 	}
 
+	activatedListeners, err := systemdActivatedListeners()
+	if err != nil {
+		return fmt.Errorf("loading systemd socket activation listeners: %w", err)
+	}
+	activatedListener, err := selectSystemdListener(activatedListeners)
+	if err != nil {
+		for _, listener := range activatedListeners {
+			_ = listener.Close()
+		}
+		return err
+	}
+	if activatedListener != nil {
+		defer func() {
+			_ = activatedListener.Close()
+		}()
+	}
+
 	os.MkdirAll(r.BackendsPath, 0750)
 	os.MkdirAll(r.ModelsPath, 0750)
 
@@ -732,8 +749,13 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// LAN, or VPN that's the historical "trusted network" deployment, but on
 	// a public IP it makes every model, gallery install, settings change, and
 	// admin endpoint reachable by anyone who can connect to the port.
+	listenAddress := r.Address
+	if activatedListener != nil {
+		listenAddress = activatedListener.Addr().String()
+	}
+
 	authConfigured := app.AuthDB() != nil || len(r.APIKeys) > 0
-	if err := requireAuthOrTrustedBind(r.Address, authConfigured, r.AllowInsecurePublicBind); err != nil {
+	if err := requireAuthOrTrustedBind(listenAddress, authConfigured, r.AllowInsecurePublicBind); err != nil {
 		return err
 	}
 
@@ -743,7 +765,11 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		return err
 	}
 
-	xlog.Info("LocalAI is started and running", "address", r.Address)
+	if activatedListener != nil {
+		appHTTP.Listener = activatedListener
+		xlog.Info("Using systemd socket activation listener", "address", listenAddress)
+	}
+	xlog.Info("LocalAI is started and running", "address", listenAddress)
 
 	// Start P2P if token was provided via CLI/env or loaded from runtime_settings.json
 	if token != "" || app.ApplicationConfig().P2PToken != "" {
@@ -762,11 +788,11 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 	// backends like PostgreSQL need to call the embeddings API during
 	// collection initialization.
 	go func() {
-		waitForServerReady(r.Address, app.ApplicationConfig().Context)
+		waitForServerReady(listenAddress, app.ApplicationConfig().Context)
 		app.StartAgentPool()
 	}()
 
-	return appHTTP.Start(r.Address)
+	return appHTTP.Start(listenAddress)
 }
 
 // waitForServerReady polls the given address until the HTTP server is

core/cli/run_socket_activation.go

@@ -0,0 +1,17 @@
+package cli
+
+import (
+	"fmt"
+	"net"
+)
+
+func selectSystemdListener(listeners []net.Listener) (net.Listener, error) {
+	switch len(listeners) {
+	case 0:
+		return nil, nil
+	case 1:
+		return listeners[0], nil
+	default:
+		return nil, fmt.Errorf("systemd socket activation requires exactly one stream listener, got %d", len(listeners))
+	}
+}

core/cli/run_socket_activation_linux.go

@@ -0,0 +1,71 @@
+//go:build linux
+
+package cli
+
+import (
+	"fmt"
+	"net"
+	"os"
+	"strconv"
+)
+
+const systemdListenFDStart = 3
+
+func systemdActivatedListeners() ([]net.Listener, error) {
+	listenPID := os.Getenv("LISTEN_PID")
+	listenFDs := os.Getenv("LISTEN_FDS")
+	if listenPID == "" && listenFDs == "" {
+		return nil, nil
+	}
+
+	defer func() {
+		for _, key := range []string{"LISTEN_PID", "LISTEN_FDS", "LISTEN_FDNAMES"} {
+			_ = os.Unsetenv(key)
+		}
+	}()
+
+	pid, err := strconv.Atoi(listenPID)
+	if err != nil {
+		return nil, fmt.Errorf("invalid LISTEN_PID %q: %w", listenPID, err)
+	}
+	count, err := strconv.Atoi(listenFDs)
+	if err != nil || count < 0 {
+		return nil, fmt.Errorf("invalid LISTEN_FDS %q", listenFDs)
+	}
+	if pid != os.Getpid() || count == 0 {
+		return nil, nil
+	}
+
+	return listenersFromSystemdFDs(systemdListenFDStart, count)
+}
+
+func listenersFromSystemdFDs(start, count int) (_ []net.Listener, err error) {
+	listeners := make([]net.Listener, 0, count)
+	defer func() {
+		if err != nil {
+			for _, listener := range listeners {
+				_ = listener.Close()
+			}
+		}
+	}()
+
+	for offset := range count {
+		fd := uintptr(start + offset)
+		file := os.NewFile(fd, fmt.Sprintf("LISTEN_FD_%d", fd))
+		if file == nil {
+			return nil, fmt.Errorf("opening systemd listener file descriptor %d", fd)
+		}
+		listener, listenerErr := net.FileListener(file)
+		closeErr := file.Close()
+		if listenerErr != nil {
+			return nil, fmt.Errorf("using systemd file descriptor %d as a stream listener: %w", fd, listenerErr)
+		}
+		if closeErr != nil {
+			_ = listener.Close()
+			return nil, fmt.Errorf("closing inherited systemd file descriptor %d: %w", fd, closeErr)
+		}
+		listeners = append(listeners, listener)
+	}
+
+	return listeners, nil
+}

core/cli/run_socket_activation_other.go

@@ -0,0 +1,9 @@
+//go:build !linux
+
+package cli
+
+import "net"
+
+func systemdActivatedListeners() ([]net.Listener, error) {
+	return nil, nil
+}

core/cli/run_socket_activation_test.go

@@ -0,0 +1,101 @@
+//go:build linux
+
+package cli
+
+import (
+	"net"
+	"os"
+	"strconv"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("selectSystemdListener", func() {
+	It("keeps normal address binding when systemd passes no listener", func() {
+		listener, err := selectSystemdListener(nil)
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listener).To(BeNil())
+	})
+
+	It("uses the single stream listener passed by systemd", func() {
+		inherited, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(inherited.Close)
+
+		listener, err := selectSystemdListener([]net.Listener{inherited})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listener).To(BeIdenticalTo(inherited))
+	})
+
+	It("rejects ambiguous activation with multiple stream listeners", func() {
+		first, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(first.Close)
+		second, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(second.Close)
+
+		listener, err := selectSystemdListener([]net.Listener{first, second})
+
+		Expect(err).To(MatchError(ContainSubstring("exactly one")))
+		Expect(listener).To(BeNil())
+	})
+})
+
+var _ = Describe("systemdActivatedListeners", func() {
+	It("turns an inherited TCP file descriptor into a working listener", func() {
+		original, err := net.Listen("tcp", "127.0.0.1:0")
+		Expect(err).NotTo(HaveOccurred())
+		file, err := original.(*net.TCPListener).File()
+		Expect(err).NotTo(HaveOccurred())
+		Expect(original.Close()).To(Succeed())
+
+		listeners, err := listenersFromSystemdFDs(int(file.Fd()), 1)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(HaveLen(1))
+		DeferCleanup(listeners[0].Close)
+
+		client, err := net.Dial("tcp", listeners[0].Addr().String())
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(client.Close)
+		server, err := listeners[0].Accept()
+		Expect(err).NotTo(HaveOccurred())
+		Expect(server.Close()).To(Succeed())
+	})
+
+	It("ignores descriptors intended for another process and clears the activation environment", func() {
+		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()+1))).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDS", "1")).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDNAMES", "localai-http")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_PID")
+			_ = os.Unsetenv("LISTEN_FDS")
+			_ = os.Unsetenv("LISTEN_FDNAMES")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_PID")).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_FDS")).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_FDNAMES")).To(BeEmpty())
+	})
+
+	It("reports malformed activation metadata instead of silently binding another socket", func() {
+		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()))).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDS", "not-a-number")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_PID")
+			_ = os.Unsetenv("LISTEN_FDS")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).To(MatchError(ContainSubstring("LISTEN_FDS")))
+		Expect(listeners).To(BeNil())
+	})
+})

core/config/backend_capabilities.go

@@ -30,6 +30,7 @@ const (
 	UsecaseFaceRecognition     = "face_recognition"
 	UsecaseSpeakerRecognition  = "speaker_recognition"
 	UsecaseTokenClassify       = "token_classify"
+	UsecaseScore               = "score"
 )
 
 // GRPCMethod identifies a Backend service RPC from backend.proto.
@@ -60,6 +61,7 @@ const (
 	MethodVoiceEmbed         GRPCMethod = "VoiceEmbed"
 	MethodVoiceAnalyze       GRPCMethod = "VoiceAnalyze"
 	MethodTokenClassify      GRPCMethod = "TokenClassify"
+	MethodScore              GRPCMethod = "Score"
 )
 
 // UsecaseInfo describes a single known_usecase value and how it maps
@@ -192,6 +194,11 @@ var UsecaseInfoMap = map[string]UsecaseInfo{
 		GRPCMethod:  MethodTokenClassify,
 		Description: "Per-token classification (NER) via the TokenClassify RPC — the PII detector tier. Declared explicitly via known_usecases; never auto-guessed, since the token-classification head is not useful as general generation or embeddings.",
 	},
+	UsecaseScore: {
+		Flag:        FLAG_SCORE,
+		GRPCMethod:  MethodScore,
+		Description: "Joint log-probability scoring of candidate continuations via the Score RPC. Declared explicitly via known_usecases and usable alongside generation usecases.",
+	},
 }
 
 // BackendCapability describes which gRPC methods and usecases a backend supports.
@@ -241,8 +248,8 @@ func referenceVoiceCloning() *VoiceCloningCapability {
 var BackendCapabilities = map[string]BackendCapability{
 	// --- LLM / text generation backends ---
 	"llama-cpp": {
-		GRPCMethods:      []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString},
-		PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision},
+		GRPCMethods:      []GRPCMethod{MethodPredict, MethodPredictStream, MethodEmbedding, MethodTokenizeString, MethodScore},
+		PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseEdit, UsecaseEmbeddings, UsecaseTokenize, UsecaseVision, UsecaseScore},
 		DefaultUsecases:  []string{UsecaseChat},
 		AcceptsImages:    true, // requires mmproj
 		Description:      "llama.cpp GGUF models — LLM inference with optional vision via mmproj",

core/config/meta/registry.go

@@ -623,6 +623,13 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Component:   "toggle",
 			Order:       89,
 		},
+		"pipeline.turn_detection.vad_window_sec": {
+			Section:     "pipeline",
+			Label:       "VAD Window (s)",
+			Description: "Widen the slice of recent audio the VAD rescans each turn-detection tick. Sized automatically from the commit silence threshold (server_vad silence window, or the semantic eagerness fallback) plus a warm-up margin — set only to widen it; values below the automatic floor are ignored.",
+			Component:   "number",
+			Order:       90,
+		},
 		"pipeline.disable_warmup": {
 			Section:     "pipeline",
 			Label:       "Disable Warmup",
@@ -630,6 +637,99 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Component:   "toggle",
 			Order:       90,
 		},
+		"pipeline.classifier.enabled": {
+			Section:     "pipeline",
+			Label:       "Classifier Mode",
+			Description: "Replace autoregressive generation with prefill-only option selection: each user turn is scored against the option list via the Score primitive and the winning option's canned reply / tool call is emitted. Built for hardware that can afford prompt processing but not decode (e.g. a Raspberry Pi).",
+			Component:   "toggle",
+			Order:       91,
+		},
+		"pipeline.classifier.options": {
+			Section:     "pipeline",
+			Label:       "Classifier Options",
+			Description: "The intents the classifier scores each turn against. Each option has an id (also the scored route label — keep it short), a description of when it applies, an optional canned spoken reply, and an optional canned tool call {name, arguments}. A tool may also declare slots ([{name, type: number|enum|string, values, default, hint}]) whose \"{{name}}\" placeholders in arguments (and, optionally, the reply) are filled by a short grammar-constrained completion when the option wins — the hybrid between prefill-only classification and full generation (requires completion in the scoring model's known_usecases). Clients can replace the list per session via session.update localai_classifier.",
+			Component:   "json-editor",
+			Order:       92,
+		},
+		"pipeline.classifier.threshold": {
+			Section:     "pipeline",
+			Label:       "Classifier Threshold",
+			Description: "Softmax-probability floor the best option must clear; below it the fallback applies. 0 always picks the argmax.",
+			Component:   "slider",
+			Min:         f64(0),
+			Max:         f64(0.99),
+			Step:        f64(0.01),
+			Order:       93,
+		},
+		"pipeline.classifier.fallback.mode": {
+			Section:     "pipeline",
+			Label:       "Classifier Fallback",
+			Description: "What happens when no option clears the threshold: complete with no output, speak the canned fallback reply, or fall through to normal (slow) generation.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "none", Label: "none (empty response)"},
+				{Value: "reply", Label: "canned reply"},
+				{Value: "generate", Label: "generate"},
+			},
+			Order: 94,
+		},
+		"pipeline.classifier.fallback.reply": {
+			Section:     "pipeline",
+			Label:       "Classifier Fallback Reply",
+			Description: "The canned reply spoken when the fallback mode is 'reply' and no option clears the threshold.",
+			Component:   "text",
+			Order:       95,
+		},
+		"pipeline.classifier.normalization": {
+			Section:     "pipeline",
+			Label:       "Classifier Normalization",
+			Description: "How option scores feed the softmax: 'raw' compares joint log-probs (default); 'mean' divides by token count, which is fairer when option ids have very different lengths.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "raw", Label: "raw (joint log-prob)"},
+				{Value: "mean", Label: "mean (per-token)"},
+			},
+			Order: 96,
+		},
+		"pipeline.classifier.history_items": {
+			Section:     "pipeline",
+			Label:       "Classifier History Items",
+			Description: "What gets scored: 0 or -1 (default) score only the latest user message; a positive N includes the trailing N conversation messages, role-labeled. Prior turns echo option names and can dominate small scoring models — only opt in with a larger scorer.",
+			Component:   "number",
+			Order:       97,
+		},
+		"pipeline.classifier.model": {
+			Section:              "pipeline",
+			Label:                "Classifier Scoring Model",
+			Description:          "Optionally score on a different model config. Empty uses the pipeline LLM — scoring runs through the same llama.cpp slot as generation and shares its prompt cache, so a separate model is rarely needed.",
+			Component:            "model-select",
+			AutocompleteProvider: ProviderModels,
+			Order:                98,
+		},
+		"pipeline.classifier.address.names": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Names",
+			Description: "Wake-word gate: only act on turns that mention one of these names as a whole word ('Drone go up', not just 'go up'). Matching is deterministic on the transcript; unaddressed turns skip scoring entirely.",
+			Component:   "string-list",
+			Order:       99,
+		},
+		"pipeline.classifier.address.mode": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Mode",
+			Description: "What to do with unaddressed turns: 'ignore' completes silently (right for ambient conversation), 'reply' speaks the address reply.",
+			Component:   "select",
+			Options: []FieldOption{
+				{Value: "ignore", Label: "ignore (stay silent)"},
+				{Value: "reply", Label: "reply (speak the address reply)"},
+			},
+			Order: 100,
+		},
+		"pipeline.classifier.address.reply": {
+			Section:     "pipeline",
+			Label:       "Classifier Address Reply",
+			Description: "Spoken when an unaddressed turn arrives in 'reply' mode.",
+			Order:       101,
+		},
 
 		// --- Functions ---
 		"function.grammar.parallel_calls": {
@@ -822,6 +922,13 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Min:         f64(0),
 			Order:       213,
 		},
+		"proxy.cache_prompt": {
+			Section:     "proxy",
+			Label:       "Proxy Anthropic Prompt Cache",
+			Description: "Inject Anthropic prompt-cache breakpoints (cache_control: ephemeral) on the stable prefix (system, tools, last message) when mode is translate and provider is anthropic. Serves the repeated prefix at the cache-read rate on multi-turn/agentic calls. No effect otherwise.",
+			Component:   "checkbox",
+			Order:       214,
+		},
 
 		// --- MITM intercept hosts ---
 		// Each host listed here is claimed by this model config; the

core/config/model_config.go

@@ -232,6 +232,15 @@ type ProxyConfig struct {
 	// means no per-request timeout (only the request context, which
 	// is bound to the client connection, applies).
 	RequestTimeoutSeconds int `yaml:"request_timeout_seconds,omitempty" json:"request_timeout_seconds,omitempty"`
+
+	// CachePrompt enables automatic Anthropic prompt-cache breakpoints
+	// (cache_control: ephemeral) on the stable prefix — system prompt,
+	// tools, and the last message block — when mode=translate and
+	// provider=anthropic. Anthropic then serves the repeated prefix at
+	// the cache-read rate (0.1x input), which sharply cuts cost on
+	// agentic/multi-turn workloads that re-send a large stable prefix.
+	// No effect for passthrough mode or non-Anthropic providers.
+	CachePrompt bool `yaml:"cache_prompt,omitempty" json:"cache_prompt,omitempty"`
 }
 
 // Proxy mode names. Validate() normalises an empty Mode to
@@ -669,6 +678,16 @@ type Pipeline struct {
 	// per session; retranscribe is server-side only. Unset keeps server_vad.
 	TurnDetection PipelineTurnDetection `yaml:"turn_detection,omitempty" json:"turn_detection,omitempty"`
 
+	// Classifier switches realtime responses to prefill-only option
+	// selection (LocalAI classifier mode): each user turn is scored
+	// against a fixed option list via the Score primitive and the winning
+	// option's canned reply / tool call is emitted, so weak hardware
+	// never pays for autoregressive decode. Nil means disabled; clients
+	// can still enable per session via session.update localai_classifier.
+	// Validated (and rejected loudly) at realtime session setup, like the
+	// pipeline model slots.
+	Classifier *PipelineClassifier `yaml:"classifier,omitempty" json:"classifier,omitempty"`
+
 	// DisableWarmup turns off eager pre-loading of the pipeline's sub-models at
 	// realtime session start. By default (false) LocalAI loads every configured
 	// sub-model backend (VAD, transcription, LLM, TTS, sound detection, voice
@@ -682,6 +701,65 @@ type Pipeline struct {
 	DisableWarmup bool `yaml:"disable_warmup,omitempty" json:"disable_warmup,omitempty"`
 }
 
+// PipelineClassifier is the YAML mirror of the realtime API's
+// localai_classifier extension (see
+// core/http/endpoints/openai/types/classifier.go, which documents the
+// field semantics and owns validation — the realtime session converts and
+// validates this block at setup).
+type PipelineClassifier struct {
+	Enabled bool `yaml:"enabled,omitempty" json:"enabled,omitempty"`
+	// Model optionally names a different config to score on. Empty uses
+	// the pipeline's llm — with slot-based Score the same process serves
+	// both scoring and generation and shares its prompt cache.
+	Model         string                      `yaml:"model,omitempty" json:"model,omitempty"`
+	Threshold     float64                     `yaml:"threshold,omitempty" json:"threshold,omitempty"`
+	Normalization string                      `yaml:"normalization,omitempty" json:"normalization,omitempty"`
+	HistoryItems  int                         `yaml:"history_items,omitempty" json:"history_items,omitempty"`
+	Fallback      *PipelineClassifierFallback `yaml:"fallback,omitempty" json:"fallback,omitempty"`
+	Options       []PipelineClassifierOption  `yaml:"options,omitempty" json:"options,omitempty"`
+	// Address gates every turn on the assistant being addressed by one of
+	// these names (wake-word behavior); see types.ClassifierAddress.
+	Address *PipelineClassifierAddress `yaml:"address,omitempty" json:"address,omitempty"`
+}
+
+// PipelineClassifierAddress mirrors types.ClassifierAddress for YAML.
+type PipelineClassifierAddress struct {
+	Names []string `yaml:"names,omitempty" json:"names,omitempty"`
+	Mode  string   `yaml:"mode,omitempty" json:"mode,omitempty"`
+	Reply string   `yaml:"reply,omitempty" json:"reply,omitempty"`
+}
+
+type PipelineClassifierOption struct {
+	ID          string                  `yaml:"id" json:"id"`
+	Description string                  `yaml:"description" json:"description"`
+	Reply       string                  `yaml:"reply,omitempty" json:"reply,omitempty"`
+	Tool        *PipelineClassifierTool `yaml:"tool,omitempty" json:"tool,omitempty"`
+}
+
+type PipelineClassifierTool struct {
+	Name string `yaml:"name" json:"name"`
+	// Arguments is a plain YAML map; the realtime session marshals it to
+	// the JSON arguments string of the emitted function call. With Slots
+	// it is a template: "{{name}}" values are filled by a constrained
+	// completion when the option wins.
+	Arguments map[string]any `yaml:"arguments,omitempty" json:"arguments,omitempty"`
+	// Slots declares the inferred arguments; see types.ClassifierSlot.
+	Slots []PipelineClassifierSlot `yaml:"slots,omitempty" json:"slots,omitempty"`
+}
+
+type PipelineClassifierSlot struct {
+	Name    string   `yaml:"name" json:"name"`
+	Type    string   `yaml:"type" json:"type"` // number | enum | string
+	Values  []string `yaml:"values,omitempty" json:"values,omitempty"`
+	Default string   `yaml:"default,omitempty" json:"default,omitempty"`
+	Hint    string   `yaml:"hint,omitempty" json:"hint,omitempty"`
+}
+
+type PipelineClassifierFallback struct {
+	Mode  string `yaml:"mode,omitempty" json:"mode,omitempty"`
+	Reply string `yaml:"reply,omitempty" json:"reply,omitempty"`
+}
+
 // PipelineCompaction configures summarize-then-drop for a realtime pipeline.
 type PipelineCompaction struct {
 	// Enabled turns summarize-then-drop on. Default false.
@@ -982,6 +1060,12 @@ type PipelineTurnDetection struct {
 	// are compared in the logs — a diagnostic for streaming/batch alignment
 	// at the cost of one extra decode per turn.
 	Retranscribe *bool `yaml:"retranscribe,omitempty" json:"retranscribe,omitempty"`
+	// VadWindowSec widens the slice of recent audio the VAD rescans each
+	// tick. The pipeline sizes it automatically from the commit silence
+	// threshold (server_vad silence window, or the semantic eagerness
+	// fallback) plus a warm-up margin; set this only to widen it further —
+	// values below the automatic floor are ignored.
+	VadWindowSec float64 `yaml:"vad_window_sec,omitempty" json:"vad_window_sec,omitempty"`
 }
 
 // TurnDetectionSemantic reports whether this pipeline defaults sessions to
@@ -1435,20 +1519,9 @@ func (c *ModelConfig) Validate() (bool, error) {
 			ProxyProviderOpenAI, ProxyProviderAnthropic)
 	}
 
-	// Score on llama-cpp bypasses the slot loop and races the
-	// llama_context against concurrent generation/embedding traffic
-	// (see backend/cpp/llama-cpp/grpc-server.cpp on Score). Reject the
-	// combination here so operators are forced to split the model.
-	// (token_classify is unaffected — it runs on the standalone
-	// privacy-filter backend, not llama-cpp.)
-	const scoreConflicts = FLAG_CHAT | FLAG_COMPLETION | FLAG_EMBEDDINGS
-	if (c.Backend == "llama-cpp" || c.Backend == "llama") &&
-		c.HasUsecases(FLAG_SCORE) && c.KnownUsecases != nil &&
-		*c.KnownUsecases&scoreConflicts != 0 {
-		return false, fmt.Errorf(
-			"known_usecases conflict on llama-cpp: score is incompatible " +
-				"with chat/completion/embeddings — split into separate model configs")
-	}
+	// Score on llama-cpp runs through the slot loop (SERVER_TASK_TYPE_SCORE,
+	// see backend/cpp/llama-cpp/patches/), so it is safe to combine with
+	// chat/completion/embeddings on one config — no conflict check needed.
 
 	// Pattern detector: validate built-in names and that each operator-defined
 	// pattern is a well-formed, anchored, bounded restricted-regex. Reject at
@@ -1576,9 +1649,10 @@ const (
 	// Marks a model as wired for the Score gRPC primitive (joint
 	// log-prob of candidate continuations under a shared prompt). Must
 	// be declared explicitly via `known_usecases: [score]` — there's
-	// no heuristic for it. On llama-cpp, Score bypasses the slot loop
-	// (direct llama_decode), so combining score with
-	// chat/completion/embeddings in one config is rejected at validation.
+	// no heuristic for it. On llama-cpp, Score runs through the slot
+	// loop (SERVER_TASK_TYPE_SCORE), so it may combine freely with
+	// chat/completion/embeddings on one config and shares the slot's
+	// prompt cache with generation.
 	FLAG_SCORE ModelConfigUsecase = 0b10000000000000000000
 
 	// Marks a model as wired for the Depth gRPC primitive (per-pixel
@@ -1691,9 +1765,9 @@ func GetUsecasesFromYAML(input []string) *ModelConfigUsecase {
 // either, they reserved the model for an internal direct-decode primitive
 // (the router classifier, or the PII NER tier). Letting GuessUsecases
 // paint chat/completion/embeddings on top would surface it in pickers it
-// was deliberately kept out of, and (on llama-cpp) reintroduce the slot
-// contention the conflict check exists to prevent. So a declared score or
-// token_classify list is authoritative.
+// was deliberately kept out of. So a declared score or token_classify
+// list is authoritative; declare the generation usecases explicitly
+// alongside score to serve both from one config.
 func (c *ModelConfig) HasUsecases(u ModelConfigUsecase) bool {
 	if c.KnownUsecases != nil {
 		if (u & *c.KnownUsecases) == u {
@@ -1883,8 +1957,8 @@ func (c *ModelConfig) GuessUsecases(u ModelConfigUsecase) bool {
 
 	if (u & FLAG_SCORE) == FLAG_SCORE {
 		// No heuristic: Score-intent is a deliberate operator choice
-		// (it reserves the model from generation traffic on llama-cpp),
-		// so HasUsecases(FLAG_SCORE) is true only when KnownUsecases
+		// (it keeps the model out of pickers it wasn't meant for), so
+		// HasUsecases(FLAG_SCORE) is true only when KnownUsecases
 		// declares it explicitly.
 		return false
 	}

core/config/model_config_loader.go

@@ -201,8 +201,8 @@ func (bcl *ModelConfigLoader) LoadModelConfigFileByNameDefaultOptions(modelName
 // survives unresolved into model loading and fails downstream — notably in
 // distributed mode with "backend name is empty". Mirrors the top-level alias
 // resolution in core/http/middleware/request.go.
-func (bcl *ModelConfigLoader) LoadResolvedModelConfig(modelName, modelPath string) (*ModelConfig, error) {
-	cfg, err := bcl.LoadModelConfigFileByName(modelName, modelPath)
+func (bcl *ModelConfigLoader) LoadResolvedModelConfig(modelName, modelPath string, opts ...ConfigLoaderOption) (*ModelConfig, error) {
+	cfg, err := bcl.LoadModelConfigFileByName(modelName, modelPath, opts...)
 	if err != nil {
 		return nil, err
 	}

core/config/model_config_loader_resolve_test.go

@@ -49,4 +49,21 @@ alias: real-llm
 		Expect(direct.Backend).To(Equal("llama-cpp"))
 		Expect(direct.Name).To(Equal("real-llm"))
 	})
+
+	It("applies loader defaults while preserving explicit model threads", func() {
+		tmpDir := GinkgoT().TempDir()
+		Expect(os.WriteFile(filepath.Join(tmpDir, "defaulted.yaml"), []byte("name: defaulted\nbackend: llama-cpp\n"), 0644)).To(Succeed())
+		Expect(os.WriteFile(filepath.Join(tmpDir, "explicit.yaml"), []byte("name: explicit\nbackend: llama-cpp\nthreads: 3\n"), 0644)).To(Succeed())
+
+		cl := config.NewModelConfigLoader(tmpDir)
+		defaulted, err := cl.LoadResolvedModelConfig("defaulted", tmpDir, config.LoadOptionThreads(11))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(defaulted.Threads).NotTo(BeNil())
+		Expect(*defaulted.Threads).To(Equal(11))
+
+		explicit, err := cl.LoadResolvedModelConfig("explicit", tmpDir, config.LoadOptionThreads(11))
+		Expect(err).NotTo(HaveOccurred())
+		Expect(explicit.Threads).NotTo(BeNil())
+		Expect(*explicit.Threads).To(Equal(3))
+	})
 })

core/config/model_config_test.go

@@ -127,21 +127,19 @@ parameters:
 			Expect(err).To(BeNil())
 			Expect(valid).To(BeTrue())
 
-			// llama-cpp configs can't mix the score usecase with
-			// chat/completion/embeddings — Score bypasses the slot loop
-			// and would race the llama_context. (token_classify is exempt:
-			// it runs on the privacy-filter backend, not llama-cpp, so the
-			// token_classify combinations below stay valid.)
+			// Score runs through the llama-cpp slot loop, so mixing the
+			// score usecase with chat/completion/embeddings on one config
+			// is valid — the slot scheduler serializes score against
+			// generation and shares the prompt cache between them.
 			scoreFlag := FLAG_SCORE | FLAG_CHAT
-			conflicting := ModelConfig{
-				Name:          "router-but-also-chat",
+			scoringChat := ModelConfig{
+				Name:          "router-and-chat",
 				Backend:       "llama-cpp",
 				KnownUsecases: &scoreFlag,
 			}
-			valid, err = conflicting.Validate()
-			Expect(valid).To(BeFalse())
-			Expect(err).To(HaveOccurred())
-			Expect(err.Error()).To(ContainSubstring("score is incompatible"))
+			valid, err = scoringChat.Validate()
+			Expect(valid).To(BeTrue())
+			Expect(err).NotTo(HaveOccurred())
 
 			scoreOnly := FLAG_SCORE
 			dedicated := ModelConfig{

core/gallery/backend_index_capabilities_test.go

@@ -1,83 +0,0 @@
-package gallery_test
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"sync"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-	"gopkg.in/yaml.v3"
-
-	"github.com/mudler/LocalAI/core/gallery"
-	"github.com/mudler/LocalAI/pkg/system"
-)
-
-// loadBackendIndex parses backend/index.yaml once for the whole suite.
-var loadBackendIndex = sync.OnceValues(func() (gallery.GalleryElements[*gallery.GalleryBackend], error) {
-	data, err := os.ReadFile(filepath.Join("..", "..", "backend", "index.yaml"))
-	if err != nil {
-		return nil, err
-	}
-	var entries gallery.GalleryElements[*gallery.GalleryBackend]
-	if err := yaml.Unmarshal(data, &entries); err != nil {
-		return nil, err
-	}
-	return entries, nil
-})
-
-var _ = Describe("backend/index.yaml capability maps", func() {
-	var entries gallery.GalleryElements[*gallery.GalleryBackend]
-
-	BeforeEach(func() {
-		var err error
-		entries, err = loadBackendIndex()
-		Expect(err).ToNot(HaveOccurred())
-		Expect(entries).ToNot(BeEmpty())
-	})
-
-	// A capability pointing at a name that does not exist is invisible until a
-	// host with exactly that capability tries to install: FindBestBackendFromMeta
-	// returns nil and the install fails with "no backend found".
-	It("resolves every capability reference to an entry in the index", func() {
-		names := map[string]struct{}{}
-		for _, e := range entries {
-			names[e.Name] = struct{}{}
-		}
-
-		dangling := []string{}
-		for _, e := range entries {
-			for capability, target := range e.CapabilitiesMap {
-				if _, ok := names[target]; !ok {
-					dangling = append(dangling, fmt.Sprintf("  %s -> %s: %q", e.Name, capability, target))
-				}
-			}
-		}
-		Expect(dangling).To(BeEmpty(), "capabilities naming a missing entry:\n%s", strings.Join(dangling, "\n"))
-	})
-
-	// vllm.cpp's CUDA kernels need the CUDA 13 toolchain (12.x nvcc cannot
-	// compile the Blackwell fp4 paths), so CUDA 12 hosts have no GPU build to
-	// install and must land on the CPU one. Assert the fallback is explicit
-	// rather than an accident of the "default" catch-all, so mapping these
-	// capabilities at a CUDA image later is a test failure and not a host that
-	// pulls kernels it cannot run.
-	DescribeTable("routes vllm-cpp hosts to the build their toolchain supports",
-		func(metaName, capability, expected string) {
-			meta := entries.FindByName(metaName)
-			Expect(meta).ToNot(BeNil())
-
-			resolved := meta.FindBestBackendFromMeta(system.NewCapabilityState(capability), entries)
-			Expect(resolved).ToNot(BeNil())
-			Expect(resolved.Name).To(Equal(expected))
-		},
-		Entry("CUDA 12 x86_64 gets the CPU build", "vllm-cpp", "nvidia-cuda-12", "cpu-vllm-cpp"),
-		Entry("CUDA 12 Jetson (AGX Orin) gets the CPU build", "vllm-cpp", "nvidia-l4t-cuda-12", "cpu-vllm-cpp"),
-		Entry("CUDA 13 Jetson (DGX Spark) gets the L4T build", "vllm-cpp", "nvidia-l4t-cuda-13", "nvidia-l4t-arm64-vllm-cpp"),
-		Entry("CUDA 13 x86_64 gets the CUDA build", "vllm-cpp", "nvidia-cuda-13", "cuda13-vllm-cpp"),
-		Entry("development CUDA 12 Jetson gets the CPU build", "vllm-cpp-development", "nvidia-l4t-cuda-12", "cpu-vllm-cpp-development"),
-		Entry("development CUDA 13 Jetson gets the L4T build", "vllm-cpp-development", "nvidia-l4t-cuda-13", "nvidia-l4t-arm64-vllm-cpp-development"),
-	)
-})

core/gallery/backends_test.go

@@ -32,6 +32,42 @@ var _ = Describe("Runtime capability-based backend selection", func() {
 		os.RemoveAll(tempDir)
 	})
 
+	It("keeps the Kokoro CPU fallback installable from the backend gallery", func() {
+		backends, err := ReadConfigFile[[]*GalleryBackend](filepath.Join("..", "..", "backend", "index.yaml"))
+		Expect(err).NotTo(HaveOccurred())
+
+		byName := make(map[string]*GalleryBackend, len(*backends))
+		for _, backend := range *backends {
+			byName[backend.Name] = backend
+		}
+
+		Expect(byName).To(HaveKey("kokoro"))
+		Expect(byName["kokoro"].CapabilitiesMap).To(HaveKeyWithValue("default", "cpu-kokoro"))
+		Expect(byName).To(HaveKey("cpu-kokoro"))
+		Expect(byName["cpu-kokoro"].URI).To(Equal("quay.io/go-skynet/local-ai-backends:latest-cpu-kokoro"))
+
+		type matrixEntry struct {
+			Backend     string `yaml:"backend"`
+			Platforms   string `yaml:"platforms"`
+			PlatformTag string `yaml:"platform-tag"`
+			TagSuffix   string `yaml:"tag-suffix"`
+		}
+		type backendMatrix struct {
+			Include []matrixEntry `yaml:"include"`
+		}
+
+		matrix, err := ReadConfigFile[backendMatrix](filepath.Join("..", "..", ".github", "backend-matrix.yml"))
+		Expect(err).NotTo(HaveOccurred())
+
+		var cpuArchitectures []string
+		for _, entry := range matrix.Include {
+			if entry.Backend == "kokoro" && entry.TagSuffix == "-cpu-kokoro" {
+				cpuArchitectures = append(cpuArchitectures, entry.Platforms+"/"+entry.PlatformTag)
+			}
+		}
+		Expect(cpuArchitectures).To(ConsistOf("linux/amd64/amd64", "linux/arm64/arm64"))
+	})
+
 	It("ListSystemBackends prefers optimal alias candidate", func() {
 		// Arrange two installed backends sharing the same alias
 		must := func(err error) { Expect(err).NotTo(HaveOccurred()) }

core/http/endpoints/localai/router_decide_test.go

@@ -136,7 +136,7 @@ type stubScorer struct {
 	labelToLogProb map[string]float64
 }
 
-func (s *stubScorer) Score(_ context.Context, _ string, candidates []string) ([]backend.CandidateScore, error) {
+func (s *stubScorer) Score(_ context.Context, _ string, _ int, candidates []string) ([]backend.CandidateScore, error) {
 	out := make([]backend.CandidateScore, len(candidates))
 	for i, c := range candidates {
 		// Candidate is the Arch-Router JSON envelope

core/http/endpoints/openai/realtime.go

@@ -30,6 +30,7 @@ import (
 	"github.com/mudler/LocalAI/core/http/endpoints/openai/turncoord"
 	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
 	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/router"
 	"github.com/mudler/LocalAI/core/templates"
 	laudio "github.com/mudler/LocalAI/pkg/audio"
 	"github.com/mudler/LocalAI/pkg/functions"
@@ -150,6 +151,12 @@ type Session struct {
 	// pairs are kept together so we never feed an orphaned tool result.
 	MaxHistoryItems int
 
+	// Classifier holds the LocalAI classifier-mode config (prefill-scored
+	// option selection instead of generation), seeded from
+	// pipeline.classifier and replaced wholesale by session.update's
+	// localai_classifier field. nil means off.
+	Classifier *types.ClassifierConfig
+
 	// Compaction settings resolved from pipeline.compaction (see resolveCompaction).
 	CompactionEnabled bool
 	CompactionTrigger int
@@ -210,14 +217,15 @@ func (s *Session) ToServer() types.SessionUnion {
 	} else {
 		return types.SessionUnion{
 			Realtime: &types.RealtimeSession{
-				ID:               s.ID,
-				Object:           "realtime.session",
-				Model:            s.Model,
-				Instructions:     s.Instructions,
-				Tools:            s.Tools,
-				ToolChoice:       s.ToolChoice,
-				MaxOutputTokens:  s.MaxOutputTokens,
-				OutputModalities: s.OutputModalities,
+				ID:                s.ID,
+				Object:            "realtime.session",
+				Model:             s.Model,
+				Instructions:      s.Instructions,
+				Tools:             s.Tools,
+				ToolChoice:        s.ToolChoice,
+				MaxOutputTokens:   s.MaxOutputTokens,
+				OutputModalities:  s.OutputModalities,
+				LocalAIClassifier: s.Classifier,
 				Audio: &types.RealtimeSessionAudio{
 					Input: &types.SessionAudioInput{
 						TurnDetection: s.TurnDetection,
@@ -279,6 +287,24 @@ type Model interface {
 	// event. Backends without live support fail with an error satisfying
 	// grpcerrors.IsLiveTranscriptionUnsupported.
 	TranscribeLive(ctx context.Context, language string, onEvent func(backend.LiveTranscriptionEvent)) (backend.LiveTranscriptionSession, error)
+	// ClassifyTurn prefill-scores each classifier option as a candidate
+	// continuation of the conversation (LocalAI classifier-mode extension)
+	// and returns the softmax distribution in option order. Runs on the
+	// pipeline's scoring model (classifier.model, defaulting to the LLM) —
+	// no autoregressive decode happens.
+	ClassifyTurn(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string) ([]router.LabelScore, error)
+	// PrewarmClassifier primes the scoring backend's prompt cache for a
+	// newly registered option list (fired async on registration) so the
+	// first turns after a session.update don't pay the option-list
+	// prefill. Best-effort and idempotent per option set.
+	PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string)
+	// FillToolArguments completes the chosen option's argument slots with a
+	// short grammar-constrained completion that continues the exact scoring
+	// prompt (so the backend's prompt cache stays warm) and returns the
+	// spliced tool-arguments JSON plus the raw slot values (for reply
+	// templating) — the hybrid between prefill-only classification and full
+	// generation.
+	FillToolArguments(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string, chosen *types.ClassifierOption) (string, map[string]string, error)
 	PredictConfig() *config.ModelConfig
 	// Warmup eagerly loads the pipeline's sub-model backends into memory so the
 	// first realtime turn doesn't pay each backend's cold-start load cost. Loads
@@ -553,6 +579,12 @@ func runRealtimeSession(application *application.Application, t Transport, model
 		SoundDetectionHopMs:     cfg.Pipeline.SoundDetectionHopMs,
 	}
 	session.CompactionEnabled, session.CompactionTrigger, session.MaxSummaryTokens, session.SummaryModel = resolveCompaction(cfg, session.MaxHistoryItems)
+	classifier, err := classifierConfigFromPipeline(cfg.Pipeline.Classifier)
+	if err != nil {
+		sendError(t, "invalid_pipeline", "pipeline classifier: "+err.Error(), "", "")
+		return
+	}
+	session.Classifier = classifier
 
 	// Single-writer response coordinator (machine M3). All response starts and
 	// cancels go through this, so the read-loop and VAD goroutine can never race
@@ -602,6 +634,10 @@ func runRealtimeSession(application *application.Application, t Transport, model
 		return
 	}
 	session.ModelInterface = m
+	// A pipeline-seeded option list gets its scoring prompt prewarmed
+	// alongside the model warm-up below, so the session's first turn
+	// doesn't pay the option-list prefill.
+	prewarmClassifier(session)
 
 	// The voice gate is built before the warm-up below so its
 	// speaker-recognition model can warm alongside the pipeline stages.
@@ -764,7 +800,9 @@ func runRealtimeSession(application *application.Application, t Transport, model
 					application.ApplicationConfig(),
 				); err != nil {
 					xlog.Error("failed to update session", "error", err)
-					sendError(t, "session_update_error", "Failed to update session", "", "")
+					// The cause is validation feedback on the client's own
+					// payload — echo it so UIs can show something actionable.
+					sendError(t, "session_update_error", fmt.Sprintf("Failed to update session: %v", err), "", "")
 					continue
 				}
 
@@ -790,7 +828,7 @@ func runRealtimeSession(application *application.Application, t Transport, model
 					buildRealtimeRoutingContext(application, session.ID),
 				); err != nil {
 					xlog.Error("failed to update session", "error", err)
-					sendError(t, "session_update_error", "Failed to update session", "", "")
+					sendError(t, "session_update_error", fmt.Sprintf("Failed to update session: %v", err), "", "")
 					continue
 				}
 
@@ -966,6 +1004,10 @@ func runRealtimeSession(application *application.Application, t Transport, model
 
 		case types.ResponseCreateEvent:
 			xlog.Debug("recv", "message", string(msg))
+			if err := validateClassifierActivation(session.ModelInterface, e.Response.LocalAIClassifier); err != nil {
+				sendError(t, "invalid_request_error", "Invalid response classifier: "+err.Error(), "", e.EventID)
+				continue
+			}
 
 			// Handle optional items to add to context
 			if len(e.Response.Input) > 0 {
@@ -1241,6 +1283,17 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 		session.ToolChoice = rt.ToolChoice
 	}
 
+	if rt.LocalAIClassifier != nil {
+		// Replace-not-merge, like tools: the client owns the whole option
+		// list. Invalid configs reject the update without touching the
+		// session's current classifier.
+		if err := validateClassifierActivation(session.ModelInterface, rt.LocalAIClassifier); err != nil {
+			return err
+		}
+		session.Classifier = rt.LocalAIClassifier
+		prewarmClassifier(session)
+	}
+
 	if rt.MaxOutputTokens != 0 {
 		session.MaxOutputTokens = rt.MaxOutputTokens
 	}
@@ -1314,6 +1367,23 @@ func decodeOpusLoop(session *Session, opusBackend grpc.Backend, done chan struct
 // it cuts the start of the utterance the next tick will detect.
 const noSpeechHoldbackSec = 0.5
 
+// vadWarmupMarginSec pads the VAD scan window beyond the largest silence the
+// commit test can need to measure. It covers silero's cold-start (the LSTM
+// state converges within a few hundred ms — the model has no longer-range
+// memory, which is why clipping is sound at all) plus sherpa's segment
+// hysteresis (min_speech 0.25s / min_silence 0.5s), which must fit inside
+// the clip for segments to open and close at all.
+const vadWarmupMarginSec = 1.0
+
+// maxTurnBufferSec bounds the raw input buffer. Without it a turn that never
+// pauses (continuous noise or speech: silero segments every tick, so the
+// no-speech clear never runs and nothing commits) grows the buffer toward the
+// 100MB append cap, and with it the per-tick copy+resample and the
+// commit-time WAV/batch decode. 90s keeps all of those trivial; only an
+// unbroken >90s turn loses head audio from a server_vad batch transcription
+// (semantic mode already consumed it incrementally via the live stream).
+const maxTurnBufferSec = 90.0
+
 // dropInspectedPrefix removes the head of the audio buffer that a VAD tick
 // inspected (the first inspected bytes), keeping the newest holdbackBytes of
 // that window plus everything appended while the tick ran — audio the VAD
@@ -1375,185 +1445,292 @@ func handleVAD(session *Session, conv *Conversation, t Transport, done chan stru
 		case <-done:
 			return
 		case <-ticker.C:
-			// Semantic mode is re-read each tick: session.update can switch
-			// turn-detection modes (and the retranscribe gate) mid-session.
-			sessionLock.Lock()
-			var sv *types.RealtimeSessionSemanticVad
-			if session.TurnDetection != nil {
-				sv = session.TurnDetection.SemanticVad
-			}
-			retranscribe := sv != nil && session.ModelConfig != nil &&
-				session.ModelConfig.Pipeline.TurnDetectionRetranscribe()
-			sessionLock.Unlock()
-
-			// The turn coordinator's data-heavy effects (OpenTurn/CommitTurn)
-			// need this tick's mode; set it before any Apply below.
-			sink.sv = sv
-
-			// session.update switched semantic -> server mid-turn: drop the
-			// orphaned live stream. This is NOT a turn abort — the turn continues
-			// under server_vad (a config change must not cut off a mid-utterance
-			// speaker), so the coordinator stays Speaking; only the orphaned live
-			// stream is closed.
-			if sv == nil && lts.open() {
-				lts.discardTurn()
-			}
-
-			session.AudioBufferLock.Lock()
-			allAudio := make([]byte, len(session.InputAudioBuffer))
-			copy(allAudio, session.InputAudioBuffer)
-			session.AudioBufferLock.Unlock()
-
-			aints := sound.BytesToInt16sLE(allAudio)
-			if len(aints) == 0 || len(aints) < int(silenceThreshold*float64(session.InputSampleRate)) {
-				continue
-			}
-
-			// Resample from InputSampleRate to 16kHz
-			aints = sound.ResampleInt16(aints, session.InputSampleRate, localSampleRate)
-
-			audioLength := float64(len(aints)) / localSampleRate
-
-			if sv != nil && lts.open() {
-				lts.feedNewAudio(aints)
-				lts.drainEvents(audioLength)
-			}
-
-			segments, err := runVAD(vadContext, session, aints)
-			if err != nil {
-				if err.Error() == "unexpected speech end" {
-					xlog.Debug("VAD cancelled")
-					continue
-				}
-				xlog.Error("failed to process audio", "error", err)
-				sendError(t, "processing_error", "Failed to process audio: "+err.Error(), "", "")
-				continue
-			}
-
-			// NOTE: the no-speech clear and the min-buffer gate above stay on
-			// the short silenceThreshold even in semantic mode — the eagerness
-			// fallback applies only to the end-of-speech commit decision, or a
-			// low eagerness would delay speech_started/barge-in by seconds.
-			if len(segments) == 0 && audioLength > silenceThreshold {
-				// "No segments" is not "no speech": silero (threshold 0.5)
-				// crosses up to a few hundred ms into a soft word onset, so
-				// the newest audio in the inspected window may be the start
-				// of a word the next tick will recognize — and more audio
-				// arrived while this tick ran. Keep both; drop only the
-				// older, confirmed-silent head, or utterance onsets get cut.
-				holdback := int(noSpeechHoldbackSec*float64(session.InputSampleRate)) * 2
-				session.AudioBufferLock.Lock()
-				session.InputAudioBuffer = dropInspectedPrefix(session.InputAudioBuffer, len(allAudio), holdback)
-				session.AudioBufferLock.Unlock()
-
-				// No-speech clear: end any open turn (Speaking -> Idle, discarding
-				// the partial). Returning to Idle is the fix for failure mode 4 —
-				// the legacy discardTurn left speechStarted true, suppressing the
-				// next onset. Idle while not speaking is a no-op.
-				if err := sink.coord.Apply(turncoord.Abort{Reason: turncoord.AbortNoSpeech}); err != nil {
-					xlog.Error("turncoord: abort(no_speech) failed", "error", err)
-				}
-				continue
-			} else if len(segments) == 0 {
-				continue
-			}
-
-			// Speech detected this tick: open the turn (Idle -> Speaking) through
-			// the coordinator. On that transition it opens the turn's live ASR
-			// stream + feeds the buffered prefix (OpenTurn), cancels any in-flight
-			// response (BargeIn, non-blocking — the VAD tick is never stalled), and
-			// emits speech_started. While already Speaking it is a no-op, so "turn
-			// open" and "speech started" can never disagree. The turn id is minted
-			// here and carried by the coordinator through to the committed event.
-			sink.onsetAudio = aints
-			if err := sink.coord.Apply(turncoord.Onset{Turn: turncoord.TurnID(generateItemID())}); err != nil {
-				xlog.Error("turncoord: onset failed", "error", err)
-			}
-
-			if sv != nil {
-				// Drain again: events produced by THIS tick's feed have
-				// usually arrived by the time runVAD returns, and leaving
-				// them for the next tick adds 300ms to every EOU-triggered
-				// commit.
-				lts.drainEvents(audioLength)
-			}
-
-			// Segment still in progress when audio ended
-			segEndTime := segments[len(segments)-1].End
-			if segEndTime == 0 {
-				continue
-			}
-
-			threshold := silenceThreshold
-			eouPending := false
-			if sv != nil {
-				eouPending = lts.eouPending(segments)
-				threshold = lts.thresholdSec(eouPending, sv)
-			}
-
-			if float32(audioLength)-segEndTime > float32(threshold) {
-				if sv != nil {
-					trigger, eouLag := lts.commitTrigger(eouPending, float64(segEndTime))
-					xlog.Info("semantic_vad: committing turn",
-						"trigger", trigger,
-						"speech_end_s", segEndTime,
-						"eou_lag_s", eouLag,
-						"silence_s", audioLength-float64(segEndTime),
-						"audio_s", audioLength)
-				}
-				// Retranscribe gate (semantic mode, EOU-triggered commits
-				// only): cross-check the streamed EOU with an offline decode
-				// of the buffered turn before committing. Runs synchronously
-				// on the tick — the engine would serialize a concurrent feed
-				// against it anyway. Timeout-triggered commits skip the gate.
-				var gated *schema.TranscriptionResult
-				if retranscribe && eouPending {
-					batch, gerr := transcribeUtterance(vadContext, sound.Int16toBytesLE(aints), session)
-					switch {
-					case gerr != nil:
-						xlog.Warn("semantic_vad: retranscribe gate failed; committing via the file path", "error", gerr)
-					case !batch.Eou:
-						xlog.Info("semantic_vad: batch decode did not confirm the streamed EOU; continuing to listen",
-							"streamed", lts.previewText(), "batch", batch.Text)
-						// The batch decode rejected the streamed EOU as a false
-						// positive: consume the recorded EOU so the next tick
-						// falls back to the eagerness window instead of
-						// re-triggering on the same token.
-						lts.eouAtSec = 0
-						continue
-					default:
-						xlog.Info("semantic_vad: batch decode confirmed the streamed EOU",
-							"streamed", lts.previewText(), "batch", batch.Text)
-						gated = batch
-					}
-				}
-
-				xlog.Debug("Detected end of speech segment")
-				session.AudioBufferLock.Lock()
-				// Keep audio appended while this tick ran — it belongs to
-				// the next turn (in any mode: nil-ing it dropped the onset
-				// of an utterance started right after a commit).
-				session.InputAudioBuffer = dropInspectedPrefix(session.InputAudioBuffer, len(allAudio), 0)
-				session.AudioBufferLock.Unlock()
-
-				// Commit the turn through the coordinator: it emits speech_stopped
-				// (EmitSpeechStopped) then the committed event, finalizes the live
-				// stream, and issues the response (CommitTurn). The committed item
-				// id is the coordinator's turn id (== the id the live captions
-				// streamed under), so the client replaces the partial text.
-				sink.commitAudio = sound.Int16toBytesLE(aints)
-				sink.commitAudioLength = audioLength
-				sink.commitRetranscribe = retranscribe
-				sink.commitGated = gated
-				// TODO: Remove prefix silence that is over TurnDetectionParams.PrefixPaddingMs
-				if err := sink.coord.Apply(turncoord.Silence{}); err != nil {
-					xlog.Error("turncoord: commit failed", "error", err)
-				}
-			}
+			vadTick(sink, silenceThreshold)
 		}
 	}
 }
 
+// vadTick runs one turn-detection inspection of the session's input buffer:
+// snapshot, resample, silero scan, live-ASR drain, and the coordinator
+// transitions that follow. Extracted from handleVAD so specs can drive turn
+// detection synchronously without the ticker (same shape as
+// classifySoundWindow).
+func vadTick(sink *turnSink, silenceThreshold float64) {
+	session := sink.session
+	t := sink.transport
+	lts := sink.lts
+	vadContext := sink.vadContext
+
+	// Semantic mode is re-read each tick: session.update can switch
+	// turn-detection modes (and the retranscribe gate) mid-session.
+	sessionLock.Lock()
+	var sv *types.RealtimeSessionSemanticVad
+	if session.TurnDetection != nil {
+		sv = session.TurnDetection.SemanticVad
+	}
+	retranscribe := sv != nil && session.ModelConfig != nil &&
+		session.ModelConfig.Pipeline.TurnDetectionRetranscribe()
+	sessionLock.Unlock()
+
+	// The turn coordinator's data-heavy effects (OpenTurn/CommitTurn)
+	// need this tick's mode; set it before any Apply below.
+	sink.sv = sv
+
+	// session.update switched semantic -> server mid-turn: drop the
+	// orphaned live stream. This is NOT a turn abort — the turn continues
+	// under server_vad (a config change must not cut off a mid-utterance
+	// speaker), so the coordinator stays Speaking; only the orphaned live
+	// stream is closed.
+	if sv == nil && lts.open() {
+		lts.discardTurn()
+	}
+
+	session.AudioBufferLock.Lock()
+	// Retention bound: drop the buffer head beyond maxTurnBufferSec so that a
+	// turn that never pauses can't grow memory, the per-tick copy+resample,
+	// or the commit-time decode without limit (this also bounds the
+	// runVAD-error path below, which can't trim). A mid-turn trim shifts
+	// every buffer-relative cursor, so the live-feed and EOU positions are
+	// rebased by the trimmed amount. Whole input-seconds only: second-aligned
+	// cuts keep the resampled tail sample-identical to the suffix of the
+	// previous whole-buffer resample, so the live feed stays gapless.
+	bytesPerSec := session.InputSampleRate * 2
+	if maxBytes := int(maxTurnBufferSec) * bytesPerSec; len(session.InputAudioBuffer) > maxBytes {
+		trimSecs := (len(session.InputAudioBuffer) - maxBytes + bytesPerSec - 1) / bytesPerSec
+		session.InputAudioBuffer = append([]byte(nil), session.InputAudioBuffer[trimSecs*bytesPerSec:]...)
+		lts.rebase(float64(trimSecs))
+		sink.lastSpeechEndSec = max(0, sink.lastSpeechEndSec-float64(trimSecs))
+	}
+	allAudio := make([]byte, len(session.InputAudioBuffer))
+	copy(allAudio, session.InputAudioBuffer)
+	session.AudioBufferLock.Unlock()
+
+	aints := sound.BytesToInt16sLE(allAudio)
+	if len(aints) == 0 || len(aints) < int(silenceThreshold*float64(session.InputSampleRate)) {
+		return
+	}
+
+	// Resample from InputSampleRate to 16kHz
+	aints = sound.ResampleInt16(aints, session.InputSampleRate, localSampleRate)
+
+	audioLength := float64(len(aints)) / localSampleRate
+
+	if sv != nil && lts.open() {
+		lts.feedNewAudio(aints)
+		lts.drainEvents(audioLength)
+	}
+
+	// Scan window: silero's recurrent state carries only a few hundred ms of
+	// context, so audio older than the largest silence the commit test can
+	// need to measure (plus warm-up margin) contributes nothing to the
+	// tail's classification — clip it instead of rescanning the whole turn
+	// every tick (~3.3ms of silero per buffered second, quadratic over a
+	// turn). Segment times are rebased back to whole-buffer coordinates so
+	// every downstream consumer (trailing-silence math, eouPending, the
+	// live-feed cursor) is untouched.
+	scan := aints
+	clipOffsetSec := 0.0
+	if maxScan := int(vadScanWindowSec(sv, silenceThreshold, session.ModelConfig) * localSampleRate); len(aints) > maxScan {
+		scan = aints[len(aints)-maxScan:]
+		clipOffsetSec = float64(len(aints)-maxScan) / localSampleRate
+	}
+
+	segments, err := runVAD(vadContext, session, scan)
+	if err != nil {
+		if err.Error() == "unexpected speech end" {
+			xlog.Debug("VAD cancelled")
+			return
+		}
+		xlog.Error("failed to process audio", "error", err)
+		sendError(t, "processing_error", "Failed to process audio: "+err.Error(), "", "")
+		return
+	}
+	for i := range segments {
+		segments[i].Start += float32(clipOffsetSec)
+		// End == 0 is the "segment still open" sentinel — leave it alone.
+		if segments[i].End != 0 {
+			segments[i].End += float32(clipOffsetSec)
+		}
+	}
+
+	// NOTE: the no-speech clear and the min-buffer gate above stay on
+	// the short silenceThreshold even in semantic mode — the eagerness
+	// fallback applies only to the end-of-speech commit decision, or a
+	// low eagerness would delay speech_started/barge-in by seconds.
+	if len(segments) == 0 {
+		// An open turn whose scan window is all silence: the turn's speech
+		// is entirely older than the clip, so the trailing silence is at
+		// least the window — which the window sizing guarantees exceeds
+		// every commit threshold. Commit with the last speech end this
+		// turn observed instead of discarding real speech as no-speech.
+		// With no clip in effect (clipOffsetSec == 0) silero really saw
+		// the whole turn, and zero segments keeps its historical meaning:
+		// the earlier onset was reclassified as noise — clear it below.
+		if _, speaking := sink.coord.State().(turncoord.Speaking); speaking &&
+			clipOffsetSec > 0 && sink.lastSpeechEndSec > 0 {
+			vadCommit(sink, retranscribe, aints, len(allAudio), audioLength, sink.lastSpeechEndSec, false)
+			return
+		}
+		if audioLength > silenceThreshold {
+			// "No segments" is not "no speech": silero (threshold 0.5)
+			// crosses up to a few hundred ms into a soft word onset, so
+			// the newest audio in the inspected window may be the start
+			// of a word the next tick will recognize — and more audio
+			// arrived while this tick ran. Keep both; drop only the
+			// older, confirmed-silent head, or utterance onsets get cut.
+			holdback := int(noSpeechHoldbackSec*float64(session.InputSampleRate)) * 2
+			session.AudioBufferLock.Lock()
+			session.InputAudioBuffer = dropInspectedPrefix(session.InputAudioBuffer, len(allAudio), holdback)
+			session.AudioBufferLock.Unlock()
+
+			// No-speech clear: end any open turn (Speaking -> Idle, discarding
+			// the partial). Returning to Idle is the fix for failure mode 4 —
+			// the legacy discardTurn left speechStarted true, suppressing the
+			// next onset. Idle while not speaking is a no-op.
+			sink.lastSpeechEndSec = 0
+			if err := sink.coord.Apply(turncoord.Abort{Reason: turncoord.AbortNoSpeech}); err != nil {
+				xlog.Error("turncoord: abort(no_speech) failed", "error", err)
+			}
+		}
+		return
+	}
+
+	// Speech detected this tick: open the turn (Idle -> Speaking) through
+	// the coordinator. On that transition it opens the turn's live ASR
+	// stream + feeds the buffered prefix (OpenTurn), cancels any in-flight
+	// response (BargeIn, non-blocking — the VAD tick is never stalled), and
+	// emits speech_started. While already Speaking it is a no-op, so "turn
+	// open" and "speech started" can never disagree. The turn id is minted
+	// here and carried by the coordinator through to the committed event.
+	sink.onsetAudio = aints
+	if err := sink.coord.Apply(turncoord.Onset{Turn: turncoord.TurnID(generateItemID())}); err != nil {
+		xlog.Error("turncoord: onset failed", "error", err)
+	}
+
+	// Track where speech last ended, in whole-buffer seconds: once these
+	// segments scroll out of the scan clip, the silence-outran-the-window
+	// commit above still needs a speech end to report. An open segment
+	// (End == 0) means speech reaches the end of the inspected audio.
+	if end := segments[len(segments)-1].End; end != 0 {
+		sink.lastSpeechEndSec = float64(end)
+	} else {
+		sink.lastSpeechEndSec = audioLength
+	}
+
+	if sv != nil {
+		// Drain again: events produced by THIS tick's feed have
+		// usually arrived by the time runVAD returns, and leaving
+		// them for the next tick adds 300ms to every EOU-triggered
+		// commit.
+		lts.drainEvents(audioLength)
+	}
+
+	// Segment still in progress when audio ended
+	segEndTime := segments[len(segments)-1].End
+	if segEndTime == 0 {
+		return
+	}
+
+	threshold := silenceThreshold
+	eouPending := false
+	if sv != nil {
+		eouPending = lts.eouPending(segments)
+		threshold = lts.thresholdSec(eouPending, sv)
+	}
+
+	if float32(audioLength)-segEndTime > float32(threshold) {
+		vadCommit(sink, retranscribe, aints, len(allAudio), audioLength, float64(segEndTime), eouPending)
+	}
+}
+
+// vadCommit runs the commit tail of a VAD tick: the semantic commit log, the
+// retranscribe gate, the buffer trim, and the coordinator's Silence event
+// (speech_stopped + committed + finalize live stream + issue the response).
+// Shared by the normal trailing-silence commit and the
+// silence-outran-the-scan-window commit.
+func vadCommit(sink *turnSink, retranscribe bool, aints []int16, inspectedBytes int, audioLength, segEndTime float64, eouPending bool) {
+	session := sink.session
+	lts := sink.lts
+
+	if sink.sv != nil {
+		trigger, eouLag := lts.commitTrigger(eouPending, segEndTime)
+		xlog.Info("semantic_vad: committing turn",
+			"trigger", trigger,
+			"speech_end_s", segEndTime,
+			"eou_lag_s", eouLag,
+			"silence_s", audioLength-segEndTime,
+			"audio_s", audioLength)
+	}
+	// Retranscribe gate (semantic mode, EOU-triggered commits
+	// only): cross-check the streamed EOU with an offline decode
+	// of the buffered turn before committing. Runs synchronously
+	// on the tick — the engine would serialize a concurrent feed
+	// against it anyway. Timeout-triggered commits skip the gate.
+	var gated *schema.TranscriptionResult
+	if retranscribe && eouPending {
+		batch, gerr := transcribeUtterance(sink.vadContext, sound.Int16toBytesLE(aints), session)
+		switch {
+		case gerr != nil:
+			xlog.Warn("semantic_vad: retranscribe gate failed; committing via the file path", "error", gerr)
+		case !batch.Eou:
+			xlog.Info("semantic_vad: batch decode did not confirm the streamed EOU; continuing to listen",
+				"streamed", lts.previewText(), "batch", batch.Text)
+			// The batch decode rejected the streamed EOU as a false
+			// positive: consume the recorded EOU so the next tick
+			// falls back to the eagerness window instead of
+			// re-triggering on the same token.
+			lts.eouAtSec = 0
+			return
+		default:
+			xlog.Info("semantic_vad: batch decode confirmed the streamed EOU",
+				"streamed", lts.previewText(), "batch", batch.Text)
+			gated = batch
+		}
+	}
+
+	xlog.Debug("Detected end of speech segment")
+	session.AudioBufferLock.Lock()
+	// Keep audio appended while this tick ran — it belongs to
+	// the next turn (in any mode: nil-ing it dropped the onset
+	// of an utterance started right after a commit).
+	session.InputAudioBuffer = dropInspectedPrefix(session.InputAudioBuffer, inspectedBytes, 0)
+	session.AudioBufferLock.Unlock()
+
+	// Commit the turn through the coordinator: it emits speech_stopped
+	// (EmitSpeechStopped) then the committed event, finalizes the live
+	// stream, and issues the response (CommitTurn). The committed item
+	// id is the coordinator's turn id (== the id the live captions
+	// streamed under), so the client replaces the partial text.
+	sink.commitAudio = sound.Int16toBytesLE(aints)
+	sink.commitAudioLength = audioLength
+	sink.commitRetranscribe = retranscribe
+	sink.commitGated = gated
+	sink.lastSpeechEndSec = 0
+	// TODO: Remove prefix silence that is over TurnDetectionParams.PrefixPaddingMs
+	if err := sink.coord.Apply(turncoord.Silence{}); err != nil {
+		xlog.Error("turncoord: commit failed", "error", err)
+	}
+}
+
+// vadScanWindowSec sizes the tail of the buffer silero inspects each tick.
+// The window must contain the largest trailing silence the commit test can
+// need to measure — server_vad's silence window, or the semantic eagerness
+// fallback (the post-EOU window is shorter) — plus vadWarmupMarginSec.
+// pipeline.turn_detection.vad_window_sec can widen it; values below the floor
+// are ignored, since a narrower window would make long silences unmeasurable
+// and turns uncommittable.
+func vadScanWindowSec(sv *types.RealtimeSessionSemanticVad, silenceThreshold float64, cfg *config.ModelConfig) float64 {
+	needed := silenceThreshold
+	if sv != nil {
+		needed = eagernessMaxSilenceSec(sv.Eagerness)
+	}
+	window := needed + vadWarmupMarginSec
+	if cfg != nil && cfg.Pipeline.TurnDetection.VadWindowSec > window {
+		window = cfg.Pipeline.TurnDetection.VadWindowSec
+	}
+	return window
+}
+
 func commitUtterance(ctx context.Context, utt []byte, session *Session, conv *Conversation, t Transport) {
 	commitUtteranceWithTranscript(ctx, utt, nil, nil, "", session, conv, t)
 }
@@ -1896,6 +2073,11 @@ func runVAD(ctx context.Context, session *Session, adata []int16) ([]schema.VADS
 	if err != nil {
 		return nil, err
 	}
+	// A backend answering with an empty message means "no speech", not a
+	// reason to panic the VAD goroutine.
+	if resp == nil {
+		return nil, nil
+	}
 
 	// If resp.Segments is empty => no speech
 	return resp.Segments, nil
@@ -2216,9 +2398,18 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 		images = append(images, m.StringImages...)
 	}
 
-	// response.created/done are emitted once per response.create by triggerResponse;
-	// every turn (including agentic recursion) shares this id.
-	responseID := r.id
+	// Classifier mode replaces autoregressive generation for the first turn
+	// of a response: prefill-only scoring picks a registered option and its
+	// canned reply/tool is emitted through the standard response protocol.
+	// Agentic follow-ups (toolTurn > 0) always generate — the option list
+	// describes user intents, not tool outputs. This branch must precede the
+	// streamed-LLM path below or streaming pipelines would bypass it.
+	if cc := resolveClassifier(session.Classifier, overrides); toolTurn == 0 && cc.Active() {
+		if classifierRespond(ctx, session, conv, t, r, cc, conversationHistory, overrides, toolTurn) {
+			return
+		}
+		// fallback.mode "generate": fall through to normal generation.
+	}
 
 	// Streamed LLM path: when the pipeline opts into LLM streaming, stream the
 	// transcript to the client as it is generated and synthesize the buffered
@@ -2371,6 +2562,30 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 	}
 
 	if finalSpeech != "" {
+		if !emitAssistantMessage(ctx, session, conv, t, r, finalSpeech, overrides) {
+			return
+		}
+	}
+
+	// Emit the parsed tool calls and (for server-side assistant tools) the
+	// follow-up turn. Shared with the streamed path so both finalize tool calls
+	// identically. The single terminal is emitted by triggerResponse.
+	emitToolCallItems(ctx, session, conv, t, r, finalToolCalls, finalSpeech != "", toolTurn)
+}
+
+// emitAssistantMessage appends an assistant item carrying finalSpeech to the
+// conversation and emits the standard response events for it —
+// output_item.added, content_part.added, audio-transcript or output-text
+// deltas, TTS audio via emitSpeech (unless the resolved modalities are
+// text-only), content_part.done and output_item.done. Shared by the buffered
+// generation path and classifier mode. Returns false when the response was
+// cancelled (barge-in) or failed — r.outcome is already recorded and the
+// caller must emit no further items.
+func emitAssistantMessage(ctx context.Context, session *Session, conv *Conversation, t Transport, r *liveResponse, finalSpeech string, overrides *types.ResponseCreateParams) bool {
+	// response.created/done are emitted once per response.create by
+	// triggerResponse; every turn (including agentic recursion) shares this id.
+	responseID := r.id
+	{
 		// Create the assistant item now that we have content
 		item := types.MessageItemUnion{
 			Assistant: &types.MessageItemAssistant{
@@ -2438,7 +2653,7 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 			if ctx.Err() != nil {
 				xlog.Debug("Response cancelled before TTS (barge-in)")
 				sendCancelledResponse()
-				return
+				return false
 			}
 
 			// Transcript of the spoken reply (the audio's text).
@@ -2468,12 +2683,12 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 				if ctx.Err() != nil {
 					xlog.Debug("TTS cancelled (barge-in)")
 					sendCancelledResponse()
-					return
+					return false
 				}
 				xlog.Error("TTS failed", "error", err)
 				sendError(t, "tts_error", fmt.Sprintf("TTS generation failed: %v", err), "", item.Assistant.ID)
 				r.outcome = outcomeFailed
-				return
+				return false
 			}
 			if !isWebRTC {
 				audioString = base64.StdEncoding.EncodeToString(pcmAudio)
@@ -2532,11 +2747,7 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 		})
 		r.addItem(item)
 	}
-
-	// Emit the parsed tool calls and (for server-side assistant tools) the
-	// follow-up turn. Shared with the streamed path so both finalize tool calls
-	// identically. The single terminal is emitted by triggerResponse.
-	emitToolCallItems(ctx, session, conv, t, r, finalToolCalls, finalSpeech != "", toolTurn)
+	return true
 }
 
 // emitToolCallItems emits the realtime function_call items for the parsed tool

core/http/endpoints/openai/realtime_classifier.go

@@ -0,0 +1,616 @@
+package openai
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/router"
+	"github.com/mudler/LocalAI/pkg/functions"
+	"github.com/mudler/xlog"
+)
+
+// Classifier mode (LocalAI extension): instead of autoregressive
+// generation, each user turn is prefill-scored against a registered option
+// list via the Score primitive and the winning option's canned reply /
+// tool call is emitted. Designed for hardware that can afford prefill but
+// not decode. See docs/content/features/openai-realtime.md.
+
+// By default only the latest user message is scored. Earlier turns in the
+// probe — the assistant's canned replies especially — echo option names
+// ("Going up." ↔ up) and verified empirically to dominate small scoring
+// models: with any prior turn present, a 1.2B model kept re-choosing the
+// previous option at p≈1.0 regardless of the new command. history_items > 0
+// opts back into context (role-labeled), for larger scoring models.
+
+// classifierConfigFromPipeline converts the YAML pipeline.classifier block
+// into the wire ClassifierConfig and validates it, so a bad option list
+// rejects the session at setup rather than misbehaving on the first turn.
+// A nil block yields a nil config (classifier off).
+func classifierConfigFromPipeline(p *config.PipelineClassifier) (*types.ClassifierConfig, error) {
+	if p == nil {
+		return nil, nil
+	}
+	cc := &types.ClassifierConfig{
+		Enabled:       &p.Enabled,
+		Threshold:     p.Threshold,
+		Normalization: p.Normalization,
+		HistoryItems:  p.HistoryItems,
+	}
+	if p.Fallback != nil {
+		cc.Fallback = &types.ClassifierFallback{Mode: p.Fallback.Mode, Reply: p.Fallback.Reply}
+	}
+	if p.Address != nil {
+		cc.Address = &types.ClassifierAddress{Names: p.Address.Names, Mode: p.Address.Mode, Reply: p.Address.Reply}
+	}
+	for _, o := range p.Options {
+		opt := types.ClassifierOption{
+			ID:          o.ID,
+			Description: o.Description,
+			Reply:       o.Reply,
+		}
+		if o.Tool != nil {
+			args := json.RawMessage(nil)
+			if o.Tool.Arguments != nil {
+				data, err := json.Marshal(o.Tool.Arguments)
+				if err != nil {
+					return nil, fmt.Errorf("option %q: marshal tool arguments: %w", o.ID, err)
+				}
+				args = data
+			}
+			opt.Tool = &types.ClassifierTool{Name: o.Tool.Name, Arguments: args}
+			for _, s := range o.Tool.Slots {
+				opt.Tool.Slots = append(opt.Tool.Slots, types.ClassifierSlot{
+					Name:    s.Name,
+					Type:    s.Type,
+					Values:  s.Values,
+					Default: s.Default,
+					Hint:    s.Hint,
+				})
+			}
+		}
+		cc.Options = append(cc.Options, opt)
+	}
+	if err := cc.Validate(); err != nil {
+		return nil, err
+	}
+	return cc, nil
+}
+
+// prewarmClassifier primes the scoring prompt cache for the session's
+// current classifier config in the background: registration returns
+// immediately, and by the time the canned mode-switch reply finishes
+// speaking, the new option list's prompt (and, on hybrid/recurrent
+// models, a rewind checkpoint at the per-turn probe boundary) is already
+// in the backend's cache. The context is deliberately detached from the
+// registering request — the warmed cache belongs to the backend, not the
+// request.
+func prewarmClassifier(session *Session) {
+	cc := session.Classifier
+	if session.ModelInterface == nil || !cc.Active() {
+		return
+	}
+	options, normalization := cc.Options, cc.Normalization
+	go func() {
+		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+		defer cancel()
+		session.ModelInterface.PrewarmClassifier(ctx, options, normalization)
+	}()
+}
+
+// resolveClassifier merges the session classifier config with a
+// response-level override: a non-nil override replaces the whole block
+// (same replace-not-merge semantics as tools), so {"enabled": false} runs
+// normal generation for one response.
+func resolveClassifier(sessionCfg *types.ClassifierConfig, overrides *types.ResponseCreateParams) *types.ClassifierConfig {
+	if overrides != nil && overrides.LocalAIClassifier != nil {
+		return overrides.LocalAIClassifier
+	}
+	return sessionCfg
+}
+
+// validateClassifierActivation verifies both the wire config and the concrete
+// backend selected to score it. Scoring capacity is reserved at model load
+// only for configs that explicitly declare the score usecase, so accepting an
+// active classifier on any other model would defer a deterministic failure to
+// the first response.
+func validateClassifierActivation(m Model, cc *types.ClassifierConfig) error {
+	if cc == nil {
+		return nil
+	}
+	if err := cc.Validate(); err != nil {
+		return err
+	}
+	if !cc.Active() {
+		return nil
+	}
+	wm, ok := m.(*wrappedModel)
+	if !ok {
+		return fmt.Errorf("classifier: the session model does not support scoring")
+	}
+	cfg := wm.scoreConfig()
+	if cfg == nil || !cfg.HasUsecases(config.FLAG_SCORE) {
+		name := ""
+		if cfg != nil {
+			name = cfg.Name
+		}
+		return fmt.Errorf("classifier: scoring model %q must declare known_usecases: [score]", name)
+	}
+	if cfg.HasRouter() {
+		return fmt.Errorf("classifier: scoring model %q is a router; configure a concrete pipeline.classifier.model", cfg.Name)
+	}
+	return nil
+}
+
+// trimClassifierHistory drops system messages (the classifier builds its
+// own option-list system prompt) and selects what gets scored.
+// historyItems <= 0 (the default): only the latest user message. Positive
+// N: the trailing N conversation messages.
+func trimClassifierHistory(history schema.Messages, historyItems int) schema.Messages {
+	conversation := make(schema.Messages, 0, len(history))
+	for _, m := range history {
+		if m.Role == string(types.MessageRoleSystem) {
+			continue
+		}
+		conversation = append(conversation, m)
+	}
+	if historyItems <= 0 {
+		for i := len(conversation) - 1; i >= 0; i-- {
+			if conversation[i].Role == string(types.MessageRoleUser) {
+				return conversation[i : i+1]
+			}
+		}
+		return nil
+	}
+	if len(conversation) > historyItems {
+		conversation = conversation[len(conversation)-historyItems:]
+	}
+	return conversation
+}
+
+// latestUserText returns the text of the most recent user message — the
+// turn the address gate inspects (earlier turns being addressed doesn't
+// make this one addressed).
+func latestUserText(messages schema.Messages) string {
+	for i := len(messages) - 1; i >= 0; i-- {
+		if messages[i].Role == string(types.MessageRoleUser) {
+			text, _ := messages[i].Content.(string)
+			return text
+		}
+	}
+	return ""
+}
+
+// mentionsAnyName reports whether text contains any of the names as a
+// case-insensitive whole word ("drone" matches "Drone, go up" but not
+// "drones").
+func mentionsAnyName(text string, names []string) bool {
+	for _, n := range names {
+		n = strings.TrimSpace(n)
+		if n == "" {
+			continue
+		}
+		re, err := regexp.Compile(`(?i)\b` + regexp.QuoteMeta(n) + `\b`)
+		if err != nil {
+			continue
+		}
+		if re.MatchString(text) {
+			return true
+		}
+	}
+	return false
+}
+
+// classifierProbe renders the trimmed history for scoring. A single user
+// message goes in verbatim — that matches the scoring format's training
+// distribution (Arch-Router scores "the user's request"). When
+// history_items opts extra turns in, every line carries a role label so
+// the scoring model can at least tell the user's request apart from the
+// assistant's replies.
+func classifierProbe(messages schema.Messages) router.Probe {
+	parts := make([]string, 0, len(messages))
+	label := len(messages) > 1
+	for _, msg := range messages {
+		text, _ := msg.Content.(string)
+		if text == "" {
+			continue // e.g. tool-call items carry no text
+		}
+		if label {
+			switch msg.Role {
+			case string(types.MessageRoleAssistant):
+				text = "Assistant: " + text
+			case "tool":
+				text = "Tool: " + text
+			default:
+				text = "User: " + text
+			}
+		}
+		parts = append(parts, text)
+	}
+	return router.Probe{Prompt: router.JoinTurns(parts), Messages: parts}
+}
+
+// classifierRespond runs one classifier-mode response: score the options,
+// emit the localai.classifier.result observability event, then either the
+// winning option's canned reply/tool, the fallback reply, nothing, or —
+// for the generate fallback — report false so the caller falls through to
+// normal generation. Runs inside the respcoord-issued response body, so
+// the single terminal stays owned by triggerResponse. Returns true when
+// the response was fully handled here.
+func classifierRespond(ctx context.Context, session *Session, conv *Conversation, t Transport, r *liveResponse, cc *types.ClassifierConfig, history schema.Messages, overrides *types.ResponseCreateParams, toolTurn int) bool {
+	msgs := trimClassifierHistory(history, cc.HistoryItems)
+	if len(msgs) == 0 {
+		xlog.Debug("realtime classifier: no scorable conversation content; skipping to generation")
+		return false
+	}
+
+	// Address gate (wake-word behavior): when configured, a turn that
+	// doesn't mention one of the assistant's names is dropped before any
+	// scoring — the check is a deterministic word match on the transcript
+	// because scoring cannot detect the missing name (command semantics
+	// dominate the softmax), and skipping the Score call keeps ambient
+	// conversation free on weak hardware.
+	if ad := cc.Address; ad != nil && !mentionsAnyName(latestUserText(msgs), ad.Names) {
+		sendEvent(t, types.ClassifierResultEvent{
+			ResponseID: r.id,
+			Scores:     []types.ClassifierScore{},
+			Threshold:  cc.Threshold,
+			Fallback:   types.ClassifierNotAddressed,
+		})
+		xlog.Debug("realtime classifier: turn does not address the assistant; dropping", "mode", ad.AddressMode())
+		if ctx.Err() != nil {
+			r.outcome = outcomeCancelled
+			return true
+		}
+		if ad.AddressMode() == types.ClassifierAddressReply && ad.Reply != "" {
+			if !emitAssistantMessage(ctx, session, conv, t, r, ad.Reply, overrides) {
+				return true
+			}
+			emitToolCallItems(ctx, session, conv, t, r, nil, true, toolTurn)
+			return true
+		}
+		// ignore: complete the response with no output items.
+		emitToolCallItems(ctx, session, conv, t, r, nil, false, toolTurn)
+		return true
+	}
+
+	// A committed turn can carry no words at all (the VAD fires on noise
+	// and the ASR transcribes nothing). Scoring an empty prompt returns a
+	// confidently arbitrary winner — measured p≈0.95 for the first option
+	// — so skip scoring entirely and treat it like a below-threshold turn.
+	var scores []router.LabelScore
+	var latency time.Duration
+	if strings.TrimSpace(classifierProbe(msgs).Prompt) != "" {
+		start := time.Now()
+		var err error
+		scores, err = session.ModelInterface.ClassifyTurn(ctx, msgs, cc.Options, cc.Normalization)
+		if err != nil {
+			if cc.FallbackMode() == types.ClassifierFallbackGenerate {
+				xlog.Warn("realtime classifier: scoring failed; falling back to generation", "error", err)
+				return false
+			}
+			sendError(t, "classifier_failed", fmt.Sprintf("classifier scoring failed: %v", err), "", "")
+			r.outcome = outcomeFailed
+			return true
+		}
+		latency = time.Since(start)
+	} else if cc.FallbackMode() == types.ClassifierFallbackGenerate {
+		xlog.Debug("realtime classifier: turn has no scorable text; falling back to generation")
+		return false
+	}
+
+	best := -1
+	for i := range scores {
+		if best < 0 || scores[i].Score > scores[best].Score {
+			best = i
+		}
+	}
+	var chosen *types.ClassifierOption
+	chosenID := ""
+	fallbackApplied := ""
+	if best >= 0 && scores[best].Score >= cc.Threshold {
+		chosen = &cc.Options[best]
+		chosenID = chosen.ID
+	} else {
+		fallbackApplied = cc.FallbackMode()
+	}
+
+	// Hybrid path: a winning option with argument slots gets them filled by
+	// a constrained completion before anything is emitted, so the result
+	// event carries the final arguments. An unrecoverable fill failure
+	// (error and no complete default set) is handled like a scoring
+	// failure.
+	filledArgs := ""
+	var fillValues map[string]string
+	var fillLatency time.Duration
+	if chosen != nil {
+		var ferr error
+		filledArgs, fillValues, fillLatency, ferr = fillChosenArguments(ctx, session, cc, msgs, chosen)
+		if ferr != nil {
+			if cc.FallbackMode() == types.ClassifierFallbackGenerate {
+				xlog.Warn("realtime classifier: slot fill failed; falling back to generation", "error", ferr)
+				return false
+			}
+			sendError(t, "classifier_failed", fmt.Sprintf("classifier slot fill failed: %v", ferr), "", "")
+			r.outcome = outcomeFailed
+			return true
+		}
+	}
+
+	evScores := make([]types.ClassifierScore, len(scores))
+	for i, s := range scores {
+		evScores[i] = types.ClassifierScore{ID: s.Label, Score: s.Score}
+	}
+	evArgs := ""
+	if chosen != nil && chosen.Tool != nil && len(chosen.Tool.Slots) > 0 {
+		evArgs = filledArgs
+	}
+	sendEvent(t, types.ClassifierResultEvent{
+		ResponseID:    r.id,
+		Scores:        evScores,
+		ChosenID:      chosenID,
+		Threshold:     cc.Threshold,
+		Fallback:      fallbackApplied,
+		LatencyMs:     latency.Milliseconds(),
+		Arguments:     evArgs,
+		FillLatencyMs: fillLatency.Milliseconds(),
+	})
+	topScore := 0.0
+	if best >= 0 {
+		topScore = scores[best].Score
+	}
+	xlog.Debug("realtime classifier: scored turn",
+		"chosen", chosenID, "top_score", topScore,
+		"threshold", cc.Threshold, "fallback", fallbackApplied,
+		"latency_ms", latency.Milliseconds(),
+		"arguments", evArgs, "fill_latency_ms", fillLatency.Milliseconds())
+
+	if fallbackApplied == types.ClassifierFallbackGenerate {
+		return false
+	}
+
+	// Barge-in may have fired during scoring.
+	if ctx.Err() != nil {
+		r.outcome = outcomeCancelled
+		return true
+	}
+
+	reply := ""
+	var toolCalls []functions.FuncCallResults
+	switch {
+	case chosen != nil:
+		// The reply may template the filled slot values ("Going forward
+		// {{distance}} {{units}}.") so what is spoken confirms what was
+		// actually inferred.
+		reply = chosen.SpliceReply(fillValues)
+		if chosen.Tool != nil {
+			toolCalls = []functions.FuncCallResults{{Name: chosen.Tool.Name, Arguments: filledArgs}}
+		}
+	case fallbackApplied == types.ClassifierFallbackReply:
+		reply = cc.Fallback.Reply
+	default:
+		// fallback "none": complete with no output items.
+	}
+
+	if reply != "" {
+		if !emitAssistantMessage(ctx, session, conv, t, r, reply, overrides) {
+			// Cancelled or failed — outcome already recorded.
+			return true
+		}
+	}
+	// Always finalize through emitToolCallItems, mirroring the generation
+	// path: it emits the function_call items (client executes canned tools
+	// and reports back via conversation.item.create) and runs server-side
+	// assistant tools inproc.
+	emitToolCallItems(ctx, session, conv, t, r, toolCalls, reply != "", toolTurn)
+	return true
+}
+
+// ---- slot filling (hybrid classify-then-complete) --------------------------
+//
+// A winning option whose tool declares slots gets its argument values from a
+// short constrained completion: the prompt is the exact scoring prompt (warm
+// in the backend's cache) continued by the chosen route JSON re-opened at the
+// first slot field, and a GBNF grammar pins everything except the slot
+// values. The generated tail is parsed back through the JSON object it
+// completes, and the values are spliced into the tool's argument template.
+
+// gbnfLiteral renders s as a GBNF quoted literal.
+func gbnfLiteral(s string) string {
+	r := strings.NewReplacer(`\`, `\\`, `"`, `\"`, "\n", `\n`)
+	return `"` + r.Replace(s) + `"`
+}
+
+// slotFillGrammar builds the grammar for the completion tail: first slot
+// value, then each further slot as a forced `, "<name>": ` literal plus its
+// value, then the closing brace.
+func slotFillGrammar(slots []types.ClassifierSlot) string {
+	var root strings.Builder
+	var rules strings.Builder
+	needNum, needStr := false, false
+	root.WriteString("root ::= ")
+	for i := range slots {
+		if i > 0 {
+			root.WriteString(" " + gbnfLiteral(`, "`+slots[i].Name+`": `) + " ")
+		}
+		fmt.Fprintf(&root, "slot%d", i)
+		fmt.Fprintf(&rules, "\nslot%d ::= ", i)
+		switch slots[i].Type {
+		case types.ClassifierSlotNumber:
+			rules.WriteString("num")
+			needNum = true
+		case types.ClassifierSlotEnum:
+			for vi, v := range slots[i].Values {
+				if vi > 0 {
+					rules.WriteString(" | ")
+				}
+				encoded, _ := json.Marshal(v) // validation rejects values JSON cannot encode
+				rules.WriteString(gbnfLiteral(string(encoded)))
+			}
+		default: // string
+			rules.WriteString("str")
+			needStr = true
+		}
+	}
+	root.WriteString(` "}"`)
+	if needNum {
+		rules.WriteString("\nnum ::= \"-\"? [0-9] [0-9]* (\".\" [0-9] [0-9]*)?")
+	}
+	if needStr {
+		rules.WriteString("\nstr ::= \"\\\"\" [^\"\\\\\\n]* \"\\\"\"")
+	}
+	return root.String() + rules.String()
+}
+
+const (
+	// Free-form values need an explicit ceiling; forced enum values and field
+	// syntax are budgeted from their actual JSON encoding below.
+	slotFillStringTokens = 64
+	slotFillNumberTokens = 32
+)
+
+// slotFillMaxTokens conservatively budgets one token per output byte for the
+// forced JSON tail, plus explicit allowances for free-form values. This avoids
+// truncating long enum values or field names while keeping string generation
+// bounded.
+func slotFillMaxTokens(slots []types.ClassifierSlot) int {
+	tokens := 1 // closing brace
+	for i := range slots {
+		if i > 0 {
+			field, _ := json.Marshal(slots[i].Name)
+			tokens += len(field) + len(`, : `)
+		}
+		switch slots[i].Type {
+		case types.ClassifierSlotNumber:
+			tokens += slotFillNumberTokens
+		case types.ClassifierSlotString:
+			tokens += slotFillStringTokens
+		case types.ClassifierSlotEnum:
+			longest := 0
+			for _, value := range slots[i].Values {
+				encoded, _ := json.Marshal(value)
+				if len(encoded) > longest {
+					longest = len(encoded)
+				}
+			}
+			tokens += longest
+		}
+	}
+	return tokens
+}
+
+// slotFillContextReserve includes both the generated tail and the continuation
+// prefix appended after the scored prompt. It intentionally over-reserves by
+// counting bytes as tokens; preserving the identical scoring prompt is more
+// important than reclaiming a handful of context tokens.
+func slotFillContextReserve(option *types.ClassifierOption) int {
+	if option == nil || option.Tool == nil || len(option.Tool.Slots) == 0 {
+		return 0
+	}
+	route, _ := json.Marshal(option.ID)
+	field, _ := json.Marshal(option.Tool.Slots[0].Name)
+	prefixBytes := len(`{"route": , : `) + len(route) + len(field)
+	return prefixBytes + slotFillMaxTokens(option.Tool.Slots)
+}
+
+// parseSlotValues closes the completed route JSON and extracts each slot's
+// value as the string form SpliceArguments expects.
+func parseSlotValues(chosenID, firstSlot, generated string, slots []types.ClassifierSlot) (map[string]string, error) {
+	idJSON, _ := json.Marshal(chosenID)
+	full := `{"route": ` + string(idJSON) + `, "` + firstSlot + `": ` + strings.TrimSpace(generated)
+	if !strings.HasSuffix(strings.TrimSpace(generated), "}") {
+		full += "}"
+	}
+	dec := json.NewDecoder(strings.NewReader(full))
+	dec.UseNumber()
+	var obj map[string]any
+	if err := dec.Decode(&obj); err != nil {
+		return nil, fmt.Errorf("classifier: slot completion %q does not parse: %w", generated, err)
+	}
+	values := make(map[string]string, len(slots))
+	for i := range slots {
+		v, ok := obj[slots[i].Name]
+		if !ok {
+			return nil, fmt.Errorf("classifier: slot completion missing %q", slots[i].Name)
+		}
+		switch tv := v.(type) {
+		case json.Number:
+			values[slots[i].Name] = tv.String()
+		case string:
+			values[slots[i].Name] = tv
+		default:
+			return nil, fmt.Errorf("classifier: slot %q has unexpected value type %T", slots[i].Name, v)
+		}
+	}
+	return values, nil
+}
+
+// fillChosenArguments resolves a winning option's tool arguments: canned
+// options pass through, slotted options run the fill completion with a
+// default-value recovery when inference fails. The slot values ride along
+// so the caller can splice them into the spoken reply too. The error return
+// is reserved for unrecoverable failures (no complete default set).
+func fillChosenArguments(ctx context.Context, session *Session, cc *types.ClassifierConfig, msgs schema.Messages, chosen *types.ClassifierOption) (args string, values map[string]string, latency time.Duration, err error) {
+	if chosen.Tool == nil {
+		return "", nil, 0, nil
+	}
+	if len(chosen.Tool.Slots) == 0 {
+		if len(chosen.Tool.Arguments) > 0 {
+			return string(chosen.Tool.Arguments), nil, 0, nil
+		}
+		return "{}", nil, 0, nil
+	}
+	start := time.Now()
+	args, values, err = session.ModelInterface.FillToolArguments(ctx, msgs, cc.Options, cc.Normalization, chosen)
+	latency = time.Since(start)
+	if err == nil {
+		return args, values, latency, nil
+	}
+	xlog.Warn("realtime classifier: slot fill failed; trying slot defaults", "option", chosen.ID, "error", err)
+	defaults, derr := chosen.Tool.SlotDefaults()
+	if derr != nil {
+		return "", nil, latency, err
+	}
+	args, derr = chosen.Tool.SpliceArguments(defaults)
+	if derr != nil {
+		return "", nil, latency, err
+	}
+	return args, defaults, latency, nil
+}
+
+// classifierPolicyDescription renders an option's scoring description,
+// appending any slot declarations so the model both weighs the parameters
+// during scoring and knows how to fill them ("assume meters…") during the
+// slot completion — the hints ride the shared system prompt, costing no
+// extra per-turn tokens.
+func classifierPolicyDescription(o *types.ClassifierOption) string {
+	if o.Tool == nil || len(o.Tool.Slots) == 0 {
+		return o.Description
+	}
+	var b strings.Builder
+	b.WriteString(o.Description)
+	b.WriteString(" — route parameters:")
+	for i := range o.Tool.Slots {
+		s := &o.Tool.Slots[i]
+		if i > 0 {
+			b.WriteString(";")
+		}
+		b.WriteString(" " + s.Name)
+		switch s.Type {
+		case types.ClassifierSlotEnum:
+			b.WriteString(" (one of: " + strings.Join(s.Values, ", ") + ")")
+		default:
+			b.WriteString(" (" + s.Type + ")")
+		}
+		if s.Hint != "" {
+			b.WriteString(", " + s.Hint)
+		}
+	}
+	return b.String()
+}

core/http/endpoints/openai/realtime_classifier_test.go

@@ -0,0 +1,739 @@
+package openai
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/router"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func classifierTestConfig(threshold float64, fallback *types.ClassifierFallback) *types.ClassifierConfig {
+	return &types.ClassifierConfig{
+		Threshold: threshold,
+		Fallback:  fallback,
+		Options: []types.ClassifierOption{
+			{
+				ID:          "up",
+				Description: "the user asks the drone to fly up",
+				Reply:       "Going up.",
+				Tool:        &types.ClassifierTool{Name: "move", Arguments: json.RawMessage(`{"direction":"up"}`)},
+			},
+			{ID: "greeting", Description: "the user greets the assistant", Reply: "Hello."},
+		},
+	}
+}
+
+func classifierTestSession(m *fakeModel) *Session {
+	return &Session{
+		ModelInterface:   m,
+		OutputModalities: []types.Modality{types.ModalityText},
+		ModelConfig:      &config.ModelConfig{},
+	}
+}
+
+var classifierTestHistory = schema.Messages{
+	{Role: "system", StringContent: "instructions", Content: "instructions"},
+	{Role: "user", StringContent: "please go up", Content: "please go up"},
+}
+
+func classifierResultEvents(t *fakeTransport) []types.ClassifierResultEvent {
+	var out []types.ClassifierResultEvent
+	for _, e := range t.events {
+		if ev, ok := e.(types.ClassifierResultEvent); ok {
+			out = append(out, ev)
+		}
+	}
+	return out
+}
+
+// replyTexts collects the assistant reply text of every completed output
+// item — what a classifier response actually "spoke".
+func replyTexts(t *fakeTransport) []string {
+	var out []string
+	for _, e := range t.events {
+		if ev, ok := e.(types.ResponseOutputTextDoneEvent); ok {
+			out = append(out, ev.Text)
+		}
+	}
+	return out
+}
+
+var _ = Describe("prewarmClassifier", func() {
+	It("prewarms an active option list in the background", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		session.Classifier = classifierTestConfig(0.35, nil)
+
+		prewarmClassifier(session)
+
+		Eventually(func() int { n, _ := m.prewarmed(); return n }).Should(Equal(1))
+		_, opts := m.prewarmed()
+		Expect(opts).To(HaveLen(len(session.Classifier.Options)))
+	})
+
+	It("does nothing without an active classifier", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		prewarmClassifier(session)
+
+		off := false
+		session.Classifier = &types.ClassifierConfig{Enabled: &off, Options: classifierTestConfig(0.35, nil).Options}
+		prewarmClassifier(session)
+
+		Consistently(func() int { n, _ := m.prewarmed(); return n }, "150ms").Should(BeZero())
+	})
+})
+
+var _ = Describe("classifierConfigFromPipeline", func() {
+	It("returns nil for an absent block", func() {
+		cc, err := classifierConfigFromPipeline(nil)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(cc).To(BeNil())
+	})
+
+	It("converts options and tool argument maps to wire form", func() {
+		cc, err := classifierConfigFromPipeline(&config.PipelineClassifier{
+			Enabled:   true,
+			Threshold: 0.4,
+			Fallback:  &config.PipelineClassifierFallback{Mode: "reply", Reply: "Say again?"},
+			Options: []config.PipelineClassifierOption{
+				{
+					ID:          "up",
+					Description: "fly up",
+					Reply:       "Going up.",
+					Tool:        &config.PipelineClassifierTool{Name: "move", Arguments: map[string]any{"direction": "up"}},
+				},
+			},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(cc.Active()).To(BeTrue())
+		Expect(cc.Threshold).To(Equal(0.4))
+		Expect(cc.Options).To(HaveLen(1))
+		Expect(string(cc.Options[0].Tool.Arguments)).To(MatchJSON(`{"direction":"up"}`))
+		Expect(cc.Fallback.Mode).To(Equal(types.ClassifierFallbackReply))
+	})
+
+	It("rejects invalid blocks via the shared validation", func() {
+		_, err := classifierConfigFromPipeline(&config.PipelineClassifier{
+			Enabled: true,
+			Options: []config.PipelineClassifierOption{
+				{ID: "a", Description: "one"},
+				{ID: "a", Description: "two"},
+			},
+		})
+		Expect(err).To(MatchError(ContainSubstring("duplicate option id")))
+	})
+})
+
+var _ = Describe("validateClassifierActivation", func() {
+	It("accepts a combined inference and score model", func() {
+		usecases := config.FLAG_CHAT | config.FLAG_SCORE
+		m := &wrappedModel{LLMConfig: &config.ModelConfig{KnownUsecases: &usecases}}
+		Expect(validateClassifierActivation(m, classifierTestConfig(0.4, nil))).To(Succeed())
+	})
+
+	It("rejects an active classifier when the model does not declare score", func() {
+		usecases := config.FLAG_CHAT
+		m := &wrappedModel{LLMConfig: &config.ModelConfig{KnownUsecases: &usecases}}
+		Expect(validateClassifierActivation(m, classifierTestConfig(0.4, nil))).To(MatchError(ContainSubstring("known_usecases")))
+	})
+
+	It("rejects a router config as the concrete scoring model", func() {
+		usecases := config.FLAG_SCORE
+		m := &wrappedModel{LLMConfig: &config.ModelConfig{
+			KnownUsecases: &usecases,
+			Router:        config.RouterConfig{Candidates: []config.RouterCandidate{{Model: "target"}}},
+		}}
+		Expect(validateClassifierActivation(m, classifierTestConfig(0.4, nil))).To(MatchError(ContainSubstring("concrete")))
+	})
+
+	It("allows disabling classification without score support", func() {
+		disabled := false
+		m := &wrappedModel{LLMConfig: &config.ModelConfig{}}
+		Expect(validateClassifierActivation(m, &types.ClassifierConfig{Enabled: &disabled})).To(Succeed())
+	})
+})
+
+var _ = Describe("resolveClassifier", func() {
+	It("uses the session config when no override is present", func() {
+		sess := classifierTestConfig(0, nil)
+		Expect(resolveClassifier(sess, nil)).To(BeIdenticalTo(sess))
+		Expect(resolveClassifier(sess, &types.ResponseCreateParams{})).To(BeIdenticalTo(sess))
+	})
+
+	It("replaces the whole config when the response overrides it", func() {
+		sess := classifierTestConfig(0, nil)
+		disabled := false
+		over := &types.ClassifierConfig{Enabled: &disabled}
+		got := resolveClassifier(sess, &types.ResponseCreateParams{LocalAIClassifier: over})
+		Expect(got).To(BeIdenticalTo(over))
+		Expect(got.Active()).To(BeFalse())
+	})
+})
+
+var _ = Describe("trimClassifierHistory", func() {
+	history := schema.Messages{
+		{Role: "system", StringContent: "sys"},
+		{Role: "user", StringContent: "one"},
+		{Role: "assistant", StringContent: "two"},
+		{Role: "user", StringContent: "three"},
+		{Role: "assistant", StringContent: "four"},
+		{Role: "user", StringContent: "five"},
+	}
+
+	It("keeps only the latest user message by default", func() {
+		// Earlier turns echo option names (canned replies) and empirically
+		// dominate small scoring models, so the default is user-turn-only.
+		got := trimClassifierHistory(history, 0)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].StringContent).To(Equal("five"))
+	})
+
+	It("keeps only the latest user message for -1", func() {
+		got := trimClassifierHistory(history, -1)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].StringContent).To(Equal("five"))
+	})
+
+	It("honors an explicit cap", func() {
+		got := trimClassifierHistory(history, 2)
+		Expect(got).To(HaveLen(2))
+		Expect(got[0].StringContent).To(Equal("four"))
+	})
+})
+
+var _ = Describe("mentionsAnyName", func() {
+	It("matches case-insensitive whole words in any position", func() {
+		Expect(mentionsAnyName("Drone, go up", []string{"drone"})).To(BeTrue())
+		Expect(mentionsAnyName("go up drone", []string{"drone"})).To(BeTrue())
+		Expect(mentionsAnyName("go up", []string{"drone"})).To(BeFalse())
+		// Whole-word: no substring matches.
+		Expect(mentionsAnyName("I like drones", []string{"drone"})).To(BeFalse())
+		// Multiple aliases and multi-word names.
+		Expect(mentionsAnyName("hey quadcopter rise", []string{"drone", "quadcopter"})).To(BeTrue())
+		Expect(mentionsAnyName("okay drone go", []string{"okay drone"})).To(BeTrue())
+	})
+})
+
+var _ = Describe("classifierProbe", func() {
+	It("renders a single user message verbatim", func() {
+		probe := classifierProbe(schema.Messages{{Role: "user", Content: "fly forward"}})
+		Expect(probe.Prompt).To(Equal("fly forward\n"))
+		Expect(probe.Messages).To(Equal([]string{"fly forward"}))
+	})
+
+	It("role-labels multi-message histories and skips text-less items", func() {
+		probe := classifierProbe(schema.Messages{
+			{Role: "user", Content: "go up"},
+			{Role: "assistant", Content: "Going up."},
+			{Role: "assistant"}, // tool-call item: no text
+			{Role: "tool", Content: "ok: moved"},
+			{Role: "user", Content: "fly forward"},
+		})
+		Expect(probe.Messages).To(Equal([]string{
+			"User: go up",
+			"Assistant: Going up.",
+			"Tool: ok: moved",
+			"User: fly forward",
+		}))
+	})
+})
+
+var _ = Describe("classifierRespond", func() {
+	It("emits the winning option's canned reply and tool call", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.9},
+			{Label: "greeting", Score: 0.1},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, classifierTestConfig(0.35, nil), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.classifyCalls).To(Equal(1))
+		// System instructions stay out of the scoring prompt.
+		for _, msg := range m.lastMessages {
+			Expect(msg.Role).ToNot(Equal("system"))
+		}
+
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].ChosenID).To(Equal("up"))
+		Expect(results[0].Fallback).To(BeEmpty())
+		Expect(results[0].Scores).To(HaveLen(2))
+		Expect(results[0].Scores[0].Score).To(BeNumerically("~", 0.9))
+
+		// Canned reply as text (text-only modality), canned tool call after it.
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(Equal(1))
+		Expect(t.countEvents(types.ServerEventTypeResponseFunctionCallArgumentsDone)).To(Equal(1))
+		var fcArgs string
+		for _, e := range t.events {
+			if done, ok := e.(types.ResponseFunctionCallArgumentsDoneEvent); ok {
+				fcArgs = done.Arguments
+			}
+		}
+		Expect(fcArgs).To(MatchJSON(`{"direction":"up"}`))
+		// Assistant reply + function_call item recorded in the conversation.
+		Expect(conv.Items).To(HaveLen(2))
+		Expect(conv.Items[0].Assistant).ToNot(BeNil())
+		Expect(conv.Items[1].FunctionCall).ToNot(BeNil())
+		Expect(conv.Items[1].FunctionCall.Name).To(Equal("move"))
+	})
+
+	It("drops unaddressed turns without scoring when the address gate is on", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{{Label: "up", Score: 0.99}}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-unaddressed"}
+		cc := classifierTestConfig(0.35, nil)
+		cc.Address = &types.ClassifierAddress{Names: []string{"drone"}}
+		history := schema.Messages{
+			{Role: "user", StringContent: "go up", Content: "go up"},
+		}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, history, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.classifyCalls).To(BeZero(), "unaddressed turns must not be scored")
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].Scores).To(BeEmpty())
+		Expect(results[0].Fallback).To(Equal(types.ClassifierNotAddressed))
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(BeZero(), "ignore mode must stay silent")
+	})
+
+	It("scores turns that address the assistant by name", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.9},
+			{Label: "greeting", Score: 0.1},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-addressed"}
+		cc := classifierTestConfig(0.35, nil)
+		cc.Address = &types.ClassifierAddress{Names: []string{"drone"}}
+		history := schema.Messages{
+			{Role: "user", StringContent: "Drone, go up", Content: "Drone, go up"},
+		}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, history, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.classifyCalls).To(Equal(1))
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].ChosenID).To(Equal("up"))
+	})
+
+	It("speaks the address reply for unaddressed turns in reply mode", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-unaddressed-reply"}
+		cc := classifierTestConfig(0.35, nil)
+		cc.Address = &types.ClassifierAddress{Names: []string{"drone"}, Mode: types.ClassifierAddressReply, Reply: "Call me Drone."}
+		history := schema.Messages{
+			{Role: "user", StringContent: "go up", Content: "go up"},
+		}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, history, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.classifyCalls).To(BeZero())
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(Equal(1))
+	})
+
+	It("applies the fallback without scoring when the turn has no words", func() {
+		// A VAD-committed turn whose transcript is empty must not be
+		// scored: an empty prompt yields a confidently arbitrary winner.
+		m := &fakeModel{classifyScores: []router.LabelScore{{Label: "up", Score: 0.99}}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-empty"}
+		history := schema.Messages{
+			{Role: "system", StringContent: "instructions", Content: "instructions"},
+			{Role: "user", StringContent: "", Content: ""},
+		}
+		cc := classifierTestConfig(0.35, &types.ClassifierFallback{Mode: types.ClassifierFallbackReply, Reply: "Say again?"})
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, history, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.classifyCalls).To(BeZero(), "an empty turn must not be scored")
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].Scores).To(BeEmpty())
+		Expect(results[0].ChosenID).To(BeEmpty())
+		Expect(results[0].Fallback).To(Equal(types.ClassifierFallbackReply))
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(Equal(1))
+		Expect(t.countEvents(types.ServerEventTypeResponseFunctionCallArgumentsDone)).To(BeZero())
+	})
+
+	It("falls through to generation for a word-less turn when the fallback is generate", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{{Label: "up", Score: 0.99}}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-empty-gen"}
+		history := schema.Messages{
+			{Role: "user", StringContent: "", Content: ""},
+		}
+		cc := classifierTestConfig(0.35, &types.ClassifierFallback{Mode: types.ClassifierFallbackGenerate})
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, history, nil, 0)
+
+		Expect(handled).To(BeFalse())
+		Expect(m.classifyCalls).To(BeZero())
+		Expect(classifierResultEvents(t)).To(BeEmpty())
+	})
+
+	It("speaks the fallback reply when no option clears the threshold", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.3},
+			{Label: "greeting", Score: 0.3},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+		cc := classifierTestConfig(0.6, &types.ClassifierFallback{Mode: types.ClassifierFallbackReply, Reply: "Say again?"})
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].ChosenID).To(BeEmpty())
+		Expect(results[0].Fallback).To(Equal(types.ClassifierFallbackReply))
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(Equal(1))
+		Expect(t.countEvents(types.ServerEventTypeResponseFunctionCallArgumentsDone)).To(BeZero())
+	})
+
+	It("completes with no output for the none fallback", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.3},
+			{Label: "greeting", Score: 0.3},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, classifierTestConfig(0.6, nil), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(r.outcome).ToNot(Equal(outcomeFailed))
+		Expect(conv.Items).To(BeEmpty())
+		Expect(t.countEvents(types.ServerEventTypeResponseOutputTextDone)).To(BeZero())
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].Fallback).To(Equal(types.ClassifierFallbackNone))
+	})
+
+	It("falls through to generation for the generate fallback", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.3},
+			{Label: "greeting", Score: 0.3},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+		cc := classifierTestConfig(0.6, &types.ClassifierFallback{Mode: types.ClassifierFallbackGenerate})
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeFalse())
+		// The distribution is still reported before falling through.
+		Expect(classifierResultEvents(t)).To(HaveLen(1))
+	})
+
+	It("fails the response when scoring errors without a generate fallback", func() {
+		m := &fakeModel{classifyErr: fmt.Errorf("backend exploded")}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, classifierTestConfig(0.35, nil), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(r.outcome).To(Equal(outcomeFailed))
+		Expect(t.countEvents(types.ServerEventTypeError)).To(Equal(1))
+	})
+
+	It("falls through to generation when scoring errors and fallback is generate", func() {
+		m := &fakeModel{classifyErr: fmt.Errorf("backend exploded")}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+		cc := classifierTestConfig(0.35, &types.ClassifierFallback{Mode: types.ClassifierFallbackGenerate})
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeFalse())
+		Expect(r.outcome).ToNot(Equal(outcomeFailed))
+	})
+
+	It("records a cancelled outcome when barge-in fires during scoring", func() {
+		m := &fakeModel{classifyScores: []router.LabelScore{
+			{Label: "up", Score: 0.9},
+			{Label: "greeting", Score: 0.1},
+		}}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+
+		handled := classifierRespond(ctx, session, conv, t, r, classifierTestConfig(0.35, nil), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(r.outcome).To(Equal(outcomeCancelled))
+		Expect(conv.Items).To(BeEmpty())
+	})
+
+	It("skips to generation when there is nothing scorable", func() {
+		m := &fakeModel{}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp1"}
+		systemOnly := schema.Messages{{Role: "system", StringContent: "instructions"}}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, classifierTestConfig(0.35, nil), systemOnly, nil, 0)
+
+		Expect(handled).To(BeFalse())
+		Expect(m.classifyCalls).To(BeZero())
+	})
+})
+
+// slottedTestConfig is classifierTestConfig with the winning option's tool
+// carrying argument slots (the hybrid classify-then-complete path).
+func slottedTestConfig(threshold float64, fallback *types.ClassifierFallback, defaults bool) *types.ClassifierConfig {
+	slots := []types.ClassifierSlot{
+		{Name: "distance", Type: types.ClassifierSlotNumber},
+		{Name: "units", Type: types.ClassifierSlotEnum, Values: []string{"m", "meters", "ft", "feet"}, Hint: "assume m when the user gives no units"},
+	}
+	if defaults {
+		slots[0].Default = "1"
+		slots[1].Default = "m"
+	}
+	return &types.ClassifierConfig{
+		Threshold: threshold,
+		Fallback:  fallback,
+		Options: []types.ClassifierOption{
+			{
+				ID:          "up",
+				Description: "the user asks the drone to fly up",
+				Reply:       "Going up {{distance}} {{units}}.",
+				Tool: &types.ClassifierTool{
+					Name:      "move",
+					Arguments: json.RawMessage(`{"direction":"up","distance":"{{distance}}","units":"{{units}}"}`),
+					Slots:     slots,
+				},
+			},
+			{ID: "greeting", Description: "the user greets the assistant", Reply: "Hello."},
+		},
+	}
+}
+
+var _ = Describe("slotFillGrammar", func() {
+	It("pins the field skeleton and frees only the slot values", func() {
+		g := slotFillGrammar([]types.ClassifierSlot{
+			{Name: "distance", Type: types.ClassifierSlotNumber},
+			{Name: "units", Type: types.ClassifierSlotEnum, Values: []string{"m", "ft"}},
+		})
+		Expect(g).To(ContainSubstring(`root ::= slot0 ", \"units\": " slot1 "}"`))
+		Expect(g).To(ContainSubstring("slot0 ::= num"))
+		Expect(g).To(ContainSubstring(`slot1 ::= "\"m\"" | "\"ft\""`))
+		Expect(g).To(ContainSubstring("num ::="))
+	})
+
+	It("JSON-encodes enum values before embedding them in the grammar", func() {
+		g := slotFillGrammar([]types.ClassifierSlot{
+			{Name: "units", Type: types.ClassifierSlotEnum, Values: []string{"quoted\"value", "line\nbreak", `back\slash`}},
+		})
+		Expect(g).To(ContainSubstring(gbnfLiteral(`"quoted\"value"`)))
+		Expect(g).To(ContainSubstring(gbnfLiteral(`"line\nbreak"`)))
+		Expect(g).To(ContainSubstring(gbnfLiteral(`"back\\slash"`)))
+	})
+
+	It("budgets forced enum and field text by encoded length", func() {
+		short := []types.ClassifierSlot{{Name: "value", Type: types.ClassifierSlotEnum, Values: []string{"m"}}}
+		long := []types.ClassifierSlot{
+			{Name: "value", Type: types.ClassifierSlotEnum, Values: []string{strings.Repeat("long-value-", 20)}},
+			{Name: strings.Repeat("field", 20), Type: types.ClassifierSlotNumber},
+		}
+		Expect(slotFillMaxTokens(long)).To(BeNumerically(">", slotFillMaxTokens(short)+200))
+	})
+
+	It("emits a string rule only when needed", func() {
+		g := slotFillGrammar([]types.ClassifierSlot{{Name: "what", Type: types.ClassifierSlotString}})
+		Expect(g).To(ContainSubstring("slot0 ::= str"))
+		Expect(g).To(ContainSubstring("str ::="))
+		Expect(g).ToNot(ContainSubstring("num ::="))
+	})
+})
+
+var _ = Describe("parseSlotValues", func() {
+	slots := []types.ClassifierSlot{
+		{Name: "distance", Type: types.ClassifierSlotNumber},
+		{Name: "units", Type: types.ClassifierSlotEnum, Values: []string{"m", "ft"}},
+	}
+
+	It("extracts values from a grammar-shaped completion", func() {
+		values, err := parseSlotValues("up", "distance", `3.5, "units": "m"}`, slots)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(values).To(Equal(map[string]string{"distance": "3.5", "units": "m"}))
+	})
+
+	It("tolerates a completion missing the closing brace", func() {
+		values, err := parseSlotValues("up", "distance", `2, "units": "ft"`, slots)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(values["distance"]).To(Equal("2"))
+	})
+
+	It("rejects completions missing a slot", func() {
+		_, err := parseSlotValues("up", "distance", `3}`, slots)
+		Expect(err).To(MatchError(ContainSubstring(`missing "units"`)))
+	})
+})
+
+var _ = Describe("classifierPolicyDescription", func() {
+	It("passes plain options through", func() {
+		o := &types.ClassifierOption{Description: "plain"}
+		Expect(classifierPolicyDescription(o)).To(Equal("plain"))
+	})
+
+	It("appends slot declarations and hints", func() {
+		cc := slottedTestConfig(0, nil, false)
+		d := classifierPolicyDescription(&cc.Options[0])
+		Expect(d).To(ContainSubstring("route parameters:"))
+		Expect(d).To(ContainSubstring("distance (number)"))
+		Expect(d).To(ContainSubstring("units (one of: m, meters, ft, feet)"))
+		Expect(d).To(ContainSubstring("assume m when the user gives no units"))
+	})
+})
+
+var _ = Describe("classifierRespond slot filling", func() {
+	It("emits the filled tool arguments and reports them in the result event", func() {
+		m := &fakeModel{
+			classifyScores: []router.LabelScore{{Label: "up", Score: 0.9}, {Label: "greeting", Score: 0.1}},
+			fillArgs:       `{"direction":"up","distance":3,"units":"meters"}`,
+			fillValues:     map[string]string{"distance": "3", "units": "meters"},
+		}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-slots"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, slottedTestConfig(0.35, nil, false), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(m.fillCalls).To(Equal(1))
+		Expect(m.lastFillChosen.ID).To(Equal("up"))
+
+		results := classifierResultEvents(t)
+		Expect(results).To(HaveLen(1))
+		Expect(results[0].ChosenID).To(Equal("up"))
+		Expect(results[0].Arguments).To(MatchJSON(`{"direction":"up","distance":3,"units":"meters"}`))
+
+		var fcArgs string
+		for _, e := range t.events {
+			if done, ok := e.(types.ResponseFunctionCallArgumentsDoneEvent); ok {
+				fcArgs = done.Arguments
+			}
+		}
+		Expect(fcArgs).To(MatchJSON(`{"direction":"up","distance":3,"units":"meters"}`))
+	})
+
+	It("splices the filled values into a templated reply", func() {
+		m := &fakeModel{
+			classifyScores: []router.LabelScore{{Label: "up", Score: 0.9}, {Label: "greeting", Score: 0.1}},
+			fillArgs:       `{"direction":"up","distance":3,"units":"meters"}`,
+			fillValues:     map[string]string{"distance": "3", "units": "meters"},
+		}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-slot-reply"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, slottedTestConfig(0.35, nil, false), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(replyTexts(t)).To(ConsistOf("Going up 3 meters."))
+	})
+
+	It("recovers with slot defaults when filling fails", func() {
+		m := &fakeModel{
+			classifyScores: []router.LabelScore{{Label: "up", Score: 0.9}, {Label: "greeting", Score: 0.1}},
+			fillErr:        fmt.Errorf("backend unavailable"),
+		}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-slot-defaults"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, slottedTestConfig(0.35, nil, true), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		var fcArgs string
+		for _, e := range t.events {
+			if done, ok := e.(types.ResponseFunctionCallArgumentsDoneEvent); ok {
+				fcArgs = done.Arguments
+			}
+		}
+		Expect(fcArgs).To(MatchJSON(`{"direction":"up","distance":1,"units":"m"}`))
+		Expect(replyTexts(t)).To(ConsistOf("Going up 1 m."), "the default-recovery reply confirms the defaults")
+	})
+
+	It("fails the response when filling fails and a slot has no default", func() {
+		m := &fakeModel{
+			classifyScores: []router.LabelScore{{Label: "up", Score: 0.9}, {Label: "greeting", Score: 0.1}},
+			fillErr:        fmt.Errorf("backend unavailable"),
+		}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-slot-fail"}
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, slottedTestConfig(0.35, nil, false), classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeTrue())
+		Expect(r.outcome).To(Equal(outcomeFailed))
+		Expect(classifierResultEvents(t)).To(BeEmpty(), "no result event for a failed fill")
+	})
+
+	It("falls back to generation on fill failure in generate mode", func() {
+		m := &fakeModel{
+			classifyScores: []router.LabelScore{{Label: "up", Score: 0.9}, {Label: "greeting", Score: 0.1}},
+			fillErr:        fmt.Errorf("backend unavailable"),
+		}
+		session := classifierTestSession(m)
+		conv := &Conversation{}
+		t := &fakeTransport{}
+		r := &liveResponse{id: "resp-slot-genfb"}
+		cc := slottedTestConfig(0.35, &types.ClassifierFallback{Mode: types.ClassifierFallbackGenerate}, false)
+
+		handled := classifierRespond(context.Background(), session, conv, t, r, cc, classifierTestHistory, nil, 0)
+
+		Expect(handled).To(BeFalse(), "generate fallback lets the caller run generation")
+	})
+})

core/http/endpoints/openai/realtime_doubles_test.go

@@ -3,11 +3,13 @@ package openai
 import (
 	"context"
 	"strings"
+	"sync"
 
 	"github.com/mudler/LocalAI/core/backend"
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
 	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/router"
 	"github.com/mudler/LocalAI/pkg/grpc/proto"
 )
 
@@ -99,11 +101,81 @@ type fakeModel struct {
 	predictResp        backend.LLMResponse
 	predictErr         error
 
+	// ClassifyTurn scripting: classifyScores is returned as the option
+	// distribution (in option order); classifyErr fails the call.
+	// classifyCalls counts invocations and lastClassifyOptions records
+	// what the handler asked to score.
+	classifyScores      []router.LabelScore
+	classifyErr         error
+	classifyCalls       int
+	lastClassifyOptions []types.ClassifierOption
+
+	// FillToolArguments scripting: fillArgs/fillValues are returned
+	// verbatim; fillErr fails the call. fillCalls counts invocations and
+	// lastFillChosen records which option's slots the handler asked to
+	// fill.
+	fillArgs       string
+	fillValues     map[string]string
+	fillErr        error
+	fillCalls      int
+	lastFillChosen *types.ClassifierOption
+
+	// PrewarmClassifier runs on a background goroutine, so its recording
+	// is mutex-guarded; specs poll prewarmCalls with Eventually.
+	prewarmMu          sync.Mutex
+	prewarmCalls       int
+	lastPrewarmOptions []types.ClassifierOption
+
+	// VAD scripting: vadFn, when set, decides per call (specs vary the
+	// answer across ticks or record the request); otherwise
+	// vadSegments/vadErr answer every call.
+	vadFn       func(*schema.VADRequest) (*schema.VADResponse, error)
+	vadSegments []schema.VADSegment
+	vadErr      error
+
 	lastMessages schema.Messages
 }
 
-func (m *fakeModel) VAD(context.Context, *schema.VADRequest) (*schema.VADResponse, error) {
-	return nil, nil
+func (m *fakeModel) PrewarmClassifier(_ context.Context, options []types.ClassifierOption, _ string) {
+	m.prewarmMu.Lock()
+	defer m.prewarmMu.Unlock()
+	m.prewarmCalls++
+	m.lastPrewarmOptions = options
+}
+
+func (m *fakeModel) prewarmed() (int, []types.ClassifierOption) {
+	m.prewarmMu.Lock()
+	defer m.prewarmMu.Unlock()
+	return m.prewarmCalls, m.lastPrewarmOptions
+}
+
+func (m *fakeModel) FillToolArguments(_ context.Context, msgs schema.Messages, options []types.ClassifierOption, _ string, chosen *types.ClassifierOption) (string, map[string]string, error) {
+	m.fillCalls++
+	m.lastFillChosen = chosen
+	if m.fillErr != nil {
+		return "", nil, m.fillErr
+	}
+	return m.fillArgs, m.fillValues, nil
+}
+
+func (m *fakeModel) ClassifyTurn(_ context.Context, msgs schema.Messages, options []types.ClassifierOption, _ string) ([]router.LabelScore, error) {
+	m.classifyCalls++
+	m.lastClassifyOptions = options
+	m.lastMessages = msgs
+	if m.classifyErr != nil {
+		return nil, m.classifyErr
+	}
+	return m.classifyScores, nil
+}
+
+func (m *fakeModel) VAD(_ context.Context, req *schema.VADRequest) (*schema.VADResponse, error) {
+	if m.vadFn != nil {
+		return m.vadFn(req)
+	}
+	if m.vadErr != nil {
+		return nil, m.vadErr
+	}
+	return &schema.VADResponse{Segments: m.vadSegments}, nil
 }
 
 func (m *fakeModel) Transcribe(context.Context, string, string, bool, bool, string) (*schema.TranscriptionResult, error) {

core/http/endpoints/openai/realtime_model.go

@@ -7,6 +7,9 @@ import (
 	"encoding/hex"
 	"encoding/json"
 	"fmt"
+	"strings"
+	"sync"
+	"time"
 
 	"github.com/mudler/LocalAI/core/application"
 	"github.com/mudler/LocalAI/core/backend"
@@ -36,12 +39,39 @@ type wrappedModel struct {
 	LLMConfig            *config.ModelConfig
 	VADConfig            *config.ModelConfig
 	SoundDetectionConfig *config.ModelConfig
+	// ScoreConfig is the classifier-mode scoring model
+	// (pipeline.classifier.model). nil falls back to LLMConfig — with
+	// slot-based Score the same process serves scoring and generation
+	// and shares its prompt cache between them.
+	ScoreConfig *config.ModelConfig
 
 	appConfig   *config.ApplicationConfig
 	modelLoader *model.ModelLoader
 	confLoader  *config.ModelConfigLoader
 	evaluator   *templates.Evaluator
 
+	// Classifier-mode memo: constructing a ScoreClassifier parses the
+	// scoring model's chat template, so reuse it while the option set is
+	// unchanged. Guarded by a mutex only because session.update can swap
+	// options while a response is in flight.
+	classifierMu   sync.Mutex
+	classifier     *router.ScoreClassifier
+	classifierKey  string
+	classifierWarn sync.Once
+	// Prewarm FIFO: a single worker drains warms in registration order —
+	// a plain mutex proved unfair under a burst of registrations (Go
+	// mutexes barge), running the most recently registered list last,
+	// long after the user's first command for it arrived. Pending
+	// duplicates coalesce (a connect-time barrage registers the same
+	// list several times), but completed warms are deliberately NOT
+	// memoized: a rewarm on a still-resident list costs one probe-sized
+	// decode, and on an evicted list it is exactly the re-prefill the
+	// next turn would otherwise pay in the foreground.
+	prewarmMu      sync.Mutex
+	prewarmQueue   []prewarmJob
+	prewarmPending map[string]bool
+	prewarmActive  bool
+
 	// Routing — populated by newModel when the application wires routing
 	// deps in. nil-safe: with classifierRegistry == nil the per-turn
 	// routing block in Predict is skipped, preserving today's "one LLM
@@ -90,6 +120,17 @@ func (m *transcriptOnlyModel) Predict(ctx context.Context, messages schema.Messa
 	return nil, fmt.Errorf("predict operation not supported in transcript-only mode")
 }
 
+func (m *transcriptOnlyModel) ClassifyTurn(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string) ([]router.LabelScore, error) {
+	return nil, fmt.Errorf("classifier mode not supported in transcript-only mode")
+}
+
+func (m *transcriptOnlyModel) FillToolArguments(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string, chosen *types.ClassifierOption) (string, map[string]string, error) {
+	return "", nil, fmt.Errorf("classifier mode not supported in transcript-only mode")
+}
+
+func (m *transcriptOnlyModel) PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string) {
+}
+
 func (m *transcriptOnlyModel) TTS(ctx context.Context, text, voice, language string) (string, *proto.Result, error) {
 	return "", nil, fmt.Errorf("TTS not supported in transcript-only mode")
 }
@@ -369,14 +410,258 @@ func (m *wrappedModel) PredictConfig() *config.ModelConfig {
 	return m.LLMConfig
 }
 
+// scoreConfig resolves the classifier-mode scoring model: the explicit
+// pipeline.classifier.model when set, else the pipeline LLM.
+func (m *wrappedModel) scoreConfig() *config.ModelConfig {
+	if m.ScoreConfig != nil {
+		return m.ScoreConfig
+	}
+	return m.LLMConfig
+}
+
+// classifierFor returns a ScoreClassifier for the given option set,
+// reusing the previous one while options and normalization are unchanged
+// (construction parses the scoring model's chat template).
+func (m *wrappedModel) classifierFor(options []types.ClassifierOption, normalization string) (*router.ScoreClassifier, error) {
+	scoreCfg := m.scoreConfig()
+	if scoreCfg == nil || !scoreCfg.HasUsecases(config.FLAG_SCORE) {
+		return nil, fmt.Errorf("classifier: scoring model must include score in known_usecases")
+	}
+	switch normalization {
+	case "", router.ScoreNormalizationRaw, router.ScoreNormalizationMean:
+	default:
+		// NewScoreClassifier panics on unknown modes; session.update
+		// validation should have rejected this — fail soft anyway.
+		return nil, fmt.Errorf("classifier: unknown normalization %q", normalization)
+	}
+	if len(options) == 0 {
+		return nil, fmt.Errorf("classifier: no options to score")
+	}
+
+	var key strings.Builder
+	key.WriteString(normalization)
+	for _, o := range options {
+		key.WriteString("\x1f")
+		key.WriteString(o.ID)
+		key.WriteString("\x1e")
+		// The policy description includes slot declarations, so keying on
+		// it also invalidates the classifier when slots change.
+		key.WriteString(classifierPolicyDescription(&o))
+	}
+
+	m.classifierMu.Lock()
+	defer m.classifierMu.Unlock()
+	if m.classifier != nil && m.classifierKey == key.String() {
+		return m.classifier, nil
+	}
+
+	cfg := m.scoreConfig()
+	policies := make([]router.ScorePolicy, 0, len(options))
+	for _, o := range options {
+		if o.ID == "" || o.Description == "" {
+			// NewScoreClassifier panics on these; validation upstream
+			// should have caught them.
+			return nil, fmt.Errorf("classifier: option with empty id or description")
+		}
+		policies = append(policies, router.ScorePolicy{Label: o.ID, Description: classifierPolicyDescription(&o)})
+	}
+
+	opts := router.ScoreClassifierOptions{
+		// The memo cache stores only label sets — a hit would return an
+		// empty distribution and blind the localai.classifier.result
+		// event, so keep it off.
+		CacheCap:      0,
+		Normalization: normalization,
+	}
+	if m.routerDeps != nil && m.routerDeps.TokenCounter != nil && cfg.ContextSize != nil {
+		opts.TokenCounter = m.routerDeps.TokenCounter(cfg.Name)
+		opts.MaxContextTokens = *cfg.ContextSize
+	}
+	for i := range options {
+		if options[i].Tool != nil && len(options[i].Tool.Slots) > 0 {
+			reserve := slotFillContextReserve(&options[i])
+			if reserve > opts.CompletionReserveTokens {
+				opts.CompletionReserveTokens = reserve
+			}
+		}
+	}
+	if m.evaluator != nil {
+		if renderer := middleware.NewTemplateRenderer(m.evaluator, cfg); renderer != nil {
+			opts.PromptRenderer = renderer
+		} else {
+			m.classifierWarn.Do(func() {
+				xlog.Warn("realtime classifier: scoring model has no Go chat template; falling back to a generic ChatML envelope, which may be off-distribution",
+					"model", cfg.Name)
+			})
+		}
+	}
+	if st := middleware.PickAssistantTurnEnd(cfg.StopWords, cfg.TemplateConfig.ChatMessage); st != "" {
+		opts.StopToken = st
+	}
+
+	scorer := backend.NewScorer(m.modelLoader, *cfg, m.appConfig)
+	m.classifier = router.NewScoreClassifier(policies, scorer, opts)
+	m.classifierKey = key.String()
+	return m.classifier, nil
+}
+
+// PrewarmClassifier primes the scoring backend's prompt cache for a newly
+// registered option list so the first real turns don't pay the prefill.
+// One throwaway score prefills the new option-list prompt and declares the
+// per-turn probe boundary, leaving the backend a rewind point (a KV
+// checkpoint on hybrid/recurrent models, which cannot rewind arbitrarily)
+// at the stable prefix every subsequent turn reuses.
+// Best-effort: errors are logged, never surfaced.
+func (m *wrappedModel) PrewarmClassifier(ctx context.Context, options []types.ClassifierOption, normalization string) {
+	classifier, err := m.classifierFor(options, normalization)
+	if err != nil {
+		xlog.Debug("realtime classifier: prewarm skipped", "error", err)
+		return
+	}
+	m.classifierMu.Lock()
+	key := m.classifierKey
+	m.classifierMu.Unlock()
+
+	m.prewarmMu.Lock()
+	defer m.prewarmMu.Unlock()
+	if m.prewarmPending == nil {
+		m.prewarmPending = make(map[string]bool)
+	}
+	if m.prewarmPending[key] {
+		return
+	}
+	m.prewarmPending[key] = true
+	m.prewarmQueue = append(m.prewarmQueue, prewarmJob{classifier: classifier, key: key, options: len(options)})
+	if !m.prewarmActive {
+		m.prewarmActive = true
+		go m.prewarmWorker()
+	}
+}
+
+type prewarmJob struct {
+	classifier *router.ScoreClassifier
+	key        string
+	options    int
+}
+
+// prewarmWorker drains queued warms one at a time, in order. One
+// throwaway score per list is enough: the scoring call itself plants the
+// backend's reuse point at the stable-prefix boundary it declares, so
+// the real turns that follow restore from it no matter how their probe
+// differs. The worker exits when the queue drains and restarts on the
+// next registration.
+func (m *wrappedModel) prewarmWorker() {
+	for {
+		m.prewarmMu.Lock()
+		if len(m.prewarmQueue) == 0 {
+			m.prewarmActive = false
+			m.prewarmMu.Unlock()
+			return
+		}
+		job := m.prewarmQueue[0]
+		m.prewarmQueue = m.prewarmQueue[1:]
+		m.prewarmMu.Unlock()
+
+		start := time.Now()
+		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
+		const probe = "warmup"
+		_, err := job.classifier.Classify(ctx, router.Probe{Prompt: probe, Messages: []string{probe}})
+		cancel()
+		if err != nil {
+			xlog.Warn("realtime classifier: prewarm scoring failed", "error", err)
+		} else {
+			xlog.Debug("realtime classifier: prewarmed scoring prompt cache",
+				"options", job.options, "latency_ms", time.Since(start).Milliseconds())
+		}
+		m.prewarmMu.Lock()
+		delete(m.prewarmPending, job.key)
+		m.prewarmMu.Unlock()
+	}
+}
+
+func (m *wrappedModel) ClassifyTurn(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string) ([]router.LabelScore, error) {
+	classifier, err := m.classifierFor(options, normalization)
+	if err != nil {
+		return nil, err
+	}
+	decision, err := classifier.Classify(ctx, classifierProbe(messages))
+	if err != nil {
+		return nil, err
+	}
+	// LabelScores is in policy-declaration order, which mirrors option
+	// order by construction.
+	if len(decision.LabelScores) != len(options) {
+		return nil, fmt.Errorf("classifier: got %d scores for %d options", len(decision.LabelScores), len(options))
+	}
+	return decision.LabelScores, nil
+}
+
+// FillToolArguments runs the hybrid slot-fill completion: the exact prompt
+// the classifier scored (rendered by the same, cached ScoreClassifier — so
+// the backend's prompt cache is warm) continued by the chosen route JSON
+// re-opened at its first slot, with a grammar pinning everything but the
+// slot values. Deterministic (temperature 0), a couple dozen tokens at
+// most.
+func (m *wrappedModel) FillToolArguments(ctx context.Context, messages schema.Messages, options []types.ClassifierOption, normalization string, chosen *types.ClassifierOption) (string, map[string]string, error) {
+	if chosen == nil || chosen.Tool == nil || len(chosen.Tool.Slots) == 0 {
+		return "", nil, fmt.Errorf("classifier: option has no slots to fill")
+	}
+	slots := chosen.Tool.Slots
+	classifier, err := m.classifierFor(options, normalization)
+	if err != nil {
+		return "", nil, err
+	}
+	prompt, err := classifier.SlotFillPrompt(classifierProbe(messages), chosen.ID, slots[0].Name)
+	if err != nil {
+		return "", nil, err
+	}
+
+	// The scoring config, narrowed to a deterministic constrained
+	// completion. The completion usecase must be declared alongside score
+	// — bootstrap-style configs use known_usecases: [chat, completion,
+	// score].
+	cfg := *m.scoreConfig()
+	if !cfg.HasUsecases(config.FLAG_COMPLETION) {
+		return "", nil, fmt.Errorf("classifier: slot filling requires completion in the scoring model's known_usecases")
+	}
+	cfg.Grammar = slotFillGrammar(slots)
+	maxTokens := slotFillMaxTokens(slots)
+	temperature := 0.0
+	cfg.Maxtokens = &maxTokens
+	cfg.Temperature = &temperature
+
+	fn, err := backend.ModelInference(ctx, prompt, nil, nil, nil, nil, m.modelLoader, &cfg, m.confLoader, m.appConfig, nil, "", "", nil, nil, nil, nil)
+	if err != nil {
+		return "", nil, fmt.Errorf("classifier: slot fill inference: %w", err)
+	}
+	resp, err := fn()
+	if err != nil {
+		return "", nil, fmt.Errorf("classifier: slot fill inference: %w", err)
+	}
+	values, err := parseSlotValues(chosen.ID, slots[0].Name, resp.Response, slots)
+	if err != nil {
+		return "", nil, err
+	}
+	args, err := chosen.Tool.SpliceArguments(values)
+	if err != nil {
+		return "", nil, err
+	}
+	return args, values, nil
+}
+
 func (m *wrappedModel) Warmup(ctx context.Context) error {
-	_, err := backend.PreloadStages(ctx, m.modelLoader, m.appConfig, []backend.PreloadStage{
+	stages := []backend.PreloadStage{
 		{Role: "vad", Cfg: m.VADConfig},
 		{Role: "transcription", Cfg: m.TranscriptionConfig},
 		{Role: "llm", Cfg: m.LLMConfig},
 		{Role: "tts", Cfg: m.TTSConfig},
 		{Role: "sound_detection", Cfg: m.SoundDetectionConfig},
-	})
+	}
+	// The scoring model is a separate stage only when it isn't the LLM.
+	if m.ScoreConfig != nil && m.ScoreConfig != m.LLMConfig {
+		stages = append(stages, backend.PreloadStage{Role: "classifier", Cfg: m.ScoreConfig})
+	}
+	_, err := backend.PreloadStages(ctx, m.modelLoader, m.appConfig, stages)
 	return err
 }
 
@@ -456,11 +741,11 @@ func modelSoundDetection(ctx context.Context, ml *model.ModelLoader, appConfig *
 // config named by pipeline.sound_detection. Returns (nil, nil) when no model
 // is configured so sound detection stays additive and never blocks session
 // setup.
-func loadSoundDetectionConfig(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader) (*config.ModelConfig, error) {
+func loadSoundDetectionConfig(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig) (*config.ModelConfig, error) {
 	if pipeline.SoundDetection == "" {
 		return nil, nil
 	}
-	cfg, err := cl.LoadResolvedModelConfig(pipeline.SoundDetection, ml.ModelPath)
+	cfg, err := cl.LoadResolvedModelConfig(pipeline.SoundDetection, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 		return nil, fmt.Errorf("failed to load sound detection config: %w", err)
 	}
@@ -471,7 +756,7 @@ func loadSoundDetectionConfig(pipeline *config.Pipeline, cl *config.ModelConfigL
 }
 
 func newTranscriptionOnlyModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig) (Model, *config.ModelConfig, error) {
-	cfgVAD, err := cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath)
+	cfgVAD, err := cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -481,7 +766,7 @@ func newTranscriptionOnlyModel(pipeline *config.Pipeline, cl *config.ModelConfig
 		return nil, nil, fmt.Errorf("failed to validate config: %w", err)
 	}
 
-	cfgSST, err := cl.LoadResolvedModelConfig(pipeline.Transcription, ml.ModelPath)
+	cfgSST, err := cl.LoadResolvedModelConfig(pipeline.Transcription, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -491,7 +776,7 @@ func newTranscriptionOnlyModel(pipeline *config.Pipeline, cl *config.ModelConfig
 		return nil, nil, fmt.Errorf("failed to validate config: %w", err)
 	}
 
-	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml)
+	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml, appConfig)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -513,7 +798,7 @@ func newTranscriptionOnlyModel(pipeline *config.Pipeline, cl *config.ModelConfig
 // speech) and is driven by client-side windowing (turn_detection none +
 // input_audio_buffer.commit) rather than the voice VAD loop.
 func newSoundDetectionOnlyModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig) (Model, error) {
-	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml)
+	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml, appConfig)
 	if err != nil {
 		return nil, err
 	}
@@ -574,7 +859,7 @@ func buildRealtimeRoutingContext(a *application.Application, sessionID string) *
 func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig, evaluator *templates.Evaluator, routing *RealtimeRoutingContext) (Model, error) {
 	xlog.Debug("Creating new model pipeline model", "pipeline", pipeline)
 
-	cfgVAD, err := cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath)
+	cfgVAD, err := cl.LoadResolvedModelConfig(pipeline.VAD, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -585,7 +870,7 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model
 	}
 
 	// TODO: Do we always need a transcription model? It can be disabled. Note that any-to-any instruction following models don't transcribe as such, so if transcription is required it is a separate process
-	cfgSST, err := cl.LoadResolvedModelConfig(pipeline.Transcription, ml.ModelPath)
+	cfgSST, err := cl.LoadResolvedModelConfig(pipeline.Transcription, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -617,7 +902,7 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model
 	xlog.Debug("Loading a wrapped model")
 
 	// Otherwise we want to return a wrapped model, which is a "virtual" model that re-uses other models to perform operations
-	cfgLLM, err := cl.LoadResolvedModelConfig(pipeline.LLM, ml.ModelPath)
+	cfgLLM, err := cl.LoadResolvedModelConfig(pipeline.LLM, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -632,7 +917,7 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model
 	applyPipelineReasoning(cfgLLM, *pipeline)
 	applyPipelineThinking(cfgLLM, *pipeline)
 
-	cfgTTS, err := cl.LoadResolvedModelConfig(pipeline.TTS, ml.ModelPath)
+	cfgTTS, err := cl.LoadResolvedModelConfig(pipeline.TTS, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 
 		return nil, fmt.Errorf("failed to load backend config: %w", err)
@@ -642,17 +927,51 @@ func newModel(pipeline *config.Pipeline, cl *config.ModelConfigLoader, ml *model
 		return nil, fmt.Errorf("failed to validate config: %w", err)
 	}
 
-	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml)
+	cfgSound, err := loadSoundDetectionConfig(pipeline, cl, ml, appConfig)
 	if err != nil {
 		return nil, err
 	}
 
+	// Classifier mode scores on its own model config when one is named;
+	// otherwise ClassifyTurn falls back to the LLM config at call time
+	// (so a client can enable classification via session.update even
+	// when the pipeline block is absent).
+	var cfgScore *config.ModelConfig
+	if pipeline.Classifier != nil && pipeline.Classifier.Model != "" {
+		cfgScore, err = cl.LoadResolvedModelConfig(pipeline.Classifier.Model, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
+		if err != nil {
+			return nil, fmt.Errorf("failed to load classifier scoring config: %w", err)
+		}
+		if valid, err := cfgScore.Validate(); !valid {
+			return nil, fmt.Errorf("failed to validate classifier scoring config: %w", err)
+		}
+		if !cfgScore.HasUsecases(config.FLAG_SCORE) {
+			return nil, fmt.Errorf("pipeline classifier: scoring model %q must declare known_usecases: [score]", cfgScore.Name)
+		}
+	}
+	if pipeline.Classifier != nil && pipeline.Classifier.Enabled {
+		effectiveScore := cfgScore
+		if effectiveScore == nil {
+			effectiveScore = cfgLLM
+		}
+		if effectiveScore.HasRouter() {
+			// A router model has no concrete backend to score on — the
+			// per-turn routing decision happens at Predict time, after
+			// classification would already have run.
+			return nil, fmt.Errorf("pipeline classifier: llm %q is a router model; set pipeline.classifier.model to a concrete scoring model", cfgLLM.Name)
+		}
+		if !effectiveScore.HasUsecases(config.FLAG_SCORE) {
+			return nil, fmt.Errorf("pipeline classifier: scoring model %q must declare known_usecases: [score]", effectiveScore.Name)
+		}
+	}
+
 	wm := &wrappedModel{
 		TTSConfig:            cfgTTS,
 		TranscriptionConfig:  cfgSST,
 		LLMConfig:            cfgLLM,
 		VADConfig:            cfgVAD,
 		SoundDetectionConfig: cfgSound,
+		ScoreConfig:          cfgScore,
 
 		confLoader:  cl,
 		modelLoader: ml,

core/http/endpoints/openai/realtime_semantic_vad.go

@@ -96,6 +96,17 @@ func newLiveTurnState(session *Session, transport Transport) *liveTurnState {
 
 func (l *liveTurnState) open() bool { return l.live != nil }
 
+// rebase shifts the turn's buffer-relative cursors after the retention trim
+// dropped trimmedSec seconds off the buffer head: fed16k indexes the
+// resampled (16 kHz) buffer, eouAtSec the buffer clock. Both floor at zero —
+// a position inside the dropped head is more than maxTurnBufferSec old, and
+// for eouAtSec zero already means "no EOU this turn", which is the right
+// reading for a token that stale.
+func (l *liveTurnState) rebase(trimmedSec float64) {
+	l.fed16k = max(0, l.fed16k-int(trimmedSec*localSampleRate))
+	l.eouAtSec = max(0, l.eouAtSec-trimmedSec)
+}
+
 // openTurn starts the turn's live stream under the caller-supplied item id. A
 // failure (most commonly the backend's typed "live transcription unsupported"
 // signal) degrades the whole session to silence-only detection — warned once,

core/http/endpoints/openai/realtime_turncoord.go

@@ -58,6 +58,14 @@ type turnSink struct {
 	commitAudioLength  float64                           // for finishTurn (flush tail)
 	commitRetranscribe bool                              // gated batch is authoritative
 	commitGated        *schema.TranscriptionResult       // retranscribe batch decode
+
+	// lastSpeechEndSec is where speech last ended this turn, in whole-buffer
+	// seconds (audioLength while the newest segment is still open). It
+	// outlives the segments scrolling out of the VAD scan clip, so the
+	// silence-outran-the-window commit still has a speech end to report.
+	// Zeroed whenever the turn leaves Speaking; rebased by the retention
+	// trim.
+	lastSpeechEndSec float64
 }
 
 func newTurnSink(session *Session, conv *Conversation, t Transport, lts *liveTurnState, vadContext context.Context, startTime time.Time) *turnSink {

core/http/endpoints/openai/realtime_vad_tick_test.go

@@ -0,0 +1,203 @@
+package openai
+
+import (
+	"context"
+	"errors"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/turncoord"
+	"github.com/mudler/LocalAI/core/schema"
+)
+
+// vadTick specs drive one synchronous turn-detection inspection at a time
+// (no ticker), the same way classifySoundWindow's specs drive the
+// sound-detection loop. The fake VAD answers in the coordinates of the audio
+// it is HANDED — i.e. scan-clip coordinates once the buffer outgrows the
+// window — exactly like the real backend.
+var _ = Describe("vadTick", func() {
+	const rate = 16000 // InputSampleRate == localSampleRate: resample is a copy
+
+	// pcm returns sec seconds of silent 16-bit PCM; content is irrelevant to
+	// the scripted VAD.
+	pcm := func(sec float64) []byte {
+		return make([]byte, int(sec*rate)*2)
+	}
+	bufferSec := func(s *Session) float64 {
+		return float64(len(s.InputAudioBuffer)) / (rate * 2)
+	}
+
+	newHarness := func(td *types.TurnDetectionUnion, m *fakeModel) (*Session, *fakeTransport, *turnSink) {
+		session := &Session{
+			TranscriptionOnly:       true, // commit stops after the transcription events
+			TurnDetection:           td,
+			InputAudioTranscription: &types.AudioTranscription{},
+			ModelConfig:             &config.ModelConfig{},
+			ModelInterface:          m,
+			InputSampleRate:         rate,
+			respSink:                newResponseSink(),
+		}
+		tr := &fakeTransport{}
+		sink := newTurnSink(session, &Conversation{}, tr, newLiveTurnState(session, tr), context.Background(), time.Now())
+		return session, tr, sink
+	}
+	serverVad := &types.TurnDetectionUnion{ServerVad: &types.ServerVad{SilenceDurationMs: 500}}
+	semanticHigh := &types.TurnDetectionUnion{SemanticVad: &types.RealtimeSessionSemanticVad{Eagerness: "high"}}
+
+	speaking := func(sink *turnSink) bool {
+		_, ok := sink.coord.State().(turncoord.Speaking)
+		return ok
+	}
+
+	It("commits a normal short turn (extraction is behavior-neutral)", func() {
+		m := &fakeModel{
+			vadSegments:     []schema.VADSegment{{Start: 0.1, End: 0.6}},
+			transcribeFinal: &schema.TranscriptionResult{Text: "go up"},
+		}
+		session, tr, sink := newHarness(serverVad, m)
+		session.InputAudioBuffer = pcm(1.4) // under the 1.5s scan window: no clip
+
+		vadTick(sink, 0.5)
+
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferSpeechStarted)).To(Equal(1))
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferSpeechStopped)).To(Equal(1))
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferCommitted)).To(Equal(1))
+		Expect(session.InputAudioBuffer).To(BeEmpty(), "commit drops the whole inspected window")
+		Expect(speaking(sink)).To(BeFalse())
+
+		session.respSink.wait()
+		Expect(tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted)).To(Equal(1))
+	})
+
+	It("hands the VAD only the scan window and rebases its answer", func() {
+		var scanned []int
+		m := &fakeModel{
+			vadFn: func(req *schema.VADRequest) (*schema.VADResponse, error) {
+				scanned = append(scanned, len(req.Audio))
+				// Clip coordinates: speech ends 0.9s into the 1.5s window,
+				// leaving 0.6s of trailing silence > the 0.5s threshold.
+				return &schema.VADResponse{Segments: []schema.VADSegment{{Start: 0.2, End: 0.9}}}, nil
+			},
+			transcribeFinal: &schema.TranscriptionResult{Text: "clipped"},
+		}
+		session, tr, sink := newHarness(serverVad, m)
+		session.InputAudioBuffer = pcm(20)
+
+		vadTick(sink, 0.5)
+
+		Expect(scanned).To(Equal([]int{int(1.5 * rate)}), "server_vad window = silence 0.5s + 1s margin")
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferCommitted)).To(Equal(1),
+			"rebased segment end (18.5+0.9) leaves 0.6s trailing silence in buffer coordinates")
+	})
+
+	It("commits when trailing silence outruns the scan window instead of discarding the turn", func() {
+		call := 0
+		m := &fakeModel{
+			vadFn: func(req *schema.VADRequest) (*schema.VADResponse, error) {
+				call++
+				if call == 1 {
+					// Speech still running at the end of the inspected audio.
+					return &schema.VADResponse{Segments: []schema.VADSegment{{Start: 0.2, End: 0}}}, nil
+				}
+				// Later ticks: the (clipped) window is all silence.
+				return &schema.VADResponse{}, nil
+			},
+			transcribeFinal: &schema.TranscriptionResult{Text: "late silence"},
+		}
+		session, tr, sink := newHarness(serverVad, m)
+		session.InputAudioBuffer = pcm(1.4)
+		vadTick(sink, 0.5)
+		Expect(speaking(sink)).To(BeTrue())
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferCommitted)).To(BeZero())
+
+		session.InputAudioBuffer = append(session.InputAudioBuffer, pcm(2.6)...) // 4s total: clip is in effect
+		vadTick(sink, 0.5)
+
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferSpeechStopped)).To(Equal(1))
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferCommitted)).To(Equal(1))
+		Expect(session.InputAudioBuffer).To(BeEmpty())
+		Expect(speaking(sink)).To(BeFalse())
+		session.respSink.wait()
+		Expect(tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted)).To(Equal(1))
+	})
+
+	It("stays bounded when segments never stop (the noise-floor pathology)", func() {
+		var maxScan int
+		m := &fakeModel{
+			vadFn: func(req *schema.VADRequest) (*schema.VADResponse, error) {
+				if len(req.Audio) > maxScan {
+					maxScan = len(req.Audio)
+				}
+				return &schema.VADResponse{Segments: []schema.VADSegment{{Start: 0.1, End: 0}}}, nil
+			},
+		}
+		session, tr, sink := newHarness(serverVad, m)
+
+		for i := 0; i < 95; i++ {
+			session.InputAudioBuffer = append(session.InputAudioBuffer, pcm(1)...)
+			vadTick(sink, 0.5)
+		}
+
+		Expect(maxScan).To(Equal(int(1.5*rate)), "VAD never rescans more than the window")
+		Expect(bufferSec(session)).To(BeNumerically("<=", maxTurnBufferSec), "retention bound holds")
+		Expect(speaking(sink)).To(BeTrue(), "the turn is neither committed nor aborted")
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferSpeechStarted)).To(Equal(1))
+		Expect(tr.countEvents(types.ServerEventTypeInputAudioBufferCommitted)).To(BeZero())
+	})
+
+	It("keeps the live feed gapless across a retention trim", func() {
+		m := &fakeModel{
+			vadFn: func(req *schema.VADRequest) (*schema.VADResponse, error) {
+				return &schema.VADResponse{Segments: []schema.VADSegment{{Start: 0.1, End: 0}}}, nil
+			},
+		}
+		session, _, sink := newHarness(semanticHigh, m)
+		session.InputAudioBuffer = pcm(2)
+		vadTick(sink, 0.5) // opens the turn + live stream, feeds the onset audio
+		Expect(m.liveOpened).To(Equal(1))
+
+		session.InputAudioBuffer = append(session.InputAudioBuffer, pcm(89)...) // 91s: over the 90s bound
+		vadTick(sink, 0.5)
+
+		Expect(bufferSec(session)).To(BeNumerically("<=", maxTurnBufferSec))
+		total := 0
+		for _, chunk := range m.liveSession.fed {
+			total += len(chunk)
+		}
+		// Everything ever buffered minus the one held-back resample-edge
+		// sample: no gap (undercount) and no re-feed (overcount) across the
+		// trim's cursor rebase.
+		Expect(total).To(Equal(91*rate-1), "fed samples = all audio seen minus the held-back tail sample")
+	})
+
+	It("bounds memory when the VAD backend keeps failing", func() {
+		m := &fakeModel{vadErr: errors.New("backend down")}
+		session, tr, sink := newHarness(serverVad, m)
+		session.InputAudioBuffer = pcm(95)
+
+		vadTick(sink, 0.5)
+
+		Expect(bufferSec(session)).To(BeNumerically("<=", maxTurnBufferSec), "retention trim runs before the VAD call")
+		Expect(tr.countEvents(types.ServerEventTypeError)).To(Equal(1))
+	})
+})
+
+var _ = Describe("vadScanWindowSec", func() {
+	It("sizes from the silence the commit test must measure, plus the warm-up margin", func() {
+		Expect(vadScanWindowSec(nil, 0.5, nil)).To(Equal(1.5))
+		Expect(vadScanWindowSec(&types.RealtimeSessionSemanticVad{Eagerness: "high"}, 0.5, nil)).To(Equal(3.0))
+		Expect(vadScanWindowSec(&types.RealtimeSessionSemanticVad{Eagerness: "low"}, 0.5, nil)).To(Equal(9.0))
+	})
+
+	It("lets vad_window_sec widen but never narrow the window", func() {
+		cfg := &config.ModelConfig{}
+		cfg.Pipeline.TurnDetection.VadWindowSec = 10
+		Expect(vadScanWindowSec(nil, 0.5, cfg)).To(Equal(10.0))
+		cfg.Pipeline.TurnDetection.VadWindowSec = 0.2
+		Expect(vadScanWindowSec(nil, 0.5, cfg)).To(Equal(1.5), "values below the floor are ignored")
+	})
+})

core/http/endpoints/openai/realtime_voicegate.go

@@ -75,7 +75,7 @@ func newVoiceGate(
 
 	// Resolved like every other pipeline sub-model (one alias hop), so an
 	// aliased voice_recognition model gets its target's backend.
-	recCfg, err := cl.LoadResolvedModelConfig(cfg.Model, ml.ModelPath)
+	recCfg, err := cl.LoadResolvedModelConfig(cfg.Model, ml.ModelPath, appConfig.ToConfigLoaderOptions()...)
 	if err != nil {
 		return nil, fmt.Errorf("voice_recognition: failed to load model %q: %w", cfg.Model, err)
 	}
@@ -261,8 +261,10 @@ func (g *voiceGate) Authorize(ctx context.Context, wavPath string) (allowed bool
 
 // decide interprets an Authorize result against the gate's when-policy and the
 // session's prior verification state.
-//   proceed:      run the LLM response for this utterance.
-//   markVerified: record a successful first-utterance verification.
+//
+//	proceed:      run the LLM response for this utterance.
+//	markVerified: record a successful first-utterance verification.
+//
 // Note: when:first AND alreadyVerified is normally handled by the caller
 // skipping Authorize entirely; if it still reaches here, proceed is true.
 func (g *voiceGate) decide(alreadyVerified, allowed bool) (proceed, markVerified bool) {

core/http/endpoints/openai/types/classifier.go

@@ -0,0 +1,470 @@
+package types
+
+import (
+	"encoding/json"
+	"fmt"
+	"regexp"
+	"slices"
+	"strconv"
+	"strings"
+)
+
+// ClassifierConfig is a LocalAI extension to the Realtime API
+// (session.localai_classifier, response.localai_classifier): instead of
+// autoregressive generation, each user turn is prefill-scored against a
+// fixed option list via the Score primitive and the winning option's canned
+// reply / tool call is emitted. Built for hardware that can afford prefill
+// but not decode (e.g. a Raspberry Pi running a small LLM).
+type ClassifierConfig struct {
+	// Enabled is a pointer so a response-level override can force
+	// classification off for one response ({"enabled": false}) without
+	// replacing the session's option list. nil means "on when options
+	// exist".
+	Enabled *bool `json:"enabled,omitempty"`
+
+	// Options the user turn is scored against. Replaced wholesale by
+	// session.update / response.create, like tools.
+	Options []ClassifierOption `json:"options,omitempty"`
+
+	// Threshold is the softmax-probability floor the best option must
+	// clear; below it the fallback applies. 0 always picks the argmax.
+	Threshold float64 `json:"threshold,omitempty"`
+
+	// Normalization selects how candidate log-probs are compared before
+	// the softmax: "raw" (default, joint log-prob) or "mean"
+	// (length-normalized) — same semantics as the router's
+	// score_normalization.
+	Normalization string `json:"normalization,omitempty"`
+
+	// HistoryItems selects what gets scored. 0 (default) and -1 score
+	// only the latest user message; a positive N includes the trailing N
+	// conversation messages, role-labeled. Prior turns echo option names
+	// (the canned replies especially) and empirically dominate small
+	// scoring models — only opt into history with a scorer large enough
+	// to weigh it.
+	HistoryItems int `json:"history_items,omitempty"`
+
+	// Fallback controls what happens when no option clears the
+	// threshold. nil behaves like {"mode": "none"}.
+	Fallback *ClassifierFallback `json:"fallback,omitempty"`
+
+	// Address, when set, gates every turn on the assistant being
+	// addressed by name ("Drone go up", not just "go up") — the
+	// wake-word pattern. The check is a deterministic word match on the
+	// transcript: scoring cannot do it (a 1.2B scorer rates "go up" as
+	// addressed=1.0 even with a dedicated addressing stage) and matching
+	// is free, so unaddressed ambient speech skips scoring entirely.
+	Address *ClassifierAddress `json:"address,omitempty"`
+}
+
+// ClassifierAddress configures name-gating for classifier mode.
+type ClassifierAddress struct {
+	// Names that count as addressing the assistant, matched as
+	// case-insensitive whole words against the latest user turn.
+	Names []string `json:"names"`
+
+	// Mode when the turn does not mention a name: "ignore" (default —
+	// the response completes silently, the right behavior for ambient
+	// conversation) or "reply" (speak Reply).
+	Mode string `json:"mode,omitempty"`
+
+	// Reply spoken in "reply" mode.
+	Reply string `json:"reply,omitempty"`
+}
+
+// Address gate modes.
+const (
+	ClassifierAddressIgnore = "ignore"
+	ClassifierAddressReply  = "reply"
+)
+
+// ClassifierNotAddressed is the ClassifierResultEvent.Fallback value for
+// turns dropped by the address gate. It is an event-only value — the
+// config fallback modes stay none|reply|generate.
+const ClassifierNotAddressed = "not_addressed"
+
+// AddressMode returns the effective address-gate mode.
+func (a *ClassifierAddress) AddressMode() string {
+	if a == nil || a.Mode == "" {
+		return ClassifierAddressIgnore
+	}
+	return a.Mode
+}
+
+// ClassifierOption is one selectable intent: what to match on
+// (Description), what to say when chosen (Reply) and, optionally, a canned
+// tool call the client executes.
+type ClassifierOption struct {
+	// ID identifies the option in results and doubles as the scored
+	// route label, so keep it short — its tokens are what the model
+	// actually scores.
+	ID string `json:"id"`
+
+	// Description tells the model when the option applies (e.g. "the
+	// user asks the drone to move or fly up/higher"). It goes into the
+	// classification system prompt.
+	Description string `json:"description"`
+
+	// Reply is the canned assistant reply spoken/emitted when the
+	// option wins. Empty means the option is silent (tool-only).
+	Reply string `json:"reply,omitempty"`
+
+	// Tool, when set, is emitted as a function_call item with these
+	// exact arguments when the option wins.
+	Tool *ClassifierTool `json:"tool,omitempty"`
+}
+
+// ClassifierTool is a canned function call. Arguments is a raw JSON
+// object; with Slots it becomes a template whose "{{name}}" placeholders
+// are filled by a short constrained completion after classification —
+// the hybrid between prefill-only classification and full generation.
+type ClassifierTool struct {
+	Name      string          `json:"name"`
+	Arguments json.RawMessage `json:"arguments,omitempty"`
+
+	// Slots declares the argument holes to fill by inference when the
+	// option wins. Number slots substitute the quoted placeholder
+	// ("{{name}}" -> 3.5) so YAML/JSON templates stay well-formed; enum
+	// and string slots substitute inside their quotes.
+	Slots []ClassifierSlot `json:"slots,omitempty"`
+}
+
+// Classifier slot types.
+const (
+	ClassifierSlotNumber = "number"
+	ClassifierSlotEnum   = "enum"
+	ClassifierSlotString = "string"
+)
+
+// ClassifierSlot is one inferred argument of a classifier tool call.
+type ClassifierSlot struct {
+	// Name of the slot; "{{name}}" in the arguments template marks where
+	// its value lands, and the model sees it as a JSON field name.
+	Name string `json:"name"`
+
+	// Type constrains the completion grammar: "number", "enum" or
+	// "string".
+	Type string `json:"type"`
+
+	// Values enumerates the admissible values for enum slots.
+	Values []string `json:"values,omitempty"`
+
+	// Default applies when inference fails outright. Enum defaults must
+	// be one of Values; number defaults must parse as a number. A slot
+	// without a default makes the whole response fall back on failure.
+	Default string `json:"default,omitempty"`
+
+	// Hint is appended to the option's description in the scoring/fill
+	// system prompt (e.g. "assume meters when the user gives no units").
+	Hint string `json:"hint,omitempty"`
+}
+
+// slotPlaceholder returns the template marker for a slot.
+func slotPlaceholder(name string) string { return "{{" + name + "}}" }
+
+// SampleValue returns a syntactically valid stand-in for template
+// validation: the default when set, otherwise a type-appropriate value.
+func (s *ClassifierSlot) SampleValue() string {
+	if s.Default != "" {
+		return s.Default
+	}
+	switch s.Type {
+	case ClassifierSlotNumber:
+		return "0"
+	case ClassifierSlotEnum:
+		if len(s.Values) > 0 {
+			return s.Values[0]
+		}
+	}
+	return "sample"
+}
+
+// SpliceArguments fills the tool's argument template with the given slot
+// values and returns the final JSON arguments string. Number values
+// replace the quoted placeholder so they land unquoted; other types are
+// JSON-string-escaped in place. The result must parse as a JSON object.
+func (t *ClassifierTool) SpliceArguments(values map[string]string) (string, error) {
+	args := "{}"
+	if len(t.Arguments) > 0 {
+		args = string(t.Arguments)
+	}
+	for i := range t.Slots {
+		s := &t.Slots[i]
+		v, ok := values[s.Name]
+		if !ok || v == "" {
+			return "", fmt.Errorf("classifier: no value for slot %q", s.Name)
+		}
+		ph := slotPlaceholder(s.Name)
+		if s.Type == ClassifierSlotNumber {
+			args = strings.ReplaceAll(args, `"`+ph+`"`, v)
+		} else {
+			esc, err := json.Marshal(v)
+			if err != nil {
+				return "", err
+			}
+			args = strings.ReplaceAll(args, ph, string(esc[1:len(esc)-1]))
+		}
+	}
+	var obj map[string]any
+	if err := json.Unmarshal([]byte(args), &obj); err != nil {
+		return "", fmt.Errorf("classifier: spliced tool arguments are not a JSON object: %w", err)
+	}
+	return args, nil
+}
+
+// SpliceReply fills "{{name}}" placeholders in the option's spoken reply
+// with the same slot values that filled the tool arguments, as plain text
+// ("Going {{distance}} {{units}}." → "Going 3 meters."), so the reply can
+// confirm what was actually inferred. Values are optional in the reply:
+// placeholders without a value stay literal, and options without slots (or
+// a nil value set) return the reply verbatim.
+func (o *ClassifierOption) SpliceReply(values map[string]string) string {
+	reply := o.Reply
+	if o.Tool == nil || len(values) == 0 {
+		return reply
+	}
+	for i := range o.Tool.Slots {
+		s := &o.Tool.Slots[i]
+		if v, ok := values[s.Name]; ok && v != "" {
+			reply = strings.ReplaceAll(reply, slotPlaceholder(s.Name), v)
+		}
+	}
+	return reply
+}
+
+// SlotDefaults returns every slot's default value, or an error naming the
+// first slot without one — the fill-failure path either recovers with a
+// complete default set or not at all.
+func (t *ClassifierTool) SlotDefaults() (map[string]string, error) {
+	values := make(map[string]string, len(t.Slots))
+	for i := range t.Slots {
+		if t.Slots[i].Default == "" {
+			return nil, fmt.Errorf("classifier: slot %q has no default", t.Slots[i].Name)
+		}
+		values[t.Slots[i].Name] = t.Slots[i].Default
+	}
+	return values, nil
+}
+
+// Classifier fallback modes.
+const (
+	// ClassifierFallbackNone completes the response with no output.
+	ClassifierFallbackNone = "none"
+	// ClassifierFallbackReply speaks/emits the canned fallback reply.
+	ClassifierFallbackReply = "reply"
+	// ClassifierFallbackGenerate falls through to normal autoregressive
+	// generation for that response.
+	ClassifierFallbackGenerate = "generate"
+)
+
+// ClassifierFallback selects the below-threshold behavior.
+type ClassifierFallback struct {
+	Mode  string `json:"mode,omitempty"`
+	Reply string `json:"reply,omitempty"`
+}
+
+// Active reports whether classification should run: explicitly enabled, or
+// enabled by default because options are present.
+func (c *ClassifierConfig) Active() bool {
+	if c == nil {
+		return false
+	}
+	if c.Enabled != nil {
+		return *c.Enabled && len(c.Options) > 0
+	}
+	return len(c.Options) > 0
+}
+
+// FallbackMode returns the effective fallback mode.
+func (c *ClassifierConfig) FallbackMode() string {
+	if c == nil || c.Fallback == nil || c.Fallback.Mode == "" {
+		return ClassifierFallbackNone
+	}
+	return c.Fallback.Mode
+}
+
+// Validate checks the invariants the scoring engine relies on. It is
+// shared by the session.update path and pipeline-config seeding so both
+// reject bad option lists the same way.
+func (c *ClassifierConfig) Validate() error {
+	if c == nil {
+		return nil
+	}
+	if c.Threshold < 0 || c.Threshold >= 1 {
+		return fmt.Errorf("classifier: threshold must be in [0,1), got %v", c.Threshold)
+	}
+	switch c.Normalization {
+	case "", "raw", "mean":
+	default:
+		return fmt.Errorf("classifier: normalization must be \"raw\" or \"mean\", got %q", c.Normalization)
+	}
+	if c.HistoryItems < -1 {
+		return fmt.Errorf("classifier: history_items must be >= -1, got %d", c.HistoryItems)
+	}
+	switch c.FallbackMode() {
+	case ClassifierFallbackNone, ClassifierFallbackReply, ClassifierFallbackGenerate:
+	default:
+		return fmt.Errorf("classifier: fallback mode must be one of none|reply|generate, got %q", c.Fallback.Mode)
+	}
+	if c.FallbackMode() == ClassifierFallbackReply && (c.Fallback == nil || c.Fallback.Reply == "") {
+		return fmt.Errorf("classifier: fallback mode \"reply\" requires a non-empty fallback reply")
+	}
+	if c.Address != nil {
+		named := false
+		for _, n := range c.Address.Names {
+			if n != "" {
+				named = true
+				break
+			}
+		}
+		if !named {
+			return fmt.Errorf("classifier: address gate requires at least one non-empty name")
+		}
+		switch c.Address.AddressMode() {
+		case ClassifierAddressIgnore, ClassifierAddressReply:
+		default:
+			return fmt.Errorf("classifier: address mode must be one of ignore|reply, got %q", c.Address.Mode)
+		}
+		if c.Address.AddressMode() == ClassifierAddressReply && c.Address.Reply == "" {
+			return fmt.Errorf("classifier: address mode \"reply\" requires a non-empty reply")
+		}
+	}
+	seen := make(map[string]struct{}, len(c.Options))
+	for i, opt := range c.Options {
+		if opt.ID == "" {
+			return fmt.Errorf("classifier: option %d has an empty id", i)
+		}
+		if _, dup := seen[opt.ID]; dup {
+			return fmt.Errorf("classifier: duplicate option id %q", opt.ID)
+		}
+		seen[opt.ID] = struct{}{}
+		if opt.Description == "" {
+			return fmt.Errorf("classifier: option %q has an empty description", opt.ID)
+		}
+		if opt.Tool != nil {
+			if opt.Tool.Name == "" {
+				return fmt.Errorf("classifier: option %q has a tool with an empty name", opt.ID)
+			}
+			if len(opt.Tool.Arguments) > 0 && len(opt.Tool.Slots) == 0 {
+				var obj map[string]any
+				if err := json.Unmarshal(opt.Tool.Arguments, &obj); err != nil {
+					return fmt.Errorf("classifier: option %q tool arguments must be a JSON object: %w", opt.ID, err)
+				}
+			}
+			if err := validateSlots(opt.Tool); err != nil {
+				return fmt.Errorf("classifier: option %q: %w", opt.ID, err)
+			}
+		}
+	}
+	return nil
+}
+
+// slotNamePattern keeps slot names safe to embed as JSON field names and
+// template placeholders without escaping.
+var slotNamePattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
+
+func validateSlots(t *ClassifierTool) error {
+	if len(t.Slots) == 0 {
+		return nil
+	}
+	args := string(t.Arguments)
+	seen := make(map[string]struct{}, len(t.Slots))
+	sample := make(map[string]string, len(t.Slots))
+	for i := range t.Slots {
+		s := &t.Slots[i]
+		if !slotNamePattern.MatchString(s.Name) {
+			return fmt.Errorf("slot %d has invalid name %q", i, s.Name)
+		}
+		if _, dup := seen[s.Name]; dup {
+			return fmt.Errorf("duplicate slot %q", s.Name)
+		}
+		seen[s.Name] = struct{}{}
+		switch s.Type {
+		case ClassifierSlotNumber:
+			if s.Default != "" {
+				if _, err := strconv.ParseFloat(s.Default, 64); err != nil {
+					return fmt.Errorf("slot %q: number default %q does not parse", s.Name, s.Default)
+				}
+			}
+		case ClassifierSlotEnum:
+			if len(s.Values) == 0 {
+				return fmt.Errorf("slot %q: enum slots need values", s.Name)
+			}
+			if slices.Contains(s.Values, "") {
+				return fmt.Errorf("slot %q: enum values must be non-empty", s.Name)
+			}
+			if s.Default != "" && !slices.Contains(s.Values, s.Default) {
+				return fmt.Errorf("slot %q: default %q is not one of its values", s.Name, s.Default)
+			}
+		case ClassifierSlotString:
+		default:
+			return fmt.Errorf("slot %q: type must be one of number|enum|string, got %q", s.Name, s.Type)
+		}
+		if !strings.Contains(args, slotPlaceholder(s.Name)) {
+			return fmt.Errorf("slot %q: arguments template does not reference {{%s}}", s.Name, s.Name)
+		}
+		sample[s.Name] = s.SampleValue()
+	}
+	// The template with type-appropriate values must produce a JSON
+	// object, catching e.g. an unquoted string placeholder up front.
+	if _, err := t.SpliceArguments(sample); err != nil {
+		return fmt.Errorf("arguments template does not splice: %w", err)
+	}
+	return nil
+}
+
+// ClassifierScore is one entry of the softmax distribution over options.
+type ClassifierScore struct {
+	ID    string  `json:"id"`
+	Score float64 `json:"score"`
+}
+
+// ClassifierResultEvent is a LocalAI extension server event
+// (localai.classifier.result) emitted once per classifier-handled response
+// — including fallbacks — before the output items, so clients can
+// visualize the decision and its confidence.
+type ClassifierResultEvent struct {
+	ServerEventBase
+
+	// The ID of the response this classification belongs to.
+	ResponseID string `json:"response_id"`
+
+	// The full softmax distribution, in option-declaration order.
+	Scores []ClassifierScore `json:"scores"`
+
+	// The winning option id, or "" when the fallback applied.
+	ChosenID string `json:"chosen_id,omitempty"`
+
+	// The threshold the winner had to clear.
+	Threshold float64 `json:"threshold"`
+
+	// The fallback mode that applied, or "" when an option was chosen.
+	Fallback string `json:"fallback,omitempty"`
+
+	// Wall-clock scoring latency.
+	LatencyMs int64 `json:"latency_ms"`
+
+	// The chosen option's final tool arguments when its slots were filled
+	// by inference (the hybrid classify-then-complete path).
+	Arguments string `json:"arguments,omitempty"`
+
+	// Wall-clock slot-fill latency; zero when the option has no slots.
+	FillLatencyMs int64 `json:"fill_latency_ms,omitempty"`
+}
+
+func (m ClassifierResultEvent) ServerEventType() ServerEventType {
+	return ServerEventTypeClassifierResult
+}
+
+func (m ClassifierResultEvent) MarshalJSON() ([]byte, error) {
+	type typeAlias ClassifierResultEvent
+	type typeWrapper struct {
+		typeAlias
+		Type ServerEventType `json:"type"`
+	}
+	shadow := typeWrapper{
+		typeAlias: typeAlias(m),
+		Type:      m.ServerEventType(),
+	}
+	return json.Marshal(shadow)
+}

core/http/endpoints/openai/types/classifier_test.go

@@ -0,0 +1,299 @@
+package types_test
+
+import (
+	"encoding/json"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
+)
+
+func validClassifier() *types.ClassifierConfig {
+	return &types.ClassifierConfig{
+		Threshold: 0.35,
+		Options: []types.ClassifierOption{
+			{
+				ID:          "up",
+				Description: "the user asks the drone to fly up",
+				Reply:       "Going up.",
+				Tool:        &types.ClassifierTool{Name: "move", Arguments: json.RawMessage(`{"direction":"up"}`)},
+			},
+			{ID: "greeting", Description: "the user greets the assistant", Reply: "Hello."},
+		},
+		Fallback: &types.ClassifierFallback{Mode: types.ClassifierFallbackReply, Reply: "Say again?"},
+	}
+}
+
+var _ = Describe("ClassifierConfig", func() {
+	Describe("JSON round-trip", func() {
+		It("survives marshal/unmarshal with all fields", func() {
+			in := validClassifier()
+			enabled := true
+			in.Enabled = &enabled
+			in.Normalization = "mean"
+			in.HistoryItems = -1
+
+			data, err := json.Marshal(in)
+			Expect(err).ToNot(HaveOccurred())
+
+			var out types.ClassifierConfig
+			Expect(json.Unmarshal(data, &out)).To(Succeed())
+			Expect(out.Enabled).ToNot(BeNil())
+			Expect(*out.Enabled).To(BeTrue())
+			Expect(out.Threshold).To(Equal(0.35))
+			Expect(out.Normalization).To(Equal("mean"))
+			Expect(out.HistoryItems).To(Equal(-1))
+			Expect(out.Options).To(HaveLen(2))
+			Expect(out.Options[0].Tool.Name).To(Equal("move"))
+			Expect(string(out.Options[0].Tool.Arguments)).To(MatchJSON(`{"direction":"up"}`))
+			Expect(out.Fallback.Mode).To(Equal("reply"))
+		})
+
+		It("is carried by RealtimeSession under localai_classifier", func() {
+			s := types.RealtimeSession{LocalAIClassifier: validClassifier()}
+			data, err := json.Marshal(s)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring(`"localai_classifier"`))
+
+			var back types.RealtimeSession
+			Expect(json.Unmarshal(data, &back)).To(Succeed())
+			Expect(back.LocalAIClassifier).ToNot(BeNil())
+			Expect(back.LocalAIClassifier.Options).To(HaveLen(2))
+		})
+
+		It("is carried by ResponseCreateParams under localai_classifier", func() {
+			var params types.ResponseCreateParams
+			Expect(json.Unmarshal([]byte(`{"localai_classifier":{"enabled":false}}`), &params)).To(Succeed())
+			Expect(params.LocalAIClassifier).ToNot(BeNil())
+			Expect(params.LocalAIClassifier.Enabled).ToNot(BeNil())
+			Expect(*params.LocalAIClassifier.Enabled).To(BeFalse())
+		})
+	})
+
+	Describe("Active", func() {
+		It("is inactive when nil", func() {
+			var c *types.ClassifierConfig
+			Expect(c.Active()).To(BeFalse())
+		})
+
+		It("defaults to active when options exist", func() {
+			Expect(validClassifier().Active()).To(BeTrue())
+		})
+
+		It("is inactive without options even when enabled", func() {
+			enabled := true
+			c := &types.ClassifierConfig{Enabled: &enabled}
+			Expect(c.Active()).To(BeFalse())
+		})
+
+		It("honors an explicit enabled=false override", func() {
+			c := validClassifier()
+			disabled := false
+			c.Enabled = &disabled
+			Expect(c.Active()).To(BeFalse())
+		})
+	})
+
+	Describe("Validate", func() {
+		It("accepts a valid config and a nil config", func() {
+			Expect(validClassifier().Validate()).To(Succeed())
+			var c *types.ClassifierConfig
+			Expect(c.Validate()).To(Succeed())
+		})
+
+		It("rejects out-of-range thresholds", func() {
+			c := validClassifier()
+			c.Threshold = 1.0
+			Expect(c.Validate()).To(MatchError(ContainSubstring("threshold")))
+			c.Threshold = -0.1
+			Expect(c.Validate()).To(MatchError(ContainSubstring("threshold")))
+		})
+
+		It("rejects unknown normalization", func() {
+			c := validClassifier()
+			c.Normalization = "zscore"
+			Expect(c.Validate()).To(MatchError(ContainSubstring("normalization")))
+		})
+
+		It("rejects history_items below -1", func() {
+			c := validClassifier()
+			c.HistoryItems = -2
+			Expect(c.Validate()).To(MatchError(ContainSubstring("history_items")))
+		})
+
+		It("rejects unknown fallback modes", func() {
+			c := validClassifier()
+			c.Fallback = &types.ClassifierFallback{Mode: "retry"}
+			Expect(c.Validate()).To(MatchError(ContainSubstring("fallback mode")))
+		})
+
+		It("rejects a reply fallback without a reply", func() {
+			c := validClassifier()
+			c.Fallback = &types.ClassifierFallback{Mode: types.ClassifierFallbackReply}
+			Expect(c.Validate()).To(MatchError(ContainSubstring("fallback reply")))
+		})
+
+		It("rejects empty and duplicate option ids", func() {
+			c := validClassifier()
+			c.Options[1].ID = ""
+			Expect(c.Validate()).To(MatchError(ContainSubstring("empty id")))
+			c.Options[1].ID = "up"
+			Expect(c.Validate()).To(MatchError(ContainSubstring("duplicate option id")))
+		})
+
+		It("rejects an option without a description", func() {
+			c := validClassifier()
+			c.Options[0].Description = ""
+			Expect(c.Validate()).To(MatchError(ContainSubstring("empty description")))
+		})
+
+		It("rejects tools with no name or non-object arguments", func() {
+			c := validClassifier()
+			c.Options[0].Tool = &types.ClassifierTool{}
+			Expect(c.Validate()).To(MatchError(ContainSubstring("empty name")))
+			c.Options[0].Tool = &types.ClassifierTool{Name: "move", Arguments: json.RawMessage(`["up"]`)}
+			Expect(c.Validate()).To(MatchError(ContainSubstring("JSON object")))
+		})
+	})
+
+	Describe("FallbackMode", func() {
+		It("defaults to none", func() {
+			Expect((&types.ClassifierConfig{}).FallbackMode()).To(Equal(types.ClassifierFallbackNone))
+			var c *types.ClassifierConfig
+			Expect(c.FallbackMode()).To(Equal(types.ClassifierFallbackNone))
+		})
+	})
+
+	Describe("ClassifierResultEvent", func() {
+		It("marshals with the localai.classifier.result type tag", func() {
+			ev := types.ClassifierResultEvent{
+				ResponseID: "resp_1",
+				Scores:     []types.ClassifierScore{{ID: "up", Score: 0.9}, {ID: "down", Score: 0.1}},
+				ChosenID:   "up",
+				Threshold:  0.35,
+				LatencyMs:  12,
+			}
+			data, err := json.Marshal(ev)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring(`"type":"localai.classifier.result"`))
+			Expect(string(data)).To(ContainSubstring(`"chosen_id":"up"`))
+			Expect(string(data)).To(ContainSubstring(`"threshold":0.35`))
+		})
+	})
+})
+
+var _ = Describe("ClassifierTool slots", func() {
+	tool := func(slots ...types.ClassifierSlot) *types.ClassifierTool {
+		return &types.ClassifierTool{
+			Name:      "move",
+			Arguments: json.RawMessage(`{"direction":"up","distance":"{{distance}}","units":"{{units}}"}`),
+			Slots:     slots,
+		}
+	}
+	numberSlot := types.ClassifierSlot{Name: "distance", Type: types.ClassifierSlotNumber, Default: "1"}
+	enumSlot := types.ClassifierSlot{Name: "units", Type: types.ClassifierSlotEnum, Values: []string{"m", "ft"}, Default: "m"}
+
+	cfgWith := func(t *types.ClassifierTool) *types.ClassifierConfig {
+		return &types.ClassifierConfig{Options: []types.ClassifierOption{{ID: "up", Description: "d", Tool: t}}}
+	}
+
+	Describe("Validate", func() {
+		It("accepts a well-formed slotted tool", func() {
+			Expect(cfgWith(tool(numberSlot, enumSlot)).Validate()).To(Succeed())
+		})
+
+		It("rejects unknown slot types", func() {
+			bad := numberSlot
+			bad.Type = "float"
+			Expect(cfgWith(tool(bad, enumSlot)).Validate()).To(MatchError(ContainSubstring("number|enum|string")))
+		})
+
+		It("rejects enum slots without values", func() {
+			bad := enumSlot
+			bad.Values = nil
+			bad.Default = ""
+			Expect(cfgWith(tool(numberSlot, bad)).Validate()).To(MatchError(ContainSubstring("need values")))
+		})
+
+		It("rejects enum defaults outside the value set", func() {
+			bad := enumSlot
+			bad.Default = "yards"
+			Expect(cfgWith(tool(numberSlot, bad)).Validate()).To(MatchError(ContainSubstring("not one of")))
+		})
+
+		It("rejects number defaults that do not parse", func() {
+			bad := numberSlot
+			bad.Default = "three"
+			Expect(cfgWith(tool(bad, enumSlot)).Validate()).To(MatchError(ContainSubstring("does not parse")))
+		})
+
+		It("rejects empty enum values that cannot be spliced", func() {
+			bad := enumSlot
+			bad.Values = []string{"m", ""}
+			Expect(cfgWith(tool(numberSlot, bad)).Validate()).To(MatchError(ContainSubstring("must be non-empty")))
+		})
+
+		It("rejects slots the template never references", func() {
+			t := tool(numberSlot, enumSlot, types.ClassifierSlot{Name: "speed", Type: types.ClassifierSlotNumber})
+			Expect(cfgWith(t).Validate()).To(MatchError(ContainSubstring("{{speed}}")))
+		})
+
+		It("rejects invalid slot names", func() {
+			bad := numberSlot
+			bad.Name = "dis tance"
+			Expect(cfgWith(tool(bad, enumSlot)).Validate()).To(MatchError(ContainSubstring("invalid name")))
+		})
+	})
+
+	Describe("SpliceArguments", func() {
+		It("substitutes numbers unquoted and strings escaped", func() {
+			args, err := tool(numberSlot, enumSlot).SpliceArguments(map[string]string{"distance": "3.5", "units": `m"eters`})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(args).To(MatchJSON(`{"direction":"up","distance":3.5,"units":"m\"eters"}`))
+		})
+
+		It("fails on missing values", func() {
+			_, err := tool(numberSlot, enumSlot).SpliceArguments(map[string]string{"distance": "3.5"})
+			Expect(err).To(MatchError(ContainSubstring(`no value for slot "units"`)))
+		})
+	})
+
+	Describe("SlotDefaults", func() {
+		It("returns every default", func() {
+			values, err := tool(numberSlot, enumSlot).SlotDefaults()
+			Expect(err).ToNot(HaveOccurred())
+			Expect(values).To(Equal(map[string]string{"distance": "1", "units": "m"}))
+		})
+
+		It("names the slot lacking a default", func() {
+			bare := numberSlot
+			bare.Default = ""
+			_, err := tool(bare, enumSlot).SlotDefaults()
+			Expect(err).To(MatchError(ContainSubstring(`"distance"`)))
+		})
+	})
+
+	Describe("SpliceReply", func() {
+		option := func(reply string, t *types.ClassifierTool) *types.ClassifierOption {
+			return &types.ClassifierOption{ID: "up", Description: "d", Reply: reply, Tool: t}
+		}
+
+		It("substitutes slot values as plain text", func() {
+			o := option("Going up {{distance}} {{units}}.", tool(numberSlot, enumSlot))
+			Expect(o.SpliceReply(map[string]string{"distance": "3.5", "units": "m"})).To(Equal("Going up 3.5 m."))
+		})
+
+		It("leaves placeholders without a value literal", func() {
+			o := option("Going up {{distance}} {{units}}.", tool(numberSlot, enumSlot))
+			Expect(o.SpliceReply(map[string]string{"distance": "3"})).To(Equal("Going up 3 {{units}}."))
+		})
+
+		It("returns the reply verbatim without slots or values", func() {
+			o := option("Going up {{distance}}.", nil)
+			Expect(o.SpliceReply(map[string]string{"distance": "3"})).To(Equal("Going up {{distance}}."))
+			slotted := option("Going up {{distance}}.", tool(numberSlot))
+			Expect(slotted.SpliceReply(nil)).To(Equal("Going up {{distance}}."))
+		})
+	})
+})

core/http/endpoints/openai/types/server_events.go

@@ -24,34 +24,38 @@ const (
 	// ServerEventTypeConversationItemSpeaker is a LocalAI extension: it reports
 	// the recognized speaker for a user audio item. OpenAI clients ignore it.
 	ServerEventTypeConversationItemSpeaker ServerEventType = "conversation.item.speaker"
-	ServerEventTypeInputAudioBufferCommitted                        ServerEventType = "input_audio_buffer.committed"
-	ServerEventTypeInputAudioBufferCleared                          ServerEventType = "input_audio_buffer.cleared"
-	ServerEventTypeInputAudioBufferSpeechStarted                    ServerEventType = "input_audio_buffer.speech_started"
-	ServerEventTypeInputAudioBufferSpeechStopped                    ServerEventType = "input_audio_buffer.speech_stopped"
-	ServerEventTypeInputAudioBufferTimeoutTriggered                 ServerEventType = "input_audio_buffer.timeout_triggered"
-	ServerEventTypeResponseCreated                                  ServerEventType = "response.created"
-	ServerEventTypeResponseDone                                     ServerEventType = "response.done"
-	ServerEventTypeResponseOutputItemAdded                          ServerEventType = "response.output_item.added"
-	ServerEventTypeResponseOutputItemDone                           ServerEventType = "response.output_item.done"
-	ServerEventTypeResponseContentPartAdded                         ServerEventType = "response.content_part.added"
-	ServerEventTypeResponseContentPartDone                          ServerEventType = "response.content_part.done"
-	ServerEventTypeResponseOutputTextDelta                          ServerEventType = "response.output_text.delta"
-	ServerEventTypeResponseOutputTextDone                           ServerEventType = "response.output_text.done"
-	ServerEventTypeResponseOutputAudioTranscriptDelta               ServerEventType = "response.output_audio_transcript.delta"
-	ServerEventTypeResponseOutputAudioTranscriptDone                ServerEventType = "response.output_audio_transcript.done"
-	ServerEventTypeResponseOutputAudioDelta                         ServerEventType = "response.output_audio.delta"
-	ServerEventTypeResponseOutputAudioDone                          ServerEventType = "response.output_audio.done"
-	ServerEventTypeResponseFunctionCallArgumentsDelta               ServerEventType = "response.function_call_arguments.delta"
-	ServerEventTypeResponseFunctionCallArgumentsDone                ServerEventType = "response.function_call_arguments.done"
-	ServerEventTypeResponseMcpCallArgumentsDelta                    ServerEventType = "response.mcp_call_arguments.delta"
-	ServerEventTypeResponseMcpCallArgumentsDone                     ServerEventType = "response.mcp_call_arguments.done"
-	ServerEventTypeResponseMcpCallInProgress                        ServerEventType = "response.mcp_call.in_progress"
-	ServerEventTypeResponseMcpCallCompleted                         ServerEventType = "response.mcp_call.completed"
-	ServerEventTypeResponseMcpCallFailed                            ServerEventType = "response.mcp_call.failed"
-	ServerEventTypeMcpListToolsInProgress                           ServerEventType = "mcp_list_tools.in_progress"
-	ServerEventTypeMcpListToolsCompleted                            ServerEventType = "mcp_list_tools.completed"
-	ServerEventTypeMcpListToolsFailed                               ServerEventType = "mcp_list_tools.failed"
-	ServerEventTypeRateLimitsUpdated                                ServerEventType = "rate_limits.updated"
+	// ServerEventTypeClassifierResult is a LocalAI extension: it carries the
+	// classifier-mode score distribution and decision for a response. OpenAI
+	// clients ignore it.
+	ServerEventTypeClassifierResult                   ServerEventType = "localai.classifier.result"
+	ServerEventTypeInputAudioBufferCommitted          ServerEventType = "input_audio_buffer.committed"
+	ServerEventTypeInputAudioBufferCleared            ServerEventType = "input_audio_buffer.cleared"
+	ServerEventTypeInputAudioBufferSpeechStarted      ServerEventType = "input_audio_buffer.speech_started"
+	ServerEventTypeInputAudioBufferSpeechStopped      ServerEventType = "input_audio_buffer.speech_stopped"
+	ServerEventTypeInputAudioBufferTimeoutTriggered   ServerEventType = "input_audio_buffer.timeout_triggered"
+	ServerEventTypeResponseCreated                    ServerEventType = "response.created"
+	ServerEventTypeResponseDone                       ServerEventType = "response.done"
+	ServerEventTypeResponseOutputItemAdded            ServerEventType = "response.output_item.added"
+	ServerEventTypeResponseOutputItemDone             ServerEventType = "response.output_item.done"
+	ServerEventTypeResponseContentPartAdded           ServerEventType = "response.content_part.added"
+	ServerEventTypeResponseContentPartDone            ServerEventType = "response.content_part.done"
+	ServerEventTypeResponseOutputTextDelta            ServerEventType = "response.output_text.delta"
+	ServerEventTypeResponseOutputTextDone             ServerEventType = "response.output_text.done"
+	ServerEventTypeResponseOutputAudioTranscriptDelta ServerEventType = "response.output_audio_transcript.delta"
+	ServerEventTypeResponseOutputAudioTranscriptDone  ServerEventType = "response.output_audio_transcript.done"
+	ServerEventTypeResponseOutputAudioDelta           ServerEventType = "response.output_audio.delta"
+	ServerEventTypeResponseOutputAudioDone            ServerEventType = "response.output_audio.done"
+	ServerEventTypeResponseFunctionCallArgumentsDelta ServerEventType = "response.function_call_arguments.delta"
+	ServerEventTypeResponseFunctionCallArgumentsDone  ServerEventType = "response.function_call_arguments.done"
+	ServerEventTypeResponseMcpCallArgumentsDelta      ServerEventType = "response.mcp_call_arguments.delta"
+	ServerEventTypeResponseMcpCallArgumentsDone       ServerEventType = "response.mcp_call_arguments.done"
+	ServerEventTypeResponseMcpCallInProgress          ServerEventType = "response.mcp_call.in_progress"
+	ServerEventTypeResponseMcpCallCompleted           ServerEventType = "response.mcp_call.completed"
+	ServerEventTypeResponseMcpCallFailed              ServerEventType = "response.mcp_call.failed"
+	ServerEventTypeMcpListToolsInProgress             ServerEventType = "mcp_list_tools.in_progress"
+	ServerEventTypeMcpListToolsCompleted              ServerEventType = "mcp_list_tools.completed"
+	ServerEventTypeMcpListToolsFailed                 ServerEventType = "mcp_list_tools.failed"
+	ServerEventTypeRateLimitsUpdated                  ServerEventType = "rate_limits.updated"
 )
 
 // ServerEvent is the interface for server events.

core/http/endpoints/openai/types/types.go

@@ -956,6 +956,11 @@ type RealtimeSession struct {
 
 	// Controls how the realtime conversation is truncated prior to model inference. The default is auto.
 	Truncation *TruncationUnion `json:"truncation,omitempty"`
+
+	// LocalAIClassifier is a LocalAI extension: prefill-scored option
+	// selection instead of autoregressive generation. Replaced wholesale
+	// on update, like tools. OpenAI clients simply never set it.
+	LocalAIClassifier *ClassifierConfig `json:"localai_classifier,omitempty"`
 }
 
 func (r RealtimeSession) Type() SessionType {
@@ -1191,6 +1196,11 @@ type ResponseCreateParams struct {
 
 	// Tools available to the model.
 	Tools []ToolUnion `json:"tools,omitempty"`
+
+	// LocalAIClassifier is a LocalAI extension: when non-nil it replaces
+	// the session's classifier config for this response only —
+	// {"enabled": false} runs normal generation once.
+	LocalAIClassifier *ClassifierConfig `json:"localai_classifier,omitempty"`
 }
 
 type Response struct {

core/http/endpoints/openai/types/types_suite_test.go

@@ -0,0 +1,13 @@
+package types_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestTypes(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Realtime types test suite")
+}

core/http/middleware/route_model.go

@@ -339,7 +339,7 @@ func buildClassifier(cfg *config.ModelConfig, deps ClassifierDeps) (router.Class
 					// classifier model MUST carry a chat template — refusing
 					// here beats silently falling back to a generic ChatML
 					// envelope the model may not have been trained on.
-					renderer := newTemplateRenderer(deps.Evaluator, classifierCfg)
+					renderer := NewTemplateRenderer(deps.Evaluator, classifierCfg)
 					if renderer == nil {
 						return nil, fmt.Errorf(
 							"router classifier score: classifier_model %q has no chat template "+
@@ -350,7 +350,7 @@ func buildClassifier(cfg *config.ModelConfig, deps ClassifierDeps) (router.Class
 					}
 					opts.PromptRenderer = renderer
 				}
-				if st := pickAssistantTurnEnd(classifierCfg.StopWords, classifierCfg.TemplateConfig.ChatMessage); st != "" {
+				if st := PickAssistantTurnEnd(classifierCfg.StopWords, classifierCfg.TemplateConfig.ChatMessage); st != "" {
 					opts.StopToken = st
 				}
 				// Token-exact conversation trim — score classifier drops the
@@ -464,7 +464,7 @@ func validateRouterPolicies(classifierName string, rc config.RouterConfig) ([]ro
 	return policies, nil
 }
 
-// newTemplateRenderer adapts the templates.Evaluator + the classifier
+// NewTemplateRenderer adapts the templates.Evaluator + the classifier
 // model's config into the router.PromptRenderer callback. The
 // resulting renderer pushes the routing system + user prompt through
 // the classifier model's full chat-template pipeline — per-role
@@ -484,7 +484,7 @@ func validateRouterPolicies(classifierName string, rc config.RouterConfig) ([]ro
 // Returns nil (forcing the score classifier's chatMLRenderer
 // fallback) when either template piece is missing — partial
 // templating would still drop content.
-func newTemplateRenderer(eval *templates.Evaluator, classifierCfg *config.ModelConfig) router.PromptRenderer {
+func NewTemplateRenderer(eval *templates.Evaluator, classifierCfg *config.ModelConfig) router.PromptRenderer {
 	if classifierCfg.TemplateConfig.Chat == "" || classifierCfg.TemplateConfig.ChatMessage == "" {
 		return nil
 	}
@@ -502,7 +502,7 @@ func newTemplateRenderer(eval *templates.Evaluator, classifierCfg *config.ModelC
 	}
 }
 
-// pickAssistantTurnEnd returns the classifier model's assistant
+// PickAssistantTurnEnd returns the classifier model's assistant
 // turn-end token — the one to suffix candidates with so the model's
 // "I'm done" signal folds into the per-candidate joint log-prob.
 //
@@ -520,7 +520,7 @@ func newTemplateRenderer(eval *templates.Evaluator, classifierCfg *config.ModelC
 //
 // When no stopwords are configured at all, return "" — caller falls
 // back to defaultStopToken (<|im_end|>) inside the score classifier.
-func pickAssistantTurnEnd(words []string, chatMessageTemplate string) string {
+func PickAssistantTurnEnd(words []string, chatMessageTemplate string) string {
 	if chatMessageTemplate != "" {
 		for _, w := range words {
 			if w != "" && strings.Contains(chatMessageTemplate, w) {

core/http/middleware/route_model_test.go

@@ -301,7 +301,7 @@ var _ = Describe("RouteModel rendered classifier prompt", func() {
 		// <|im_end|> first even though the actual Llama-3 assistant
 		// turn-end is <|eot_id|>. The naive "stopwords[0]" pick would
 		// suffix candidates with <|im_end|> — a token Llama-3 never
-		// emits at turn end. pickAssistantTurnEnd should scan the
+		// emits at turn end. PickAssistantTurnEnd should scan the
 		// chat_message template and recognise <|eot_id|> as the real
 		// turn-end.
 		writeLlama3StyleClassifierModel(modelDir, "arch-router")
@@ -340,7 +340,7 @@ type stubScorer struct {
 	lastCandidates []string
 }
 
-func (s *stubScorer) Score(_ context.Context, prompt string, candidates []string) ([]backend.CandidateScore, error) {
+func (s *stubScorer) Score(_ context.Context, prompt string, _ int, candidates []string) ([]backend.CandidateScore, error) {
 	s.lastPrompt = prompt
 	s.lastCandidates = append([]string(nil), candidates...)
 	out := make([]backend.CandidateScore, len(candidates))
@@ -498,7 +498,7 @@ template:
 // writeLlama3StyleClassifierModel writes a classifier model mirroring
 // gallery/llama3-instruct.yaml — stopwords defensively list <|im_end|>
 // first even though the assistant turn-end is actually <|eot_id|>.
-// Exercises pickAssistantTurnEnd's template scan: the right token is
+// Exercises PickAssistantTurnEnd's template scan: the right token is
 // the one that appears in chat_message, not the one at position 0.
 func writeLlama3StyleClassifierModel(modelDir, name string) {
 	body := `name: ` + name + `
@@ -524,7 +524,7 @@ template:
 
 // writePartialClassifierModel writes a classifier model that has the
 // outer Chat template but no ChatMessage — exercises the
-// newTemplateRenderer "refuse partial templating" branch, which makes
+// NewTemplateRenderer "refuse partial templating" branch, which makes
 // buildClassifier reject the router with a missing-template error.
 func writePartialClassifierModel(modelDir, name string) {
 	body := `name: ` + name + `

core/http/react-ui/e2e/activity-page.spec.js

@@ -0,0 +1,417 @@
+import { test, expect } from './coverage-fixtures.js'
+
+const stub = (page, { operations = [], history = [] } = {}) => Promise.all([
+  page.route('**/api/operations', (route) => route.fulfill({
+    contentType: 'application/json',
+    body: JSON.stringify({ operations }),
+  })),
+  page.route('**/api/operations/history', (route) => route.fulfill({
+    contentType: 'application/json',
+    body: JSON.stringify({ operations: history }),
+  })),
+])
+
+test('lists live operations and cancels one from a labelled button', async ({ page }) => {
+  await stub(page, {
+    operations: [{
+      id: 'gemma-3-27b-it',
+      name: 'gemma-3-27b-it',
+      jobID: 'job-gemma',
+      progress: 22,
+      taskType: 'installation',
+      isBackend: false,
+      isQueued: false,
+      isDeletion: false,
+      cancellable: true,
+      phase: 'downloading',
+    }],
+  })
+
+  let cancelledPath = ''
+  await page.route('**/api/operations/job-gemma/cancel', (route) => {
+    cancelledPath = new URL(route.request().url()).pathname
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+
+  await page.goto('/app/activity')
+
+  const card = page.locator('.operation-card').filter({ hasText: 'gemma-3-27b-it' })
+  await expect(card).toBeVisible()
+  await expect(card).toContainText('22%')
+
+  await card.locator('.operation-card__cancel').click()
+  expect(cancelledPath).toBe('/api/operations/job-gemma/cancel')
+})
+
+test('separates an unacknowledged failure from the record', async ({ page }) => {
+  await stub(page, {
+    operations: [{
+      id: 'sherpa-onnx',
+      name: 'sherpa-onnx',
+      jobID: 'job-sherpa',
+      progress: 0,
+      taskType: 'installation',
+      isBackend: true,
+      isQueued: false,
+      isDeletion: false,
+      cancellable: false,
+      error: 'no space left on device',
+    }],
+    history: [{
+      id: 'bark-cpp',
+      name: 'bark-cpp',
+      jobID: 'job-bark',
+      isBackend: true,
+      taskType: 'installation',
+      outcome: 'failed',
+      error: 'checksum mismatch',
+      startedAt: '2026-07-28T13:40:00Z',
+      finishedAt: '2026-07-28T13:41:00Z',
+    }],
+  })
+
+  await page.goto('/app/activity')
+
+  // Live and unacknowledged: a card that needs a decision.
+  await expect(page.locator('.operation-card--error')).toContainText('sherpa-onnx')
+  // Dismissed earlier: a row in the record.
+  await expect(page.locator('.activity-row')).toContainText('bark-cpp')
+})
+
+test('a failure never appears in both In progress and Needs attention', async ({ page }) => {
+  // Section membership has to be unambiguous: the same job showing twice makes
+  // the two failure paths (retry / dismiss) impossible to reason about.
+  await stub(page, {
+    operations: [
+      {
+        id: 'model-a',
+        name: 'model-a',
+        jobID: 'job-a',
+        progress: 40,
+        taskType: 'installation',
+        isBackend: false,
+        isQueued: false,
+        isDeletion: false,
+        cancellable: true,
+      },
+      {
+        id: 'sherpa-onnx',
+        name: 'sherpa-onnx',
+        jobID: 'job-sherpa',
+        progress: 0,
+        taskType: 'installation',
+        isBackend: true,
+        isQueued: false,
+        isDeletion: false,
+        cancellable: false,
+        error: 'no space left on device',
+      },
+    ],
+  })
+
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.operation-card')).toHaveCount(2)
+  await expect(page.locator('.operation-card').filter({ hasText: 'sherpa-onnx' })).toHaveCount(1)
+})
+
+test('retrying a failed backend install dismisses it before reinstalling', async ({ page }) => {
+  // Order is load-bearing: a bare reinstall overwrites the opcache entry
+  // without going through recordTerminal, so the failure would never reach the
+  // record.
+  await stub(page, {
+    operations: [{
+      id: 'sherpa-onnx',
+      name: 'sherpa-onnx',
+      fullName: 'localai@sherpa-onnx',
+      jobID: 'job-sherpa',
+      progress: 0,
+      taskType: 'installation',
+      isBackend: true,
+      isQueued: false,
+      isDeletion: false,
+      cancellable: false,
+      error: 'no space left on device',
+    }],
+  })
+
+  const calls = []
+  await page.route('**/api/operations/job-sherpa/dismiss', (route) => {
+    calls.push('dismiss')
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+  await page.route('**/api/backends/install/**', (route) => {
+    calls.push(new URL(route.request().url()).pathname)
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+
+  await page.goto('/app/activity')
+
+  await page.locator('.operation-card__retry').click()
+
+  await expect.poll(() => calls).toEqual(['dismiss', '/api/backends/install/localai@sherpa-onnx'])
+})
+
+test('retry dismisses the job it was pressed on, not another sharing its id', async ({ page }) => {
+  // /api/operations strips the "node:<id>:" prefix, so a local install and a
+  // node-scoped install of one backend arrive with the same id and different
+  // jobIDs. Dismissing by id retired whichever came first, which both left the
+  // acted-on failure live and silently retired an unrelated one.
+  const failed = (over) => ({
+    id: 'sherpa-onnx',
+    name: 'sherpa-onnx',
+    fullName: 'sherpa-onnx',
+    progress: 0,
+    taskType: 'installation',
+    isBackend: true,
+    isQueued: false,
+    isDeletion: false,
+    cancellable: false,
+    error: 'no space left on device',
+    ...over,
+  })
+  await stub(page, {
+    operations: [
+      failed({ jobID: 'job-local' }),
+      failed({ jobID: 'job-node', nodeID: 'node-1' }),
+    ],
+  })
+
+  const calls = []
+  await page.route('**/api/operations/*/dismiss', (route) => {
+    calls.push(new URL(route.request().url()).pathname)
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+  await page.route('**/api/nodes/*/backends/install', (route) => {
+    calls.push(new URL(route.request().url()).pathname)
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+
+  await page.goto('/app/activity')
+
+  // Nothing on screen tells the two cards apart, which is the point: they
+  // share a name and an id, and only the jobID behind each one differs. The
+  // node-scoped job is second in the payload, so it is the second card.
+  await expect(page.locator('.operation-card')).toHaveCount(2)
+  await page.locator('.operation-card').nth(1).locator('.operation-card__retry').click()
+
+  await expect.poll(() => calls).toEqual([
+    '/api/operations/job-node/dismiss',
+    '/api/nodes/node-1/backends/install',
+  ])
+})
+
+test('the dismiss control also acts on the job it belongs to', async ({ page }) => {
+  // Same hazard as retry: the card's X passed the display id too.
+  const failed = (over) => ({
+    id: 'sherpa-onnx',
+    name: 'sherpa-onnx',
+    fullName: 'sherpa-onnx',
+    progress: 0,
+    taskType: 'installation',
+    isBackend: true,
+    isQueued: false,
+    isDeletion: false,
+    cancellable: false,
+    error: 'no space left on device',
+    ...over,
+  })
+  await stub(page, {
+    operations: [
+      failed({ jobID: 'job-local' }),
+      failed({ jobID: 'job-node', nodeID: 'node-1' }),
+    ],
+  })
+
+  const dismissed = []
+  await page.route('**/api/operations/*/dismiss', (route) => {
+    dismissed.push(new URL(route.request().url()).pathname)
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.operation-card')).toHaveCount(2)
+  await page.locator('.operation-card').nth(1).locator('.operation-card__hide').click()
+
+  await expect.poll(() => dismissed).toEqual(['/api/operations/job-node/dismiss'])
+})
+
+test('a filter matching nothing does not claim the instance is empty', async ({ page }) => {
+  // Three model records on file: telling the user nothing has ever run, while
+  // the header counts those same three, is simply false.
+  await stub(page, {
+    history: [1, 2, 3].map((n) => ({
+      id: `model-${n}`,
+      name: `model-${n}`,
+      jobID: `job-${n}`,
+      isBackend: false,
+      taskType: 'installation',
+      outcome: 'completed',
+      startedAt: '2026-07-28T13:40:00Z',
+      finishedAt: '2026-07-28T13:40:20Z',
+    })),
+  })
+
+  await page.goto('/app/activity')
+  await expect(page.locator('.activity-row')).toHaveCount(3)
+
+  await page.locator('.activity-chip', { hasText: 'Backends' }).click()
+
+  await expect(page.locator('.activity-empty--filtered')).toBeVisible()
+  await expect(page.locator('.activity-empty')).not.toContainText('No operations since startup')
+  // And the way back out is on screen.
+  await page.locator('.activity-empty--filtered button').click()
+  await expect(page.locator('.activity-row')).toHaveCount(3)
+})
+
+test('the summary drops a zero count instead of reporting it', async ({ page }) => {
+  await stub(page, {
+    operations: [{
+      id: 'model-a',
+      name: 'model-a',
+      jobID: 'job-a',
+      progress: 40,
+      taskType: 'installation',
+      isBackend: false,
+      isQueued: false,
+      isDeletion: false,
+      cancellable: true,
+    }],
+  })
+
+  await page.goto('/app/activity')
+
+  const supporting = page.locator('.page-header__supporting')
+  await expect(supporting).toHaveText('1 operation running.')
+  await expect(supporting).not.toContainText('0')
+})
+
+test('a cancelled deletion reports the cancellation, not a removal', async ({ page }) => {
+  await stub(page, {
+    history: [{
+      id: 'model-a',
+      name: 'model-a',
+      jobID: 'job-a',
+      isBackend: false,
+      taskType: 'deletion',
+      outcome: 'cancelled',
+      startedAt: '2026-07-28T13:40:00Z',
+      finishedAt: '2026-07-28T13:40:02Z',
+    }],
+  })
+
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.activity-row')).toContainText('cancelled')
+  await expect(page.locator('.activity-row')).not.toContainText('removed')
+})
+
+test('an implausible or zero duration never reaches the row', async ({ page }) => {
+  await stub(page, {
+    history: [
+      {
+        id: 'zero-span',
+        name: 'zero-span',
+        jobID: 'job-zero',
+        isBackend: false,
+        taskType: 'installation',
+        outcome: 'completed',
+        // recordTerminal seeds StartedAt = FinishedAt and only overwrites it
+        // with a real stamp, so this is an ordinary arrival.
+        startedAt: '2026-07-28T13:40:00Z',
+        finishedAt: '2026-07-28T13:40:00Z',
+      },
+      {
+        id: 'zero-stamp',
+        name: 'zero-stamp',
+        jobID: 'job-stamp',
+        isBackend: false,
+        taskType: 'installation',
+        outcome: 'completed',
+        startedAt: '0001-01-01T00:00:00Z',
+        finishedAt: '2026-07-28T13:41:00Z',
+      },
+    ],
+  })
+
+  await page.goto('/app/activity')
+
+  const zeroSpan = page.locator('.activity-row').filter({ hasText: 'zero-span' })
+  await expect(zeroSpan).toContainText('installed in < 1s')
+
+  // A zero-value Go stamp is not a duration. The row says what happened and
+  // stops, rather than stating a span of millennia as fact.
+  const zeroStamp = page.locator('.activity-row').filter({ hasText: 'zero-stamp' })
+  await expect(zeroStamp).toContainText('installed')
+  await expect(zeroStamp).not.toContainText('installed in')
+})
+
+test('a failed removal offers no retry, because retry only means install', async ({ page }) => {
+  await stub(page, {
+    operations: [{
+      id: 'model-a',
+      name: 'model-a',
+      fullName: 'model-a',
+      jobID: 'job-a',
+      progress: 0,
+      taskType: 'deletion',
+      isBackend: false,
+      isQueued: false,
+      isDeletion: true,
+      cancellable: false,
+      error: 'file is busy',
+    }],
+  })
+
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.operation-card--error')).toBeVisible()
+  await expect(page.locator('.operation-card__retry')).toHaveCount(0)
+  // And it must not claim an install was attempted.
+  await expect(page.locator('.operation-card--error')).not.toContainText('install')
+})
+
+test('filters the record down to backends', async ({ page }) => {
+  await stub(page, {
+    history: [
+      {
+        id: 'gemma-3-27b-it',
+        name: 'gemma-3-27b-it',
+        jobID: 'job-gemma',
+        isBackend: false,
+        taskType: 'installation',
+        outcome: 'completed',
+        startedAt: '2026-07-28T13:40:00Z',
+        finishedAt: '2026-07-28T13:41:30Z',
+      },
+      {
+        id: 'bark-cpp',
+        name: 'bark-cpp',
+        jobID: 'job-bark',
+        isBackend: true,
+        taskType: 'installation',
+        outcome: 'completed',
+        startedAt: '2026-07-28T13:40:00Z',
+        finishedAt: '2026-07-28T13:40:20Z',
+      },
+    ],
+  })
+
+  await page.goto('/app/activity')
+  await expect(page.locator('.activity-row')).toHaveCount(2)
+
+  await page.locator('.activity-chip', { hasText: 'Backends' }).click()
+
+  await expect(page.locator('.activity-row')).toHaveCount(1)
+  await expect(page.locator('.activity-row')).toContainText('bark-cpp')
+})
+
+test('shows the empty state when nothing has run', async ({ page }) => {
+  await stub(page)
+
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.page-title')).toBeVisible()
+  await expect(page.locator('.activity-empty')).toBeVisible()
+})

core/http/react-ui/e2e/model-artifact-operation.spec.js

@@ -1,6 +1,6 @@
 import { test, expect } from './coverage-fixtures.js'
 
-test('operations bar shows managed model acquisition phase and bytes', async ({ page }) => {
+test('operations strip shows managed model acquisition phase and bytes', async ({ page }) => {
   await page.route('**/api/operations', (route) => route.fulfill({
     contentType: 'application/json',
     body: JSON.stringify({
@@ -14,7 +14,6 @@ test('operations bar shows managed model acquisition phase and bytes', async ({
         isDeletion: false,
         isBackend: false,
         isQueued: false,
-        isCancelled: false,
         cancellable: true,
         phase: 'downloading',
         currentBytes: 1073741824,
@@ -22,19 +21,12 @@ test('operations bar shows managed model acquisition phase and bytes', async ({
       }],
     }),
   }))
-  let cancelledPath = ''
-  await page.route('**/api/operations/artifact-job-123/cancel', (route) => {
-    cancelledPath = new URL(route.request().url()).pathname
-    return route.fulfill({ contentType: 'application/json', body: '{}' })
-  })
 
   await page.goto('/app/models')
-  const operation = page.locator('.operation-item').filter({ hasText: 'qwen-asr' })
-  await expect(operation).toContainText('Downloading model files')
-  await expect(operation).toContainText('1 GB / 4 GB')
-  await expect(operation.locator('.operation-progress')).toHaveText('45%')
-  await expect(operation.locator('.operation-bar')).toHaveAttribute('style', /width: 45%/)
-
-  await operation.getByTitle('Cancel').click()
-  expect(cancelledPath).toBe('/api/operations/artifact-job-123/cancel')
+  const strip = page.locator('.operations-strip')
+  await expect(strip.locator('.operations-strip__name')).toHaveText('qwen-asr')
+  await expect(strip).toContainText('Downloading')
+  await expect(strip).toContainText('1 GB / 4 GB')
+  await expect(strip.locator('.operations-strip__pct')).toHaveText('45%')
+  await expect(strip.locator('.operations-strip__fill')).toHaveAttribute('style', /width: 45%/)
 })

core/http/react-ui/e2e/nodes-per-node-backend-actions.spec.js

@@ -102,6 +102,9 @@ test.describe('Nodes page — per-node backend actions', () => {
     await mockDistributedNodes(page)
     await openNodeDetail(page)
 
+    await expect(page.locator('.node-detail__metrics')).toContainText('RAM')
+    await expect(page.locator('.node-detail__metrics')).toContainText('3.7 GB / 7.5 GB')
+
     // Negative: the old, ambiguous wording must not be used.
     await expect(page.locator('button[title="Reinstall backend"]')).toHaveCount(0)
     await expect(page.locator('button[title="Reinstall backend"] i.fa-sync-alt')).toHaveCount(0)

core/http/react-ui/e2e/nodes-roster.spec.js

@@ -26,6 +26,23 @@ test.describe('Nodes roster header', () => {
 })
 
 test.describe('Nodes roster panels', () => {
+  test('shows used and total system RAM reported by a worker', async ({ page }) => {
+    await mockCluster(page, [
+      {
+        id: 'n1',
+        name: 'alpha',
+        node_type: 'backend',
+        address: '10.0.0.1:50051',
+        status: 'healthy',
+        total_ram: 8_000_000_000,
+        available_ram: 3_000_000_000,
+      },
+    ])
+
+    await page.goto('/app/nodes')
+    await expect(page.locator('.node-panel').filter({ hasText: 'alpha' })).toContainText('RAM 4.7 GB / 7.5 GB', { timeout: 15_000 })
+  })
+
   test('shows model chips without clicking and filters by type', async ({ page }) => {
     await page.route('**/api/nodes', r => r.fulfill({ status: 200, contentType: 'application/json', body: JSON.stringify([
       { id: 'n1', name: 'alpha', node_type: 'backend', address: '10.0.0.1:50051', status: 'healthy' },

core/http/react-ui/e2e/operations-strip.spec.js

@@ -0,0 +1,216 @@
+import { test, expect } from './coverage-fixtures.js'
+
+const op = (over = {}) => ({
+  id: 'model-a',
+  name: 'model-a',
+  fullName: 'model-a',
+  jobID: 'job-a',
+  progress: 40,
+  taskType: 'installation',
+  isDeletion: false,
+  isBackend: false,
+  isQueued: false,
+  cancellable: true,
+  ...over,
+})
+
+const stubOperations = (page, operations) =>
+  page.route('**/api/operations', (route) => route.fulfill({
+    contentType: 'application/json',
+    body: JSON.stringify({ operations }),
+  }))
+
+test('renders exactly one row for four concurrent operations', async ({ page }) => {
+  await stubOperations(page, [
+    op({ id: 'model-a', name: 'model-a', jobID: 'job-a', progress: 10 }),
+    op({ id: 'model-b', name: 'model-b', jobID: 'job-b', progress: 40 }),
+    op({ id: 'model-c', name: 'model-c', jobID: 'job-c', progress: 70 }),
+    op({ id: 'model-d', name: 'model-d', jobID: 'job-d', progress: 90 }),
+  ])
+
+  await page.goto('/app/models')
+
+  // One row, never four. The stacked bar is what this replaces.
+  await expect(page.locator('.operations-strip')).toHaveCount(1)
+  // The API sorts by progress ascending, so the least advanced op leads.
+  await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
+  await expect(page.locator('.operations-strip__more')).toContainText('3')
+  await expect(page.locator('.operations-strip__more')).toHaveAttribute('href', /\/app\/activity$/)
+})
+
+test('a failure takes the strip over a running install', async ({ page }) => {
+  await stubOperations(page, [
+    op({ id: 'model-a', name: 'model-a', jobID: 'job-a', progress: 10 }),
+    op({ id: 'sherpa-onnx', name: 'sherpa-onnx', jobID: 'job-f', isBackend: true, error: 'no space left on device' }),
+  ])
+
+  await page.goto('/app/models')
+
+  await expect(page.locator('.operations-strip__name')).toHaveText('sherpa-onnx')
+  await expect(page.locator('.operations-strip')).toContainText('no space left on device')
+})
+
+test('a hidden strip comes back when a different operation becomes primary', async ({ page }) => {
+  // Hiding must not be able to silence a later failure, so the hidden state is
+  // keyed by job rather than being a blanket mute.
+  //
+  // The swap is driven by the test rather than by a poll count: a count would
+  // race the click, and a click landing on the failure would take the dismiss
+  // path and fail this test for an unrelated reason.
+  let swapped = false
+  await page.route('**/api/operations', (route) => {
+    const operations = swapped
+      ? [op({ id: 'sherpa-onnx', name: 'sherpa-onnx', jobID: 'job-f', error: 'no space left on device' })]
+      : [op({ id: 'model-a', name: 'model-a', jobID: 'job-a' })]
+    return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
+  })
+
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
+
+  await page.locator('.operations-strip__hide').click()
+  await expect(page.locator('.operations-strip')).toHaveCount(0)
+
+  // The poller swaps in a different job, which must re-render the strip.
+  swapped = true
+  await expect(page.locator('.operations-strip__name')).toHaveText('sherpa-onnx', { timeout: 10_000 })
+})
+
+test('hiding a running operation does not silence that same job failing', async ({ page }) => {
+  // A job keeps its jobID when it fails, so keying the hidden state on the job
+  // alone would let a user mute the very failure they need to see.
+  let failing = false
+  await page.route('**/api/operations', (route) => {
+    const operations = failing
+      ? [op({ error: 'no space left on device' })]
+      : [op()]
+    return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
+  })
+
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip__name')).toHaveText('model-a')
+
+  await page.locator('.operations-strip__hide').click()
+  await expect(page.locator('.operations-strip')).toHaveCount(0)
+
+  failing = true
+  await expect(page.locator('.operations-strip')).toContainText('no space left on device', { timeout: 10_000 })
+})
+
+test('a long error message does not widen the page', async ({ page }) => {
+  // An install error is arbitrarily long text, and the strip sits above every
+  // page: if it cannot shrink, every page under it gets a horizontal scrollbar.
+  await stubOperations(page, [op({ error: `disk write failed: ${'x'.repeat(180)}` })])
+
+  await page.setViewportSize({ width: 1280, height: 800 })
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip')).toBeVisible()
+
+  const widths = await page.evaluate(() => ({
+    scroll: document.documentElement.scrollWidth,
+    client: document.documentElement.clientWidth,
+  }))
+  expect(widths.scroll).toBeLessThanOrEqual(widths.client)
+})
+
+test('progress is exposed to assistive tech as a named progressbar', async ({ page }) => {
+  // The percentage text is aria-hidden so the live region stops re-announcing
+  // the strip once a second; the value has to reach assistive tech some other
+  // way, and a progressbar is read on demand rather than announced.
+  await stubOperations(page, [op({ progress: 45 })])
+
+  await page.goto('/app/models')
+  const bar = page.locator('.operations-strip__track')
+  await expect(bar).toHaveAttribute('role', 'progressbar')
+  await expect(bar).toHaveAttribute('aria-valuenow', '45')
+  await expect(bar).toHaveAttribute('aria-valuemin', '0')
+  await expect(bar).toHaveAttribute('aria-valuemax', '100')
+  await expect(bar).toHaveAttribute('aria-label', /model-a/)
+})
+
+test('an operation waiting for the worker says queued, not installing', async ({ page }) => {
+  // The real payload for an admitted-but-unstarted op: phase "queued", no
+  // progress. It used to arrive with isQueued false, so the one state the
+  // strip has a clock icon for never appeared and a queued install claimed to
+  // be running.
+  await stubOperations(page, [op({ isQueued: true, phase: 'queued', progress: 0 })])
+
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip')).toContainText('Queued')
+  await expect(page.locator('.operations-strip')).not.toContainText('Installing')
+  await expect(page.locator('.operations-strip__pct')).toHaveCount(0)
+})
+
+test('cancelling the last operation does not announce it as installed', async ({ page }) => {
+  // Cancelling deletes the operation server side, so the strip sees exactly
+  // what it sees when an install finishes: the operation stops being listed.
+  // The completion hold used to take that for success and put a green
+  // "Installed model model-a" on screen for four seconds after the user
+  // called it off.
+  let cancelled = false
+  await page.route('**/api/operations', (route) => route.fulfill({
+    contentType: 'application/json',
+    body: JSON.stringify({ operations: cancelled ? [] : [op()] }),
+  }))
+  await page.route('**/api/operations/history', (route) => route.fulfill({
+    contentType: 'application/json',
+    body: JSON.stringify({ operations: [] }),
+  }))
+  await page.route('**/api/operations/job-a/cancel', (route) => {
+    cancelled = true
+    return route.fulfill({ contentType: 'application/json', body: '{"success":true}' })
+  })
+
+  // Cancel is on the card, never on the strip, so the page is where the user
+  // does this. The strip sits above it the whole time.
+  await page.goto('/app/activity')
+  await expect(page.locator('.operations-strip')).toContainText('Installing model')
+
+  await page.locator('.operation-card__cancel').click()
+
+  // The poll that empties the page is the same render that would raise the
+  // completion hold, so the strip has to be counted the instant the card
+  // goes. A retrying assertion would simply wait the four-second hold out and
+  // pass on the regression it is here to catch.
+  await expect(page.locator('.operation-card')).toHaveCount(0)
+  expect(await page.locator('.operations-strip').count()).toBe(0)
+
+  // And it must not turn up a moment later either.
+  await page.waitForTimeout(1500)
+  expect(await page.locator('.operations-strip').count()).toBe(0)
+})
+
+test('a completed removal does not announce an install', async ({ page }) => {
+  // The API drops an operation the instant it finishes, so the completion
+  // phrase is rendered from the operation the strip was already holding.
+  let removed = false
+  await page.route('**/api/operations', (route) => {
+    const operations = removed ? [] : [op({ isDeletion: true, progress: 0 })]
+    return route.fulfill({ contentType: 'application/json', body: JSON.stringify({ operations }) })
+  })
+
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip')).toContainText('Removing model')
+
+  removed = true
+  await expect(page.locator('.operations-strip')).toContainText('Removed model', { timeout: 10_000 })
+  await expect(page.locator('.operations-strip')).not.toContainText('Installed')
+})
+
+test('the hide button hides the strip without cancelling', async ({ page }) => {
+  await stubOperations(page, [op()])
+
+  let cancelCalled = false
+  await page.route('**/api/operations/*/cancel', (route) => {
+    cancelCalled = true
+    return route.fulfill({ contentType: 'application/json', body: '{}' })
+  })
+
+  await page.goto('/app/models')
+  await expect(page.locator('.operations-strip')).toBeVisible()
+
+  await page.locator('.operations-strip__hide').click()
+
+  await expect(page.locator('.operations-strip')).toHaveCount(0)
+  expect(cancelCalled).toBe(false)
+})

core/http/react-ui/e2e/page-render-smoke.spec.js

@@ -19,6 +19,7 @@ const PAGES = [
   ['/app/studio', 'Studio'],
   ['/app/manage', 'Manage'],
   ['/app/backends', 'Backends'],
+  ['/app/activity', 'Activity'],
   ['/app/settings', 'Settings'],
   ['/app/nodes', 'Nodes'],
   ['/app/scheduling', 'Scheduling'],

core/http/react-ui/e2e/real-binary-activity.spec.js

@@ -0,0 +1,73 @@
+import { test, expect } from '@playwright/test'
+
+// Runs against a REAL local-ai binary with NO route stubbing.
+//
+// Every other spec here stubs /api/operations. That is how a payload the server
+// could not actually emit (isDeletion:true on a live operation) stayed green
+// through an entire review while the UI rendered a removal as an install. These
+// assertions are only worth anything because the data came from the real handler.
+//
+//   make build
+//   ./local-ai run --address 127.0.0.1:8089 --models-path /tmp/lai-e2e
+//   cd core/http/react-ui
+//   LOCALAI_REAL_BINARY=1 PLAYWRIGHT_EXTERNAL_SERVER=1 PW_WORKERS=1 \
+//     npx playwright test e2e/real-binary-activity.spec.js
+//
+// Skipped by default so CI, which runs the stub server, is unaffected.
+test.skip(!process.env.LOCALAI_REAL_BINARY, 'needs a real local-ai on 127.0.0.1:8089')
+test.describe.configure({ mode: 'serial' })
+
+const MISSING = 'definitely-not-a-real-model-xyz'
+
+test('a real failed install lands in Needs attention with the real error and a Retry', async ({ page, request }) => {
+  // Self-contained: the gallery resolver rejects an unknown name, so this fails
+  // fast without downloading anything.
+  await request.post(`/api/models/install/${MISSING}`)
+
+  await page.goto('/app/activity')
+  await expect(page.locator('.page-title')).toBeVisible()
+
+  const card = page.locator('.operation-card--error').filter({ hasText: MISSING })
+  await expect(card).toBeVisible({ timeout: 20_000 })
+  await expect(card).toContainText('no model found with name')
+  // Retry is offered for a failed install; it is gated off for a failed removal.
+  await expect(card.locator('.operation-card__retry')).toBeVisible()
+})
+
+test('the strip shows the real failure on one line and does not widen the page', async ({ page }) => {
+  await page.goto('/app/activity')
+
+  const strip = page.locator('.operations-strip')
+  await expect(strip).toHaveCount(1)
+  await expect(strip).toHaveClass(/operations-strip--error/)
+  await expect(strip.locator('.operations-strip__name')).toHaveText(MISSING)
+
+  const box = await page.evaluate(() => ({
+    scroll: document.documentElement.scrollWidth,
+    client: document.documentElement.clientWidth,
+  }))
+  expect(box.scroll).toBeLessThanOrEqual(box.client)
+})
+
+test('dismissing moves the failure into the record instead of destroying it', async ({ page }) => {
+  await page.goto('/app/activity')
+
+  const card = page.locator('.operation-card--error').filter({ hasText: MISSING })
+  await expect(card).toBeVisible()
+  await card.locator('.operation-card__hide').click()
+
+  await expect(page.locator('.operation-card--error')).toHaveCount(0, { timeout: 15_000 })
+  const row = page.locator('.activity-row').filter({ hasText: MISSING })
+  await expect(row).toBeVisible({ timeout: 15_000 })
+  await expect(row).toContainText('failed')
+})
+
+test('Clear history empties the record against the real store', async ({ page }) => {
+  await page.goto('/app/activity')
+
+  await expect(page.locator('.activity-row').first()).toBeVisible()
+  await page.getByRole('button', { name: /clear history/i }).click()
+
+  await expect(page.locator('.activity-row')).toHaveCount(0, { timeout: 15_000 })
+  await expect(page.locator('.activity-empty')).toBeVisible()
+})

core/http/react-ui/public/locales/de/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "System",
     "subtitle": "Verwalten Sie installierte Modelle und Backends"

core/http/react-ui/public/locales/de/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "Start",
@@ -55,7 +56,8 @@
     "system": "System",
     "settings": "Einstellungen",
     "api": "API",
-    "middleware": "Middleware"
+    "middleware": "Middleware",
+    "activity": "Aktivität"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/en/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "System",
     "subtitle": "Manage installed models and backends"

core/http/react-ui/public/locales/en/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "Home",
@@ -56,7 +57,8 @@
     "swarm": "Swarm",
     "system": "System",
     "settings": "Settings",
-    "api": "API"
+    "api": "API",
+    "activity": "Activity"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/es/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "Sistema",
     "subtitle": "Administra modelos y backends instalados"

core/http/react-ui/public/locales/es/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "Inicio",
@@ -55,7 +56,8 @@
     "system": "Sistema",
     "settings": "Configuración",
     "api": "API",
-    "middleware": "Middleware"
+    "middleware": "Middleware",
+    "activity": "Actividad"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/id/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "Sistem",
     "subtitle": "Kelola model dan backend yang terinstal"

core/http/react-ui/public/locales/id/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Kluster",
     "observability": "Observabilitas",
     "access": "Akses",
-    "system": "Sistem"
+    "system": "Sistem",
+    "activity": "Activity"
   },
   "items": {
     "home": "Beranda",
@@ -55,7 +56,8 @@
     "swarm": "Swarm",
     "system": "Sistem",
     "settings": "Pengaturan",
-    "api": "API"
+    "api": "API",
+    "activity": "Aktivitas"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/it/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "Sistema",
     "subtitle": "Gestisci modelli e backend installati"

core/http/react-ui/public/locales/it/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "Home",
@@ -55,7 +56,8 @@
     "system": "Sistema",
     "settings": "Impostazioni",
     "api": "API",
-    "middleware": "Middleware"
+    "middleware": "Middleware",
+    "activity": "Attività"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/ko/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "시스템",
     "subtitle": "설치된 모델과 백엔드를 관리합니다"

core/http/react-ui/public/locales/ko/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "홈",
@@ -55,7 +56,8 @@
     "swarm": "Swarm",
     "system": "시스템",
     "settings": "설정",
-    "api": "API"
+    "api": "API",
+    "activity": "활동"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/public/locales/zh-CN/admin.json

@@ -1,4 +1,84 @@
 {
+  "activity": {
+    "title": "Activity",
+    "supporting": "Installs, downloads and removals on this instance.",
+    "hide": "Hide",
+    "moveToHistory": "Move to history",
+    "moreCount": "{{count}} more",
+    "waitingForInstaller": "waiting for the installer",
+    "progressLabel": "Progress for {{name}}",
+    "toNode": "to {{node}}",
+    "nodesDone": "{{done}} of {{total}} nodes done",
+    "timeLeft": "{{value}} left",
+    "cancel": "Cancel",
+    "cancelLabel": "Cancel {{name}}",
+    "retry": "Retry",
+    "retryLabel": "Retry {{name}}",
+    "nodeCount": "{{count}} nodes",
+    "showNodes": "Show {{count}} nodes",
+    "hideNodes": "Hide per-node detail",
+    "node": {
+      "done": "Done",
+      "failed": "Failed",
+      "queued": "Queued",
+      "workerBusy": "Worker busy",
+      "downloading": "Downloading"
+    },
+    "kind": {
+      "model": "model",
+      "backend": "backend"
+    },
+    "verb": {
+      "installing": "Installing {{kind}}",
+      "installed": "Installed {{kind}}",
+      "removed": "Removed {{kind}}",
+      "staged": "Staged model",
+      "failed": "Couldn't install {{kind}}",
+      "queued": "Queued",
+      "staging": "Staging model",
+      "removing": "Removing {{kind}}",
+      "failedRemoval": "Couldn't remove {{kind}}",
+      "failedStaging": "Couldn't stage model"
+    },
+    "phase": {
+      "resolving": "Resolving files",
+      "downloading": "Downloading",
+      "verifying": "Verifying",
+      "committing": "Finalizing",
+      "persisting": "Saving configuration"
+    },
+    "clearHistory": "Clear history",
+    "inProgress": "In progress",
+    "needsAttention": "Needs attention",
+    "record": "Record",
+    "historyNote": "Keeps the last 50 operations. History is in memory and resets when LocalAI restarts.",
+    "emptyTitle": "No operations since startup",
+    "emptyBody": "Model and backend installs appear here while they run and stay as a record afterwards. History is kept in memory, so it resets when LocalAI restarts.",
+    "browseModels": "Browse models",
+    "viewInModels": "View in Models",
+    "viewInBackends": "View in Backends",
+    "rowInstalled": "installed in {{duration}}",
+    "rowFailed": "failed: {{error}}",
+    "rowCancelled": "cancelled",
+    "rowRemoved": "removed",
+    "retryFailed": "Retry failed: {{message}}",
+    "filter": {
+      "all": "All",
+      "models": "Models",
+      "backends": "Backends",
+      "cluster": "Cluster"
+    },
+    "summaryRunning_one": "{{count}} operation running.",
+    "summaryRunning_other": "{{count}} operations running.",
+    "summaryFailed_one": "{{count}} operation needs attention.",
+    "summaryFailed_other": "{{count}} operations need attention.",
+    "summaryQuiet_one": "Nothing running. {{count}} operation since startup.",
+    "summaryQuiet_other": "Nothing running. {{count}} operations since startup.",
+    "summaryIdle": "Nothing running.",
+    "emptyFiltered": "No operations match this filter.",
+    "showAll": "Show all",
+    "rowInstalledPlain": "installed"
+  },
   "manage": {
     "title": "系统",
     "subtitle": "管理已安装的模型和后端"

core/http/react-ui/public/locales/zh-CN/nav.json

@@ -23,7 +23,8 @@
     "cluster": "Cluster",
     "observability": "Observability",
     "access": "Access",
-    "system": "System"
+    "system": "System",
+    "activity": "Activity"
   },
   "items": {
     "home": "首页",
@@ -55,7 +56,8 @@
     "system": "系统",
     "settings": "设置",
     "api": "API",
-    "middleware": "Middleware"
+    "middleware": "Middleware",
+    "activity": "活动"
   },
   "footer": {
     "github": "GitHub",

core/http/react-ui/src/App.css

@@ -13,6 +13,11 @@
   min-height: 100dvh;
   display: flex;
   flex-direction: column;
+  /* A flex item's automatic minimum is its content's minimum, so without this
+     any single wide descendant (a long install error in the operations strip,
+     a wide table on a phone) drags the whole column past the viewport and
+     every page gets a horizontal scrollbar. */
+  min-width: 0;
   transition: margin-left var(--duration-normal) var(--ease-default);
 }
 
@@ -663,41 +668,151 @@
   to { transform: rotate(360deg); }
 }
 
-/* Operations bar */
-.operations-bar {
-  background: var(--color-bg-secondary);
-  border-bottom: 1px solid var(--color-border-subtle);
-  padding: var(--spacing-xs) var(--spacing-md);
-}
-
-.operation-text {
-  font-family: var(--font-mono);
-}
-.operation-progress {
-  font-variant-numeric: tabular-nums;
-}
-
-.operation-item {
-  display: flex;
-  align-items: center;
-  gap: var(--spacing-md);
-  padding: var(--spacing-xs) 0;
-  flex-wrap: wrap;
-}
-
-.operation-info {
+/* Operations strip: always exactly one line. Anything that does not fit here
+   belongs on /app/activity. */
+.operations-strip {
   display: flex;
   align-items: center;
   gap: var(--spacing-sm);
-  flex: 2 1 0;
+  min-height: 40px;
+  padding: var(--spacing-xs) var(--spacing-md);
+  background: var(--color-bg-secondary);
+  border-bottom: 1px solid var(--color-border-subtle);
+  border-left: 2px solid var(--color-primary);
+  font-size: 0.8125rem;
+  /* An install error is arbitrarily long text. This clips it; the shrinking
+     is done by min-width on .main-content above and on __detail below. */
+  overflow: hidden;
+}
+.operations-strip--error { border-left-color: var(--color-error); background: var(--color-error-light); }
+.operations-strip--queued { border-left-color: var(--color-text-disabled); }
+.operations-strip--staging { border-left-color: var(--color-info); }
+.operations-strip--removing { border-left-color: var(--color-warning); }
+.operations-strip--done { border-left-color: var(--color-success); }
+
+.operations-strip__icon { flex: none; font-size: 0.8125rem; }
+.operations-strip--error .operations-strip__icon { color: var(--color-error); }
+.operations-strip--queued .operations-strip__icon { color: var(--color-text-muted); }
+.operations-strip--staging .operations-strip__icon { color: var(--color-info); }
+.operations-strip--removing .operations-strip__icon { color: var(--color-warning); }
+.operations-strip--done .operations-strip__icon { color: var(--color-success); }
+
+.operations-strip__spinner {
+  width: 13px;
+  height: 13px;
+  flex: none;
+  border-radius: 50%;
+  border: 2px solid var(--color-primary-light);
+  border-top-color: var(--color-primary);
+  animation: operationsStripSpin 0.9s linear infinite;
+}
+@keyframes operationsStripSpin { to { transform: rotate(360deg); } }
+
+.operations-strip__verb { flex: none; color: var(--color-text-secondary); white-space: nowrap; }
+.operations-strip__name {
+  font-family: var(--font-mono);
+  font-weight: 500;
+  white-space: nowrap;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  /* overflow: hidden zeroes the automatic minimum size, so without a floor the
+     name loses the shrink contest to a long error and the identity of the
+     thing that broke is the first casualty. */
+  min-width: 12ch;
+}
+.operations-strip__sep { flex: none; color: var(--color-border-strong); }
+/* min-width and overflow are what let these shrink: a nowrap flex item's
+   automatic minimum size is otherwise its full content width. */
+.operations-strip__detail,
+.operations-strip__bytes {
+  flex: 0 1 auto;
   min-width: 0;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  color: var(--color-text-muted);
+  font-size: 0.75rem;
+  white-space: nowrap;
+}
+.operations-strip__bytes { font-variant-numeric: tabular-nums; }
+.operations-strip__spacer { flex: 1 1 auto; min-width: var(--spacing-xs); }
+.operations-strip__pct {
+  flex: none;
+  font-family: var(--font-mono);
+  font-variant-numeric: tabular-nums;
+  font-size: 0.75rem;
+  font-weight: 500;
+  color: var(--color-primary);
+}
+.operations-strip__track {
+  flex: 0 0 132px;
+  height: 3px;
+  border-radius: var(--radius-full);
+  background: var(--color-surface-sunken);
+  overflow: hidden;
+}
+.operations-strip__fill {
+  display: block;
+  height: 100%;
+  border-radius: var(--radius-full);
+  background: var(--color-primary);
+  transition: width var(--duration-slow) var(--ease-spring);
+}
+.operations-strip--staging .operations-strip__fill { background: var(--color-info); }
+
+.operations-strip__more {
+  display: inline-flex;
+  align-items: center;
+  justify-content: center;
+  flex: none;
+  min-height: 28px;
+  padding: 0 var(--spacing-sm);
+  border-radius: var(--radius-full);
+  border: 1px solid var(--color-primary-border);
+  background: var(--color-primary-light);
+  color: var(--color-primary);
+  font-size: 0.6875rem;
+  font-weight: 600;
+  white-space: nowrap;
+  text-decoration: none;
+}
+.operations-strip__more:hover { background: var(--color-primary); color: var(--color-primary-text); }
+.operations-strip__more--neutral {
+  background: transparent;
+  border-color: var(--color-border-default);
+  color: var(--color-text-secondary);
 }
 
-.operation-info > .operation-text {
-  flex: 1 1 auto;
-  min-width: 0;
+.operations-strip__hide {
+  flex: none;
+  width: 28px;
+  height: 28px;
+  display: grid;
+  place-items: center;
+  border: 0;
+  border-radius: var(--radius-sm);
+  background: transparent;
+  color: var(--color-text-muted);
+  cursor: pointer;
+}
+.operations-strip__hide:hover { background: var(--color-bg-hover); color: var(--color-text-primary); }
+
+/* Narrow screens drop the prose, never the name, the percentage or the
+   counter. */
+@media (max-width: 640px) {
+  .operations-strip__verb,
+  .operations-strip__detail,
+  .operations-strip__bytes,
+  .operations-strip__sep,
+  .operations-strip__track { display: none; }
+  .operations-strip__name { max-width: 45vw; }
 }
 
+@media (prefers-reduced-motion: reduce) {
+  .operations-strip__spinner { animation: none; }
+  .operations-strip__fill { transition: none; }
+}
+
+/* Row-level install indicator, used by the Models and Backends tables. */
 .operation-spinner {
   width: 16px;
   height: 16px;
@@ -710,19 +825,6 @@
   display: inline-block;
 }
 
-.operation-text {
-  font-size: 0.8125rem;
-  color: var(--color-text-secondary);
-  overflow: hidden;
-  text-overflow: ellipsis;
-}
-
-.operation-progress {
-  font-size: 0.75rem;
-  color: var(--color-primary);
-  font-weight: 500;
-}
-
 .operation-bar-container {
   flex: 0 1 160px;
   min-width: 80px;
@@ -760,38 +862,7 @@
   white-space: nowrap;
 }
 
-.operation-cancel {
-  flex-shrink: 0;
-  background: none;
-  border: none;
-  color: var(--color-text-muted);
-  cursor: pointer;
-  padding: 4px 6px;
-  font-size: 0.875rem;
-}
-.operation-cancel:hover {
-  color: var(--color-error);
-}
-
-/* Operations bar: per-node breakdown (multi-worker installs) */
-.operation-expand {
-  background: none;
-  border: none;
-  color: var(--color-text-muted);
-  cursor: pointer;
-  padding: 0 var(--spacing-xs);
-  font-size: var(--text-xs);
-  display: inline-flex;
-  align-items: center;
-  gap: 0.25rem;
-}
-.operation-expand:hover {
-  color: var(--color-text-primary);
-}
-.operation-expand-label {
-  font-size: var(--text-xs);
-}
-
+/* Per-node breakdown of a multi-worker install. */
 .operation-nodes-list {
   list-style: none;
   margin: var(--spacing-xs) 0 0;
@@ -5563,20 +5634,6 @@ button.collapsible-header:focus-visible {
     border-right: 0;
     margin-inline: calc(-1 * var(--spacing-md));
   }
-
-  /* Operations toasts: scroll horizontally instead of wrapping */
-  .operations-bar {
-    overflow-x: auto;
-    flex-wrap: nowrap;
-    -webkit-overflow-scrolling: touch;
-  }
-  .operation-item { flex-shrink: 0; }
-  .operation-text {
-    max-width: 60vw;
-    overflow: hidden;
-    text-overflow: ellipsis;
-    white-space: nowrap;
-  }
 }
 
 /* Reduced motion — disable non-essential transitions for users who
@@ -5584,7 +5641,7 @@ button.collapsible-header:focus-visible {
 @media (prefers-reduced-motion: reduce) {
   .sidebar,
   .page-transition,
-  .operations-bar,
+  .operations-strip,
   .page,
   .main-content {
     transition: none !important;
@@ -9645,3 +9702,224 @@ button.collapsible-header:focus-visible {
   .variant-row__info,
   .variant-row__action { transition: none; }
 }
+
+/* Live operation card on /app/activity. Status is carried by the icon and the
+   tag rather than a coloured rail, so a page of cards does not read as stripes. */
+.operation-card {
+  background: var(--color-bg-secondary);
+  border: 1px solid var(--color-border-subtle);
+  border-radius: var(--radius-md);
+  overflow: hidden;
+}
+.operation-card--error { border-color: var(--color-error-border); background: var(--color-error-light); }
+.operation-card__main { display: flex; align-items: center; gap: var(--spacing-sm); padding: var(--spacing-sm); }
+.operation-card__icon { flex: none; }
+.operation-card__icon--error { color: var(--color-error); }
+.operation-card__icon--staging { color: var(--color-info); }
+.operation-card__icon--removing { color: var(--color-warning); }
+.operation-card__spinner {
+  width: 14px;
+  height: 14px;
+  flex: none;
+  border-radius: 50%;
+  border: 2px solid var(--color-primary-light);
+  border-top-color: var(--color-primary);
+  animation: operationsStripSpin 0.9s linear infinite;
+}
+.operation-card__body { flex: 1; min-width: 0; display: flex; flex-direction: column; gap: 4px; }
+.operation-card__title { display: flex; align-items: center; gap: var(--spacing-xs); flex-wrap: wrap; }
+.operation-card__name { font-family: var(--font-mono); font-weight: 500; }
+.operation-card__tag {
+  font-size: 0.625rem;
+  letter-spacing: 0.07em;
+  text-transform: uppercase;
+  font-weight: 600;
+  padding: 1px var(--spacing-xs);
+  border-radius: var(--radius-sm);
+  border: 1px solid var(--color-border-default);
+  color: var(--color-text-muted);
+}
+.operation-card__tag--model { color: var(--color-primary); border-color: var(--color-primary-border); background: var(--color-primary-light); }
+.operation-card__tag--backend { color: var(--color-info); border-color: var(--color-info-border); background: var(--color-info-light); }
+.operation-card__tag--cluster { color: var(--color-warning); border-color: var(--color-warning-border); background: var(--color-warning-light); }
+.operation-card__sub { display: flex; align-items: center; gap: var(--spacing-xs); flex-wrap: wrap; font-size: 0.75rem; color: var(--color-text-muted); }
+.operation-card__bytes { font-variant-numeric: tabular-nums; }
+/* The legacy installer message embeds an absolute file path, so it is both long
+   and a single unbreakable token. Left unclamped it wraps to three lines and
+   becomes the largest thing on the card, which is how it looked against a real
+   download. One line, ellipsised, full text in the title. */
+.operation-card__message {
+  min-width: 0;
+  flex: 1 1 0;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+}
+.operation-card__verb { color: var(--color-text-secondary); }
+/* A Go error is arbitrarily long and often multi-line. Clamped so it cannot
+   push the rest of the card off screen; the full text is in the title. */
+.operation-card__error {
+  color: var(--color-error);
+  font-family: var(--font-mono);
+  /* A flex item defaults to min-width:auto, so one unbroken token (a URL
+     inside a Go error) would refuse to shrink and push the sub row past the
+     card. The vertical clamp below cannot help with that. */
+  min-width: 0;
+  display: -webkit-box;
+  -webkit-box-orient: vertical;
+  -webkit-line-clamp: 2;
+  line-clamp: 2;
+  overflow: hidden;
+}
+.operation-card__track { height: 4px; border-radius: var(--radius-full); background: var(--color-surface-sunken); overflow: hidden; }
+.operation-card__fill { display: block; height: 100%; border-radius: var(--radius-full); background: var(--color-primary); transition: width var(--duration-slow) var(--ease-spring); }
+.operation-card__actions { display: flex; align-items: center; gap: var(--spacing-xs); flex: none; }
+.operation-card__pct { font-family: var(--font-mono); font-variant-numeric: tabular-nums; font-weight: 600; color: var(--color-primary); }
+.operation-card__hide {
+  width: 28px;
+  height: 28px;
+  display: grid;
+  place-items: center;
+  border: 0;
+  border-radius: var(--radius-sm);
+  background: transparent;
+  color: var(--color-text-muted);
+  cursor: pointer;
+}
+.operation-card__hide:hover { background: var(--color-bg-hover); color: var(--color-text-primary); }
+.operation-card__nodes-toggle {
+  display: flex;
+  align-items: center;
+  gap: var(--spacing-xs);
+  width: 100%;
+  min-height: 28px;
+  padding: var(--spacing-xs) var(--spacing-sm);
+  border: 0;
+  border-top: 1px solid var(--color-border-subtle);
+  background: transparent;
+  color: var(--color-text-muted);
+  font-size: 0.6875rem;
+  cursor: pointer;
+}
+.operation-card__nodes-toggle:hover { color: var(--color-text-secondary); background: var(--color-bg-hover); }
+
+/* The node list carries its own inset here. It was written for the old bar,
+   whose parent supplied one, and the card only pads its main row. */
+.operation-card .operation-nodes-list { padding: 0 var(--spacing-sm) var(--spacing-xs); }
+/* One rule between the card body and the node block, not two: when the
+   disclosure is there, it is already the separator. */
+.operation-card__nodes-toggle + .operation-nodes-list { margin-top: 0; border-top: 0; }
+.operation-card .operation-node-error {
+  max-width: 40ch;
+  overflow: hidden;
+  text-overflow: ellipsis;
+  white-space: nowrap;
+}
+
+@media (prefers-reduced-motion: reduce) {
+  .operation-card__spinner { animation: none; }
+  .operation-card__fill { transition: none; }
+  .operation-card .operation-node-bar { transition: none; }
+}
+
+/* ── Activity page ────────────────────────────────────────────────────────── */
+.activity-page { display: flex; flex-direction: column; gap: var(--spacing-md); }
+.activity-filters { display: flex; gap: var(--spacing-xs); flex-wrap: wrap; }
+.activity-chip {
+  min-height: 28px;
+  padding: 0 var(--spacing-sm);
+  border-radius: var(--radius-full);
+  border: 1px solid var(--color-border-default);
+  background: transparent;
+  color: var(--color-text-muted);
+  font-size: 0.75rem;
+  cursor: pointer;
+}
+.activity-chip[aria-pressed="true"] {
+  background: var(--color-primary-light);
+  border-color: var(--color-primary-border);
+  color: var(--color-primary);
+  font-weight: 500;
+}
+.activity-section { display: flex; flex-direction: column; gap: var(--spacing-sm); }
+.activity-section__title {
+  display: flex;
+  align-items: baseline;
+  gap: var(--spacing-xs);
+  margin: 0;
+  font-size: 0.75rem;
+  font-weight: 600;
+  letter-spacing: 0.08em;
+  text-transform: uppercase;
+  color: var(--color-text-secondary);
+}
+.activity-section__count { font-family: var(--font-mono); color: var(--color-text-muted); font-weight: 400; }
+.activity-rows {
+  background: var(--color-bg-secondary);
+  border: 1px solid var(--color-border-subtle);
+  border-radius: var(--radius-md);
+  overflow: hidden;
+}
+.activity-row {
+  display: grid;
+  grid-template-columns: 16px 1fr auto auto;
+  gap: var(--spacing-sm);
+  align-items: center;
+  min-height: 38px;
+  padding: var(--spacing-xs) var(--spacing-sm);
+  border-bottom: 1px solid var(--color-border-subtle);
+  font-size: 0.8125rem;
+}
+.activity-row:last-child { border-bottom: 0; }
+.activity-row:hover { background: var(--color-bg-hover); }
+.activity-row__icon--completed { color: var(--color-success); }
+.activity-row__icon--failed { color: var(--color-error); }
+.activity-row__icon--cancelled { color: var(--color-warning); }
+.activity-row__name { font-family: var(--font-mono); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+.activity-row__name small { font-family: var(--font-sans); color: var(--color-text-muted); margin-left: var(--spacing-xs); }
+.activity-row__when { color: var(--color-text-muted); font-size: 0.75rem; font-variant-numeric: tabular-nums; }
+.activity-row__action { color: var(--color-primary); font-size: 0.75rem; text-decoration: none; white-space: nowrap; }
+.activity-row__action:hover { text-decoration: underline; }
+.activity-note { font-size: 0.75rem; color: var(--color-text-muted); margin: 0; }
+.activity-empty {
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+  gap: var(--spacing-xs);
+  padding: var(--spacing-xl) var(--spacing-md);
+  text-align: center;
+}
+.activity-empty__icon { font-size: 1.5rem; color: var(--color-text-disabled); margin-bottom: var(--spacing-xs); }
+.activity-empty__title { margin: 0; font-weight: 500; color: var(--color-text-secondary); }
+.activity-empty__body { margin: 0; max-width: 46ch; font-size: 0.8125rem; color: var(--color-text-muted); }
+/* A chip matching nothing is a smaller event than an empty instance, and gets
+   less of the page so the chip row above stays the obvious thing to change. */
+.activity-empty--filtered { padding: var(--spacing-lg) var(--spacing-md); }
+
+.nav-badge {
+  min-width: 18px;
+  height: 18px;
+  padding: 0 var(--spacing-xs);
+  border-radius: var(--radius-full);
+  background: var(--color-primary);
+  color: var(--color-primary-text);
+  font-family: var(--font-mono);
+  font-size: 0.625rem;
+  font-weight: 700;
+  line-height: 18px;
+  text-align: center;
+  font-variant-numeric: tabular-nums;
+}
+.nav-badge--error { background: var(--color-error); color: var(--color-bg-secondary); }
+/* Collapsed, the label is gone and the item centres its icon: an inline badge
+   would shove that icon off the rail's axis. Pinning it to the corner keeps the
+   count visible without moving anything else. */
+.sidebar.collapsed .nav-badge {
+  position: absolute;
+  top: 2px;
+  right: 4px;
+  min-width: 16px;
+  height: 16px;
+  line-height: 16px;
+  font-size: 0.5625rem;
+}

core/http/react-ui/src/components/OperationCard.jsx

@@ -0,0 +1,233 @@
+import { useId, useState } from 'react'
+import { useTranslation } from 'react-i18next'
+import { formatBytes } from '../utils/format'
+
+const phaseKeys = {
+  resolving: 'activity.phase.resolving',
+  downloading: 'activity.phase.downloading',
+  verifying: 'activity.phase.verifying',
+  committing: 'activity.phase.committing',
+  persisting: 'activity.phase.persisting',
+}
+
+const nodeStatusKeys = {
+  success: 'activity.node.done',
+  error: 'activity.node.failed',
+  queued: 'activity.node.queued',
+  running_on_worker: 'activity.node.workerBusy',
+  downloading: 'activity.node.downloading',
+}
+
+// etaSeconds is derived by OperationsContext from the byte delta between
+// polls. It is absent until two samples exist, and absent for every operation
+// when it is absent for any byte-tracked one.
+function formatEta(seconds) {
+  if (!Number.isFinite(seconds) || seconds <= 0) return ''
+  if (seconds < 60) return `${seconds}s`
+  const minutes = Math.round(seconds / 60)
+  if (minutes < 60) return `${minutes} min`
+  return `${Math.floor(minutes / 60)}h ${minutes % 60}m`
+}
+
+export default function OperationCard({ operation, onCancel, onDismiss, onRetry }) {
+  const { t } = useTranslation('admin')
+  const nodes = Array.isArray(operation.nodes) ? operation.nodes : []
+  // Holds only what the user chose. The default has to stay a live
+  // expression: an operation appears in /api/operations as soon as it is
+  // admitted, but its nodes are filled in later, when the fan-out starts
+  // reporting. State seeded at mount would latch on the empty list.
+  const [nodesOpenOverride, setNodesOpenOverride] = useState(null)
+  const listId = useId()
+
+  const failed = Boolean(operation.error)
+  const name = operation.name || operation.id
+  const kind = operation.isBackend ? t('activity.kind.backend') : t('activity.kind.model')
+
+  // Same chain as the one-line strip, so the two never describe one job
+  // differently. Without it a removal and an install render identically: a
+  // deletion has no phase, no bytes and no nodes to tell them apart.
+  let icon
+  let verb
+  if (failed) {
+    icon = <i className="fas fa-circle-exclamation operation-card__icon operation-card__icon--error" aria-hidden="true" />
+    // The failure phrase has to name the work that actually failed. A removal
+    // or a staging job reported as a failed install describes the opposite of
+    // what happened, and would make the missing Retry button look like a bug.
+    if (operation.isDeletion) verb = t('activity.verb.failedRemoval', { kind })
+    else if (operation.taskType === 'staging') verb = t('activity.verb.failedStaging')
+    else verb = t('activity.verb.failed', { kind })
+  } else if (operation.isQueued) {
+    icon = <i className="fas fa-clock operation-card__icon" aria-hidden="true" />
+    verb = t('activity.verb.queued')
+  } else if (operation.taskType === 'staging') {
+    icon = <i className="fas fa-cloud-arrow-up operation-card__icon operation-card__icon--staging" aria-hidden="true" />
+    verb = t('activity.verb.staging')
+  } else if (operation.isDeletion) {
+    icon = <i className="fas fa-trash operation-card__icon operation-card__icon--removing" aria-hidden="true" />
+    verb = t('activity.verb.removing', { kind })
+  } else {
+    icon = <span className="operation-card__spinner" aria-hidden="true" />
+    verb = t('activity.verb.installing', { kind })
+  }
+
+  const byteLabel = Number.isFinite(operation.currentBytes) && Number.isFinite(operation.totalBytes) && operation.totalBytes > 0
+    ? `${formatBytes(operation.currentBytes)} / ${formatBytes(operation.totalBytes)}`
+    : ''
+  const phaseKey = phaseKeys[operation.phase]
+  const etaLabel = formatEta(operation.etaSeconds)
+  // Same call the strip makes, for the same reason: a failed operation
+  // stopped where it broke and a queued one has not moved, so neither has a
+  // bar worth drawing.
+  const showProgress = !failed && !operation.isQueued && operation.progress > 0
+  const canCancel = operation.cancellable && !failed
+  // Retrying means reconstructing an install call out of the operation, which
+  // is page knowledge. The card offers the button only when the page handed it
+  // a handler, so the control can never be present with nothing behind it.
+  const canRetry = failed && typeof onRetry === 'function'
+
+  // One node needs no disclosure: the single row is the whole story, and a
+  // count-less string would render "Show 1 nodes".
+  const showNodesToggle = nodes.length > 1
+  const nodesOpen = nodesOpenOverride ?? (nodes.length > 0 && nodes.length <= 4)
+  const showNodesList = nodes.length > 0 && (nodesOpen || !showNodesToggle)
+
+  return (
+    <div className={`operation-card${failed ? ' operation-card--error' : ''}`}>
+      <div className="operation-card__main">
+        {icon}
+
+        <div className="operation-card__body">
+          <div className="operation-card__title">
+            <span className="operation-card__name">{name}</span>
+            <span className={`operation-card__tag operation-card__tag--${operation.isBackend ? 'backend' : 'model'}`}>
+              {kind}
+            </span>
+            {nodes.length > 1 && (
+              <span className="operation-card__tag operation-card__tag--cluster">
+                {t('activity.nodeCount', { count: nodes.length })}
+              </span>
+            )}
+          </div>
+
+          <div className="operation-card__sub">
+            <span className="operation-card__verb">{verb}</span>
+            {operation.nodeName && <span>{t('activity.toNode', { node: operation.nodeName })}</span>}
+            {failed && <span className="operation-card__error" title={operation.error}>{operation.error}</span>}
+            {!failed && phaseKey && <span>{t(phaseKey)}</span>}
+            {/* Phases and byte counters exist only on the managed-artifact
+                path, so a legacy files: gallery model and every backend
+                install would otherwise say nothing beyond the verb. The
+                server's own message is the only detail those jobs have. It is
+                skipped while queued because there it is just "queued", which
+                the line below already says in the user's language. */}
+            {!failed && !phaseKey && !operation.isQueued && operation.message && (
+              <span className="operation-card__message" title={operation.message}>{operation.message}</span>
+            )}
+            {!failed && operation.isQueued && <span>{t('activity.waitingForInstaller')}</span>}
+            {!failed && byteLabel && <span className="operation-card__bytes">{byteLabel}</span>}
+            {!failed && etaLabel && <span className="operation-card__bytes">{t('activity.timeLeft', { value: etaLabel })}</span>}
+          </div>
+
+          {showProgress && (
+            <div
+              className="operation-card__track"
+              role="progressbar"
+              aria-valuenow={Math.round(operation.progress)}
+              aria-valuemin={0}
+              aria-valuemax={100}
+              aria-label={t('activity.progressLabel', { name })}
+            >
+              <span className="operation-card__fill" style={{ width: `${operation.progress}%` }} />
+            </div>
+          )}
+        </div>
+
+        <div className="operation-card__actions">
+          {showProgress && <span className="operation-card__pct" aria-hidden="true">{Math.round(operation.progress)}%</span>}
+          {canCancel && (
+            // A page of cards would otherwise hand a screen reader a list of
+            // identical "Cancel" buttons with nothing to tell them apart.
+            <button
+              type="button"
+              className="btn btn-sm btn-danger operation-card__cancel"
+              onClick={() => onCancel?.(operation.jobID)}
+              aria-label={t('activity.cancelLabel', { name })}
+            >
+              {t('activity.cancel')}
+            </button>
+          )}
+          {canRetry && (
+            <button
+              type="button"
+              className="btn btn-sm btn-secondary operation-card__retry"
+              onClick={() => onRetry(operation)}
+              aria-label={t('activity.retryLabel', { name })}
+            >
+              {t('activity.retry')}
+            </button>
+          )}
+          {failed && (
+            <button
+              type="button"
+              className="operation-card__hide"
+              onClick={() => onDismiss?.(operation.jobID)}
+              title={t('activity.moveToHistory')}
+              aria-label={t('activity.moveToHistory')}
+            >
+              <i className="fas fa-xmark" aria-hidden="true" />
+            </button>
+          )}
+        </div>
+      </div>
+
+      {/* The disclosure sits above what it discloses: a control that follows
+          its own region reads backwards to anyone moving through the page. */}
+      {showNodesToggle && (
+        <button
+          type="button"
+          className="operation-card__nodes-toggle"
+          aria-expanded={nodesOpen}
+          aria-controls={listId}
+          onClick={() => setNodesOpenOverride(!nodesOpen)}
+        >
+          <i className={`fas fa-chevron-${nodesOpen ? 'up' : 'down'}`} aria-hidden="true" />
+          {nodesOpen ? t('activity.hideNodes') : t('activity.showNodes', { count: nodes.length })}
+        </button>
+      )}
+
+      {/* Hidden rather than unmounted while collapsed, so the toggle's
+          aria-controls always points at something that exists. */}
+      {nodes.length > 0 && (
+        <ul className="operation-nodes-list" id={listId} hidden={!showNodesList}>
+          {nodes.map((node) => (
+            <li key={node.node_id} className={`operation-node operation-node-${node.status}`}>
+              <span className={`operation-node-status operation-node-status-${node.status}`}>
+                {/* An unmapped status is shown as it arrived: inventing
+                    "queued" for it would report a state the node is not in. */}
+                {nodeStatusKeys[node.status] ? t(nodeStatusKeys[node.status]) : node.status}
+              </span>
+              <span className="operation-node-name">{node.node_name || node.node_id}</span>
+              {node.file_name && (
+                <span className="operation-node-file" title={node.file_name}>{node.file_name}</span>
+              )}
+              {(node.current || node.total) && (
+                <span className="operation-node-bytes">{node.current || '?'} / {node.total || '?'}</span>
+              )}
+              {node.percentage > 0 && (
+                <span className="operation-node-pct">{Math.round(node.percentage)}%</span>
+              )}
+              {node.error && (
+                <span className="operation-node-error" title={node.error}>{node.error}</span>
+              )}
+              {node.percentage > 0 && node.percentage < 100 && (
+                <div className="operation-node-bar-container">
+                  <div className="operation-node-bar" style={{ width: `${node.percentage}%` }} />
+                </div>
+              )}
+            </li>
+          ))}
+        </ul>
+      )}
+    </div>
+  )
+}

core/http/react-ui/src/components/OperationsBar.jsx

@@ -1,176 +1,200 @@
-import { useState } from 'react'
+import { useEffect, useRef, useState } from 'react'
+// eslint-plugin-react is not configured here, so eslint cannot see that a
+// JSX-only import is used.
+// eslint-disable-next-line no-unused-vars
+import { Link } from 'react-router-dom'
+import { useTranslation } from 'react-i18next'
 import { useOperations } from '../hooks/useOperations'
 import { formatBytes } from '../utils/format'
 
-const artifactPhaseLabels = {
-  resolving: 'Resolving model files',
-  downloading: 'Downloading model files',
-  verifying: 'Verifying model files',
-  committing: 'Finalizing model installation',
-  persisting: 'Saving model configuration',
+const artifactPhaseKeys = {
+  resolving: 'activity.phase.resolving',
+  downloading: 'activity.phase.downloading',
+  verifying: 'activity.phase.verifying',
+  committing: 'activity.phase.committing',
+  persisting: 'activity.phase.persisting',
 }
 
-const nodeStatusLabels = {
-  success: 'Done',
-  error: 'Failed',
-  queued: 'Queued',
-  running_on_worker: 'Worker busy',
-  downloading: 'Downloading',
-}
+// How long a finished operation stays on screen. The API drops an operation
+// the instant it succeeds, so without this a fast install is a flicker.
+const SUCCESS_HOLD_MS = 4000
 
-const runningOnWorkerTooltip = 'NATS round-trip timed out, but the worker is still installing in the background. The reconciler will confirm completion.'
+// An unacknowledged failure outranks any progress; otherwise the API's own
+// sort (progress ascending) already puts the operation that gates the batch
+// first, and it is the most stable choice across polls.
+//
+// The strip is the only surface that picks one operation out of many. The
+// Activity page shows all of them, partitioned into failed and running, so it
+// has no primary to agree with.
+function primaryOperation(operations) {
+  if (!operations || operations.length === 0) return null
+  return operations.find((op) => op.error) || operations[0]
+}
 
 export default function OperationsBar() {
-  const { operations, cancelOperation, dismissFailedOp } = useOperations()
-  const [expanded, setExpanded] = useState({})
+  const { t } = useTranslation('admin')
+  const { operations, dismissFailedOp, wasCancelled } = useOperations()
+  // Which operation the user hid. Keyed by job so a different operation
+  // becoming primary brings the strip back: hiding must never be able to
+  // silence a later failure.
+  const [hiddenJobID, setHiddenJobID] = useState(null)
+  const [finished, setFinished] = useState(null)
+  const previousRef = useRef(null)
 
-  if (operations.length === 0) return null
+  const primary = primaryOperation(operations)
 
-  const toggle = (key) => setExpanded((m) => ({ ...m, [key]: !m[key] }))
+  useEffect(() => {
+    const previous = previousRef.current
+    previousRef.current = primary
+
+    // The previous primary is gone from the live list and it was not failing:
+    // it completed. Hold it on screen briefly, then drop it.
+    //
+    // Unless the user cancelled it. Cancelling deletes the operation server
+    // side, so a cancel and a completion are the same event here: the
+    // operation simply stops being listed. Nothing in the payload separates
+    // them, which is why this asks the page whether it issued the cancel.
+    // Without that, cancelling the last running install put a green
+    // "Installed model X" on screen for four seconds.
+    if (previous && !primary && !previous.error && !wasCancelled(previous.jobID)) {
+      setFinished(previous)
+      const timer = setTimeout(() => setFinished(null), SUCCESS_HOLD_MS)
+      return () => clearTimeout(timer)
+    }
+    if (primary) setFinished(null)
+    return undefined
+  }, [primary, wasCancelled])
+
+  const shown = primary || finished
+  if (!shown) return null
+  // A job keeps its jobID when it turns into a failure, so hiding the running
+  // operation would otherwise swallow that same job's error. Hiding is a way
+  // to get on with your work, never a way to opt out of bad news.
+  if (hiddenJobID === shown.jobID && !shown.error) return null
+
+  const extra = Math.max(0, operations.length - 1)
+  const isFinished = !primary
+  const phaseKey = artifactPhaseKeys[shown.phase]
+  const byteLabel = Number.isFinite(shown.currentBytes) && Number.isFinite(shown.totalBytes) && shown.totalBytes > 0
+    ? `${formatBytes(shown.currentBytes)} / ${formatBytes(shown.totalBytes)}`
+    : ''
+  const kind = shown.isBackend ? t('activity.kind.backend') : t('activity.kind.model')
+
+  let modifier = ''
+  let icon = null
+  let verb = ''
+  if (isFinished) {
+    modifier = 'operations-strip--done'
+    icon = <i className="fas fa-check operations-strip__icon" aria-hidden="true" />
+    // The completion phrase has to match the work that just ended: a removal
+    // that reports "Installed" reads as the opposite of what happened.
+    if (shown.isDeletion) verb = t('activity.verb.removed', { kind })
+    else if (shown.taskType === 'staging') verb = t('activity.verb.staged')
+    else verb = t('activity.verb.installed', { kind })
+  } else if (shown.error) {
+    modifier = 'operations-strip--error'
+    icon = <i className="fas fa-circle-exclamation operations-strip__icon" aria-hidden="true" />
+    // Same split as the card, so the two surfaces never describe one failed
+    // job differently: a removal reported as a failed install is the opposite
+    // of what happened.
+    if (shown.isDeletion) verb = t('activity.verb.failedRemoval', { kind })
+    else if (shown.taskType === 'staging') verb = t('activity.verb.failedStaging')
+    else verb = t('activity.verb.failed', { kind })
+  } else if (shown.isQueued) {
+    modifier = 'operations-strip--queued'
+    icon = <i className="fas fa-clock operations-strip__icon" aria-hidden="true" />
+    verb = t('activity.verb.queued')
+  } else if (shown.taskType === 'staging') {
+    modifier = 'operations-strip--staging'
+    icon = <i className="fas fa-cloud-arrow-up operations-strip__icon" aria-hidden="true" />
+    verb = t('activity.verb.staging')
+  } else if (shown.isDeletion) {
+    modifier = 'operations-strip--removing'
+    icon = <i className="fas fa-trash operations-strip__icon" aria-hidden="true" />
+    verb = t('activity.verb.removing', { kind })
+  } else {
+    icon = <span className="operations-strip__spinner" aria-hidden="true" />
+    verb = t('activity.verb.installing', { kind })
+  }
+
+  // A fanned-out backend install rolls its nodes up into one phrase. Without
+  // this the strip would report one node's phase as if it were the whole job,
+  // and the per-node list is what the Activity page is for.
+  const nodes = Array.isArray(shown.nodes) ? shown.nodes : []
+  const nodesDone = nodes.filter((node) => node.status === 'success').length
+  const nodeRollup = nodes.length > 1
+    ? t('activity.nodesDone', { done: nodesDone, total: nodes.length })
+    : ''
+
+  const detail = shown.error
+    || nodeRollup
+    || (shown.taskType === 'staging' && shown.nodeName ? t('activity.toNode', { node: shown.nodeName }) : '')
+    || (phaseKey ? t(phaseKey) : '')
+    || (shown.isQueued ? t('activity.waitingForInstaller') : '')
+
+  // A finished, failed or not-yet-started operation has no progress worth a
+  // bar: the first is over, the second stopped where it broke and the third
+  // has not moved.
+  const showProgress = !isFinished && !shown.error && !shown.isQueued && shown.progress > 0
+
+  const onHide = () => {
+    // A failure is dismissed server side, which moves it into the record.
+    // Anything else is hidden locally: the work carries on and the sidebar
+    // count still shows it.
+    if (shown.error) {
+      dismissFailedOp(shown.jobID)
+      return
+    }
+    setHiddenJobID(shown.jobID)
+    setFinished(null)
+  }
 
   return (
-    <div className="operations-bar">
-      {operations.map(op => {
-        const key = op.jobID || op.id
-        const nodes = Array.isArray(op.nodes) ? op.nodes : []
-        const canExpand = nodes.length > 1
-        const isOpen = !!expanded[key]
-        const phaseLabel = artifactPhaseLabels[op.phase]
-        const byteLabel = Number.isFinite(op.currentBytes) && Number.isFinite(op.totalBytes) && op.totalBytes > 0
-          ? `${formatBytes(op.currentBytes)} / ${formatBytes(op.totalBytes)}`
-          : ''
-        return (
-        <div key={key} className="operation-item">
-          <div className="operation-info">
-            {op.error ? (
-              <i className="fas fa-circle-exclamation" style={{ color: 'var(--color-error)', marginRight: 'var(--spacing-xs)' }} />
-            ) : op.isCancelled ? (
-              <i className="fas fa-ban" style={{ color: 'var(--color-warning)', marginRight: 'var(--spacing-xs)' }} />
-            ) : op.isDeletion ? (
-              <i className="fas fa-trash" style={{ color: 'var(--color-error)', marginRight: 'var(--spacing-xs)' }} />
-            ) : (
-              <div className="operation-spinner" />
-            )}
-            <span className="operation-text">
-              {op.error ? (
-                <>
-                  Failed to install {op.isBackend ? 'backend' : 'model'}: {op.name || op.id}
-                  <span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
-                    ({op.error})
-                  </span>
-                </>
-              ) : op.taskType === 'staging' ? (
-                <>
-                  <i className="fas fa-cloud-arrow-up" style={{ marginRight: 'var(--spacing-xs)' }} />
-                  Staging model: {op.name}{op.nodeName ? ` → ${op.nodeName}` : ''}
-                </>
-              ) : (
-                <>
-                  {op.isDeletion ? 'Removing' : 'Installing'}{' '}
-                  {op.isBackend ? 'backend' : 'model'}: {op.name || op.id}
-                </>
-              )}
-            </span>
-            {!op.error && op.isQueued && (
-              <span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
-                (Queued)
-              </span>
-            )}
-            {!op.error && op.isCancelled && (
-              <span style={{ fontSize: '0.75rem', color: 'var(--color-warning)', marginLeft: 'var(--spacing-xs)' }}>
-                Cancelling...
-              </span>
-            )}
-            {!op.error && phaseLabel && !op.isCancelled && (
-              <span className="operation-phase" style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
-                {phaseLabel}
-              </span>
-            )}
-            {!op.error && byteLabel && !op.isCancelled && (
-              <span className="operation-bytes" style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
-                {byteLabel}
-              </span>
-            )}
-            {!op.error && op.message && !phaseLabel && !op.isQueued && !op.isCancelled && (
-              <span style={{ fontSize: '0.75rem', color: 'var(--color-text-muted)', marginLeft: 'var(--spacing-xs)' }}>
-                {op.message}
-              </span>
-            )}
-            {!op.error && op.progress !== undefined && op.progress > 0 && (
-              <span className="operation-progress">{Math.round(op.progress)}%</span>
-            )}
-          </div>
-          {!op.error && op.progress !== undefined && op.progress > 0 && (
-            <div className="operation-bar-container">
-              <div className="operation-bar" style={{ width: `${op.progress}%` }} />
-            </div>
-          )}
-          {op.error ? (
-            <button
-              className="operation-cancel"
-              onClick={() => dismissFailedOp(op.id)}
-              title="Dismiss"
-            >
-              <i className="fas fa-xmark" />
-            </button>
-          ) : op.cancellable && !op.isCancelled ? (
-            <button
-              className="operation-cancel"
-              onClick={() => cancelOperation(op.jobID)}
-              title="Cancel"
-            >
-              <i className="fas fa-xmark" />
-            </button>
-          ) : null}
-          {canExpand && (
-            <button
-              type="button"
-              className="operation-expand"
-              onClick={() => toggle(key)}
-              aria-expanded={isOpen}
-              title={isOpen ? 'Hide per-node detail' : `Show ${nodes.length} nodes`}
-            >
-              <i className={`fas fa-chevron-${isOpen ? 'up' : 'down'}`} />
-              <span className="operation-expand-label">{nodes.length} nodes</span>
-            </button>
-          )}
-          {canExpand && isOpen && (
-            <ul className="operation-nodes-list">
-              {nodes.map((n) => (
-                <li key={n.node_id} className={`operation-node operation-node-${n.status}`}>
-                  <span
-                    className={`operation-node-status operation-node-status-${n.status}`}
-                    title={n.status === 'running_on_worker' ? runningOnWorkerTooltip : undefined}
-                  >
-                    {nodeStatusLabels[n.status] || n.status}
-                  </span>
-                  <span className="operation-node-name">{n.node_name || n.node_id}</span>
-                  {n.file_name && <span className="operation-node-file">{n.file_name}</span>}
-                  {(n.current || n.total) && (
-                    <span className="operation-node-bytes">
-                      {n.current || '?'} / {n.total || '?'}
-                    </span>
-                  )}
-                  {n.percentage > 0 && (
-                    <span className="operation-node-pct">{Math.round(n.percentage)}%</span>
-                  )}
-                  {n.error && (
-                    <span className="operation-node-error" title={n.error}>
-                      {n.error.length > 80 ? n.error.slice(0, 80) + '...' : n.error}
-                    </span>
-                  )}
-                  {n.percentage > 0 && n.percentage < 100 && (
-                    <div className="operation-node-bar-container">
-                      <div className="operation-node-bar" style={{ width: `${n.percentage}%` }} />
-                    </div>
-                  )}
-                </li>
-              ))}
-            </ul>
-          )}
-        </div>
-        )
-      })}
+    // role="status" already implies a polite live region. It deliberately does
+    // not cover the percentage, which changes every poll and would have a
+    // screen reader re-reading the whole strip once a second.
+    <div className={`operations-strip ${modifier}`.trim()} role="status">
+      {icon}
+      <span className="operations-strip__verb">{verb}</span>
+      <span className="operations-strip__name">{shown.name || shown.id}</span>
+      {detail && <span className="operations-strip__sep" aria-hidden="true">·</span>}
+      {detail && <span className="operations-strip__detail">{detail}</span>}
+      {byteLabel && !shown.error && <span className="operations-strip__bytes">{byteLabel}</span>}
+      <span className="operations-strip__spacer" />
+      {showProgress && (
+        <>
+          <span className="operations-strip__pct" aria-hidden="true">{Math.round(shown.progress)}%</span>
+          {/* A progressbar carries the value without a live region's chatter:
+              its updates are readable on demand rather than announced. */}
+          <span
+            className="operations-strip__track"
+            role="progressbar"
+            aria-valuenow={Math.round(shown.progress)}
+            aria-valuemin={0}
+            aria-valuemax={100}
+            aria-label={t('activity.progressLabel', { name: shown.name || shown.id })}
+          >
+            <span className="operations-strip__fill" style={{ width: `${shown.progress}%` }} />
+          </span>
+        </>
+      )}
+      {extra > 0 && (
+        <Link
+          className={`operations-strip__more${shown.error ? ' operations-strip__more--neutral' : ''}`}
+          to="/app/activity"
+        >
+          {t('activity.moreCount', { count: extra })}
+        </Link>
+      )}
+      <button
+        type="button"
+        className="operations-strip__hide"
+        onClick={onHide}
+        title={shown.error ? t('activity.moveToHistory') : t('activity.hide')}
+        aria-label={shown.error ? t('activity.moveToHistory') : t('activity.hide')}
+      >
+        <i className="fas fa-xmark" aria-hidden="true" />
+      </button>
     </div>
   )
 }

core/http/react-ui/src/components/Sidebar.jsx

@@ -8,6 +8,7 @@ import { useBranding } from '../contexts/BrandingContext'
 import { apiUrl } from '../utils/basePath'
 import { preloadRoute } from '../router'
 import { consoles, firstVisiblePath, consolePaths } from './console/consoleConfig'
+import { useOperations } from '../hooks/useOperations'
 
 const COLLAPSED_KEY = 'localai_sidebar_collapsed'
 const SECTIONS_KEY = 'localai_sidebar_sections'
@@ -83,6 +84,7 @@ export default function Sidebar({ isOpen, onClose }) {
   })
   const [openSections, setOpenSections] = useState(loadSectionState)
   const { isAdmin, authEnabled, user, logout, hasFeature } = useAuth()
+  const { operations } = useOperations()
   const branding = useBranding()
   const navigate = useNavigate()
   const location = useLocation()
@@ -160,6 +162,12 @@ export default function Sidebar({ isOpen, onClose }) {
   // Shared shape for the console gating helpers (consoleConfig.js).
   const auth = { isAdmin, authEnabled, hasFeature, features }
 
+  // One badge, on the always-visible sidebar entry. The console rail only
+  // exists while the user is on an Operate route and can be collapsed, so
+  // badging the rail item instead would let the count disappear entirely.
+  const failedOps = operations.filter((op) => op.error).length
+  const activeOps = operations.length
+
   // Inline sections (Create) carry no gating; a plain filterItem pass suffices.
   const getVisibleSectionItems = (section) => section.items.filter(filterItem)
 
@@ -249,6 +257,11 @@ export default function Sidebar({ isOpen, onClose }) {
                 >
                   <i className={`${config.icon} nav-icon`} />
                   <span className="nav-label">{label}</span>
+                  {config.groups.some(g => g.items.some(i => i.badge === 'operations')) && activeOps > 0 && (
+                    <span className={`nav-badge${failedOps > 0 ? ' nav-badge--error' : ''}`}>
+                      {failedOps > 0 ? failedOps : activeOps}
+                    </span>
+                  )}
                 </NavLink>
               </div>
             )

core/http/react-ui/src/components/console/consoleConfig.js

@@ -56,6 +56,12 @@ export const operateConsole = {
         { path: '/app/voice-library', icon: 'fas fa-wave-square', labelKey: 'items.voiceLibrary', adminOnly: true },
       ],
     },
+    {
+      titleKey: 'operate.activity',
+      items: [
+        { path: '/app/activity', icon: 'fas fa-download', labelKey: 'items.activity', adminOnly: true, badge: 'operations' },
+      ],
+    },
     {
       titleKey: 'operate.cluster',
       items: [

core/http/react-ui/src/components/nodes/NodePanel.jsx

@@ -9,6 +9,7 @@ export default function NodePanel({ node, models = [], onApprove, onDrain, onRes
   const isAgent = node.node_type === 'agent'
   const open = () => navigate(`/app/nodes/${node.id}`)
   const usedVRAM = node.total_vram && node.available_vram != null ? node.total_vram - node.available_vram : null
+  const usedRAM = node.total_ram && node.available_ram != null ? node.total_ram - node.available_ram : null
 
   return (
     <div className="node-panel">
@@ -45,6 +46,9 @@ export default function NodePanel({ node, models = [], onApprove, onDrain, onRes
               {node.total_vram > 0 && (
                 <span className="cell-mono">VRAM {formatVRAM(usedVRAM) || '0'} / {formatVRAM(node.total_vram)}</span>
               )}
+              {node.total_ram > 0 && (
+                <span className="cell-mono">RAM {formatVRAM(usedRAM) || '0'} / {formatVRAM(node.total_ram)}</span>
+              )}
               <span className="cell-mono">{node.in_flight_count || 0} in-flight</span>
             </div>
             <div className="node-panel__models">

core/http/react-ui/src/contexts/OperationsContext.jsx

@@ -1,4 +1,4 @@
-import { createContext, useContext, useState, useEffect, useCallback, useRef } from 'react'
+import { createContext, useContext, useState, useEffect, useCallback, useMemo, useRef } from 'react'
 import { operationsApi } from '../utils/api'
 import { useAuth } from '../context/AuthContext'
 
@@ -12,6 +12,11 @@ function serializeOps(ops) {
 
 const OperationsContext = createContext(null)
 
+// How long a cancelled job is remembered. It only has to outlive the poll that
+// notices the operation left the list; a session that cancels all day must not
+// accumulate job IDs.
+const CANCELLED_MEMORY_MS = 60_000
+
 // Single shared poller for /api/operations. Before this provider existed,
 // each useOperations() call ran its own setInterval; with OperationsBar
 // always mounted plus the per-page consumers (Models, Backends, Chat), the
@@ -21,9 +26,43 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
   const [operations, setOperations] = useState([])
   const [loading, setLoading] = useState(true)
   const [error, setError] = useState(null)
+  const [history, setHistory] = useState([])
+  const [historyLoading, setHistoryLoading] = useState(false)
   const { isAdmin } = useAuth()
   const intervalRef = useRef(null)
   const lastSerializedRef = useRef('[]')
+  const liveIDsRef = useRef(new Set())
+  // Jobs cancelled from this tab, by job ID. The cancel endpoint removes the
+  // operation immediately, so on the next poll the only thing the UI can
+  // observe is that the operation is gone, which is exactly what finishing
+  // looks like. Nothing in the payload distinguishes them (a cancelled
+  // operation is never listed), so the side that issued the cancel is the only
+  // one that can remember it.
+  const cancelledRef = useRef(new Map())
+
+  // History is fetched on demand, never on the poll interval: it only changes
+  // when an operation finishes, and the Activity page is the only consumer.
+  const fetchHistory = useCallback(async () => {
+    if (!isAdmin) return
+    setHistoryLoading(true)
+    try {
+      const data = await operationsApi.history()
+      setHistory(data?.operations || [])
+    } catch (err) {
+      setError((prev) => (prev === err.message ? prev : err.message))
+    } finally {
+      setHistoryLoading(false)
+    }
+  }, [isAdmin])
+
+  const clearHistory = useCallback(async () => {
+    try {
+      await operationsApi.clearHistory()
+      setHistory([])
+    } catch (err) {
+      setError(err.message)
+    }
+  }, [])
 
   const fetchOperations = useCallback(async () => {
     if (!isAdmin) {
@@ -40,13 +79,38 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
         setOperations(ops)
       }
 
+      // An operation leaving the live list is the one moment the record can
+      // have changed. Refetching here keeps the page correct without polling
+      // a second endpoint every second.
+      //
+      // Tracked by identity rather than by count: during a batch install one
+      // operation finishing in the same second another starts leaves the
+      // length unchanged, and a count comparison would miss the completion.
+      const liveIDs = new Set(ops.map((op) => op.jobID || op.id))
+      let departed = false
+      for (const id of liveIDsRef.current) {
+        if (!liveIDs.has(id)) {
+          departed = true
+          break
+        }
+      }
+      liveIDsRef.current = liveIDs
+      if (departed) {
+        fetchHistory()
+      }
+
+      const cutoff = Date.now() - CANCELLED_MEMORY_MS
+      for (const [id, at] of cancelledRef.current) {
+        if (at < cutoff) cancelledRef.current.delete(id)
+      }
+
       setError((prev) => (prev === null ? prev : null))
     } catch (err) {
       setError((prev) => (prev === err.message ? prev : err.message))
     } finally {
       setLoading((prev) => (prev ? false : prev))
     }
-  }, [isAdmin])
+  }, [isAdmin, fetchHistory])
 
   useEffect(() => {
     if (!isAdmin) return
@@ -63,29 +127,106 @@ export function OperationsProvider({ children, pollInterval = 1000 }) {
   const cancelOperation = useCallback(async (jobID) => {
     try {
       await operationsApi.cancel(jobID)
+      // Recorded before the refetch: that refetch is the one that sees the
+      // operation gone, and a consumer reacting to the disappearance has to
+      // find the cancel already remembered or it will call it a success.
+      cancelledRef.current.set(jobID, Date.now())
       await fetchOperations()
     } catch (err) {
       setError(err.message)
     }
   }, [fetchOperations])
 
-  const dismissFailedOp = useCallback(async (opId) => {
+  // Whether this tab cancelled the job. Read by the strip to tell "the last
+  // operation finished" from "the user called it off": both look identical in
+  // /api/operations, which lists neither.
+  const wasCancelled = useCallback((jobID) => cancelledRef.current.has(jobID), [])
+
+  // Takes the jobID, never the display id. /api/operations strips the
+  // "node:<nodeID>:" prefix before emitting, so a local install and a
+  // node-scoped install of the same backend arrive as two distinct jobs
+  // sharing one id: looking the job up by id could dismiss the wrong one,
+  // leaving the failure the user acted on live and silently retiring another.
+  const dismissFailedOp = useCallback(async (jobID) => {
+    if (!jobID) return
     try {
-      const op = operations.find((o) => o.id === opId)
-      if (op?.jobID) {
-        await operationsApi.dismiss(op.jobID)
-        await fetchOperations()
-      }
+      await operationsApi.dismiss(jobID)
+      await fetchOperations()
     } catch {
       // Ignore dismiss errors
     }
-  }, [operations, fetchOperations])
+  }, [fetchOperations])
+
+  // Time remaining is derived, not reported. We keep the previous
+  // (bytes, timestamp) sample per job and estimate from the delta.
+  //
+  // All or nothing on purpose: an estimate needs two samples, and one card
+  // showing "11 min left" while its neighbours show nothing reads as a
+  // rendering bug rather than as missing data.
+  const samplesRef = useRef(new Map())
+  const operationsWithEta = useMemo(() => {
+    const now = Date.now()
+    const samples = samplesRef.current
+    const seen = new Set()
+
+    const withEta = operations.map((op) => {
+      const key = op.jobID || op.id
+      seen.add(key)
+      const current = op.currentBytes
+      const total = op.totalBytes
+      if (!Number.isFinite(current) || !Number.isFinite(total) || total <= 0) return op
+
+      const previous = samples.get(key)
+      samples.set(key, { bytes: current, at: now })
+      if (!previous || current <= previous.bytes) return op
+
+      const bytesPerMs = (current - previous.bytes) / Math.max(1, now - previous.at)
+      if (bytesPerMs <= 0) return op
+      return { ...op, etaSeconds: Math.round((total - current) / bytesPerMs / 1000) }
+    })
+
+    // Drop samples for jobs that finished, so the map cannot grow forever.
+    for (const key of samples.keys()) {
+      if (!seen.has(key)) samples.delete(key)
+    }
+
+    // All or nothing: if any operation still transferring has no estimate yet,
+    // nobody shows one this tick.
+    //
+    // Only operations actually downloading get a vote. Every other phase
+    // reports bytes but stops advancing them: verifying hashes a finished file
+    // while the counter sits below the multi-file total, and committing sits
+    // pinned at the total. Both can last minutes, and counting them would
+    // blank every other operation's estimate for that whole window.
+    //
+    // The byte clauses are not redundant with the phase clause: a producer can
+    // report downloading with bytes already at the total. The undefined-phase
+    // arm keeps today's behaviour for producers that do not report a phase,
+    // which in practice do not report totalBytes either.
+    const tracked = withEta.filter(
+      (op) =>
+        Number.isFinite(op.totalBytes) &&
+        op.totalBytes > 0 &&
+        Number.isFinite(op.currentBytes) &&
+        op.currentBytes < op.totalBytes &&
+        (op.phase === undefined || op.phase === 'downloading')
+    )
+    if (tracked.length > 0 && tracked.some((op) => op.etaSeconds === undefined)) {
+      return withEta.map(({ etaSeconds: _etaSeconds, ...op }) => op)
+    }
+    return withEta
+  }, [operations])
 
   const value = {
-    operations,
+    operations: operationsWithEta,
     loading,
     error,
+    history,
+    historyLoading,
+    fetchHistory,
+    clearHistory,
     cancelOperation,
+    wasCancelled,
     dismissFailedOp,
     refetch: fetchOperations,
   }

core/http/react-ui/src/pages/Activity.jsx

@@ -0,0 +1,269 @@
+import { useCallback, useEffect, useMemo, useState } from 'react'
+import { useTranslation } from 'react-i18next'
+import { useOperations } from '../hooks/useOperations'
+import { modelsApi, backendsApi, nodesApi } from '../utils/api'
+// eslint-plugin-react is not configured here, so eslint cannot see that an
+// import used only inside JSX is used at all. Link, PageHeader and
+// OperationCard are each referenced from JSX only.
+// eslint-disable-next-line no-unused-vars
+import { Link, useOutletContext } from 'react-router-dom'
+// eslint-disable-next-line no-unused-vars
+import PageHeader from '../components/PageHeader'
+// eslint-disable-next-line no-unused-vars
+import OperationCard from '../components/OperationCard'
+
+const FILTERS = [
+  { id: 'all', labelKey: 'activity.filter.all' },
+  { id: 'models', labelKey: 'activity.filter.models' },
+  { id: 'backends', labelKey: 'activity.filter.backends' },
+  { id: 'cluster', labelKey: 'activity.filter.cluster' },
+]
+
+function matchesFilter(entry, filter) {
+  if (filter === 'all') return true
+  if (filter === 'models') return !entry.isBackend && entry.taskType !== 'staging'
+  if (filter === 'backends') return Boolean(entry.isBackend)
+  // Cluster covers anything scoped to a node: staged files and node-scoped
+  // backend installs.
+  return entry.taskType === 'staging' || Boolean(entry.nodeID) || (Array.isArray(entry.nodes) && entry.nodes.length > 0)
+}
+
+const outcomeIcon = {
+  completed: 'fas fa-check',
+  failed: 'fas fa-circle-exclamation',
+  cancelled: 'fas fa-ban',
+}
+
+function timeOfDay(iso) {
+  const date = new Date(iso)
+  if (Number.isNaN(date.getTime())) return ''
+  return date.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
+}
+
+// Beyond this the elapsed time is not a duration, it is a broken start stamp.
+// A zero-value Go time reaching the page renders as a span of millennia, which
+// the row would state as fact; the page is the last place that can refuse to.
+const MAX_PLAUSIBLE_DURATION_SECONDS = 24 * 60 * 60
+
+// Returns '' when the elapsed time cannot be trusted, which the caller renders
+// as a duration-less phrase rather than as "installed in " with nothing after
+// it. recordTerminal seeds StartedAt = FinishedAt and only overwrites it with a
+// real stamp, so a zero span is an ordinary arrival and gets a floor instead.
+function durationLabel(record) {
+  const started = new Date(record.startedAt).getTime()
+  const finished = new Date(record.finishedAt).getTime()
+  if (Number.isNaN(started) || Number.isNaN(finished) || finished < started) return ''
+  const seconds = Math.round((finished - started) / 1000)
+  if (seconds > MAX_PLAUSIBLE_DURATION_SECONDS) return ''
+  if (seconds < 1) return '< 1s'
+  if (seconds < 60) return `${seconds}s`
+  return `${Math.floor(seconds / 60)}m ${seconds % 60}s`
+}
+
+// Cancellation is tested before the task type on purpose: a deletion cancelled
+// mid-flight must report the cancellation, not "removed". recordTerminal
+// produces exactly that pair, and the row's ban icon would otherwise sit beside
+// text claiming work that never happened.
+function recordSummary(record, t) {
+  if (record.outcome === 'failed') return t('activity.rowFailed', { error: record.error })
+  if (record.outcome === 'cancelled') return t('activity.rowCancelled')
+  if (record.taskType === 'deletion') return t('activity.rowRemoved')
+  const duration = durationLabel(record)
+  return duration ? t('activity.rowInstalled', { duration }) : t('activity.rowInstalledPlain')
+}
+
+// Retry only ever means "install this again". A failed deletion would need the
+// delete endpoint and a staging operation is driven by the router rather than
+// by a user action, so neither is retryable from here.
+function isRetryable(op) {
+  return Boolean(op.error) && !op.isDeletion && op.taskType !== 'staging'
+}
+
+export default function Activity() {
+  const { t } = useTranslation('admin')
+  const outlet = useOutletContext()
+  const addToast = outlet?.addToast
+  const { operations, history, fetchHistory, clearHistory, cancelOperation, dismissFailedOp } = useOperations()
+  const [filter, setFilter] = useState('all')
+
+  useEffect(() => { fetchHistory() }, [fetchHistory])
+
+  const retryOperation = useCallback(async (op) => {
+    // Dismiss before reinstalling, never after: the reinstall reuses the same
+    // opcache key, and overwriting a failed entry in place skips recordTerminal
+    // so the failure would never reach the record. Dismissing first is what
+    // puts it there.
+    //
+    // By jobID, because the guarantee only holds while both calls address the
+    // same job. Two ops can share an id (a local and a node-scoped install of
+    // one backend), and dismissing by id could retire the other one instead,
+    // leaving this failure to be overwritten in place by the reinstall below.
+    await dismissFailedOp(op.jobID)
+    // fullName is the gallery-qualified id the install endpoints expect;
+    // `name` has the repo prefix stripped for display. Node-scoped ops already
+    // had their prefix removed server side, so fullName is the bare slug there.
+    const target = op.fullName || op.id
+    try {
+      if (op.nodeID) {
+        await nodesApi.installBackend(op.nodeID, target)
+      } else if (op.isBackend) {
+        await backendsApi.install(target)
+      } else {
+        // The variant is not on the payload YET, so a pinned model retries as
+        // an auto-select: someone who chose a specific quant, watched it fail
+        // at 90% and pressed Retry gets a different build, with nothing on
+        // screen saying so. Worth closing, and close to closed: ui_api.go
+        // already reads ?variant= at enqueue and stores it on the ManagementOp,
+        // so it only has to reach the /api/operations payload and this call.
+        // Until then Retry stays, because nothing here distinguishes a pinned
+        // install from an unpinned one and dropping it would cost every model
+        // the button, including the common plain install that hit a network
+        // error.
+        await modelsApi.install(target)
+      }
+    } catch (err) {
+      addToast?.(t('activity.retryFailed', { message: err.message }), 'error')
+    }
+  }, [dismissFailedOp, addToast, t])
+
+  const live = useMemo(
+    () => operations.filter((op) => !op.error && matchesFilter(op, filter)),
+    [operations, filter],
+  )
+  const failing = useMemo(
+    () => operations.filter((op) => op.error && matchesFilter(op, filter)),
+    [operations, filter],
+  )
+  const records = useMemo(
+    () => history.filter((entry) => matchesFilter(entry, filter)),
+    [history, filter],
+  )
+
+  // The header describes the instance, not the current chip, which is what the
+  // Clear-history button beside it already does. A filtered count here would
+  // report "Nothing running" while two model installs were running just
+  // offscreen; the filtered view explains itself through the sections and the
+  // filtered empty state instead.
+  //
+  // "Nothing running" must also not be said while a failure is waiting for a
+  // decision, so both counts get a clause. Each clause is dropped when its
+  // count is zero rather than rendered as a literal 0: the ordinary happy path
+  // would otherwise put "0 needs attention" under the page title on every
+  // render, which reads as a report about failures rather than the absence of
+  // one.
+  const runningTotal = operations.filter((op) => !op.error).length
+  const failingTotal = operations.length - runningTotal
+  const summaryClauses = []
+  if (runningTotal > 0) summaryClauses.push(t('activity.summaryRunning', { count: runningTotal }))
+  if (failingTotal > 0) summaryClauses.push(t('activity.summaryFailed', { count: failingTotal }))
+  let supporting
+  if (summaryClauses.length > 0) supporting = summaryClauses.join(' ')
+  else if (history.length > 0) supporting = t('activity.summaryQuiet', { count: history.length })
+  // Saying "0 operations since startup" directly above "No operations since
+  // startup" states the same nothing twice.
+  else supporting = t('activity.summaryIdle')
+
+  return (
+    <div className="page page--wide activity-page">
+      <PageHeader
+        title={t('activity.title')}
+        supporting={supporting}
+        actions={history.length > 0 ? (
+          <button type="button" className="btn btn-secondary" onClick={clearHistory}>
+            {t('activity.clearHistory')}
+          </button>
+        ) : null}
+      />
+
+      <div className="activity-filters">
+        {FILTERS.map((entry) => (
+          <button
+            key={entry.id}
+            type="button"
+            className="activity-chip"
+            aria-pressed={filter === entry.id}
+            onClick={() => setFilter(entry.id)}
+          >
+            {t(entry.labelKey)}
+          </button>
+        ))}
+      </div>
+
+      {live.length > 0 && (
+        <section className="activity-section">
+          <h2 className="activity-section__title">
+            {t('activity.inProgress')} <span className="activity-section__count">{live.length}</span>
+          </h2>
+          {live.map((op) => (
+            <OperationCard key={op.jobID || op.id} operation={op} onCancel={cancelOperation} />
+          ))}
+        </section>
+      )}
+
+      {failing.length > 0 && (
+        <section className="activity-section">
+          <h2 className="activity-section__title">
+            {t('activity.needsAttention')} <span className="activity-section__count">{failing.length}</span>
+          </h2>
+          {failing.map((op) => (
+            <OperationCard
+              key={op.jobID || op.id}
+              operation={op}
+              onDismiss={dismissFailedOp}
+              onRetry={isRetryable(op) ? retryOperation : undefined}
+            />
+          ))}
+        </section>
+      )}
+
+      {records.length > 0 && (
+        <section className="activity-section">
+          <h2 className="activity-section__title">
+            {t('activity.record')} <span className="activity-section__count">{records.length}</span>
+          </h2>
+          <div className="activity-rows">
+            {records.map((record) => (
+              <div key={record.jobID} className="activity-row">
+                <i
+                  className={`${outcomeIcon[record.outcome] || 'fas fa-check'} activity-row__icon activity-row__icon--${record.outcome}`}
+                  aria-hidden="true"
+                />
+                <span className="activity-row__name">
+                  {record.name}
+                  <small>{recordSummary(record, t)}</small>
+                </span>
+                <span className="activity-row__when">{timeOfDay(record.finishedAt)}</span>
+                <Link className="activity-row__action" to={record.isBackend ? '/app/backends' : '/app/models'}>
+                  {record.isBackend ? t('activity.viewInBackends') : t('activity.viewInModels')}
+                </Link>
+              </div>
+            ))}
+          </div>
+          <p className="activity-note">{t('activity.historyNote')}</p>
+        </section>
+      )}
+
+      {/* A chip that matches nothing is not an empty system. Telling someone
+          with three model installs on record that nothing has ever run, while
+          the line above them counts those same three, is simply false. */}
+      {live.length === 0 && failing.length === 0 && records.length === 0 && (
+        filter === 'all' ? (
+          <div className="activity-empty">
+            <i className="fas fa-download activity-empty__icon" aria-hidden="true" />
+            <p className="activity-empty__title">{t('activity.emptyTitle')}</p>
+            <p className="activity-empty__body">{t('activity.emptyBody')}</p>
+            <Link className="btn btn-primary" to="/app/models">{t('activity.browseModels')}</Link>
+          </div>
+        ) : (
+          <div className="activity-empty activity-empty--filtered">
+            <i className="fas fa-filter activity-empty__icon" aria-hidden="true" />
+            <p className="activity-empty__title">{t('activity.emptyFiltered')}</p>
+            <button type="button" className="btn btn-secondary" onClick={() => setFilter('all')}>
+              {t('activity.showAll')}
+            </button>
+          </div>
+        )
+      )}
+    </div>
+  )
+}