chore: ⬆️ Update vllm-project/vllm cu130 wheel to 0.29.0 (#11954)

⬆️ Update vllm-project/vllm cu130 wheel

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>

.agents/ai-coding-assistants.md

@@ -49,6 +49,40 @@ AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
 A human reviewer owns the contribution; the AI's involvement is recorded
 via `Assisted-by` (see below).
 
+### Exception: automation operated by a maintainer
+
+The rule above addresses the common case, an AI assistant helping a human
+contributor who then signs off. It does not fit automation that a
+maintainer runs themselves, which opens pull requests with no human
+submitter to sign. Applied literally there, nothing ever signs and the
+DCO check blocks the pull request permanently.
+
+A maintainer-operated bot MUST therefore add a `Signed-off-by` trailer
+naming **the maintainer who operates it**, not the bot and not the model:
+
+```
+Assisted-by: Codex:gpt-5
+Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
+```
+
+This is not the AI certifying the DCO. The maintainer is, exactly as they
+do for a commit they typed by hand: they configured the automation, they
+own its output, and they take responsibility for it when they merge it.
+The `Assisted-by` trailer still records that a model produced the code, so
+the provenance trail is unchanged.
+
+The exception is narrow and does not widen the rule for anyone else:
+
+- It applies only to automation a LocalAI maintainer operates and whose
+  output that maintainer reviews before merge.
+- The sign-off names a real person who accepts DCO responsibility.
+- An AI assistant helping an outside contributor still MUST NOT sign off.
+  That contributor adds their own trailer.
+- A bot MUST NOT sign off on behalf of anyone other than its operator, and
+  MUST NOT add a trailer for a contributor whose branch it pushes to. If
+  automation contributes to someone else's branch, it leaves the sign-off
+  to that contributor.
+
 ## Attribution
 
 When AI tools contribute to LocalAI development, proper attribution helps

.agents/api-endpoints-and-auth.md

@@ -236,6 +236,58 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
+## Control-plane database health metrics
+
+In distributed mode the frontend registers three OpenTelemetry gauges over the
+PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`,
+wired in `core/application/distributed.go`). They reach `/metrics` through the
+same Prometheus exporter as the rest of the API metrics.
+
+| Metric | Meaning | Page when |
+|--------|---------|-----------|
+| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
+| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
+| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
+
+A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
+While it grows, autovacuum can reclaim nothing anywhere in the database no
+matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
+registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
+The fix is to find the transaction holding the horizon open and clear it:
+
+```sql
+SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query
+FROM pg_stat_activity
+WHERE backend_xmin IS NOT NULL
+ORDER BY age(backend_xmin) DESC;
+```
+
+Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the
+bloated tables once the horizon has moved.
+
+**A healthy-looking xmin age does not on its own prove the horizon is free.**
+The gauge reads `pg_stat_activity`, which only sees live backends. Two other
+things pin the very same horizon and are invisible there, so either one can hold
+vacuum back while the gauge reads 0:
+
+```sql
+SELECT gid, prepared, database, transaction FROM pg_prepared_xacts;
+SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;
+```
+
+An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
+and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
+before concluding that a bloated table has some other cause.
+
+Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
+translate into database load. Failed and timed-out samples cost the same interval
+as successful ones, so a database that is already struggling is not retried on
+every scrape. A failed sample reports the last good values rather than failing the
+scrape, because these gauges matter most when the database is struggling. Before
+the first successful sample the gauges are absent rather than zero, since a zero
+xmin age would read as a healthy horizon: alert on `absent()` too if you need to
+distinguish "healthy" from "never sampled".
+
 ## Advertising surfaces — where to register a new capability
 
 Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.

.agents/ds4-backend.md

@@ -77,6 +77,56 @@ spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with
 budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash
 on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental).
 
+## CUDA architecture (do not build without one)
+
+`backend/cpp/ds4/Makefile` drives upstream's **object targets** directly
+(`$(MAKE) -C ds4 ds4.o ds4_cuda.o ...`), which bypasses upstream's own guard:
+its `cuda` target refuses to build unless `CUDA_ARCH` is set, and offers
+`cuda-spark` (sm_121, DGX Spark / GB10) and `cuda-generic` (native) instead.
+Built with no `-arch`, nvcc targets its default architecture and the kernels run
+as JIT'd PTX. On GB10 that silently corrupted every prefill batch of >=128
+tokens - the model emitted text unrelated to the prompt and never closed its
+thinking block, so `content` came back empty - and cost close to two orders of
+magnitude of prefill throughput (4.21 t/s vs 325.70 t/s, same box, same model).
+Short prompts stayed correct, which is why it went unnoticed.
+
+The Makefile therefore picks a gencode list from `CUDA_MAJOR_VERSION` (a build
+arg the backend matrix already declares, forwarded by `Dockerfile.ds4`) and
+`uname -m`, and passes it as `NVCC_ARCH_FLAGS` to the sub-make. Upstream's
+`CUDA_ARCH` accepts a single value, so it cannot express the fat binary the
+shipped images need; a command-line assignment beats its `:=`. An empty
+`CUDA_MAJOR_VERSION` falls back to upstream's `native` for local developer
+builds, and an unrecognised one is a hard error - no CI runner has a GPU, so a
+silent `native` there is exactly the failure mode this guards against.
+
+`DS4_CUDA_HAVE_MXF4` is deliberately unset: upstream defines it only for
+single-arch sm_120/sm_121 builds and guards it with a plain `#ifdef` rather than
+`__CUDA_ARCH__`, so it cannot be combined with older archs. It gates an optional
+MXFP4 indexer fast path whose `#ifndef` branch returns 0, so omitting it costs
+speed, not correctness.
+
+### Verifying a build
+
+Check which flags a configuration resolves to, without compiling anything:
+
+```
+make -C backend/cpp/ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 NATIVE=false \
+  --eval='show: ; @echo [$(DS4_ARCH_MAKEVARS)]' show
+```
+
+Do not use `make -n` for this: the recipe is `+$(MAKE) ...`, and the `+` prefix
+makes it run even under `-n`.
+
+Then exercise the failure mode itself against a built backend. It only appears
+above one prefill batch, so the ordinary `predict` spec cannot catch it:
+
+```
+BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
+BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
+BACKEND_TEST_CAPS=health,load,predict,long_prefill \
+go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
+```
+
 ## Build matrix
 
 | Build | Where | Notes |

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,7 +5,7 @@ This PR fixes #
 **Notes for Reviewers**
 
 
-**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
+**[Signed commits](../CONTRIBUTING.md#commit-messages)**
 - [ ] Yes, I signed my commits.
 - [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
  

.github/bump_vllm_metal.sh

@@ -3,9 +3,9 @@
 # darwin (Apple Silicon) install path. The macOS/Metal build
 # (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
 # version-locked to a specific vLLM source release. install.sh derives that vLLM
-# version at build time from vllm-metal's own installer at the pinned
-# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
-# which bumps the Linux cu130 wheel pin.
+# version, and the wheel asset name, at build time from the pinned tag, so there
+# is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, which bumps the
+# Linux cu130 wheel pin.
 #
 # This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
@@ -23,15 +23,20 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
     exit 1
 fi
 
-# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
-# /releases/latest returns the newest one (with its cp312 wheel asset).
+# vllm-metal ships frequent .dev releases, flagged as prereleases, alongside the
+# stable ones. /releases/latest skips the prereleases and returns the newest
+# stable tag, which is what darwin should pin: upstream deletes and re-cuts .dev
+# tags, and a pin to a deleted tag 404s the whole build.
 LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 
-# The coupled vLLM source version lives in vllm-metal's installer at that tag.
-NEW_VLLM_VERSION=$(gh_curl \
-    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
+# The coupled vLLM release lives in .github/vllm-release-tag.commit at that tag
+# (since vllm-metal 0.28); releases predating that file pinned it inline in their
+# own install.sh. The extractor reads both forms.
+NEW_VLLM_VERSION=$( { gh_curl \
+    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/.github/vllm-release-tag.commit" \
+    || gh_curl "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh"; } \
     | "$(dirname "${BASH_SOURCE[0]}")/../scripts/lib/extract-vllm-metal-version.sh")
 
 if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then

.github/workflows/bump_deps.yaml

@@ -166,7 +166,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update ${{ matrix.repository }}'
           title: 'chore: :arrow_up: Update ${{ matrix.repository }} to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/${{ matrix.variable }}"
+          branch: "bump/${{ matrix.variable }}"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -203,7 +203,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
           title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_VERSION"
+          branch: "bump/VLLM_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -241,6 +241,6 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
           title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/VLLM_METAL_VERSION"
+          branch: "bump/VLLM_METAL_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true

.github/workflows/test.yml

@@ -80,8 +80,13 @@ jobs:
             coverage/coverage.out
             coverage/coverage.html
           if-no-files-found: ignore
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true
@@ -125,8 +130,13 @@ jobs:
           export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
           PATH="$PATH:$HOME/go/bin" make protogen-go
           PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-aio.yml

@@ -77,8 +77,13 @@ jobs:
       - name: Test
         run: |
             PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-e2e.yml

@@ -63,8 +63,13 @@ jobs:
       - name: Test Backend E2E
         run: |
           PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-pii-ner-e2e.yml

@@ -88,8 +88,13 @@ jobs:
       # CPU and runs the token_classify capability spec (byte-offset contract).
       - name: Run live PII NER backend E2E
         run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-ui-e2e.yml

@@ -75,8 +75,13 @@ jobs:
           path: core/http/react-ui/coverage/
           if-no-files-found: ignore
           retention-days: 7
+      # tmate keeps the runner busy until the 6 hour job limit, so a single
+      # failure costs a whole runner slot. Only open a session when someone
+      # asked for one by labelling the pull request `ci-debug`, and cap the
+      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() }}
+        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
+        timeout-minutes: 30
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

AGENTS.md

@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w
 
 LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
 
-- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
+- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
 - **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
 - **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
 - **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.

CONTRIBUTING.md

@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis
 
 The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
 
-- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
+- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
 - **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
 - **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
 

Makefile

@@ -34,6 +34,11 @@ TEST_FLAKES?=5
 RANDOM := $(shell bash -c 'echo $$RANDOM')
 
 VERSION?=$(shell git describe --always --tags || echo "dev" )
+# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git
+# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged
+# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0.
+# Without this the packaged launcher reports itself as version 0.0.0 (#11673).
+LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0")
 # go tool nm ./local-ai | grep Commit
 LD_FLAGS?=-s -w
 override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)"
@@ -235,7 +240,7 @@ test-ci-scripts:
 ## pure stdlib on purpose so they run without any backend venv; the list is
 ## explicit because their siblings (model_identity_test) import grpc and the
 ## generated protobufs, which only exist inside a built backend.
-PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test
+PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test temp_utils_test
 test-python-helpers:
 	cd backend/python/common && python3 -m unittest $(PYTHON_HELPER_TESTS)
 
@@ -388,9 +393,17 @@ test-e2e: build-mock-backend build-cloud-proxy-backend prepare-e2e run-e2e-image
 	$(MAKE) teardown-e2e
 	docker rmi localai-tests
 
+# `docker stop` returns as soon as the container exits, but Docker reaps a
+# `--rm` container asynchronously after that. The `docker rmi localai-tests` in
+# test-e2e then loses the race against the reaper and fails on a still
+# referenced image, turning a green suite red. Removing the container ourselves
+# is synchronous, so the image reference is gone before we return. It also
+# covers the case where nothing is running, which `docker stop` could not
+# because it rejects an empty argument list.
 teardown-e2e:
 	rm -rf $(TEST_DIR) || true
-	docker stop $$(docker ps -q --filter ancestor=localai-tests)
+	@CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \
+	if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi
 
 ########################################################
 ## Integration and unit tests
@@ -1622,7 +1635,7 @@ site-serve: site
 build-launcher-darwin:
 	rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
 	mkdir -p dist
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
 	mv cmd/launcher/LocalAI.app dist/LocalAI.app
 	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
 
@@ -1649,4 +1662,4 @@ release-launcher-darwin: notarize-launcher-darwin
 	@echo "dist/LocalAI.dmg is ready"
 
 build-launcher-linux:
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz

backend/Dockerfile.ds4

@@ -10,6 +10,7 @@ FROM ${BASE_IMAGE} AS builder
 ARG BUILD_TYPE
 ARG TARGETARCH
 ARG TARGETVARIANT
+ARG CUDA_MAJOR_VERSION
 
 ENV BUILD_TYPE=${BUILD_TYPE} \
     DEBIAN_FRONTEND=noninteractive \
@@ -35,7 +36,8 @@ RUN apt-get update && \
 COPY . /LocalAI
 
 RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
+    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} \
+        CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} NATIVE=false grpc-server package
 
 FROM scratch
 COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=89a0e9803380880305e9e1b83c93614f9df2c893
+AUDIO_CPP_VERSION?=fa5aaac9266a98c68f8a5c9fcd1ba6ff65875416
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ds4/CMakeLists.txt

@@ -84,9 +84,10 @@ elseif(DS4_GPU STREQUAL "cpu")
     set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()
 
-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. Link
-# them regardless of DS4_GPU.
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. Link them regardless of DS4_GPU.
+list(APPEND DS4_OBJS "${DS4_DIR}/ds4_image.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=8db89fe083ae4d17c9a2428ccd29803d3ae8f577
+# Upstream pin lives below as DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=8db89fe083ae4d17c9a2428ccd29803d3ae8f577
+DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,21 +18,83 @@ UNAME_S := $(shell uname -s)
 
 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
 
-# Upstream splits distributed inference, tensor-parallel transport, the SSD
-# expert cache, and layer placement into GPU-agnostic translation units. They
-# are shared by every GPU mode, so append them unconditionally below.
+# nvcc must be told the target architecture explicitly for a cublas build, and
+# this is not a tuning knob. Upstream's Makefile leaves CUDA_ARCH empty and its
+# `cuda` target REFUSES to build without one, offering `cuda-spark`
+# (CUDA_ARCH=sm_121) and `cuda-generic` (CUDA_ARCH=native) instead. We drive its
+# object targets directly, which bypasses that guard: nvcc then compiles with no
+# -arch at all, and the kernels run as JIT'd PTX for its default architecture.
+# On GB10 (sm_121) that silently produced corrupt inference output above a
+# ~128-token prefill batch and ~77x slower prefill (4.21 t/s vs 325.70 t/s,
+# measured on the same box with the same model). No CI runner has a GPU, so
+# `native` has nothing to enumerate there.
+#
+# Upstream's CUDA_ARCH takes a SINGLE value (see its sm_120/sm_121 special cases
+# and the `-arch=$(CUDA_ARCH)` fallback), so it cannot express the fat binary
+# these images need. NVCC_ARCH_FLAGS is overridden instead: a command-line
+# assignment wins over the `:=` in upstream's Makefile, and its NVCCFLAGS
+# expands whatever we pass.
+#
+# The architecture lists are copied from backend/go/vllm-cpp/Makefile rather
+# than invented, so the two CUDA images cover the same GPUs: amd64 datacenter +
+# consumer, and l4t/arm64 covering Orin (87), Thor (110) and GB10 (121a).
+#
+# -DDS4_CUDA_HAVE_MXF4=1 is deliberately NOT set. Upstream only defines it for
+# single-arch sm_120/sm_121 builds and guards the code with a plain #ifdef
+# rather than __CUDA_ARCH__, so it cannot be combined with older archs in one
+# fat binary. It gates an optional MXFP4 indexer fast path whose #ifndef branch
+# returns 0 and falls back to the generic path, so omitting it costs some speed
+# on GB10, not correctness. Revisit if upstream adds __CUDA_ARCH__ guards.
+#
+# An EMPTY CUDA_MAJOR_VERSION means a local developer build, not CI: fall back
+# to upstream's own `native` handling, which needs a GPU present but is what a
+# developer building on their own machine wants. Both variables are `?=` so an
+# explicit value on the command line always wins.
+UNAME_M := $(shell uname -m)
+CUDA_MAJOR_VERSION ?=
+ifeq ($(BUILD_TYPE),cublas)
+ifeq ($(CUDA_MAJOR_VERSION),13)
+ifeq ($(UNAME_M),aarch64)
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_87,code=sm_87 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_110,code=sm_110 \
+                           -gencode arch=compute_121a,code=sm_121a
+else
+    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_80,code=sm_80 \
+                           -gencode arch=compute_86,code=sm_86 \
+                           -gencode arch=compute_89,code=sm_89 \
+                           -gencode arch=compute_90a,code=sm_90a \
+                           -gencode arch=compute_100a,code=sm_100a \
+                           -gencode arch=compute_103a,code=sm_103a \
+                           -gencode arch=compute_120a,code=sm_120a \
+                           -gencode arch=compute_121a,code=sm_121a
+endif
+    DS4_ARCH_MAKEVARS := NVCC_ARCH_FLAGS="$(DS4_NVCC_ARCH_FLAGS)"
+else ifeq ($(CUDA_MAJOR_VERSION),)
+    # Local build: let upstream resolve the host GPU.
+    DS4_ARCH_MAKEVARS := CUDA_ARCH=native
+else
+    $(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (13 does). Leave it empty for a native build, or pass DS4_NVCC_ARCH_FLAGS explicitly.)
+endif
+endif
+
+# Upstream splits image preprocessing, distributed inference, tensor-parallel
+# transport, the SSD expert cache, and layer placement into GPU-agnostic
+# translation units. They are shared by every GPU mode, so append them
+# unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
         cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
         cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
 else ifeq ($(UNAME_S),Darwin)
     CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
     # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
     CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 ifneq ($(NATIVE),true)
@@ -57,11 +119,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 $(DS4_OBJ_TARGET)
+	+$(MAKE) -C ds4 $(DS4_ARCH_MAKEVARS) $(DS4_OBJ_TARGET)
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 grpc-server: ds4/ds4.o

backend/cpp/ds4/dsml_parser.cpp

@@ -92,7 +92,8 @@ std::string json_escape(const std::string &in) {
 
 } // namespace
 
-DsmlParser::DsmlParser() = default;
+DsmlParser::DsmlParser(bool starts_in_thinking)
+    : state_(starts_in_thinking ? State::THINK : State::TEXT) {}
 
 bool DsmlParser::IsInDsmlStructural() const {
     switch (state_) {

backend/cpp/ds4/dsml_parser.h

@@ -17,7 +17,9 @@ struct ParserEvent {
 // Streaming parser. Stateless across instances; one per Predict call.
 class DsmlParser {
 public:
-    DsmlParser();
+    // The chat prompt may already contain the opening thinking marker, so the
+    // generated text can begin directly with reasoning bytes.
+    explicit DsmlParser(bool starts_in_thinking = false);
 
     // Feed a chunk of raw model-emitted text. Appends classified events to
     // `out`. May buffer the tail of `chunk` internally if it looks like a
@@ -43,7 +45,7 @@ public:
 
 private:
     enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
-    State state_ = State::TEXT;
+    State state_;
     std::string buf_;
     std::string current_tool_name_;
     int tool_index_ = -1;

backend/cpp/ds4/dsml_parser_test.cpp

@@ -0,0 +1,133 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for the DSML streaming parser.
+//
+// The repository's backend/cpp/run-unit-tests.sh harness compiles each
+// *_test.cpp as a single translation unit, so include the implementation here.
+
+#include "dsml_parser.cpp"
+
+#include <cstdio>
+#include <string>
+#include <type_traits>
+#include <vector>
+
+namespace {
+
+struct ParsedText {
+    std::string content;
+    std::string reasoning;
+};
+
+int failures = 0;
+
+void check_equal(const std::string &got, const std::string &want,
+                 const char *name) {
+    if (got == want) return;
+    std::fprintf(stderr, "FAIL %s: got \"%s\", want \"%s\"\n",
+                 name, got.c_str(), want.c_str());
+    failures++;
+}
+
+void collect_text(const std::vector<ds4cpp::ParserEvent> &events,
+                  ParsedText *parsed) {
+    for (const auto &event : events) {
+        if (event.type == ds4cpp::ParserEvent::CONTENT) {
+            parsed->content += event.text;
+        } else if (event.type == ds4cpp::ParserEvent::REASONING) {
+            parsed->reasoning += event.text;
+        }
+    }
+}
+
+ParsedText parse_chunks(ds4cpp::DsmlParser *parser,
+                        const std::vector<std::string> &chunks) {
+    ParsedText parsed;
+    for (const auto &chunk : chunks) {
+        std::vector<ds4cpp::ParserEvent> events;
+        parser->Feed(chunk, events);
+        collect_text(events, &parsed);
+    }
+    std::vector<ds4cpp::ParserEvent> events;
+    parser->Flush(events);
+    collect_text(events, &parsed);
+    return parsed;
+}
+
+template <typename Parser>
+void test_reasoning_opened_by_prompt() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL reasoning_opened_by_prompt: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need to calculate factorial recursively.</think>Here is the answer."});
+        check_equal(parsed.reasoning,
+                    "We need to calculate factorial recursively.",
+                    "reasoning_opened_by_prompt:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "reasoning_opened_by_prompt:content");
+    }
+}
+
+template <typename Parser>
+Parser text_parser() {
+    if constexpr (std::is_constructible_v<Parser, bool>) {
+        return Parser(false);
+    } else {
+        return Parser();
+    }
+}
+
+void test_reasoning_disabled() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(&parser, {"Here is the answer."});
+    check_equal(parsed.reasoning, "", "reasoning_disabled:reasoning");
+    check_equal(parsed.content, "Here is the answer.",
+                "reasoning_disabled:content");
+}
+
+void test_explicit_think_tag() {
+    auto parser = text_parser<ds4cpp::DsmlParser>();
+    ParsedText parsed = parse_chunks(
+        &parser, {"<think>reasoning</think>answer"});
+    check_equal(parsed.reasoning, "reasoning", "explicit_think_tag:reasoning");
+    check_equal(parsed.content, "answer", "explicit_think_tag:content");
+}
+
+template <typename Parser>
+void test_split_think_close_marker() {
+    if constexpr (!std::is_constructible_v<Parser, bool>) {
+        std::fprintf(stderr,
+                     "FAIL split_think_close_marker: parser cannot start in thinking state\n");
+        failures++;
+    } else {
+        Parser parser(true);
+        ParsedText parsed = parse_chunks(
+            &parser,
+            {"We need ", "to calculate ", "factorial", "</thi", "nk>",
+             "Here is ", "the answer."});
+        check_equal(parsed.reasoning, "We need to calculate factorial",
+                    "split_think_close_marker:reasoning");
+        check_equal(parsed.content, "Here is the answer.",
+                    "split_think_close_marker:content");
+    }
+}
+
+} // namespace
+
+int main() {
+    test_reasoning_opened_by_prompt<ds4cpp::DsmlParser>();
+    test_reasoning_disabled();
+    test_explicit_think_tag();
+    test_split_think_close_marker<ds4cpp::DsmlParser>();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all dsml_parser checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}

backend/cpp/ds4/generation_limits.h

@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+#include <algorithm>
+
+namespace ds4cpp {
+
+inline int EffectiveGenerationLimit(int requested, int context_size,
+                                    int session_position) {
+  const int limit = requested > 0 ? requested : 256;
+  const int room = context_size - session_position;
+  if (room <= 1) return 0;
+  return std::min(limit, room - 1);
+}
+
+inline int RemainingGenerationBudget(int effective_limit, int produced) {
+  if (effective_limit <= produced) return 0;
+  return effective_limit - produced;
+}
+
+inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance,
+                                       int buffer_capacity) {
+  if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0;
+  return std::min({remaining, draft_allowance + 1, buffer_capacity});
+}
+
+} // namespace ds4cpp

backend/cpp/ds4/generation_limits_test.cpp

@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: MIT
+
+#include "generation_limits.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+void check_equal(int got, int want, const char *name) {
+  if (got == want) return;
+  std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want);
+  failures++;
+}
+
+// Mutation caught: treating omitted or negative max_tokens as unlimited instead
+// of preserving DS4's legacy 256-token default.
+void test_nonpositive_uses_legacy_default_when_space_permits() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256,
+              "zero max_tokens uses legacy default");
+  check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256,
+              "negative max_tokens uses legacy default");
+}
+
+// Mutation caught: applying the legacy default without clamping it to the
+// post-prefill context room and reserved slot.
+void test_legacy_default_is_clamped_by_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199,
+              "legacy default is context-clamped");
+}
+
+// Mutation caught: allowing an explicitly large request to overrun the
+// post-prefill context boundary.
+void test_large_positive_limit_is_clamped_to_context() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667,
+              "large positive is context-clamped");
+}
+
+// Mutation caught: replacing every positive request with the legacy default
+// rather than preserving a smaller configured limit.
+void test_smaller_positive_limit_is_preserved() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64,
+              "smaller positive is preserved");
+}
+
+// Mutation caught: consuming the final context slot instead of reserving it as
+// required by DS4's generation loop.
+void test_no_usable_room_returns_zero() {
+  check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0,
+              "one remaining context slot is not usable");
+}
+
+// Mutation caught: sending the original generation limit to a later
+// speculative cycle instead of subtracting tokens already produced.
+void test_remaining_budget_accounts_for_produced_tokens() {
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6,
+              "remaining budget subtracts produced tokens");
+  check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0,
+              "remaining budget never becomes negative");
+}
+
+// Mutation caught: giving speculative evaluation capacity beyond either the
+// output budget, the draft allowance plus its first target token, or the fixed
+// accepted-token buffer.
+void test_speculative_capacity_obeys_all_bounds() {
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3,
+              "capacity respects remaining output budget");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5,
+              "capacity includes one target token beyond draft allowance");
+  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6,
+              "capacity respects fixed buffer");
+}
+
+} // namespace
+
+int main() {
+  test_nonpositive_uses_legacy_default_when_space_permits();
+  test_legacy_default_is_clamped_by_context();
+  test_large_positive_limit_is_clamped_to_context();
+  test_smaller_positive_limit_is_preserved();
+  test_no_usable_room_returns_zero();
+  test_remaining_budget_accounts_for_produced_tokens();
+  test_speculative_capacity_obeys_all_bounds();
+
+  if (failures == 0) {
+    std::fprintf(stderr, "all generation limit checks passed\n");
+    return 0;
+  }
+  std::fprintf(stderr, "%d check(s) failed\n", failures);
+  return 1;
+}

backend/cpp/ds4/grpc-server.cpp

@@ -10,7 +10,9 @@
 
 #include "dsml_parser.h"   // populated in Task 12
 #include "dsml_renderer.h" // populated in Task 16
+#include "generation_limits.h"
 #include "kv_cache.h"      // populated in Task 17
+#include "request_lifecycle.h"
 
 extern "C" {
 #include "ds4.h"
@@ -35,6 +37,7 @@ extern "C" {
 #include <mutex>
 #include <string>
 #include <thread>
+#include <utility>
 #include <vector>
 
 using grpc::Server;
@@ -69,6 +72,21 @@ int g_route_timeout_sec = 60;
 
 std::atomic<Server *> g_server{nullptr};
 
+static bool server_context_cancelled(void *ud) {
+    return static_cast<ServerContext *>(ud)->IsCancelled();
+}
+
+static void set_session_cancel(void *target, ds4cpp::CancelCallback callback,
+                               void *userdata) noexcept {
+    ds4_session_set_cancel(static_cast<ds4_session *>(target), callback, userdata);
+}
+
+static bool request_should_continue(ds4cpp::RequestLifecycle *request,
+                                    ServerContext *context) {
+    request->ObserveContextCancellation(context->IsCancelled());
+    return request->ShouldContinue();
+}
+
 // Parse a "key:value" option string. Returns empty when no colon.
 static std::pair<std::string, std::string> split_option(const std::string &opt) {
     auto colon = opt.find(':');
@@ -238,37 +256,58 @@ static bool apply_engine_option(ds4_engine_options *opt, const std::string &key,
 
 // When acting as a distributed coordinator, block until the worker route
 // covers all layers (ds4_session_distributed_route_ready == 1) or the timeout
-// elapses. Returns an empty string on success, or an error message to return
-// to the client. No-op when not distributed.
+// elapses. No-op when not distributed.
 //
 // Takes the g_engine_mu lock by reference and RELEASES it during each poll
 // sleep. The wait can span up to g_route_timeout_sec seconds while workers
 // connect; holding g_engine_mu the whole time would block the Status/Health
 // readiness probes (they also lock g_engine_mu), making LocalAI's loader treat
 // a still-starting worker as hung.
-static std::string wait_route_ready(std::unique_lock<std::mutex> &lock) {
-    if (!g_distributed) return "";
+struct RouteWaitResult {
+    ds4cpp::RouteWaitDecision decision;
+    std::string error;
+};
+
+static RouteWaitResult wait_route_ready(std::unique_lock<std::mutex> &lock,
+                                        ServerContext *context) {
+    if (!g_distributed) return {ds4cpp::RouteWaitDecision::Ready, ""};
     char err[256] = {0};
     const int deadline_polls = g_route_timeout_sec * 10; // 100ms per poll
     for (int i = 0; i <= deadline_polls; ++i) {
         int ready = ds4_session_distributed_route_ready(g_session, err, sizeof(err));
-        if (ready == 1) return "";
-        if (ready < 0) {
-            return std::string("ds4 distributed route error: ") +
-                   (err[0] ? err : "unknown");
+        switch (ds4cpp::DecideRouteWait(ready, context->IsCancelled())) {
+        case ds4cpp::RouteWaitDecision::Ready:
+            return {ds4cpp::RouteWaitDecision::Ready, ""};
+        case ds4cpp::RouteWaitDecision::Error:
+            return {ds4cpp::RouteWaitDecision::Error,
+                    std::string("ds4 distributed route error: ") +
+                        (err[0] ? err : "unknown")};
+        case ds4cpp::RouteWaitDecision::Cancelled:
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        case ds4cpp::RouteWaitDecision::Pending:
+            break;
         }
+        if (i == deadline_polls) break;
         // Release the lock while sleeping so Status/Health and other RPCs can
         // interleave during worker startup.
         lock.unlock();
         struct timespec ts = {0, 100L * 1000L * 1000L}; // 100ms
         nanosleep(&ts, nullptr);
         lock.lock();
+        if (context->IsCancelled()) {
+            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+        }
         // A concurrent Free() may have torn down the engine while we slept.
         if (!g_engine || !g_session) {
-            return "ds4: model unloaded while waiting for distributed route";
+            return {ds4cpp::RouteWaitDecision::Error,
+                    "ds4: model unloaded while waiting for distributed route"};
         }
     }
-    return "ds4 distributed route incomplete: workers not connected (layers uncovered)";
+    if (context->IsCancelled()) {
+        return {ds4cpp::RouteWaitDecision::Cancelled, ""};
+    }
+    return {ds4cpp::RouteWaitDecision::Error,
+            "ds4 distributed route incomplete: workers not connected (layers uncovered)"};
 }
 
 static void append_token_text(ds4_engine *engine, int token, std::string &out) {
@@ -341,9 +380,9 @@ static void collect_done(void *) {}
 struct StreamCtx {
     ds4_engine *engine;
     ServerWriter<backend::Reply> *writer;
+    ds4cpp::RequestLifecycle *request;
     ds4cpp::DsmlParser parser;
     int tokens;
-    bool aborted;
     // Track which tool indices we've seen TOOL_START for, so subsequent
     // ARGS deltas can elide the redundant id/name fields.
     std::vector<bool> tool_started;
@@ -351,7 +390,7 @@ struct StreamCtx {
 
 static void stream_emit(void *ud, int token) {
     auto *s = static_cast<StreamCtx *>(ud);
-    if (s->aborted) return;
+    if (!s->request->ShouldContinue()) return;
     if (token == ds4_token_eos(s->engine)) return;
     size_t len = 0;
     const char *text = ds4_token_text(s->engine, token, &len);
@@ -401,7 +440,7 @@ static void stream_emit(void *ud, int token) {
     reply.set_message(chunk);
     reply.set_tokens(1);
     if (any_field) {
-        if (!s->writer->Write(reply)) s->aborted = true;
+        s->request->ObserveStreamWrite(s->writer->Write(reply));
     }
     s->tokens++;
 }
@@ -757,21 +796,30 @@ public:
         return GStatus::OK;
     }
 
-    GStatus Predict(ServerContext *, const backend::PredictOptions *request,
+    GStatus Predict(ServerContext *context, const backend::PredictOptions *request,
                    backend::Reply *reply) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        CollectCtx collect = {
+            g_engine, "", ds4cpp::DsmlParser(starts_in_thinking),
+            reply, 0, {}, "", ""};
+        ds4cpp::RequestLifecycle lifecycle;
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit; // future: skip prompt prefix if hit covers full prompt
@@ -783,15 +831,27 @@ public:
         // Either way g_session advances so the disk KV cache picks up a
         // real checkpoint after the call (see maybe_save_cache below).
         char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
         int prompt_len = prompt.len;
         ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
             while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                 SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -806,13 +866,20 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         collect_emit(&collect, accepted[j]);
                         if (++produced >= n_predict) { stop = true; break; }
@@ -821,12 +888,26 @@ public:
                 } else {
                     collect_emit(&collect, first);
                     if (++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
-            collect_done(&collect);
         }
+
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        if (!terminal.should_finalize) {
+            if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
+                return GStatus(StatusCode::INTERNAL,
+                              std::string("ds4 generation failed: ") + err);
+            }
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
+        if (generation_started) collect_done(&collect);
         maybe_save_cache(cache_key);
 
         // Flush any buffered parser state.
@@ -834,7 +915,7 @@ public:
         collect.parser.Flush(events);
         apply_events(&collect, events);
 
-        if (rc != 0) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
@@ -857,21 +938,30 @@ public:
         return GStatus::OK;
     }
 
-    GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
+    GStatus PredictStream(ServerContext *context, const backend::PredictOptions *request,
                          ServerWriter<backend::Reply> *writer) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
-            return GStatus(StatusCode::UNAVAILABLE, route_err);
+        RouteWaitResult route = wait_route_ready(lock, context);
+        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
+            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
+        }
+        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
+            return GStatus(StatusCode::UNAVAILABLE, route.error);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
-        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        StreamCtx s = {g_engine, writer, {}, 0, false, {}};
+        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
+        const bool starts_in_thinking = think_enabled &&
+            request->usetokenizertemplate() && request->messages_size() > 0;
+        ds4cpp::RequestLifecycle lifecycle;
+        StreamCtx s = {
+            g_engine, writer, &lifecycle,
+            ds4cpp::DsmlParser(starts_in_thinking), 0, {}};
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit;
@@ -879,14 +969,26 @@ public:
         // Manual loop on g_session - see Predict() above for the rationale.
         // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
         char err[256] = {0};
-        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        int rc;
+        {
+            ds4cpp::CancelCallbackScope cancel_scope(
+                g_session, set_session_cancel, server_context_cancelled, context);
+            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
+        }
         ds4_tokens_free(&prompt);
-        if (rc == 0) {
+        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
+            lifecycle.ObserveContextCancellation(true);
+        }
+        const bool generation_started = rc == 0;
+        if (generation_started) {
+            const int n_predict = ds4cpp::EffectiveGenerationLimit(
+                request->tokens(), ds4_session_ctx(g_session),
+                ds4_session_pos(g_session));
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
-            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
-            while (produced < n_predict && !s.aborted) {
+            while (produced < n_predict) {
+                if (!request_should_continue(&lifecycle, context)) break;
                 SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -900,50 +1002,77 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    int cap = std::min(kAcceptedMax, draft_max + 1);
+                    const int remaining = ds4cpp::RemainingGenerationBudget(
+                        n_predict, produced);
+                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
+                        remaining, draft_max, kAcceptedMax);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, draft_max, eos,
+                        g_session, first, remaining, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
+                        if (!request_should_continue(&lifecycle, context)) {
+                            stop = true;
+                            break;
+                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         stream_emit(&s, accepted[j]);
-                        if (s.aborted) { stop = true; break; }
+                        if (!lifecycle.ShouldContinue()) { stop = true; break; }
                         if (++produced >= n_predict) { stop = true; break; }
                     }
                     if (stop) break;
                 } else {
                     stream_emit(&s, first);
-                    if (s.aborted || ++produced >= n_predict) break;
+                    if (!lifecycle.ShouldContinue() || ++produced >= n_predict) break;
+                    if (!request_should_continue(&lifecycle, context)) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
-            stream_done(&s);
         }
-        maybe_save_cache(cache_key);
 
-        // Flush parser state.
-        std::vector<ds4cpp::ParserEvent> events;
-        s.parser.Flush(events);
-        if (!events.empty() && !s.aborted) {
-            backend::Reply reply;
-            auto *delta = reply.add_chat_deltas();
-            for (const auto &e : events) {
-                if (e.type == ds4cpp::ParserEvent::CONTENT) {
-                    delta->set_content(delta->content() + e.text);
-                } else if (e.type == ds4cpp::ParserEvent::REASONING) {
-                    delta->set_reasoning_content(delta->reasoning_content() + e.text);
+        request_should_continue(&lifecycle, context);
+        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
+            !lifecycle.ShouldFinalize());
+        terminal = ds4cpp::RunPostlude(
+            terminal,
+            [&]() {
+                ds4cpp::DsmlParser staged_parser = s.parser;
+                std::vector<ds4cpp::ParserEvent> events;
+                staged_parser.Flush(events);
+                bool write_succeeded = true;
+                if (!events.empty()) {
+                    backend::Reply reply;
+                    auto *delta = reply.add_chat_deltas();
+                    for (const auto &e : events) {
+                        if (e.type == ds4cpp::ParserEvent::CONTENT) {
+                            delta->set_content(delta->content() + e.text);
+                        } else if (e.type == ds4cpp::ParserEvent::REASONING) {
+                            delta->set_reasoning_content(
+                                delta->reasoning_content() + e.text);
+                        }
+                    }
+                    write_succeeded = s.writer->Write(reply);
                 }
-            }
-            s.writer->Write(reply);
-        }
+                lifecycle.ObserveStreamWrite(write_succeeded);
+                request_should_continue(&lifecycle, context);
+                if (!lifecycle.ShouldFinalize()) return false;
+                s.parser = std::move(staged_parser);
+                if (generation_started) stream_done(&s);
+                return true;
+            },
+            [&]() { maybe_save_cache(cache_key); });
 
-        if (rc != 0 && !s.aborted) {
+        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
+        if (terminal.cause == ds4cpp::TerminalCause::Cancelled) {
+            return GStatus(StatusCode::CANCELLED,
+                          "ds4 request cancelled");
+        }
         return GStatus::OK;
     }
 

backend/cpp/ds4/request_lifecycle.h

@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: MIT
+#pragma once
+
+namespace ds4cpp {
+
+using CancelCallback = bool (*)(void *);
+using CancelSetter = void (*)(void *, CancelCallback, void *) noexcept;
+
+class CancelCallbackScope {
+public:
+    CancelCallbackScope(void *target, CancelSetter setter,
+                        CancelCallback callback, void *userdata) noexcept
+        : target_(target), setter_(setter) {
+        setter_(target_, callback, userdata);
+    }
+
+    ~CancelCallbackScope() noexcept {
+        setter_(target_, nullptr, nullptr);
+    }
+
+    CancelCallbackScope(const CancelCallbackScope &) = delete;
+    CancelCallbackScope &operator=(const CancelCallbackScope &) = delete;
+
+private:
+    void *target_;
+    CancelSetter setter_;
+};
+
+enum class RouteWaitDecision {
+    Pending,
+    Ready,
+    Error,
+    Cancelled,
+};
+
+inline RouteWaitDecision DecideRouteWait(int route_status, bool cancelled) {
+    if (cancelled) return RouteWaitDecision::Cancelled;
+    if (route_status > 0) return RouteWaitDecision::Ready;
+    if (route_status < 0) return RouteWaitDecision::Error;
+    return RouteWaitDecision::Pending;
+}
+
+enum class TerminalCause {
+    Success,
+    Cancelled,
+    EngineError,
+};
+
+inline TerminalCause DecideTerminalCause(bool sync_interrupted,
+                                         bool engine_error,
+                                         bool abandoned) {
+    if (sync_interrupted) return TerminalCause::Cancelled;
+    if (engine_error) return TerminalCause::EngineError;
+    if (abandoned) return TerminalCause::Cancelled;
+    return TerminalCause::Success;
+}
+
+struct TerminalDecision {
+    TerminalCause cause;
+    bool should_finalize;
+};
+
+inline TerminalDecision ResolveTerminalDecision(bool sync_interrupted,
+                                                 bool engine_error,
+                                                 bool abandoned) {
+    return {
+        DecideTerminalCause(sync_interrupted, engine_error, abandoned),
+        !sync_interrupted && !abandoned,
+    };
+}
+
+template <typename Finalize, typename Persist>
+TerminalDecision RunPostlude(TerminalDecision terminal,
+                             Finalize transactional_finalize,
+                             Persist persist) {
+    if (!terminal.should_finalize) return terminal;
+    if (!transactional_finalize()) {
+        terminal.should_finalize = false;
+        if (terminal.cause != TerminalCause::EngineError) {
+            terminal.cause = TerminalCause::Cancelled;
+        }
+        return terminal;
+    }
+    persist();
+    return terminal;
+}
+
+class RequestLifecycle {
+public:
+    void ObserveContextCancellation(bool cancelled) {
+        context_cancelled_ = context_cancelled_ || cancelled;
+    }
+
+    void ObserveStreamWrite(bool succeeded) {
+        stream_write_aborted_ = stream_write_aborted_ || !succeeded;
+    }
+
+    bool ShouldContinue() const {
+        return !context_cancelled_ && !stream_write_aborted_;
+    }
+
+    bool ShouldFinalize() const {
+        return ShouldContinue();
+    }
+
+private:
+    bool context_cancelled_ = false;
+    bool stream_write_aborted_ = false;
+};
+
+} // namespace ds4cpp

backend/cpp/ds4/request_lifecycle_test.cpp

@@ -0,0 +1,414 @@
+// SPDX-License-Identifier: MIT
+// Standalone regression tests for DS4 request cancellation policy.
+
+#include "request_lifecycle.h"
+
+#include <cstdio>
+
+namespace {
+
+int failures = 0;
+
+struct FakeCancelTarget {
+    ds4cpp::CancelCallback callback = nullptr;
+    void *userdata = nullptr;
+    int installs = 0;
+    int clears = 0;
+};
+
+struct PostludeCounts {
+    int finalize_attempts = 0;
+    int finalize_commits = 0;
+    int cache_persists = 0;
+    bool cache_followed_commit = true;
+};
+
+ds4cpp::TerminalDecision run_fake_postlude(
+        ds4cpp::TerminalDecision terminal, bool finalize_succeeds,
+        PostludeCounts *counts) {
+    return ds4cpp::RunPostlude(
+        terminal,
+        [=]() {
+            counts->finalize_attempts++;
+            if (!finalize_succeeds) return false;
+            counts->finalize_commits++;
+            return true;
+        },
+        [=]() {
+            counts->cache_followed_commit = counts->finalize_commits == 1;
+            counts->cache_persists++;
+        });
+}
+
+bool fake_cancel(void *) {
+    return false;
+}
+
+void fake_set_cancel(void *target, ds4cpp::CancelCallback callback,
+                     void *userdata) noexcept {
+    auto *fake = static_cast<FakeCancelTarget *>(target);
+    fake->callback = callback;
+    fake->userdata = userdata;
+    if (callback) {
+        fake->installs++;
+    } else {
+        fake->clears++;
+    }
+}
+
+void check(bool condition, const char *name) {
+    if (condition) return;
+    std::fprintf(stderr, "FAIL %s\n", name);
+    failures++;
+}
+
+// Production mutation caught: treating an active request as abandoned would
+// skip its parser finalization and cache save.
+void test_active_request_continues_and_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    check(request.ShouldContinue(), "active:continue");
+    check(request.ShouldFinalize(), "active:finalize");
+}
+
+// Production mutation caught: omitting the ServerContext cancellation branch
+// would continue decoding and finalize a partial response.
+void test_context_cancellation_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveContextCancellation(true);
+
+    check(!request.ShouldContinue(), "context_cancelled:stop");
+    check(!request.ShouldFinalize(), "context_cancelled:no_finalize");
+}
+
+// Production mutation caught: ignoring ServerWriter::Write failure would keep
+// streaming and finalize a response whose client has gone away.
+void test_stream_write_abort_stops_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+
+    request.ObserveStreamWrite(false);
+
+    check(!request.ShouldContinue(), "write_abort:stop");
+    check(!request.ShouldFinalize(), "write_abort:no_finalize");
+}
+
+// Production mutation caught: combining cancellation and write failure with
+// AND would fail to stop when either signal occurs on its own.
+void test_cancellation_and_write_abort_are_independent_or_conditions() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    cancelled.ObserveStreamWrite(true);
+
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveContextCancellation(false);
+    write_aborted.ObserveStreamWrite(false);
+
+    check(!cancelled.ShouldContinue(), "or:context_only");
+    check(!write_aborted.ShouldContinue(), "or:write_only");
+}
+
+// Production mutation caught: treating an incomplete distributed route as an
+// error would return before workers have time to connect.
+void test_route_wait_pending() {
+    check(ds4cpp::DecideRouteWait(0, false) ==
+              ds4cpp::RouteWaitDecision::Pending,
+          "route_wait:pending");
+}
+
+// Production mutation caught: failing to recognize a complete route would
+// keep a ready inference request in the polling loop.
+void test_route_wait_ready() {
+    check(ds4cpp::DecideRouteWait(1, false) ==
+              ds4cpp::RouteWaitDecision::Ready,
+          "route_wait:ready");
+}
+
+// Production mutation caught: ignoring a route probe error would poll until a
+// misleading timeout instead of returning UNAVAILABLE promptly.
+void test_route_wait_error() {
+    check(ds4cpp::DecideRouteWait(-1, false) ==
+              ds4cpp::RouteWaitDecision::Error,
+          "route_wait:error");
+}
+
+// Production mutation caught: omitting cancellation from route waiting would
+// leave an abandoned request blocked until the distributed timeout.
+void test_route_wait_cancellation() {
+    check(ds4cpp::DecideRouteWait(0, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancelled");
+}
+
+// Production mutation caught: checking route errors before cancellation would
+// report UNAVAILABLE for a request the client already abandoned.
+void test_route_wait_cancellation_precedes_error() {
+    check(ds4cpp::DecideRouteWait(-1, true) ==
+              ds4cpp::RouteWaitDecision::Cancelled,
+          "route_wait:cancellation_precedence");
+}
+
+// Production mutation caught: classifying a successful active request as a
+// terminal failure would suppress its normal response finalization.
+void test_terminal_success() {
+    check(ds4cpp::DecideTerminalCause(false, false, false) ==
+              ds4cpp::TerminalCause::Success,
+          "terminal:success");
+}
+
+// Production mutation caught: treating DS4's cooperative sync interruption
+// as an ordinary engine error would return INTERNAL instead of CANCELLED.
+void test_terminal_sync_interruption_is_cancelled() {
+    check(ds4cpp::DecideTerminalCause(true, true, true) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:sync_interrupted");
+}
+
+// Production mutation caught: treating every nonzero engine result as client
+// abandonment would hide genuine DS4 failures behind CANCELLED.
+void test_terminal_engine_error() {
+    check(ds4cpp::DecideTerminalCause(false, true, false) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error");
+}
+
+// Production mutation caught: ignoring an rc==0 context cancellation would
+// finalize and cache an abandoned request.
+void test_terminal_context_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:context_abandonment");
+}
+
+// Production mutation caught: ignoring an rc==0 stream write failure would
+// finalize and cache an abandoned streaming request.
+void test_terminal_write_abandonment() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, false, !request.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::Cancelled,
+          "terminal:write_abandonment");
+}
+
+// Production mutation caught: checking late cancellation or write failure
+// before a determined ordinary DS4 error would replace INTERNAL with CANCELLED.
+void test_terminal_engine_error_precedes_late_abandonment() {
+    ds4cpp::RequestLifecycle cancelled;
+    cancelled.ObserveContextCancellation(true);
+    ds4cpp::RequestLifecycle write_aborted;
+    write_aborted.ObserveStreamWrite(false);
+
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !cancelled.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_cancellation");
+    check(ds4cpp::DecideTerminalCause(
+              false, true, !write_aborted.ShouldFinalize()) ==
+              ds4cpp::TerminalCause::EngineError,
+          "terminal:engine_error_precedes_write_abort");
+}
+
+// Production mutation caught: using status precedence alone to gate side
+// effects would finalize and persist an engine-error request abandoned later.
+void test_abandoned_engine_error_keeps_internal_without_finalizing() {
+    ds4cpp::RequestLifecycle request;
+    request.ObserveContextCancellation(true);
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:abandoned_engine_error_status");
+    check(!terminal.should_finalize,
+          "terminal_decision:abandoned_engine_error_no_finalize");
+}
+
+// Production mutation caught: suppressing side effects for every engine error
+// would change the existing finalization and cache behavior of active failures.
+void test_active_engine_error_still_finalizes() {
+    ds4cpp::RequestLifecycle request;
+
+    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
+        false, true, !request.ShouldFinalize());
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "terminal_decision:active_engine_error_status");
+    check(terminal.should_finalize,
+          "terminal_decision:active_engine_error_finalize");
+}
+
+// Production mutation caught: persisting before committed finalization would
+// cache a state whose final buffered stream reply was never completed.
+void test_postlude_active_success_commits_then_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Success,
+          "postlude:success_outcome");
+    check(terminal.should_finalize, "postlude:success_committed");
+    check(counts.finalize_attempts == 1, "postlude:success_attempts");
+    check(counts.finalize_commits == 1, "postlude:success_commits");
+    check(counts.cache_persists == 1, "postlude:success_cache");
+    check(counts.cache_followed_commit, "postlude:success_cache_order");
+}
+
+// Production mutation caught: starting the postlude for an already-cancelled
+// request would flush buffered parser state or persist an abandoned session.
+void test_postlude_cancellation_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Cancelled, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:cancelled_outcome");
+    check(counts.finalize_attempts == 0, "postlude:cancelled_attempts");
+    check(counts.finalize_commits == 0, "postlude:cancelled_commits");
+    check(counts.cache_persists == 0, "postlude:cancelled_cache");
+}
+
+// Production mutation caught: committing the live parser or cache after a
+// failed final Write would publish an abandoned streaming postlude.
+void test_postlude_finalize_failure_cancels_without_commit_or_cache() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::Success, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
+          "postlude:write_failure_outcome");
+    check(!terminal.should_finalize, "postlude:write_failure_not_committed");
+    check(counts.finalize_attempts == 1, "postlude:write_failure_attempts");
+    check(counts.finalize_commits == 0, "postlude:write_failure_commits");
+    check(counts.cache_persists == 0, "postlude:write_failure_cache");
+}
+
+// Production mutation caught: skipping the postlude for every engine error
+// would change active internal-error finalization and cache behavior.
+void test_postlude_active_engine_error_finalizes_and_persists() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_outcome");
+    check(counts.finalize_attempts == 1, "postlude:engine_error_attempts");
+    check(counts.finalize_commits == 1, "postlude:engine_error_commits");
+    check(counts.cache_persists == 1, "postlude:engine_error_cache");
+    check(counts.cache_followed_commit, "postlude:engine_error_cache_order");
+}
+
+// Production mutation caught: replacing every failed transactional finalize
+// with cancellation would hide an already-determined engine error.
+void test_postlude_engine_error_finalize_failure_preserves_internal() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, true}, false, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:engine_error_write_failure_outcome");
+    check(!terminal.should_finalize,
+          "postlude:engine_error_write_failure_not_committed");
+    check(counts.finalize_attempts == 1,
+          "postlude:engine_error_write_failure_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:engine_error_write_failure_commits");
+    check(counts.cache_persists == 0,
+          "postlude:engine_error_write_failure_cache");
+}
+
+// Production mutation caught: status precedence must not grant side-effect
+// permission to an engine-error request that was also abandoned.
+void test_postlude_abandoned_engine_error_skips_all_side_effects() {
+    PostludeCounts counts;
+
+    ds4cpp::TerminalDecision terminal = run_fake_postlude(
+        {ds4cpp::TerminalCause::EngineError, false}, true, &counts);
+
+    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
+          "postlude:abandoned_engine_error_outcome");
+    check(counts.finalize_attempts == 0,
+          "postlude:abandoned_engine_error_attempts");
+    check(counts.finalize_commits == 0,
+          "postlude:abandoned_engine_error_commits");
+    check(counts.cache_persists == 0,
+          "postlude:abandoned_engine_error_cache");
+}
+
+// Production mutation caught: failing to install the request callback would
+// make DS4 prompt synchronization unable to observe client cancellation.
+void test_cancel_callback_scope_installs_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+        check(target.callback == fake_cancel, "cancel_scope:callback_installed");
+        check(target.userdata == &request_context, "cancel_scope:userdata_installed");
+        check(target.installs == 1, "cancel_scope:installed_once");
+    }
+}
+
+// Production mutation caught: failing to clear the callback at every scope
+// exit would leave DS4 pointing at a destroyed stack-owned ServerContext.
+void test_cancel_callback_scope_clears_callback() {
+    FakeCancelTarget target;
+    int request_context = 42;
+
+    {
+        ds4cpp::CancelCallbackScope scope(
+            &target, fake_set_cancel, fake_cancel, &request_context);
+    }
+
+    check(target.callback == nullptr, "cancel_scope:callback_cleared");
+    check(target.userdata == nullptr, "cancel_scope:userdata_cleared");
+    check(target.clears == 1, "cancel_scope:cleared_once");
+}
+
+} // namespace
+
+int main() {
+    test_active_request_continues_and_finalizes();
+    test_context_cancellation_stops_without_finalizing();
+    test_stream_write_abort_stops_without_finalizing();
+    test_cancellation_and_write_abort_are_independent_or_conditions();
+    test_route_wait_pending();
+    test_route_wait_ready();
+    test_route_wait_error();
+    test_route_wait_cancellation();
+    test_route_wait_cancellation_precedes_error();
+    test_terminal_success();
+    test_terminal_sync_interruption_is_cancelled();
+    test_terminal_engine_error();
+    test_terminal_context_abandonment();
+    test_terminal_write_abandonment();
+    test_terminal_engine_error_precedes_late_abandonment();
+    test_abandoned_engine_error_keeps_internal_without_finalizing();
+    test_active_engine_error_still_finalizes();
+    test_postlude_active_success_commits_then_persists();
+    test_postlude_cancellation_skips_all_side_effects();
+    test_postlude_finalize_failure_cancels_without_commit_or_cache();
+    test_postlude_active_engine_error_finalizes_and_persists();
+    test_postlude_engine_error_finalize_failure_preserves_internal();
+    test_postlude_abandoned_engine_error_skips_all_side_effects();
+    test_cancel_callback_scope_installs_callback();
+    test_cancel_callback_scope_clears_callback();
+
+    if (failures == 0) {
+        std::fprintf(stderr, "all request_lifecycle checks passed\n");
+        return 0;
+    }
+    std::fprintf(stderr, "%d check(s) failed\n", failures);
+    return 1;
+}

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=15dddc60b3fc937a9e2a210359ecce392ccdf446
+IK_LLAMA_VERSION?=3e416d7f5a9d4cc3195e8171dbf891541ca59c6a
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=d7bd3bfcad3e29c7e49fd26f38c79ee3e9a3fd6b
+LLAMA_VERSION?=434ddbbc0e30522e897670681e503b797c12b7c1
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/prepare.sh

@@ -15,6 +15,30 @@ if [ -d "patches" ]; then
     done
 fi
 
+## Apple RDMA link fixup.
+
+## ggml-rpc hands Apple's librdma to the linker with
+## target_link_options(ggml-rpc PRIVATE "LINKER:-weak_library,..."). Link options are not
+## a usage requirement of a static library, so in our BUILD_SHARED_LIBS=OFF build the flag
+## dies with libggml-rpc.a and every ibv_* symbol transport-apple.cpp reaches for comes out
+## undefined when grpc-server and ggml-rpc-server link. Re-declare the same weak link as
+## INTERFACE so it travels to whoever links the static library.
+##
+## Guarded on the marker so a second prepare.sh over the same checkout is a no-op, and on
+## GGML_RPC_RDMA_APPLE so forks that branched before the Apple RDMA transport (turboquant,
+## bonsai) are left alone.
+RPC_CMAKE=llama.cpp/ggml/src/ggml-rpc/CMakeLists.txt
+if [ -f "$RPC_CMAKE" ] && grep -q "GGML_RPC_RDMA_APPLE" "$RPC_CMAKE" && ! grep -q "LOCALAI_RDMA_IFACE" "$RPC_CMAKE"; then
+    echo "==> ggml-rpc carries the Apple RDMA transport, re-declaring its weak librdma link as INTERFACE"
+    cat >> "$RPC_CMAKE" <<'EOF'
+
+# LOCALAI_RDMA_IFACE: added by backend/cpp/llama-cpp/prepare.sh
+if (GGML_RPC_RDMA AND APPLE AND NOT BUILD_SHARED_LIBS)
+    target_link_options(ggml-rpc INTERFACE "LINKER:-weak_library,${RDMA_LIB}")
+endif()
+EOF
+fi
+
 for file in $(ls llama.cpp/tools/server/); do
     cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
 done

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=ae4474dd8306384a0e697183d863dfc52e69a2fb
+CRISPASR_VERSION?=301acd87b036764973b8bfba71e0a21818036d33
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/crispasr/gocrispasr.go

@@ -615,10 +615,10 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 		return fmt.Errorf("crispasr: tempfile: %w", err)
 	}
 	dst := tmp.Name()
+	defer func() { _ = os.Remove(dst) }()
 	if err := tmp.Close(); err != nil {
 		return fmt.Errorf("crispasr: close tempfile: %w", err)
 	}
-	defer func() { _ = os.Remove(dst) }()
 
 	if err := writeWAV(dst, pcm, w.sampleRate); err != nil {
 		return err

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=739992d10bf9472c46dcd4622b14d2b20766c58d
+DEPTHANYTHING_VERSION?=14f7461d1f704761a038ac9f50dbde8fdb7275e2
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/locate-anything-cpp/golocateanythingcpp.go

@@ -11,6 +11,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"unsafe"
 
 	"github.com/mudler/LocalAI/pkg/grpc/base"
@@ -109,30 +110,25 @@ func (r *LocateAnythingCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, e
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: a text prompt is required (open-vocabulary detection)")
 	}
 
-	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to decode base64 image: %w", err)
 	}
-
-	tmpFile, err := os.CreateTemp("", "locate-anything-*.img")
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to create temp file: %w", err)
-	}
-	defer func() { _ = os.Remove(tmpFile.Name()) }()
-
-	if _, err := tmpFile.Write(imgData); err != nil {
-		_ = tmpFile.Close()
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to write temp file: %w", err)
-	}
-	if err := tmpFile.Close(); err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to close temp file: %w", err)
+	if len(imgData) == 0 {
+		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: decoded image is empty")
 	}
 
 	// mode 0 = hybrid (Parallel Box Decoding). The JSON return value is unused:
 	// structured detections are read via the accessor functions. Still must
 	// free the returned string.
-	jsonPtr := CapiLocatePath(r.handle, tmpFile.Name(), prompt, 0)
+	jsonPtr := CapiLocateBuffer(
+		r.handle,
+		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
+		uintptr(len(imgData)),
+		prompt,
+		0,
+	)
+	runtime.KeepAlive(imgData)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}

backend/go/locate-anything-cpp/golocateanythingcpp_unit_test.go

@@ -0,0 +1,54 @@
+package main
+
+import (
+	"encoding/base64"
+	"path/filepath"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("LocateAnythingCpp detection input", func() {
+	It("detects from memory when the temporary directory is unavailable", func() {
+		originalLocateBuffer := CapiLocateBuffer
+		originalLocatePath := CapiLocatePath
+		originalGetNDetections := CapiGetNDetections
+		defer func() {
+			CapiLocateBuffer = originalLocateBuffer
+			CapiLocatePath = originalLocatePath
+			CapiGetNDetections = originalGetNDetections
+		}()
+
+		image := []byte("encoded-image")
+		var receivedData uintptr
+		var receivedLength uintptr
+		CapiLocateBuffer = func(_ uintptr, data uintptr, length uintptr, _ string, _ int32) uintptr {
+			receivedData = data
+			receivedLength = length
+			return 0
+		}
+		CapiLocatePath = func(_ uintptr, _ string, _ string, _ int32) uintptr {
+			Fail("path-based detection must not be called")
+			return 0
+		}
+		CapiGetNDetections = func(uintptr) int32 { return 0 }
+		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
+
+		result, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{
+			Src:    base64.StdEncoding.EncodeToString(image),
+			Prompt: "the object",
+		})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(result.Detections).To(BeEmpty())
+		Expect(receivedData).NotTo(BeZero())
+		Expect(receivedLength).To(Equal(uintptr(len(image))))
+	})
+
+	It("rejects an empty decoded image", func() {
+		_, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{Prompt: "the object"})
+
+		Expect(err).To(MatchError("locate-anything-cpp: decoded image is empty"))
+	})
+})

backend/go/nemo-speech-cpp/Makefile

@@ -12,7 +12,7 @@
 # runs 'make -C backend/go/$(BACKEND) build' and then copies package/), so it
 # has to produce the binary and the package, not just the shared libraries.
 
-NEMO_SPEECH_VERSION?=4f9676226f667d14608487df744f375db87127f8
+NEMO_SPEECH_VERSION?=a5b6953c4a579a2bbd1c0913ad8a85c2a4d99953
 NEMO_SPEECH_REPO?=https://github.com/NVIDIA/NeMo-Speech.cpp
 
 GOCMD?=go

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
+OMNIVOICE_VERSION?=040c8b344d8c670ce1475194751d119b5ef82c78
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/rfdetr-cpp/gorfdetrcpp.go

@@ -10,6 +10,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime"
 	"strconv"
 	"unsafe"
 
@@ -102,24 +103,12 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: model not loaded")
 	}
 
-	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to decode base64 image: %w", err)
 	}
-
-	tmpFile, err := os.CreateTemp("", "rfdetr-*.img")
-	if err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to create temp file: %w", err)
-	}
-	defer func() { _ = os.Remove(tmpFile.Name()) }()
-
-	if _, err := tmpFile.Write(imgData); err != nil {
-		_ = tmpFile.Close()
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to write temp file: %w", err)
-	}
-	if err := tmpFile.Close(); err != nil {
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to close temp file: %w", err)
+	if len(imgData) == 0 {
+		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: decoded image is empty")
 	}
 
 	threshold := opts.Threshold
@@ -127,10 +116,18 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		threshold = 0.5
 	}
 
-	// JSON output from detect_path is unused: we read structured detections via
+	// JSON output from the detection ABI is unused: we read structured detections via
 	// the accessor functions. Still must free the returned string.
 	var jsonPtr uintptr
-	rc := CapiDetectPath(r.handle, tmpFile.Name(), threshold, uint32(defaultTopK), &jsonPtr)
+	rc := CapiDetectBuffer(
+		r.handle,
+		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
+		uintptr(len(imgData)),
+		threshold,
+		uint32(defaultTopK),
+		&jsonPtr,
+	)
+	runtime.KeepAlive(imgData)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}

backend/go/rfdetr-cpp/gorfdetrcpp_unit_test.go

@@ -0,0 +1,56 @@
+package main
+
+import (
+	"encoding/base64"
+	"path/filepath"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("RFDetrCpp detection input", func() {
+	It("detects from memory when the temporary directory is unavailable", func() {
+		originalDetectBuffer := CapiDetectBuffer
+		originalDetectPath := CapiDetectPath
+		originalFreeString := CapiFreeString
+		originalGetNDetections := CapiGetNDetections
+		defer func() {
+			CapiDetectBuffer = originalDetectBuffer
+			CapiDetectPath = originalDetectPath
+			CapiFreeString = originalFreeString
+			CapiGetNDetections = originalGetNDetections
+		}()
+
+		image := []byte("encoded-image")
+		var receivedData uintptr
+		var receivedLength uintptr
+		CapiDetectBuffer = func(_ uintptr, data uintptr, length uintptr, _ float32, _ uint32, _ *uintptr) int32 {
+			receivedData = data
+			receivedLength = length
+			return 0
+		}
+		CapiDetectPath = func(_ uintptr, _ string, _ float32, _ uint32, _ *uintptr) int32 {
+			Fail("path-based detection must not be called")
+			return -1
+		}
+		CapiFreeString = func(uintptr) {}
+		CapiGetNDetections = func(uintptr) int32 { return 0 }
+		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
+
+		result, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{
+			Src: base64.StdEncoding.EncodeToString(image),
+		})
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(result.Detections).To(BeEmpty())
+		Expect(receivedData).NotTo(BeZero())
+		Expect(receivedLength).To(Equal(uintptr(len(image))))
+	})
+
+	It("rejects an empty decoded image", func() {
+		_, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{})
+
+		Expect(err).To(MatchError("rfdetr-cpp: decoded image is empty"))
+	})
+})

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=be0e34480dada95f8ce9a021bbb95c5de85d67c7
+STABLEDIFFUSION_GGML_VERSION?=d04e8950c1ec8d30248cbe996682b3182fb1adf6
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/stablediffusion-ggml/cpp/gosd.cpp

@@ -401,7 +401,6 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     const char *params_backend_arg = "";
     const char *rpc_servers_arg = "";
     const char *max_vram_arg = "";
-    bool stream_layers = false;
 
     int n_threads = threads;
     enum sd_type_t wtype = SD_TYPE_COUNT;
@@ -510,7 +509,10 @@ int load_model(const char *model, char *model_path, char* options[], int threads
         if (!strcmp(optname, "params_backend")) params_backend_arg = strdup(optval);
         if (!strcmp(optname, "rpc_servers")) rpc_servers_arg = strdup(optval);
         if (!strcmp(optname, "max_vram")) max_vram_arg = strdup(optval);
-        if (!strcmp(optname, "stream_layers")) stream_layers = (strcmp(optval, "true") == 0 || strcmp(optval, "1") == 0);
+        if (!strcmp(optname, "stream_layers")) {
+            // Retained as a no-op for existing configurations. Upstream now
+            // selects segmented weight streaming automatically.
+        }
 
         // vae_decode_only is still accepted for backwards compatibility with
         // existing gallery configs, but upstream dropped the option (the model
@@ -650,11 +652,9 @@ int load_model(const char *model, char *model_path, char* options[], int threads
             ctx_params.rpc_servers = env_rpc_servers;
         }
     }
-    // max_vram: GiB budget or per-backend spec for graph-cut segmented param
-    // offload ("0" = disabled, "-1" = auto). stream_layers only has effect when
-    // max_vram is set.
+    // max_vram is an optional GiB budget or per-backend spec for automatic
+    // graph-cut execution. A zero value uses the live free-VRAM budget.
     if (strlen(max_vram_arg) > 0) ctx_params.max_vram = max_vram_arg;
-    ctx_params.stream_layers = stream_layers;
     ctx_params.diffusion_flash_attn = diffusion_flash_attn;
     ctx_params.tae_preview_only = tae_preview_only;
     ctx_params.diffusion_conv_direct = diffusion_conv_direct;
@@ -1144,17 +1144,25 @@ static uint8_t* load_and_resize_image(const char* path, int target_width, int ta
 // Write sd.cpp's audio buffer to a temp WAV file (IEEE float, interleaved).
 // sd_audio_t.data is planar (all channel 0 samples, then channel 1, etc.) — we
 // interleave on the fly so ffmpeg's standard wav demuxer can read it directly.
-// Returns 0 on success and fills wav_path (must be at least 64 bytes).
+// Returns 0 on success and fills wav_path.
 static int write_planar_float_wav(const sd_audio_t* a, char* wav_path, size_t wav_path_sz) {
     if (!a || !a->data || a->sample_count == 0 || a->channels == 0 || a->sample_rate == 0) {
         return -1;
     }
 
-    snprintf(wav_path, wav_path_sz, "/tmp/gosd-audio-XXXXXX.wav");
+    const char* temp_dir = getenv("TMPDIR");
+    if (!temp_dir || temp_dir[0] == '\0') {
+        temp_dir = "/tmp";
+    }
+    int path_len = snprintf(wav_path, wav_path_sz, "%s/gosd-audio-XXXXXX.wav", temp_dir);
+    if (path_len < 0 || (size_t)path_len >= wav_path_sz) {
+        fprintf(stderr, "temporary directory path is too long\n");
+        return -1;
+    }
     int fd = mkstemps(wav_path, 4);
     if (fd < 0) { perror("mkstemps wav"); return -1; }
     FILE* f = fdopen(fd, "wb");
-    if (!f) { perror("fdopen wav"); close(fd); return -1; }
+    if (!f) { perror("fdopen wav"); close(fd); unlink(wav_path); return -1; }
 
     uint64_t frames = a->sample_count;
     uint32_t channels = a->channels;
@@ -1221,7 +1229,7 @@ static int ffmpeg_mux_raw_to_mp4(sd_image_t* frames, int num_frames, int fps,
     snprintf(fps_str, sizeof(fps_str), "%d", fps);
 
     // Optional audio: write a temp WAV file if the model produced audio.
-    char wav_path[64] = {0};
+    char wav_path[4096] = {0};
     bool have_audio = false;
     if (audio && audio->data && audio->sample_count > 0 && audio->channels > 0 && audio->sample_rate > 0) {
         if (write_planar_float_wav(audio, wav_path, sizeof(wav_path)) == 0) {
@@ -1438,4 +1446,3 @@ int unload() {
     free_sd_ctx(sd_c);
     return 0;
 }
-

backend/go/vllm-cpp/Makefile

@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e
 
 # vllm.cpp version
 VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
-VLLM_CPP_VERSION?=150b37852c123f7855fb219b37347572ca9427e7
+VLLM_CPP_VERSION?=6bf3abb580982f4fd2e4525ef37802ee0ce28981
 
 # MLX GEMM provider (darwin/metal only; see the metal branch below for why).
 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun

backend/go/vllm-cpp/options.go

@@ -128,9 +128,40 @@ func parseOptions(opts *pb.ModelOptions) loadOptions {
 	lo := loadOptions{}
 	applyOptionsList(&lo, opts.GetOptions())
 	applyEngineArgs(&lo, opts.GetEngineArgs())
+	applyDraftModelOption(&lo, opts.GetOptions())
 	return lo
 }
 
+// applyDraftModelOption binds a managed companion snapshot after engine_args
+// has supplied the speculative document. Companion paths do not exist until
+// LocalAI materializes the artifact, so they must replace the gallery's static
+// repository reference without disturbing the method or token budget.
+func applyDraftModelOption(lo *loadOptions, options []string) {
+	if strings.TrimSpace(lo.speculativeConfig) == "" {
+		return
+	}
+	var draftModel string
+	for _, option := range options {
+		key, value, found := strings.Cut(option, ":")
+		if found && strings.TrimSpace(key) == "draft_model" {
+			draftModel = strings.TrimSpace(value)
+		}
+	}
+	if draftModel == "" {
+		return
+	}
+
+	var spec map[string]any
+	if err := json.Unmarshal([]byte(lo.speculativeConfig), &spec); err != nil {
+		return
+	}
+	spec["model"] = draftModel
+	encoded, err := json.Marshal(spec)
+	if err == nil {
+		lo.speculativeConfig = string(encoded)
+	}
+}
+
 // applyOptionsList reads the legacy free-form "key:value" list. strings.Cut
 // splits on the FIRST colon only, so a JSON object value survives intact.
 func applyOptionsList(lo *loadOptions, options []string) {

backend/go/vllm-cpp/options_test.go

@@ -0,0 +1,38 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+var _ = Describe("managed DFlash companion options", func() {
+	It("replaces only the draft model in an existing speculative configuration", func() {
+		managedPath := ".artifacts/huggingface/0123456789abcdef/snapshot"
+		lo := parseOptions(&pb.ModelOptions{
+			Options: []string{"draft_model:" + managedPath},
+			EngineArgs: `{
+				"speculative_config": {
+					"method": "dflash",
+					"model": "Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw",
+					"num_speculative_tokens": 7
+				}
+			}`,
+		})
+
+		Expect(lo.speculativeConfig).To(MatchJSON(`{
+			"method": "dflash",
+			"model": ".artifacts/huggingface/0123456789abcdef/snapshot",
+			"num_speculative_tokens": 7
+		}`))
+	})
+
+	It("ignores a draft companion when speculative decoding is not configured", func() {
+		lo := parseOptions(&pb.ModelOptions{
+			Options: []string{"draft_model:.artifacts/huggingface/0123456789abcdef/snapshot"},
+		})
+
+		Expect(lo.speculativeConfig).To(BeEmpty())
+	})
+})

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=978113305b2ead22249b881deafa131dc8884911
+WHISPER_CPP_VERSION?=c44b60b8053bbf2a5c1e014f11323fb3f2485177
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/python/chatterbox/backend.py

@@ -19,6 +19,7 @@ import grpc
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
+from temp_utils import cleanup_paths
 
 import tempfile
 
@@ -115,11 +116,6 @@ def merge_audio_files(audio_files, output_path, sample_rate):
     # Save the merged audio
     ta.save(output_path, merged_waveform, sample_rate)
     
-    # Clean up temporary files
-    for audio_file in audio_files:
-        if os.path.exists(audio_file):
-            os.remove(audio_file)
-
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24
 
 # If MAX_WORKERS are specified in the environment use it, otherwise default to 1
@@ -226,19 +222,20 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 text_chunks = split_text_at_word_boundary(request.text, max_length=250)
                 print(f"Splitting text into chunks of 250 characters: {len(text_chunks)}", file=sys.stderr)
                 # Generate audio for each chunk
-                temp_audio_files = []
-                for i, chunk in enumerate(text_chunks):
-                    # Generate audio for this chunk
-                    wav = self.model.generate(chunk, **kwargs)
-                    
-                    # Create temporary file for this chunk
-                    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
-                    temp_file.close()
-                    ta.save(temp_file.name, wav, self.model.sr)
-                    temp_audio_files.append(temp_file.name)
-                
-                # Merge all audio files
-                merge_audio_files(temp_audio_files, request.dst, self.model.sr)
+                with cleanup_paths() as temp_audio_files:
+                    for i, chunk in enumerate(text_chunks):
+                        # Generate audio for this chunk
+                        wav = self.model.generate(chunk, **kwargs)
+
+                        # Register ownership before saving so a partial write is
+                        # removed too when generation or encoding fails.
+                        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+                        temp_file.close()
+                        temp_audio_files.append(temp_file.name)
+                        ta.save(temp_file.name, wav, self.model.sr)
+
+                    # Merge all audio files
+                    merge_audio_files(temp_audio_files, request.dst, self.model.sr)
             else:
                 # Generate audio using ChatterboxTTS for short text
                 wav = self.model.generate(request.text, **kwargs)

backend/python/common/python_utils.py

@@ -37,6 +37,46 @@ def parse_options(options_list):
     return opts
 
 
+def attach_media_parts(messages_dicts, n_images=0, n_videos=0):
+    """Rebuild the last user message as content *parts* carrying media markers.
+
+    Backends that let the tokenizer do the templating hand plain string content
+    to ``apply_chat_template``, but a chat template only emits the model's own
+    media tokens (``<|vision_start|><|image_pad|><|vision_end|>`` for the
+    Qwen-VL family, and the equivalents elsewhere) when the content is a list
+    of parts. Without those markers the engine's multimodal processor finds
+    nothing to substitute and silently discards the pixels, even though they
+    were forwarded correctly out of band.
+
+    Returns a new list whose last user message has
+    ``[{"type": "image"} * n_images, {"type": "video"} * n_videos, text]`` as
+    its content, or ``None`` when there is nothing to attach - no media, no
+    user turn, or content that is already a list of parts - so the caller can
+    keep using the original string-content list.
+    """
+    if not n_images and not n_videos:
+        return None
+    idx = next(
+        (
+            i
+            for i in reversed(range(len(messages_dicts)))
+            if messages_dicts[i].get("role") == "user"
+        ),
+        None,
+    )
+    if idx is None:
+        return None
+    text = messages_dicts[idx].get("content") or ""
+    if not isinstance(text, str):
+        return None
+    parts = [{"type": "image"}] * n_images + [{"type": "video"}] * n_videos
+    if text:
+        parts.append({"type": "text", "text": text})
+    patched = list(messages_dicts)
+    patched[idx] = dict(patched[idx], content=parts)
+    return patched
+
+
 def messages_to_dicts(proto_messages):
     """Convert proto ``Message`` objects to dicts suitable for ``apply_chat_template``.
 

backend/python/common/python_utils_test.py

@@ -14,7 +14,7 @@ import json
 import types
 import unittest
 
-from python_utils import messages_to_dicts, parse_options
+from python_utils import attach_media_parts, messages_to_dicts, parse_options
 
 
 def _msg(**fields):
@@ -118,5 +118,63 @@ class TestMessagesToDicts(unittest.TestCase):
         self.assertNotIn("tool_calls", out[0])
 
 
+class TestAttachMediaParts(unittest.TestCase):
+    def test_image_marker_added_to_last_user_turn(self):
+        messages = [
+            {"role": "system", "content": "be brief"},
+            {"role": "user", "content": "first"},
+            {"role": "assistant", "content": "ok"},
+            {"role": "user", "content": "how high is the water?"},
+        ]
+        out = attach_media_parts(messages, n_images=1)
+        self.assertEqual(
+            out[3]["content"],
+            [{"type": "image"}, {"type": "text", "text": "how high is the water?"}],
+        )
+        # Earlier turns and the input list itself are untouched.
+        self.assertEqual(out[:3], messages[:3])
+        self.assertEqual(messages[3]["content"], "how high is the water?")
+
+    def test_counts_and_order_images_then_videos(self):
+        out = attach_media_parts(
+            [{"role": "user", "content": "describe"}], n_images=2, n_videos=1
+        )
+        self.assertEqual(
+            out[0]["content"],
+            [
+                {"type": "image"},
+                {"type": "image"},
+                {"type": "video"},
+                {"type": "text", "text": "describe"},
+            ],
+        )
+
+    def test_empty_text_yields_media_only_parts(self):
+        out = attach_media_parts([{"role": "user", "content": ""}], n_images=1)
+        self.assertEqual(out[0]["content"], [{"type": "image"}])
+
+    def test_other_message_keys_are_preserved(self):
+        out = attach_media_parts(
+            [{"role": "user", "content": "hi", "name": "bob"}], n_images=1
+        )
+        self.assertEqual(out[0]["name"], "bob")
+
+    def test_no_media_is_a_no_op(self):
+        self.assertIsNone(attach_media_parts([{"role": "user", "content": "hi"}]))
+
+    def test_no_user_turn_is_a_no_op(self):
+        self.assertIsNone(
+            attach_media_parts([{"role": "system", "content": "hi"}], n_images=1)
+        )
+
+    def test_content_already_parts_is_a_no_op(self):
+        self.assertIsNone(
+            attach_media_parts(
+                [{"role": "user", "content": [{"type": "text", "text": "hi"}]}],
+                n_images=1,
+            )
+        )
+
+
 if __name__ == "__main__":
     unittest.main()

backend/python/common/temp_utils.py

@@ -0,0 +1,36 @@
+import base64
+import contextlib
+import os
+import tempfile
+
+
+@contextlib.contextmanager
+def materialize_base64(data, suffix=""):
+    """Materialize base64 data for a path-only library and always remove it."""
+    descriptor, path = tempfile.mkstemp(prefix="localai-media-", suffix=suffix)
+    try:
+        with os.fdopen(descriptor, "wb") as output:
+            descriptor = None
+            output.write(base64.b64decode(data))
+        yield path
+    finally:
+        if descriptor is not None:
+            os.close(descriptor)
+        try:
+            os.remove(path)
+        except OSError:
+            pass
+
+
+@contextlib.contextmanager
+def cleanup_paths():
+    """Collect temporary paths and remove them on success or failure."""
+    paths = []
+    try:
+        yield paths
+    finally:
+        for path in paths:
+            try:
+                os.remove(path)
+            except OSError:
+                pass

backend/python/common/temp_utils_test.py

@@ -0,0 +1,41 @@
+import os
+import tempfile
+import unittest
+from unittest import mock
+
+from temp_utils import cleanup_paths, materialize_base64
+
+
+class MaterializeBase64Test(unittest.TestCase):
+    def test_removes_materialized_file_after_success(self):
+        with tempfile.TemporaryDirectory() as directory:
+            with mock.patch.object(tempfile, "tempdir", directory):
+                with materialize_base64("aGVsbG8=", suffix=".data") as path:
+                    with open(path, "rb") as materialized:
+                        self.assertEqual(materialized.read(), b"hello")
+                self.assertFalse(os.path.exists(path))
+
+    def test_removes_materialized_file_when_consumer_fails(self):
+        with tempfile.TemporaryDirectory() as directory:
+            with mock.patch.object(tempfile, "tempdir", directory):
+                with self.assertRaisesRegex(RuntimeError, "decode failed"):
+                    with materialize_base64("aGVsbG8="):
+                        raise RuntimeError("decode failed")
+                self.assertEqual(os.listdir(directory), [])
+
+
+class CleanupPathsTest(unittest.TestCase):
+    def test_removes_every_registered_path_after_failure(self):
+        with tempfile.TemporaryDirectory() as directory:
+            paths = [os.path.join(directory, name) for name in ("one.wav", "two.wav")]
+            with self.assertRaisesRegex(RuntimeError, "merge failed"):
+                with cleanup_paths() as registered:
+                    for path in paths:
+                        open(path, "wb").close()
+                        registered.append(path)
+                    raise RuntimeError("merge failed")
+            self.assertEqual(os.listdir(directory), [])
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/common/template/requirements-intel.txt

@@ -1,4 +1,4 @@
 --extra-index-url https://download.pytorch.org/whl/xpu
-torch==2.13.0+xpu
+torch==2.14.0+xpu
 oneccl_bind_pt==2.8.0+xpu
 optimum[openvino]

backend/python/common/template/requirements.txt

@@ -1,3 +1,3 @@
-grpcio==1.82.1
+grpcio==1.83.1
 protobuf
 grpcio-tools

backend/python/coqui/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.0
+grpcio==1.83.1
 protobuf
 certifi
 packaging==26.3

backend/python/diffusers/backend.py

@@ -122,6 +122,21 @@ from diffusers.schedulers import (
     UniPCMultistepScheduler,
 )
 
+def select_device(request_cuda, device_option, cuda_available, xpu, mps_available):
+    """Pick the pipeline device. An explicit `device:` model option wins;
+    otherwise CUDA is used whenever torch reports it available (ROCm
+    builds included) or the model config forces it with `cuda: true`,
+    keeping the pre-existing XPU/MPS overrides. CPU is the fallback, not
+    the default."""
+    if device_option:
+        return device_option
+    device = "cuda" if (request_cuda or cuda_available) else "cpu"
+    if xpu:
+        device = "xpu"
+    if mps_available:
+        device = "mps"
+    return device
+
 def is_float(s):
     """Check if a string can be converted to float."""
     try:
@@ -627,12 +642,13 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 # modify LoraAdapter to be relative to modelFileBase
                 request.LoraAdapter = os.path.join(request.ModelPath, request.LoraAdapter)
 
-            device = "cpu" if not request.CUDA else "cuda"
-            if XPU:
-                device = "xpu"
-            mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
-            if mps_available:
-                device = "mps"
+            device = select_device(
+                request.CUDA,
+                self.options.pop("device", None),
+                torch.cuda.is_available(),
+                XPU,
+                hasattr(torch.backends, "mps") and torch.backends.mps.is_available(),
+            )
             self.device = device
             if request.LoraAdapter:
                 # Check if its a local file and not a directory ( we load lora differently for a safetensor file )
@@ -800,12 +816,12 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             image = image.resize((1024, 576))
 
             generator = torch.manual_seed(request.seed)
-            frames = self.pipe(image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0]
+            frames = self.pipe(image=image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0]
             export_to_video(frames, request.dst, fps=FPS)
             return backend_pb2.Result(message="Media generated successfully", success=True)
 
         if self.txt2vid:
-            video_frames = self.pipe(prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames
+            video_frames = self.pipe(prompt=prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames
             export_to_video(video_frames, request.dst)
             return backend_pb2.Result(message="Media generated successfully", success=True)
 
@@ -868,7 +884,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         else:
             # pass the kwargs dictionary to the self.pipe method
             image = self.pipe(
-                prompt,
+                prompt=prompt,
                 guidance_scale=self.cfg_scale,
                 **kwargs
             ).images[0]

backend/python/diffusers/test.py

@@ -7,6 +7,7 @@ import time
 from unittest.mock import patch, MagicMock
 
 # Import dynamic loader for testing (these don't need gRPC)
+import backend
 import diffusers_dynamic_loader as loader
 from diffusers import DiffusionPipeline, StableDiffusionPipeline
 
@@ -373,3 +374,74 @@ class TestGenerateImageOptionsKwargsMerge(unittest.TestCase):
         finally:
             os.unlink(src_file.name)
             os.unlink(dst_file.name)
+
+    def test_text_to_image_prompt_is_passed_by_keyword(self):
+        """Test compatibility with pipelines that take image before prompt."""
+        import os
+        import tempfile
+
+        from PIL import Image
+
+        from backend import BackendServicer
+
+        class Flux2CompatiblePipeline:
+            """Model the FLUX.2 call signature: image is before prompt."""
+
+            def __call__(self, image=None, prompt=None, **kwargs):
+                if prompt is None:
+                    raise ValueError("prompt was not passed by keyword")
+                self.prompt = prompt
+                self.kwargs = kwargs
+                return MagicMock(images=[Image.new("RGB", (4, 4))])
+
+        pipeline = Flux2CompatiblePipeline()
+        svc = BackendServicer.__new__(BackendServicer)
+        svc.pipe = pipeline
+        svc.cfg_scale = 7.5
+        svc.controlnet = None
+        svc.img2vid = False
+        svc.txt2vid = False
+        svc.clip_skip = 0
+        svc.PipelineType = "Flux2KleinPipeline"
+        svc.options = {}
+
+        with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as dst_file:
+            dst_path = dst_file.name
+
+        try:
+            request = MagicMock()
+            request.positive_prompt = "a red apple on a wooden table"
+            request.negative_prompt = ""
+            request.step = 4
+            request.seed = 0
+            request.width = 0
+            request.height = 0
+            request.src = ""
+            request.ref_images = []
+            request.dst = dst_path
+
+            svc.GenerateImage(request, context=None)
+
+            self.assertEqual(pipeline.prompt, request.positive_prompt)
+            self.assertEqual(pipeline.kwargs["num_inference_steps"], 4)
+        finally:
+            os.unlink(dst_path)
+
+
+class TestDeviceSelection(unittest.TestCase):
+    """Unit tests for backend.select_device (no GPU required)."""
+
+    def test_autodetect_cuda(self):
+        self.assertEqual(backend.select_device(False, None, True, False, False), "cuda")
+
+    def test_cpu_fallback(self):
+        self.assertEqual(backend.select_device(False, None, False, False, False), "cpu")
+
+    def test_forced_cuda(self):
+        self.assertEqual(backend.select_device(True, None, False, False, False), "cuda")
+
+    def test_device_option_wins(self):
+        self.assertEqual(backend.select_device(True, "cpu", True, True, True), "cpu")
+
+    def test_mps_overrides(self):
+        self.assertEqual(backend.select_device(False, None, True, False, True), "mps")

backend/python/longcat-video/README.md

@@ -11,7 +11,7 @@ RPC. It supports:
   systems such as NVIDIA DGX Spark.
 
 Install the `longcat-video` or `longcat-video-avatar-1.5` recipe from the
-LocalAI Model Gallery. See the [LongCat user guide](../../../docs/content/features/longcat-video.md)
+LocalAI Model Gallery. LongCat video backend
 for Studio and API examples, hardware requirements, and manual configuration.
 
 The upstream source is pinned in `Makefile` and patched at build time. The

backend/python/longcat-video/backend.py

@@ -6,6 +6,7 @@ import datetime
 import gc
 import math
 import os
+import shutil
 import signal
 import subprocess
 import sys
@@ -888,6 +889,13 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
     def _release_model(self):
         self.pipeline = None
         self.model_kind = None
+        try:
+            if hasattr(self, "dist") and self.dist.is_initialized():
+                self.dist.destroy_process_group()
+        finally:
+            if self._dist_store_dir is not None:
+                shutil.rmtree(self._dist_store_dir, ignore_errors=True)
+                self._dist_store_dir = None
         gc.collect()
         if hasattr(self, "torch") and self.torch.cuda.is_available():
             self.torch.cuda.empty_cache()

backend/python/qwen-asr/backend.py

@@ -18,6 +18,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
 from model_utils import resolve_model_reference
+from device_utils import device_map_for, select_device
 
 
 
@@ -95,13 +96,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         return backend_pb2.Reply(message=bytes("OK", 'utf-8'))
 
     def LoadModel(self, request, context):
-        if torch.cuda.is_available():
-            device = "cuda"
-        else:
-            device = "cpu"
-        mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
-        if mps_available:
-            device = "mps"
+        device = select_device(torch)
         if not torch.cuda.is_available() and request.CUDA:
             return backend_pb2.Result(success=False, message="CUDA is not available")
 
@@ -123,7 +118,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         model_path, local_only = resolve_model_reference(
             request, "Qwen/Qwen3-ASR-1.7B"
         )
-        default_dtype = torch.bfloat16 if self.device == "cuda" else torch.float32
+        default_dtype = torch.bfloat16 if self.device in ("cuda", "xpu") else torch.float32
         load_dtype = default_dtype
         if "torch_dtype" in self.options:
             d = str(self.options["torch_dtype"]).lower()
@@ -145,12 +140,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         if attn_implementation is not None and isinstance(attn_implementation, str):
             attn_implementation = attn_implementation.strip() or None
 
-        if self.device == "mps":
-            device_map = None
-        elif self.device == "cuda":
-            device_map = "cuda:0"
-        else:
-            device_map = "cpu"
+        device_map = device_map_for(self.device)
 
         load_kwargs = dict(
             dtype=load_dtype,
@@ -423,4 +413,4 @@ if __name__ == "__main__":
     parser = argparse.ArgumentParser(description="Run the gRPC server.")
     parser.add_argument("--addr", default="localhost:50051", help="The address to bind the server to.")
     args = parser.parse_args()
-    serve(args.addr)
+    serve(args.addr)

backend/python/qwen-asr/device_utils.py

@@ -0,0 +1,18 @@
+def select_device(torch_module):
+    mps = getattr(getattr(torch_module, "backends", None), "mps", None)
+    if mps is not None and mps.is_available():
+        return "mps"
+    if torch_module.cuda.is_available():
+        return "cuda"
+    xpu = getattr(torch_module, "xpu", None)
+    if xpu is not None and xpu.is_available():
+        return "xpu"
+    return "cpu"
+
+
+def device_map_for(device):
+    if device == "mps":
+        return None
+    if device in ("cuda", "xpu"):
+        return f"{device}:0"
+    return "cpu"

backend/python/qwen-asr/device_utils_test.py

@@ -0,0 +1,58 @@
+import unittest
+
+from device_utils import device_map_for, select_device
+
+
+class Availability:
+    def __init__(self, available):
+        self._available = available
+
+    def is_available(self):
+        return self._available
+
+
+class TorchStub:
+    def __init__(self, *, cuda=False, mps=False, xpu=False):
+        self.cuda = Availability(cuda)
+        self.backends = type("Backends", (), {"mps": Availability(mps)})()
+        self.xpu = Availability(xpu)
+
+
+class SelectDeviceTest(unittest.TestCase):
+    def test_preserves_cuda_selection(self):
+        torch_module = TorchStub(cuda=True)
+
+        self.assertEqual(select_device(torch_module), "cuda")
+
+    def test_preserves_mps_selection(self):
+        torch_module = TorchStub(mps=True)
+
+        self.assertEqual(select_device(torch_module), "mps")
+
+    def test_selects_xpu_when_intel_gpu_is_available(self):
+        torch_module = TorchStub(xpu=True)
+
+        self.assertEqual(select_device(torch_module), "xpu")
+
+    def test_falls_back_to_cpu(self):
+        torch_module = TorchStub()
+
+        self.assertEqual(select_device(torch_module), "cpu")
+
+
+class DeviceMapTest(unittest.TestCase):
+    def test_preserves_cuda_model_placement(self):
+        self.assertEqual(device_map_for("cuda"), "cuda:0")
+
+    def test_preserves_mps_model_placement(self):
+        self.assertIsNone(device_map_for("mps"))
+
+    def test_places_the_model_on_the_first_xpu(self):
+        self.assertEqual(device_map_for("xpu"), "xpu:0")
+
+    def test_preserves_cpu_model_placement(self):
+        self.assertEqual(device_map_for("cpu"), "cpu")
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/rerankers/requirements.txt

@@ -1,3 +1,3 @@
-grpcio==1.82.1
+grpcio==1.83.1
 protobuf
 certifi

backend/python/sglang/backend.py

@@ -40,6 +40,7 @@ import grpc
 
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
+from python_utils import attach_media_parts
 from grpc_auth import get_auth_interceptors
 from model_utils import resolve_model_reference
 
@@ -90,6 +91,14 @@ except Exception:
 
 
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24
+
+# proto3 has no field presence, so an explicit 0 is indistinguishable from
+# "unset" and the zero-filter below would drop it. These two fields have a
+# meaningful zero a caller can actually intend: temperature 0 is greedy
+# decoding, and 0 is a valid seed. Silently substituting a default for either
+# turns a reproducible request into a random one.
+_EXPLICIT_ZERO_FIELDS = ("Temperature", "Seed")
+
 MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1'))
 
 
@@ -323,7 +332,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             if not hasattr(request, proto_field):
                 continue
             value = getattr(request, proto_field)
-            if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
+            if proto_field not in _EXPLICIT_ZERO_FIELDS and value in (None, 0, 0.0, [], False, ""):
                 continue
             # repeated fields come back as RepeatedScalarContainer — convert
             if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)):
@@ -367,6 +376,24 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         if _thinking in ("true", "false"):
             template_kwargs["enable_thinking"] = (_thinking == "true")
 
+        # sglang locates the attached images/videos by scanning the rendered
+        # prompt for the model's own media token, so the template has to be
+        # given content *parts* - string content renders a prompt with no
+        # placeholder and the media are dropped without a word (#11621).
+        media_dicts = attach_media_parts(
+            messages_dicts, len(request.Images), len(request.Videos)
+        )
+        if media_dicts is not None:
+            try:
+                return self.tokenizer.apply_chat_template(media_dicts, **template_kwargs)
+            except Exception as e:
+                # A text-only template cannot iterate content parts; fall
+                # through to the text-only prompt instead of failing.
+                print(
+                    f"chat template rejected multimodal content parts: {e!r}",
+                    file=sys.stderr,
+                )
+
         try:
             return self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs)
         except TypeError:
@@ -374,10 +401,67 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 messages_dicts, tokenize=False, add_generation_prompt=True,
             )
 
-    def _make_parsers(self, request):
+    def _new_reasoning_parser(self, stream_reasoning: bool, prompt: str = "",
+                              grammar_constrained: bool = False):
+        """Build a ReasoningParser for one request, or None.
+
+        Reasoning templates come in two flavours. Some let the model emit the
+        opening tag, others put it into the *prompt* — Qwen3's template appends
+        ``<think>`` when thinking is on, so the completion starts straight in
+        the reasoning block and only the closing ``</think>`` ever shows up.
+        sglang's detector keys off the opening tag, so in that second case it
+        classifies the whole completion as normal content and
+        ``reasoning_content`` stays empty.
+
+        sglang's own OpenAI server covers this with
+        ``template_manager.force_reasoning``; this backend has no template
+        manager, so it derives the same signal from the rendered prompt.
+        ``force_reasoning`` is only passed when we mean True, leaving detector
+        defaults (e.g. DeepSeek-R1's built-in True) untouched.
+
+        ``grammar_constrained`` suppresses the prefill heuristic. A structured
+        decoding constraint applies from the first token, so the model cannot
+        emit the closing tag even though the template opened the block: the
+        whole completion is schema output and belongs in ``content``. Forcing
+        there files the answer as reasoning and leaves content empty. sglang's
+        own server keeps the two apart for the same reason — its grammar
+        backend owns the reasoning prefix when a reasoning parser is set.
+        """
+        if grammar_constrained:
+            prompt = ""
+
+        if not (HAS_REASONING_PARSERS and self.reasoning_parser_name):
+            return None
+
+        kwargs = {
+            "model_type": self.reasoning_parser_name,
+            "stream_reasoning": stream_reasoning,
+        }
+        try:
+            parser = ReasoningParser(**kwargs)
+        except Exception as e:
+            print(f"ReasoningParser init failed: {e!r}", file=sys.stderr)
+            return None
+
+        start = getattr(getattr(parser, "detector", None), "think_start_token", None)
+        if start and prompt and prompt.rstrip().endswith(start):
+            try:
+                parser = ReasoningParser(force_reasoning=True, **kwargs)
+            except TypeError:
+                # sglang without the force_reasoning kwarg: keep the default
+                # parser rather than failing the request.
+                pass
+            except Exception as e:
+                print(
+                    f"ReasoningParser(force_reasoning=True) failed: {e!r}",
+                    file=sys.stderr,
+                )
+
+        return parser
+
+    def _make_parsers(self, request, prompt: str = ""):
         """Construct fresh per-request parser instances (stateful)."""
         tool_parser = None
-        reasoning_parser = None
 
         if HAS_TOOL_PARSERS and self.tool_parser_name and request.Tools:
             try:
@@ -389,14 +473,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             except Exception as e:
                 print(f"FunctionCallParser init failed: {e!r}", file=sys.stderr)
 
-        if HAS_REASONING_PARSERS and self.reasoning_parser_name:
-            try:
-                reasoning_parser = ReasoningParser(
-                    model_type=self.reasoning_parser_name,
-                    stream_reasoning=True,
-                )
-            except Exception as e:
-                print(f"ReasoningParser init failed: {e!r}", file=sys.stderr)
+        reasoning_parser = self._new_reasoning_parser(
+            True, prompt, bool(getattr(request, "Grammar", "")),
+        )
 
         return tool_parser, reasoning_parser
 
@@ -404,7 +483,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         sampling_params = self._build_sampling_params(request)
         prompt = self._build_prompt(request)
 
-        tool_parser, reasoning_parser = self._make_parsers(request)
+        tool_parser, reasoning_parser = self._make_parsers(request, prompt)
 
         image_data = list(request.Images) if request.Images else None
         video_data = list(request.Videos) if request.Videos else None
@@ -500,15 +579,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         final_tool_calls: List[backend_pb2.ToolCallDelta] = []
 
         if not streaming:
-            final_reasoning_parser = None
-            if HAS_REASONING_PARSERS and self.reasoning_parser_name:
-                try:
-                    final_reasoning_parser = ReasoningParser(
-                        model_type=self.reasoning_parser_name,
-                        stream_reasoning=False,
-                    )
-                except Exception:
-                    final_reasoning_parser = None
+            final_reasoning_parser = self._new_reasoning_parser(
+                False, prompt, bool(getattr(request, "Grammar", "")),
+            )
 
             if final_reasoning_parser is not None:
                 try:

backend/python/sglang/test.py

@@ -128,11 +128,66 @@ class TestSglangHelpers(unittest.TestCase):
         self.assertNotIn("enable_thinking", kwargs_for({}))
         self.assertIs(kwargs_for({"enable_thinking": "FALSE"})["enable_thinking"], False)
 
-    def test_explicit_zero_temperature_is_preserved(self):
-        """Temperature=0 is valid greedy decoding, not an unset value."""
+    def test_reasoning_parser_forced_when_template_prefills_think_tag(self):
+        """Qwen3's template puts ``<think>`` in the prompt, so the completion
+        never contains it. Without force_reasoning the detector treats the whole
+        completion as normal text and reasoning_content stays empty."""
+        servicer = self._servicer()
+        servicer.reasoning_parser_name = "qwen3"
+
+        # What the model actually emits when the prompt ends in "<think>".
+        completion = "adding two and two</think>4"
+
+        forced = servicer._new_reasoning_parser(False, prompt="user: hi\n<think>\n")
+        reasoning, content = forced.parse_non_stream(completion)
+        self.assertEqual(reasoning, "adding two and two")
+        self.assertEqual(content, "4")
+
+        # No prefilled tag in the prompt: detector default, unchanged behaviour.
+        unforced = servicer._new_reasoning_parser(False, prompt="user: hi\n")
+        reasoning, content = unforced.parse_non_stream(completion)
+        self.assertFalse(reasoning)
+        self.assertEqual(content, completion)
+
+    def test_reasoning_parser_not_forced_when_thinking_is_off(self):
+        """Thinking off means no ``<think>`` in the prompt either, so the answer
+        must not be swallowed into reasoning_content."""
+        servicer = self._servicer()
+        servicer.reasoning_parser_name = "qwen3"
+
+        parser = servicer._new_reasoning_parser(False, prompt="user: primes?\n")
+        reasoning, content = parser.parse_non_stream("2,3,5,7,11")
+        self.assertFalse(reasoning)
+        self.assertEqual(content, "2,3,5,7,11")
+
+    def test_grammar_constrained_output_is_not_forced_into_reasoning(self):
+        """Structured decoding applies from the first token, so the model cannot
+        emit the closing tag even though the template opened the block. The whole
+        completion is schema output and must stay in content."""
+        servicer = self._servicer()
+        servicer.reasoning_parser_name = "qwen3"
+
+        schema_out = '{"findings": [{"line": 42, "issue": "off-by-one"}]}'
+        parser = servicer._new_reasoning_parser(
+            False, prompt="audit this\n<think>\n", grammar_constrained=True,
+        )
+        reasoning, content = parser.parse_non_stream(schema_out)
+        self.assertFalse(reasoning)
+        self.assertEqual(content, schema_out)
+
+    def test_reasoning_parser_absent_without_configured_parser(self):
+        servicer = self._servicer()
+        servicer.reasoning_parser_name = None
+        self.assertIsNone(servicer._new_reasoning_parser(False, prompt="<think>"))
+
+    def test_explicit_zero_temperature_and_seed_are_preserved(self):
+        """Temperature=0 is greedy decoding and 0 is a valid seed — neither is
+        an unset value. A dropped seed turns a reproducible request random."""
         from types import SimpleNamespace
 
         servicer = self._servicer()
+        import sys as _sys
+        _SEED_KEY_FOR_TEST = _sys.modules["backend"]._SEED_KEY
         request = SimpleNamespace(
             Temperature=0,
             N=0,
@@ -154,8 +209,12 @@ class TestSglangHelpers(unittest.TestCase):
 
         params = servicer._build_sampling_params(request)
         self.assertEqual(params["temperature"], 0)
-        # Other protobuf-default scalar fields must remain filtered.
+        self.assertEqual(params[_SEED_KEY_FOR_TEST], 0)
+        # Other protobuf-default scalar fields must remain filtered. top_k=0 in
+        # particular is not a value sglang accepts (-1 disables it), so it must
+        # keep falling through to the engine default.
         self.assertNotIn("top_p", params)
+        self.assertNotIn("top_k", params)
 
 
 if __name__ == "__main__":

backend/python/vllm-omni/backend.py

@@ -19,7 +19,6 @@ import base64
 import io
 import json
 import gc
-import tempfile
 
 from PIL import Image
 import torch
@@ -34,6 +33,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
 from model_utils import resolve_model_reference
+from temp_utils import materialize_base64
 from vllm_utils import parse_options, messages_to_dicts, setup_parsers
 
 
@@ -118,13 +118,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             return video_to_ndarrays(video_path, num_frames=16)
         # Try base64 decode
         try:
-            timestamp = str(int(time.time() * 1000))
-            p = os.path.join(tempfile.gettempdir(), f"vl-{timestamp}.data")
-            with open(p, "wb") as f:
-                f.write(base64.b64decode(video_path))
-            video = VideoAsset(name=p).np_ndarrays
-            os.remove(p)
-            return video
+            with materialize_base64(video_path, suffix=".data") as path:
+                return VideoAsset(name=path).np_ndarrays
         except:
             return None
 
@@ -136,15 +131,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             return (audio_signal.astype(np.float32), sr)
         # Try base64 decode
         try:
-            audio_data = base64.b64decode(audio_path)
-            # Save to temp file and load
-            timestamp = str(int(time.time() * 1000))
-            p = os.path.join(tempfile.gettempdir(), f"audio-{timestamp}.wav")
-            with open(p, "wb") as f:
-                f.write(audio_data)
-            audio_signal, sr = librosa.load(p, sr=16000)
-            os.remove(p)
-            return (audio_signal.astype(np.float32), sr)
+            with materialize_base64(audio_path, suffix=".wav") as path:
+                audio_signal, sr = librosa.load(path, sr=16000)
+                return (audio_signal.astype(np.float32), sr)
         except:
             return None
 

backend/python/vllm/backend.py

@@ -10,7 +10,6 @@ import os
 import json
 import time
 import gc
-import tempfile
 from typing import List
 from PIL import Image
 
@@ -20,8 +19,10 @@ import backend_pb2_grpc
 import grpc
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
+from python_utils import attach_media_parts
 from grpc_auth import get_auth_interceptors
 from model_utils import resolve_model_reference
+from temp_utils import materialize_base64
 from vllm_utils import apply_options_to_engine_args, normalize_option_key
 
 from vllm.engine.arg_utils import AsyncEngineArgs
@@ -60,6 +61,12 @@ except ImportError:
 
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24
 
+# proto3 has no field presence, so an explicit 0 is indistinguishable from
+# "unset". These two fields have a meaningful zero a caller can intend:
+# temperature 0 is greedy decoding, and 0 is a valid seed.
+_EXPLICIT_ZERO_FIELDS = ("Temperature", "Seed")
+
+
 # If MAX_WORKERS are specified in the environment use it, otherwise default to 1
 MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1'))
 
@@ -553,11 +560,80 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         for request_field, param_field in request_to_sampling_params.items():
             if hasattr(request, request_field):
                 value = getattr(request, request_field)
-                if request_field == "Temperature" or value not in (None, 0, [], False):
+                # See _EXPLICIT_ZERO_FIELDS: temperature 0 is greedy decoding
+                # and 0 is a valid seed, so neither may be filtered out.
+                if request_field in _EXPLICIT_ZERO_FIELDS or value not in (None, 0, [], False):
                     setattr(sampling_params, param_field, value)
 
         return sampling_params
 
+    def _new_reasoning_parser(self, chat_template_kwargs):
+        """Build the reasoning parser, telling it whether thinking is on.
+
+        vLLM's newer parser engines decide their *initial state* from
+        ``chat_template_kwargs``: ``Qwen3Parser`` reads
+        ``chat_template_kwargs["enable_thinking"]`` and defaults to ``True``,
+        starting in the REASONING state. Constructed without it, a completion
+        produced with thinking disabled is classified as reasoning end to end,
+        and the answer is reported in both ``reasoning_content`` and
+        ``content``.
+
+        vLLM's own OpenAI server forwards the request's chat template kwargs
+        here; this backend renders the template itself, so it forwards the
+        same dict. Older parsers do not accept the argument — fall back to the
+        plain constructor for those.
+        """
+        try:
+            return self.reasoning_parser_cls(
+                self.tokenizer, chat_template_kwargs=chat_template_kwargs or {},
+            )
+        except TypeError:
+            return self.reasoning_parser_cls(self.tokenizer)
+
+    @staticmethod
+    def _split_reasoning(rp, generated_text, prompt, reasoning, content):
+        """Decide what the reasoning parser's output actually means.
+
+        Covers the *older* parser shape, which has no initial state to set:
+        ``BaseThinkingReasoningParser.extract_reasoning`` documents its own
+        fallback — "For models that may not generate start token, assume the
+        reasoning content is always at the start." When no end token is
+        present it returns *everything* as reasoning and ``None`` as content,
+        which is right for a truncated reasoning run and wrong for a
+        completion that never contained reasoning at all.
+
+        Taking ``None`` content to mean "keep the raw text" then duplicates
+        the answer into both fields.
+
+        The prompt says which case it is. A template with thinking on leaves
+        the reasoning block open (the prompt ends with the start token); with
+        thinking off it closes the block in the prompt, so the completion is
+        plain content. Parsers that expose no token pair (the engine-based
+        adapters, which take the ``chat_template_kwargs`` route above) keep
+        the parser's verdict unchanged.
+        """
+        start = getattr(rp, "start_token", None)
+        end = getattr(rp, "end_token", None)
+
+        if end and end in generated_text:
+            # The parser split on the end token. Empty content here means the
+            # model stopped right after it, not that parsing failed.
+            return reasoning or "", content or ""
+
+        if not start:
+            # Unknown token layout — keep the previous behaviour rather than
+            # guess.
+            return reasoning or "", content if content is not None else generated_text
+
+        if not (start in generated_text or (prompt or "").rstrip().endswith(start)):
+            # No end token and the block was never open: the "reasoning starts
+            # at the beginning" fallback does not apply to this completion.
+            return "", generated_text
+
+        # Block was open and the end token never arrived — reasoning ran out of
+        # budget. It is all reasoning, and there is no answer to report.
+        return reasoning or "", content or ""
+
     async def _predict(self, request, context, streaming=False):
         # Build the sampling parameters
         sampling_params = self._build_sampling_params(request)
@@ -572,6 +648,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
         # Extract image paths and process images
         prompt = request.Prompt
+        # Kept in scope: the reasoning parser needs to know which chat
+        # template kwargs produced this prompt.
+        template_kwargs = {}
 
         image_paths = request.Images
         image_data = [self.load_image(img_path) for img_path in image_paths]
@@ -582,7 +661,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         # If tokenizer template is enabled and messages are provided instead of prompt, apply the tokenizer template
         if not request.Prompt and request.UseTokenizerTemplate and request.Messages:
             messages_dicts = self._messages_to_dicts(request.Messages)
-            template_kwargs = {"tokenize": False, "add_generation_prompt": True}
+            template_kwargs.update({"tokenize": False, "add_generation_prompt": True})
 
             # Pass tools for tool calling
             if request.Tools:
@@ -595,13 +674,33 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             if _thinking in ("true", "false"):
                 template_kwargs["enable_thinking"] = (_thinking == "true")
 
-            try:
-                prompt = self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs)
-            except TypeError:
-                # Some tokenizers don't support tools/enable_thinking kwargs — retry without them
-                prompt = self.tokenizer.apply_chat_template(
-                    messages_dicts, tokenize=False, add_generation_prompt=True
-                )
+            # vLLM substitutes multi_modal_data into the model's own media
+            # token, so the template has to be given content *parts* - string
+            # content renders a prompt with no placeholder and the media are
+            # dropped without a word (#11621).
+            prompt = None
+            media_dicts = attach_media_parts(
+                messages_dicts, len(image_data), len(video_data)
+            )
+            if media_dicts is not None:
+                try:
+                    prompt = self.tokenizer.apply_chat_template(media_dicts, **template_kwargs)
+                except Exception as e:
+                    # A text-only template cannot iterate content parts; fall
+                    # through to the text-only prompt instead of failing.
+                    print(
+                        f"chat template rejected multimodal content parts: {e!r}",
+                        file=sys.stderr,
+                    )
+
+            if prompt is None:
+                try:
+                    prompt = self.tokenizer.apply_chat_template(messages_dicts, **template_kwargs)
+                except TypeError:
+                    # Some tokenizers don't support tools/enable_thinking kwargs — retry without them
+                    prompt = self.tokenizer.apply_chat_template(
+                        messages_dicts, tokenize=False, add_generation_prompt=True
+                    )
 
         # Generate text using the LLM engine
         request_id = random_uuid()
@@ -757,10 +856,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
         if self.reasoning_parser_cls:
             try:
-                rp = self.reasoning_parser_cls(self.tokenizer)
+                rp = self._new_reasoning_parser(template_kwargs)
                 r, c = rp.extract_reasoning(generated_text, request=None)
-                reasoning_content = r or ""
-                content = c if c is not None else generated_text
+                reasoning_content, content = self._split_reasoning(
+                    rp, generated_text, prompt, r, c,
+                )
             except Exception as e:
                 print(f"Reasoning parser error: {e}", file=sys.stderr)
 
@@ -905,13 +1005,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             Video: The loaded video.
         """
         try:
-            timestamp = str(int(time.time() * 1000))  # Generate timestamp
-            p = os.path.join(tempfile.gettempdir(), f"vl-{timestamp}.data")
-            with open(p, "wb") as f:
-                f.write(base64.b64decode(video_path))
-            video = VideoAsset(name=p).np_ndarrays
-            os.remove(p)
-            return video
+            with materialize_base64(video_path, suffix=".data") as path:
+                return VideoAsset(name=path).np_ndarrays
         except Exception as e:
             print(f"Error loading video {video_path}: {e}", file=sys.stderr)
             return None

backend/python/vllm/install.sh

@@ -119,14 +119,18 @@ if [ "$(uname -s)" = "Darwin" ]; then
     # can rewrite it. Darwin therefore follows vllm-metal and can lag the Linux
     # vllm pin (requirements-cublas13-after.txt, bumped independently against
     # vllm/vllm) until vllm-metal supports a newer vLLM.
-    VLLM_METAL_VERSION="v0.3.0.dev20260818075955"
+    VLLM_METAL_VERSION="v0.28.0"
 
     # The coupled vLLM source version is whatever this vllm-metal release builds
-    # against. Derive it from
-    # the PINNED tag rather than hardcoding a second value that could drift. The
-    # tag is immutable, so this stays reproducible across rebuilds.
-    VLLM_VERSION=$(curl -fsSL "https://raw.githubusercontent.com/vllm-project/vllm-metal/${VLLM_METAL_VERSION}/install.sh" \
-        | "$backend_dir/../../../scripts/lib/extract-vllm-metal-version.sh")
+    # against. Derive it from the PINNED tag rather than hardcoding a second value
+    # that could drift. The tag is immutable, so this stays reproducible across
+    # rebuilds. Since vllm-metal 0.28 the coupling is declared in
+    # .github/vllm-release-tag.commit; older releases pinned it inline in their
+    # own install.sh, so fall back to that. The extractor reads both forms.
+    _vllm_metal_raw="https://raw.githubusercontent.com/vllm-project/vllm-metal/${VLLM_METAL_VERSION}"
+    VLLM_VERSION=$( { curl -fsSL "${_vllm_metal_raw}/.github/vllm-release-tag.commit" \
+        || curl -fsSL "${_vllm_metal_raw}/install.sh"; } \
+        | "$backend_dir/../../../scripts/lib/extract-vllm-metal-version.sh" || true)
     if [ -z "${VLLM_VERSION}" ]; then
         echo "ERROR: could not derive the vLLM version from vllm-metal ${VLLM_METAL_VERSION}" >&2
         exit 1
@@ -153,10 +157,18 @@ if [ "$(uname -s)" = "Darwin" ]; then
     # 2) Install the prebuilt vllm-metal wheel for the PINNED release. It pulls
     #    mlx / mlx-metal as deps and registers the `metal` platform plugin that
     #    backend.py resolves to at engine-init time. Build the release-asset URL
-    #    deterministically (tag + the cp312/arm64 wheel name) rather than querying
-    #    api.github.com, whose unauthenticated rate limit (60/hr per IP) 403s on
-    #    shared CI runners. The wheel version is the tag without its leading 'v'.
-    _metal_wheel="vllm_metal-${VLLM_METAL_VERSION#v}-cp312-cp312-macosx_11_0_arm64.whl"
+    #    from the release's OWN asset listing rather than composing it from a
+    #    hardcoded platform tag: upstream raised its macOS deployment target
+    #    (macosx_11_0 -> macosx_15_0) and every composed URL started to 404.
+    #    expanded_assets is the plain release page, not api.github.com, whose
+    #    unauthenticated rate limit (60/hr per IP) 403s on shared CI runners.
+    #    The wheel version is the tag without its leading 'v'.
+    _metal_wheel=$(curl -fsSL "https://github.com/vllm-project/vllm-metal/releases/expanded_assets/${VLLM_METAL_VERSION}" \
+        | grep -oE "vllm_metal-${VLLM_METAL_VERSION#v}-cp312-cp312-[A-Za-z0-9_]+\.whl" | head -1 || true)
+    if [ -z "${_metal_wheel}" ]; then
+        echo "ERROR: no cp312 wheel asset on vllm-metal release ${VLLM_METAL_VERSION}" >&2
+        exit 1
+    fi
     _metal_wheel_url="https://github.com/vllm-project/vllm-metal/releases/download/${VLLM_METAL_VERSION}/${_metal_wheel}"
     echo "Installing vllm-metal wheel: ${_metal_wheel_url}"
     uv pip install "${_metal_wheel_url}"

backend/python/vllm/requirements-cublas13-after.txt

@@ -3,8 +3,8 @@
 # on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
 # instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
 # so uv consults this index alongside PyPI.
---extra-index-url https://wheels.vllm.ai/0.28.0/cu130
+--extra-index-url https://wheels.vllm.ai/0.29.0/cu130
 # VERSION COUPLING: darwin/Apple-Silicon builds use vllm-metal (see install.sh),
 # which pins this exact vLLM version. Bumping vllm here means coordinating with a
 # vllm-metal release that supports the new version, or macOS/Metal builds break.
-vllm==0.28.0
+vllm==0.29.0

backend/python/vllm/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.0
+grpcio==1.83.1
 protobuf
 certifi
 setuptools

backend/python/vllm/test.py

@@ -121,16 +121,18 @@ class TestBackendServicer(unittest.TestCase):
         finally:
             self.tearDown()
 
-    def test_explicit_zero_temperature_is_preserved(self):
-        """Temperature=0 is valid greedy decoding, not an unset value."""
+    def test_explicit_zero_temperature_and_seed_are_preserved(self):
+        """Temperature=0 is greedy decoding and 0 is a valid seed — neither is
+        an unset value. A dropped seed turns a reproducible request random."""
         import sys, os
         sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
         from backend import BackendServicer
 
         servicer = BackendServicer()
-        request = backend_pb2.PredictOptions(Prompt="hello", Temperature=0)
+        request = backend_pb2.PredictOptions(Prompt="hello", Temperature=0, Seed=0)
         sampling_params = servicer._build_sampling_params(request)
         self.assertEqual(sampling_params.temperature, 0)
+        self.assertEqual(sampling_params.seed, 0)
         # Other protobuf-default scalar fields must remain filtered.
         self.assertEqual(sampling_params.top_p, 0.9)
 
@@ -549,3 +551,116 @@ class TestStreamingToolParser(unittest.TestCase):
             intermediate, ["Hello ", "world", "!"],
             f"plain streaming changed; got {intermediate!r}",
         )
+
+
+class TestReasoningSplit(unittest.TestCase):
+    """Server-less tests for BackendServicer._split_reasoning.
+
+    vLLM's BaseThinkingReasoningParser returns the whole completion as
+    reasoning and None as content whenever the end token is missing. Taken
+    literally that duplicates a thinking-disabled answer into both fields.
+    """
+
+    class _Parser:
+        start_token = "<think>"
+        end_token = "</think>"
+
+    def _split(self, generated, prompt, reasoning, content):
+        import sys, os
+        sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+        from backend import BackendServicer
+        return BackendServicer._split_reasoning(
+            self._Parser(), generated, prompt, reasoning, content,
+        )
+
+    def test_thinking_off_is_not_duplicated_into_reasoning(self):
+        """No tags anywhere: the answer is content, and only content."""
+        r, c = self._split(
+            "391", "user: 17*23?\n<think>\n\n</think>\n\n",
+            reasoning="391", content=None,
+        )
+        self.assertEqual(r, "")
+        self.assertEqual(c, "391")
+
+    def test_prefilled_start_tag_keeps_truncated_reasoning(self):
+        """Prompt left the block open and the end token never arrived
+        (budget exhausted): that really is all reasoning."""
+        r, c = self._split(
+            "thinking and thinking", "user: hi\n<think>\n",
+            reasoning="thinking and thinking", content=None,
+        )
+        self.assertEqual(r, "thinking and thinking")
+        self.assertEqual(c, "")
+
+    def test_end_token_present_keeps_parser_split(self):
+        r, c = self._split(
+            "adding two and two</think>4", "user: hi\n<think>\n",
+            reasoning="adding two and two", content="4",
+        )
+        self.assertEqual(r, "adding two and two")
+        self.assertEqual(c, "4")
+
+    def test_stop_right_after_end_token_yields_empty_content(self):
+        """Content must not fall back to the raw text — that would put the
+        reasoning into the answer."""
+        r, c = self._split(
+            "reasoned</think>", "user: hi\n<think>\n",
+            reasoning="reasoned", content=None,
+        )
+        self.assertEqual(r, "reasoned")
+        self.assertEqual(c, "")
+
+    def test_unknown_token_layout_keeps_previous_behaviour(self):
+        class _Bare:
+            pass
+        import sys, os
+        sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+        from backend import BackendServicer
+        r, c = BackendServicer._split_reasoning(
+            _Bare(), "raw", "prompt", "raw", None,
+        )
+        self.assertEqual(r, "raw")
+        self.assertEqual(c, "raw")
+
+
+class TestReasoningParserConstruction(unittest.TestCase):
+    """The parser must learn whether thinking was on for this request.
+
+    vLLM's engine-based parsers (Qwen3Parser and friends) read
+    chat_template_kwargs["enable_thinking"] and default to True, so a parser
+    built without it treats a thinking-disabled completion as pure reasoning.
+    """
+
+    def _servicer(self):
+        import sys, os
+        sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+        from backend import BackendServicer
+        s = BackendServicer()
+        s.tokenizer = object()
+        return s
+
+    def test_chat_template_kwargs_are_forwarded(self):
+        seen = {}
+
+        class _Parser:
+            def __init__(self, tokenizer, **kwargs):
+                seen.update(kwargs)
+
+        s = self._servicer()
+        s.reasoning_parser_cls = _Parser
+        s._new_reasoning_parser({"enable_thinking": False})
+        self.assertEqual(
+            seen.get("chat_template_kwargs"), {"enable_thinking": False},
+        )
+
+    def test_parser_without_the_kwarg_still_builds(self):
+        """Older parsers take only the tokenizer — must not break them."""
+        class _Old:
+            def __init__(self, tokenizer):
+                self.tokenizer = tokenizer
+
+        s = self._servicer()
+        s.reasoning_parser_cls = _Old
+        self.assertIsInstance(
+            s._new_reasoning_parser({"enable_thinking": False}), _Old,
+        )

backend/python/whisperx/backend.py

@@ -16,6 +16,7 @@ import grpc
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
+from transcript_utils import require_diarization_token, seconds_to_nanoseconds
 
 
 
@@ -81,6 +82,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         import whisperx
         from whisperx.diarize import DiarizationPipeline
 
+        try:
+            require_diarization_token(request.diarize, self.hf_token)
+        except ValueError as err:
+            context.abort(grpc.StatusCode.FAILED_PRECONDITION, str(err))
+
         resultSegments = []
         text = ""
         try:
@@ -117,8 +123,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             # Build result segments
             for idx, seg in enumerate(transcript["segments"]):
                 seg_text = seg.get("text", "")
-                start = int(seg.get("start", 0))
-                end = int(seg.get("end", 0))
+                start = seconds_to_nanoseconds(seg.get("start", 0))
+                end = seconds_to_nanoseconds(seg.get("end", 0))
                 speaker = seg.get("speaker", "")
 
                 resultSegments.append(backend_pb2.TranscriptSegment(

backend/python/whisperx/test_transcript_utils.py

@@ -0,0 +1,25 @@
+import unittest
+
+import transcript_utils
+
+
+class TestTranscriptUtils(unittest.TestCase):
+    def test_diarization_requires_hugging_face_token(self):
+        with self.assertRaisesRegex(
+            ValueError,
+            "HF_TOKEN is required for WhisperX diarization",
+        ):
+            transcript_utils.require_diarization_token(True, None)
+
+    def test_diarization_does_not_require_token_when_disabled(self):
+        transcript_utils.require_diarization_token(False, None)
+
+    def test_seconds_are_serialized_as_nanoseconds(self):
+        self.assertEqual(
+            transcript_utils.seconds_to_nanoseconds(3.25),
+            3_250_000_000,
+        )
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/whisperx/transcript_utils.py

@@ -0,0 +1,12 @@
+"""Helpers for WhisperX transcript responses."""
+
+
+def require_diarization_token(diarize, token):
+    """Reject diarization when WhisperX cannot load its gated pipeline."""
+    if diarize and not token:
+        raise ValueError("HF_TOKEN is required for WhisperX diarization")
+
+
+def seconds_to_nanoseconds(seconds):
+    """Convert WhisperX timestamps to the duration unit used by LocalAI."""
+    return int(seconds * 1_000_000_000)

cmd/launcher/internal/launcher.go

@@ -24,10 +24,17 @@ import (
 
 // Config represents the launcher configuration
 type Config struct {
-	ModelsPath      string            `json:"models_path"`
-	BackendsPath    string            `json:"backends_path"`
-	Address         string            `json:"address"`
-	AutoStart       bool              `json:"auto_start"`
+	ModelsPath   string `json:"models_path"`
+	BackendsPath string `json:"backends_path"`
+	Address      string `json:"address"`
+	// AutoStart controls whether the launcher starts the LocalAI server as
+	// soon as the launcher itself opens (and right after a fresh install).
+	// Unset means enabled: launching the app must yield a serving endpoint,
+	// which is what the quickstart docs promise. The JSON key is deliberately
+	// not the legacy "auto_start": that field was never honored nor exposed
+	// in any UI, so every existing launcher.json carries an unintentional
+	// false that would keep auto-start permanently off (#11673).
+	AutoStart       *bool             `json:"auto_start_server"`
 	StartOnBoot     bool              `json:"start_on_boot"`
 	LogLevel        string            `json:"log_level"`
 	EnvironmentVars map[string]string `json:"environment_vars"`
@@ -122,9 +129,6 @@ func (l *Launcher) Initialize() error {
 		log.Printf("Warning: failed to cleanup partial downloads: %v", err)
 	}
 
-	if l.config.StartOnBoot {
-		l.StartLocalAI()
-	}
 	// Set default paths if not configured (only if not already loaded from config)
 	if l.config.ModelsPath == "" {
 		homeDir, _ := os.UserHomeDir()
@@ -156,6 +160,12 @@ func (l *Launcher) Initialize() error {
 		log.Printf("Setting default ShowWelcome: true")
 	}
 
+	if l.config.AutoStart == nil {
+		enabled := true
+		l.config.AutoStart = &enabled
+		log.Printf("Setting default AutoStart: true")
+	}
+
 	// Create directories
 	os.MkdirAll(l.config.ModelsPath, 0755)
 	os.MkdirAll(l.config.BackendsPath, 0755)
@@ -177,6 +187,11 @@ func (l *Launcher) Initialize() error {
 				l.showDownloadLocalAIDialog()
 			}
 		})
+	} else if l.ShouldAutoStartServer() {
+		// The launcher is a tray-only app: without this the user launches it,
+		// sees no window and no server, and concludes it does nothing (#11673).
+		log.Printf("Auto-starting LocalAI server")
+		l.autoStartServer()
 	}
 
 	// Check for updates periodically
@@ -185,6 +200,35 @@ func (l *Launcher) Initialize() error {
 	return nil
 }
 
+// ShouldAutoStartServer reports whether the launcher should start the server
+// without user interaction: at launcher startup and right after a fresh
+// install. Defaults to enabled; StartOnBoot forces a start even when
+// auto-start was explicitly disabled, preserving its historical behavior.
+func (l *Launcher) ShouldAutoStartServer() bool {
+	if l.config == nil {
+		return false
+	}
+	if l.config.StartOnBoot {
+		return true
+	}
+	return l.config.AutoStart == nil || *l.config.AutoStart
+}
+
+// autoStartServer starts LocalAI in the background and surfaces failures
+// through the systray error dialog: during an auto-start there is no visible
+// window for a regular error dialog to attach to.
+func (l *Launcher) autoStartServer() {
+	go func() {
+		if err := l.StartLocalAI(); err != nil {
+			log.Printf("Failed to auto-start LocalAI: %v", err)
+			l.updateStatus(fmt.Sprintf("Failed to start LocalAI: %v", err))
+			if l.systray != nil {
+				l.systray.showStartupErrorDialog(err)
+			}
+		}
+	}()
+}
+
 // StartLocalAI starts the LocalAI server
 func (l *Launcher) StartLocalAI() error {
 	if l.isRunning {
@@ -644,14 +688,22 @@ func (l *Launcher) showDownloadError(title, message string) {
 // after a fresh install (no LocalAI binary present yet).
 func (l *Launcher) showDownloadProgress(version, title string) {
 	l.showDownloadProgressWindow(version, title, func(win fyne.Window) {
-		dialog.ShowConfirm("Installation Complete",
-			"LocalAI has been downloaded and installed successfully. You can now start LocalAI from the launcher.",
+		message := "LocalAI has been downloaded and installed successfully. You can now start LocalAI from the launcher."
+		if l.ShouldAutoStartServer() {
+			message = "LocalAI has been downloaded and installed successfully. It will start now: manage it and open the WebUI from the system tray icon."
+		}
+		dialog.ShowConfirm("Installation Complete", message,
 			func(bool) {
 				win.Close()
 				l.updateStatus("LocalAI installed successfully")
 				if l.systray != nil {
 					l.systray.recreateMenu()
 				}
+				// A fresh install should end with a running server, not with
+				// the user hunting for a start button in the tray (#11673).
+				if l.ShouldAutoStartServer() && !l.isRunning {
+					l.autoStartServer()
+				}
 			}, win)
 	})
 }

cmd/launcher/internal/launcher_test.go

@@ -1,6 +1,7 @@
 package launcher_test
 
 import (
+	"encoding/json"
 	"os"
 	"path/filepath"
 	"strings"
@@ -55,7 +56,8 @@ var _ = Describe("Launcher", func() {
 			Expect(err).ToNot(HaveOccurred())
 
 			config := launcherInstance.GetConfig()
-			Expect(config.ShowWelcome).To(BeTrue())
+			Expect(config.ShowWelcome).ToNot(BeNil())
+			Expect(*config.ShowWelcome).To(BeTrue())
 			Expect(config.Address).To(Equal("127.0.0.1:8080"))
 			Expect(config.LogLevel).To(Equal("info"))
 		})
@@ -177,13 +179,53 @@ var _ = Describe("Launcher", func() {
 
 			assertFlagValue("--generated-content-path", filepath.Join(dataPath, "generated"))
 			assertFlagValue("--upload-path", filepath.Join(dataPath, "uploads"))
-			// The bug was the server resolving these to shared /tmp paths.
+			// The bug was the server resolving these to its shared /tmp
+			// defaults. Only reject those specific paths: on Linux the test's
+			// own temp directory legitimately lives under /tmp.
 			for _, a := range args {
-				Expect(a).ToNot(HavePrefix("/tmp/"), "run args must not reference shared /tmp paths, got %s", a)
+				Expect(a).ToNot(HavePrefix("/tmp/generated"), "run args must not reference the shared /tmp generated-content default, got %s", a)
+				Expect(a).ToNot(HavePrefix("/tmp/upload"), "run args must not reference the shared /tmp upload default, got %s", a)
 			}
 		})
 	})
 
+	// Regression for "Mac dmg launcher launches nothing" (issue #11673): the
+	// launcher created empty log files and served nothing because nothing ever
+	// started the server unless the unrelated "start on system boot" option was
+	// enabled. Launching the app must yield a serving endpoint by default.
+	Describe("ShouldAutoStartServer", func() {
+		It("should auto-start by default when nothing is configured", func() {
+			Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue())
+		})
+
+		It("should respect an explicit opt-out", func() {
+			config := launcherInstance.GetConfig()
+			err := json.Unmarshal([]byte(`{"auto_start_server": false}`), config)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(launcherInstance.ShouldAutoStartServer()).To(BeFalse())
+		})
+
+		It("should still auto-start when StartOnBoot is set even if auto-start is off", func() {
+			config := launcherInstance.GetConfig()
+			err := json.Unmarshal([]byte(`{"auto_start_server": false, "start_on_boot": true}`), config)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue())
+		})
+
+		It("should ignore the legacy auto_start key older launchers persisted as false", func() {
+			// Old launchers marshaled the never-honored AutoStart field as
+			// "auto_start": false into every launcher.json. That stale value
+			// carries no user intent and must not disable auto-start.
+			config := launcherInstance.GetConfig()
+			err := json.Unmarshal([]byte(`{"auto_start": false}`), config)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(launcherInstance.ShouldAutoStartServer()).To(BeTrue())
+		})
+	})
+
 	Describe("Logs", func() {
 		It("should return empty logs initially", func() {
 			logs := launcherInstance.GetLogs()
@@ -210,13 +252,38 @@ var _ = Describe("Launcher", func() {
 	})
 })
 
+// Regression for the welcome window suppressing itself (part of issue
+// #11673): the "don't show this welcome window again" checkbox was
+// initialized with the ShowWelcome value itself, so on the very first
+// showing it came up checked AND its change callback persisted
+// ShowWelcome=false, hiding the welcome window forever.
+var _ = Describe("WelcomeDontShowAgainChecked", func() {
+	It("should be unchecked when the welcome window is enabled", func() {
+		show := true
+		config := &launcher.Config{ShowWelcome: &show}
+		Expect(launcher.WelcomeDontShowAgainChecked(config)).To(BeFalse())
+	})
+
+	It("should be checked when the user opted out", func() {
+		show := false
+		config := &launcher.Config{ShowWelcome: &show}
+		Expect(launcher.WelcomeDontShowAgainChecked(config)).To(BeTrue())
+	})
+
+	It("should be unchecked when the preference is unset", func() {
+		Expect(launcher.WelcomeDontShowAgainChecked(&launcher.Config{})).To(BeFalse())
+		Expect(launcher.WelcomeDontShowAgainChecked(nil)).To(BeFalse())
+	})
+})
+
 var _ = Describe("Config", func() {
 	It("should have proper JSON tags", func() {
+		autoStart := true
 		config := &launcher.Config{
 			ModelsPath:      "/test/models",
 			BackendsPath:    "/test/backends",
 			Address:         ":8080",
-			AutoStart:       true,
+			AutoStart:       &autoStart,
 			LogLevel:        "info",
 			EnvironmentVars: map[string]string{"TEST": "value"},
 		}
@@ -224,7 +291,7 @@ var _ = Describe("Config", func() {
 		Expect(config.ModelsPath).To(Equal("/test/models"))
 		Expect(config.BackendsPath).To(Equal("/test/backends"))
 		Expect(config.Address).To(Equal(":8080"))
-		Expect(config.AutoStart).To(BeTrue())
+		Expect(*config.AutoStart).To(BeTrue())
 		Expect(config.LogLevel).To(Equal("info"))
 		Expect(config.EnvironmentVars).To(HaveKeyWithValue("TEST", "value"))
 	})

cmd/launcher/internal/ui.go

@@ -34,6 +34,7 @@ type LauncherUI struct {
 	backendsPathEntry *widget.Entry
 	addressEntry      *widget.Entry
 	logLevelSelect    *widget.Select
+	autoStartCheck    *widget.Check
 	startOnBootCheck  *widget.Check
 
 	// Environment Variables
@@ -75,6 +76,7 @@ func NewLauncherUI() *LauncherUI {
 		backendsPathEntry: widget.NewEntry(),
 		addressEntry:      widget.NewEntry(),
 		logLevelSelect:    widget.NewSelect([]string{"error", "warn", "info", "debug", "trace"}, nil),
+		autoStartCheck:    widget.NewCheck("Start LocalAI when the launcher opens", nil),
 		startOnBootCheck:  widget.NewCheck("Start LocalAI on system boot", nil),
 		logText:           widget.NewMultiLineEntry(),
 		progressBar:       widget.NewProgressBar(),
@@ -117,6 +119,7 @@ func (ui *LauncherUI) createConfigTab() *fyne.Container {
 			widget.NewLabel("Log Level:"),
 			ui.logLevelSelect,
 		),
+		ui.autoStartCheck,
 		ui.startOnBootCheck,
 	))
 
@@ -401,6 +404,8 @@ func (ui *LauncherUI) saveConfiguration() {
 	config.BackendsPath = ui.backendsPathEntry.Text
 	config.Address = ui.addressEntry.Text
 	config.LogLevel = ui.logLevelSelect.Selected
+	autoStart := ui.autoStartCheck.Checked
+	config.AutoStart = &autoStart
 	config.StartOnBoot = ui.startOnBootCheck.Checked
 
 	// Ensure environment variables are included in the configuration
@@ -583,6 +588,7 @@ func (ui *LauncherUI) LoadConfiguration() {
 	ui.backendsPathEntry.SetText(config.BackendsPath)
 	ui.addressEntry.SetText(config.Address)
 	ui.logLevelSelect.SetSelected(config.LogLevel)
+	ui.autoStartCheck.SetChecked(config.AutoStart == nil || *config.AutoStart)
 	ui.startOnBootCheck.SetChecked(config.StartOnBoot)
 
 	// Load environment variables
@@ -616,6 +622,14 @@ func (ui *LauncherUI) UpdateRunningState(isRunning bool) {
 	})
 }
 
+// WelcomeDontShowAgainChecked reports the initial state of the welcome
+// window's "don't show this welcome window again" checkbox for the given
+// config: checked only when the user has already opted out of the welcome
+// window.
+func WelcomeDontShowAgainChecked(config *Config) bool {
+	return config != nil && config.ShowWelcome != nil && !*config.ShowWelcome
+}
+
 // ShowWelcomeWindow displays the welcome window with helpful information
 func (ui *LauncherUI) ShowWelcomeWindow() {
 	if ui.launcher == nil || ui.launcher.window == nil {
@@ -677,19 +691,20 @@ Getting Started:
 			ui.openURL("https://discord.gg/XgwjKptP7Z")
 		})
 
-		// Checkbox to disable welcome window
-		dontShowAgainCheck := widget.NewCheck("Don't show this welcome window again", func(checked bool) {
+		// Checkbox to disable welcome window. The initial state is applied
+		// BEFORE the change callback is attached: SetChecked fires OnChanged,
+		// and letting the initialization itself persist a ShowWelcome flip is
+		// exactly the bug that suppressed this window forever after its first
+		// showing (#11673).
+		dontShowAgainCheck := widget.NewCheck("Don't show this welcome window again", nil)
+		dontShowAgainCheck.SetChecked(WelcomeDontShowAgainChecked(ui.launcher.GetConfig()))
+		dontShowAgainCheck.OnChanged = func(checked bool) {
 			if ui.launcher != nil {
 				config := ui.launcher.GetConfig()
 				v := !checked
 				config.ShowWelcome = &v
 				ui.launcher.SetConfig(config)
 			}
-		})
-
-		config := ui.launcher.GetConfig()
-		if config.ShowWelcome != nil {
-			dontShowAgainCheck.SetChecked(*config.ShowWelcome)
 		}
 
 		// Close button

core/application/distributed.go

@@ -15,6 +15,7 @@ import (
 	"github.com/mudler/LocalAI/core/services/distributed"
 	"github.com/mudler/LocalAI/core/services/jobs"
 	"github.com/mudler/LocalAI/core/services/messaging"
+	"github.com/mudler/LocalAI/core/services/monitoring"
 	"github.com/mudler/LocalAI/core/services/nodes"
 	"github.com/mudler/LocalAI/core/services/nodes/prefixcache"
 	"github.com/mudler/LocalAI/core/services/storage"
@@ -162,6 +163,17 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 	}
 	xlog.Info("Node registry initialized")
 
+	// Bound durable heartbeat writes: a beat that only carries a fresher
+	// timestamp is what turned backend_nodes into a 460 MB six-row table.
+	registry.SetHeartbeatCheckpoint(cfg.Distributed.NodeHeartbeatCheckpointOrDefault())
+
+	// Measure the vacuum horizon. The 42 days it stayed open went unnoticed
+	// because no gauge reported it until models started failing to load.
+	if err := monitoring.RegisterControlPlaneDBMetrics(authDB, 30*time.Second); err != nil {
+		// Metrics are diagnostic; a failure here must not stop the frontend.
+		xlog.Warn("Control-plane database metrics unavailable", "error", err)
+	}
+
 	// Let scheduling rules be keyed by a model alias. The registry resolves a
 	// rule's name through the config loader to find the model it governs, so an
 	// operator can pin placement to a stable name like "production" and have it

core/cli/run.go

@@ -182,6 +182,8 @@ type RunCMD struct {
 	BackendUpgradeTimeout        string `env:"LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT" help:"NATS round-trip timeout for backend.upgrade requests (default 15m)." group:"distributed"`
 	ModelLoadTimeout             string `env:"LOCALAI_NATS_MODEL_LOAD_TIMEOUT" help:"Fixed gRPC deadline for the remote LoadModel call sent to a worker node once its backend is installed and model files are staged. Unset (the default), the deadline is derived from the checkpoint size instead: 5m plus 20s per GiB, capped at 6h, so multi-tens-of-GB diffusion/video checkpoints get the minutes they need without a fixed cliff. Set this only to pin a specific budget; the value is used verbatim, including when it is shorter than the derived one." group:"distributed"`
 	ModelLoadWait                string `env:"LOCALAI_MODEL_LOAD_WAIT" help:"How long an inference request waits for a model that is still cold-loading onto a worker before it is answered with 503, a Retry-After header and live staging progress (default 60s). The request is served the moment the model becomes ready, so a model already most of the way staged needs no client retry. Set to 0 to wait as long as the load takes — only safe when no ingress or load balancer with an idle timeout sits in front." group:"distributed"`
+	StaleNodeThreshold           string `env:"LOCALAI_STALE_NODE_THRESHOLD" help:"How long a worker node may go without a durable heartbeat before the health monitor marks it offline (default 5m). Because a beat that only carries a fresher timestamp is held back by --node-heartbeat-checkpoint, this must stay comfortably wider than that interval; raise both together. Dead-node detection through the per-model gRPC health check and through request-time failure is unaffected by this knob." group:"distributed"`
+	NodeHeartbeatCheckpoint      string `env:"LOCALAI_NODE_HEARTBEAT_CHECKPOINT" help:"Minimum gap between durable heartbeat writes for a worker node (default 60s). A beat that only carries a fresher timestamp is dropped until this interval elapses; every field is compared against the value last written, so a node's first beat, a changed total VRAM/total disk/GPU vendor, and a free VRAM/RAM/disk reading that has moved more than 256 MiB from the written value all still write immediately, and a node that is not active is never suppressed. Set below the worker heartbeat interval to write on every beat." group:"distributed"`
 	NatsAccountSeed              string `env:"LOCALAI_NATS_ACCOUNT_SEED" help:"NATS account signing seed (SU...) used to mint per-node worker JWTs at registration" group:"distributed"`
 	NatsServiceJWT               string `env:"LOCALAI_NATS_SERVICE_JWT" help:"NATS user JWT for the frontend (and agent workers) to publish control-plane messages" group:"distributed"`
 	NatsServiceSeed              string `env:"LOCALAI_NATS_SERVICE_SEED" help:"NATS user signing seed (SU...) paired with LOCALAI_NATS_SERVICE_JWT" group:"distributed"`
@@ -397,6 +399,20 @@ func (r *RunCMD) Run(ctx *cliContext.Context) error {
 		}
 		opts = append(opts, config.WithModelLoadWait(d))
 	}
+	if r.StaleNodeThreshold != "" {
+		d, err := parseDistributedDuration("LOCALAI_STALE_NODE_THRESHOLD", r.StaleNodeThreshold)
+		if err != nil {
+			return err
+		}
+		opts = append(opts, config.WithStaleNodeThreshold(d))
+	}
+	if r.NodeHeartbeatCheckpoint != "" {
+		d, err := parseDistributedDuration("LOCALAI_NODE_HEARTBEAT_CHECKPOINT", r.NodeHeartbeatCheckpoint)
+		if err != nil {
+			return err
+		}
+		opts = append(opts, config.WithNodeHeartbeatCheckpoint(d))
+	}
 	if r.RegistrationToken != "" {
 		opts = append(opts, config.WithRegistrationToken(r.RegistrationToken))
 	}

core/cli/worker/labels.go

@@ -10,7 +10,7 @@ func ParseNodeLabels(input string) map[string]string {
 	if input == "" {
 		return labels
 	}
-	for _, pair := range strings.Split(input, ",") {
+	for pair := range strings.SplitSeq(input, ",") {
 		pair = strings.TrimSpace(pair)
 		if k, v, ok := strings.Cut(pair, "="); ok {
 			labels[strings.TrimSpace(k)] = strings.TrimSpace(v)

core/config/distributed_config.go

@@ -57,12 +57,13 @@ type DistributedConfig struct {
 	StorageSecretKey string // --storage-secret-key / LOCALAI_STORAGE_SECRET_KEY
 
 	// Timeout configuration (all have sensible defaults — zero means use default)
-	MCPToolTimeout      time.Duration // MCP tool execution timeout (default 360s)
-	MCPDiscoveryTimeout time.Duration // MCP discovery timeout (default 60s)
-	WorkerWaitTimeout   time.Duration // Max wait for healthy worker at startup (default 5m)
-	DrainTimeout        time.Duration // Time to wait for in-flight requests during drain (default 30s)
-	HealthCheckInterval time.Duration // Health monitor check interval (default 15s)
-	StaleNodeThreshold  time.Duration // Time before a node is considered stale (default 60s)
+	MCPToolTimeout          time.Duration // MCP tool execution timeout (default 360s)
+	MCPDiscoveryTimeout     time.Duration // MCP discovery timeout (default 60s)
+	WorkerWaitTimeout       time.Duration // Max wait for healthy worker at startup (default 5m)
+	DrainTimeout            time.Duration // Time to wait for in-flight requests during drain (default 30s)
+	HealthCheckInterval     time.Duration // Health monitor check interval (default 15s)
+	StaleNodeThreshold      time.Duration // Time before a node is considered stale (default 5m)
+	NodeHeartbeatCheckpoint time.Duration // Minimum gap between durable heartbeat writes (default 60s, 0 = every beat)
 	// DisablePerModelHealthCheck turns off the health monitor's per-model
 	// gRPC probe. When enabled (the default), the monitor pings each model's
 	// gRPC address and removes stale node_models rows whose backend has
@@ -165,16 +166,17 @@ func (c DistributedConfig) Validate() error {
 	c.NatsAuthConfig().WarnIfInsecure(true)
 	// Check for negative durations
 	for name, d := range map[string]time.Duration{
-		FlagMCPToolTimeout:        c.MCPToolTimeout,
-		FlagMCPDiscoveryTimeout:   c.MCPDiscoveryTimeout,
-		FlagWorkerWaitTimeout:     c.WorkerWaitTimeout,
-		FlagDrainTimeout:          c.DrainTimeout,
-		FlagHealthCheckInterval:   c.HealthCheckInterval,
-		FlagStaleNodeThreshold:    c.StaleNodeThreshold,
-		FlagMCPCIJobTimeout:       c.MCPCIJobTimeout,
-		FlagBackendInstallTimeout: c.BackendInstallTimeout,
-		FlagBackendUpgradeTimeout: c.BackendUpgradeTimeout,
-		FlagModelLoadTimeout:      c.ModelLoadTimeout,
+		FlagMCPToolTimeout:          c.MCPToolTimeout,
+		FlagMCPDiscoveryTimeout:     c.MCPDiscoveryTimeout,
+		FlagWorkerWaitTimeout:       c.WorkerWaitTimeout,
+		FlagDrainTimeout:            c.DrainTimeout,
+		FlagHealthCheckInterval:     c.HealthCheckInterval,
+		FlagStaleNodeThreshold:      c.StaleNodeThreshold,
+		FlagNodeHeartbeatCheckpoint: c.NodeHeartbeatCheckpoint,
+		FlagMCPCIJobTimeout:         c.MCPCIJobTimeout,
+		FlagBackendInstallTimeout:   c.BackendInstallTimeout,
+		FlagBackendUpgradeTimeout:   c.BackendUpgradeTimeout,
+		FlagModelLoadTimeout:        c.ModelLoadTimeout,
 	} {
 		if d < 0 {
 			return fmt.Errorf("%s must not be negative", name)
@@ -337,6 +339,27 @@ func WithModelLoadWait(d time.Duration) AppOption {
 	}
 }
 
+// WithStaleNodeThreshold sets how long a node may go without a durable
+// heartbeat before the health monitor marks it offline. It has to be raised
+// alongside WithNodeHeartbeatCheckpoint: a checkpoint interval wider than this
+// threshold makes every healthy node look dead the moment its beats start
+// being suppressed.
+func WithStaleNodeThreshold(d time.Duration) AppOption {
+	return func(o *ApplicationConfig) {
+		o.Distributed.StaleNodeThreshold = d
+	}
+}
+
+// WithNodeHeartbeatCheckpoint bounds durable heartbeat writes. A zero d is
+// deliberately not special-cased into "unbounded": NodeHeartbeatCheckpointOrDefault
+// reads zero as unset, and an operator who wants a write per beat sets a value
+// below the worker's heartbeat interval instead.
+func WithNodeHeartbeatCheckpoint(d time.Duration) AppOption {
+	return func(o *ApplicationConfig) {
+		o.Distributed.NodeHeartbeatCheckpoint = d
+	}
+}
+
 var EnableAutoApproveNodes = func(o *ApplicationConfig) {
 	o.Distributed.AutoApproveNodes = true
 }
@@ -391,17 +414,18 @@ func WithModelSchedulingConfigPath(path string) AppOption {
 // them as constants prevents the string from drifting from the actual
 // flag a future rename would produce.
 const (
-	FlagMCPToolTimeout        = "mcp-tool-timeout"
-	FlagMCPDiscoveryTimeout   = "mcp-discovery-timeout"
-	FlagWorkerWaitTimeout     = "worker-wait-timeout"
-	FlagDrainTimeout          = "drain-timeout"
-	FlagHealthCheckInterval   = "health-check-interval"
-	FlagStaleNodeThreshold    = "stale-node-threshold"
-	FlagMCPCIJobTimeout       = "mcp-ci-job-timeout"
-	FlagBackendInstallTimeout = "backend-install-timeout"
-	FlagBackendUpgradeTimeout = "backend-upgrade-timeout"
-	FlagModelLoadTimeout      = "model-load-timeout"
-	FlagModelLoadWait         = "model-load-wait"
+	FlagMCPToolTimeout          = "mcp-tool-timeout"
+	FlagMCPDiscoveryTimeout     = "mcp-discovery-timeout"
+	FlagWorkerWaitTimeout       = "worker-wait-timeout"
+	FlagDrainTimeout            = "drain-timeout"
+	FlagHealthCheckInterval     = "health-check-interval"
+	FlagStaleNodeThreshold      = "stale-node-threshold"
+	FlagNodeHeartbeatCheckpoint = "node-heartbeat-checkpoint"
+	FlagMCPCIJobTimeout         = "mcp-ci-job-timeout"
+	FlagBackendInstallTimeout   = "backend-install-timeout"
+	FlagBackendUpgradeTimeout   = "backend-upgrade-timeout"
+	FlagModelLoadTimeout        = "model-load-timeout"
+	FlagModelLoadWait           = "model-load-wait"
 	// FlagDiskHeadroomCheck names the disk-headroom toggle. It is quoted in
 	// the warning the check emits while disabled, so the operator reading a
 	// log line knows exactly which knob produced it.
@@ -410,16 +434,22 @@ const (
 
 // Defaults for distributed timeouts.
 const (
-	DefaultMCPToolTimeout        = 360 * time.Second
-	DefaultMCPDiscoveryTimeout   = 60 * time.Second
-	DefaultWorkerWaitTimeout     = 5 * time.Minute
-	DefaultDrainTimeout          = 30 * time.Second
-	DefaultHealthCheckInterval   = 15 * time.Second
-	DefaultStaleNodeThreshold    = 60 * time.Second
-	DefaultMCPCIJobTimeout       = 10 * time.Minute
-	DefaultBackendInstallTimeout = 15 * time.Minute
-	DefaultBackendUpgradeTimeout = 15 * time.Minute
-	DefaultModelLoadTimeout      = 5 * time.Minute
+	DefaultMCPToolTimeout      = 360 * time.Second
+	DefaultMCPDiscoveryTimeout = 60 * time.Second
+	DefaultWorkerWaitTimeout   = 5 * time.Minute
+	DefaultDrainTimeout        = 30 * time.Second
+	DefaultHealthCheckInterval = 15 * time.Second
+	// A beat that only refreshes the timestamp is now dropped until the
+	// checkpoint interval elapses, so the persisted column is up to one
+	// interval stale by design. The threshold covers that plus jitter.
+	// A genuinely dead node is still caught sooner by the per-model gRPC
+	// health check and by request-time failure, neither of which reads this.
+	DefaultStaleNodeThreshold      = 5 * time.Minute
+	DefaultNodeHeartbeatCheckpoint = 60 * time.Second
+	DefaultMCPCIJobTimeout         = 10 * time.Minute
+	DefaultBackendInstallTimeout   = 15 * time.Minute
+	DefaultBackendUpgradeTimeout   = 15 * time.Minute
+	DefaultModelLoadTimeout        = 5 * time.Minute
 	// DefaultModelLoadWait is how long a request waits for a cold-loading model
 	// before it is answered with 503 and live progress. Chosen to sit under the
 	// idle timeout of typical ingress/LB defaults, so the answer comes from
@@ -519,6 +549,14 @@ func (c DistributedConfig) StaleNodeThresholdOrDefault() time.Duration {
 	return cmp.Or(c.StaleNodeThreshold, DefaultStaleNodeThreshold)
 }
 
+// NodeHeartbeatCheckpointOrDefault returns the configured interval or the
+// default. A configured zero is indistinguishable from unset here, which is
+// intentional: cmp.Or falls back to the default, and an operator who wants a
+// write per beat sets a value below the heartbeat interval instead.
+func (c DistributedConfig) NodeHeartbeatCheckpointOrDefault() time.Duration {
+	return cmp.Or(c.NodeHeartbeatCheckpoint, DefaultNodeHeartbeatCheckpoint)
+}
+
 // MCPCIJobTimeoutOrDefault returns the configured MCP CI job timeout or the default.
 func (c DistributedConfig) MCPCIJobTimeoutOrDefault() time.Duration {
 	return cmp.Or(c.MCPCIJobTimeout, DefaultMCPCIJobTimeout)

core/config/distributed_config_test.go

@@ -47,6 +47,27 @@ var _ = Describe("DistributedConfig backend NATS timeouts", func() {
 	})
 })
 
+// Heartbeat checkpointing makes last_heartbeat up to one checkpoint interval
+// stale by design, which is why the threshold defaults to 5 minutes. An
+// operator who widens the checkpoint has to widen this to match, so it has to
+// be reachable from the CLI rather than being a compile-time constant.
+var _ = Describe("DistributedConfig stale node threshold", func() {
+	It("defaults to 5 minutes, wide enough to cover a suppressed beat", func() {
+		Expect(config.DistributedConfig{}.StaleNodeThresholdOrDefault()).
+			To(Equal(5 * time.Minute))
+		Expect(config.DefaultStaleNodeThreshold).
+			To(BeNumerically(">", config.DefaultNodeHeartbeatCheckpoint),
+				"a threshold at or below the checkpoint interval marks healthy, "+
+					"beating nodes offline every cycle")
+	})
+
+	It("is configurable, so a widened checkpoint can be matched", func() {
+		o := config.NewApplicationConfig(config.WithStaleNodeThreshold(20 * time.Minute))
+		Expect(o.Distributed.StaleNodeThreshold).To(Equal(20 * time.Minute))
+		Expect(o.Distributed.StaleNodeThresholdOrDefault()).To(Equal(20 * time.Minute))
+	})
+})
+
 var _ = Describe("DistributedConfig flag-name constants", func() {
 	// Pin the kebab-case strings so a rename of the Go field name (or a
 	// CLI flag naming convention change) forces the constant to update,
@@ -62,6 +83,7 @@ var _ = Describe("DistributedConfig flag-name constants", func() {
 		Entry("drain timeout", config.FlagDrainTimeout, "drain-timeout"),
 		Entry("health check interval", config.FlagHealthCheckInterval, "health-check-interval"),
 		Entry("stale node threshold", config.FlagStaleNodeThreshold, "stale-node-threshold"),
+		Entry("node heartbeat checkpoint", config.FlagNodeHeartbeatCheckpoint, "node-heartbeat-checkpoint"),
 		Entry("MCP CI job timeout", config.FlagMCPCIJobTimeout, "mcp-ci-job-timeout"),
 		Entry("backend install timeout", config.FlagBackendInstallTimeout, "backend-install-timeout"),
 		Entry("backend upgrade timeout", config.FlagBackendUpgradeTimeout, "backend-upgrade-timeout"),

core/config/meta/constants.go

@@ -99,3 +99,12 @@ var DiffusersSchedulerOptions = []FieldOption{
 	{Value: "heun", Label: "Heun"},
 	{Value: "unipc", Label: "UniPC"},
 }
+
+// SystemMessagesAfterFirstOptions are the values of template.system_messages_after_first:
+// how system messages that appear after the first turn are handled before the chat
+// template runs (empty = pass through unchanged, which strict Jinja templates reject).
+var SystemMessagesAfterFirstOptions = []FieldOption{
+	{Value: "", Label: "Pass through (default)"},
+	{Value: "merge", Label: "Merge into the first system message"},
+	{Value: "user", Label: "Forward as user messages"},
+}

core/config/meta/registry.go

@@ -382,6 +382,14 @@ func DefaultRegistry() map[string]FieldMetaOverride {
 			Description: "Use the chat template from the model's tokenizer config",
 			Order:       44,
 		},
+		"template.system_messages_after_first": {
+			Section:     "templates",
+			Label:       "System Messages After First",
+			Description: "How system messages that appear after the first turn are handled before templating: merge into the first system message, or forward as user messages. Empty passes them through unchanged, which strict Jinja templates reject.",
+			Component:   "select",
+			Options:     SystemMessagesAfterFirstOptions,
+			Order:       45,
+		},
 		// Router section template — kept in the templates UI section
 		// (rather than the router section under "other") so operators
 		// editing prompt shapes find all template-typed fields in one

core/config/model_config.go

@@ -1351,6 +1351,16 @@ type TemplateConfig struct {
 	// that can use the tokenizers specified in the JSON config files of the models
 	UseTokenizerTemplate bool `yaml:"use_tokenizer_template,omitempty" json:"use_tokenizer_template,omitempty"`
 
+	// SystemMessagesAfterFirst controls what happens to system-role messages that
+	// appear after the leading system block. Some tokenizer chat templates (e.g.
+	// Qwen3.8 / Flash-Next) raise "System message must be at the beginning" for
+	// them, while agent frameworks (cogito tool selection, adjustment prompts)
+	// legitimately append system instructions mid-conversation.
+	//   ""/"error": pass through unchanged (template decides)
+	//   "merge":    fold them into the leading system message
+	//   "user":     forward them as user-role instructions (keeps their position)
+	SystemMessagesAfterFirst string `yaml:"system_messages_after_first,omitempty" json:"system_messages_after_first,omitempty"`
+
 	// JoinChatMessagesByCharacter is a string that will be used to join chat messages together.
 	// It defaults to \n
 	JoinChatMessagesByCharacter *string `yaml:"join_chat_messages_by_character,omitempty" json:"join_chat_messages_by_character,omitempty"`

core/gallery/empty_base_install_test.go

@@ -236,8 +236,15 @@ var _ = Describe("InstallModelFromGallery with an empty base config", func() {
 		Expect(install(e.Name, gallery.GalleryModel{})).To(Succeed())
 		cfg := installedConfig(e.Name)
 		Expect(cfg["name"]).To(Equal(e.Name))
-		// The catalog's own overrides, verbatim, laid over the empty base.
-		Expect(cfg["parameters"]).To(Equal(e.Overrides["parameters"]))
+		// The catalog's own overrides, laid over the empty base. parameters is
+		// checked key by key rather than as a whole map: the install also merges
+		// the model family's inference defaults into it, and what matters here is
+		// that the authored keys survive that.
+		authored, ok := e.Overrides["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		for key, want := range authored {
+			Expect(cfg["parameters"]).To(HaveKeyWithValue(key, want))
+		}
 		Expect(cfg["known_usecases"]).To(Equal(e.Overrides["known_usecases"]))
 	})
 })

core/gallery/exl3_gallery_test.go

@@ -0,0 +1,101 @@
+package gallery_test
+
+import (
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"gopkg.in/yaml.v3"
+)
+
+func remarshal(value any, target any) error {
+	data, err := yaml.Marshal(value)
+	if err != nil {
+		return err
+	}
+	return yaml.Unmarshal(data, target)
+}
+
+var _ = Describe("EXL3 gallery entries", func() {
+	It("pins the four full repositories and configures the Qwen DFlash companion", func() {
+		entries, err := gallery.ReadConfigFile[[]gallery.GalleryModel](filepath.Join("..", "..", "gallery", "index.yaml"))
+		Expect(err).ToNot(HaveOccurred())
+
+		byName := make(map[string]gallery.GalleryModel, len(*entries))
+		for _, entry := range *entries {
+			byName[entry.Name] = entry
+		}
+
+		expected := map[string]struct {
+			repo     string
+			revision string
+		}{
+			"qwen3.8-27b-exl3-vllm-cpp": {
+				repo: "Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw", revision: "19441ac874c4018295da848e250f23511361cda4",
+			},
+			"qwen3.8-27b-dflash2-exl3-vllm-cpp": {
+				repo: "Mia-AiLab/Qwen3.8-27B-EXL3-3.5bpw", revision: "19441ac874c4018295da848e250f23511361cda4",
+			},
+			"deepseek-v4-flash-spark-exl3-vllm-cpp": {
+				repo: "0xSero/deepseek-v4-flash-0731-spark", revision: "ce5ff0f1efb2e184aafc759d281bfae47d3a359c",
+			},
+			"deepseek-v4-flash-exl3-3bpw-vllm-cpp": {
+				repo: "0xSero/DeepSeek-V4-Flash-0731-EXL3-3.0bpw", revision: "e0bf84ac76a5100e8790c22ad10b70b1e2d06d71",
+			},
+		}
+
+		for name, want := range expected {
+			entry, found := byName[name]
+			Expect(found).To(BeTrue(), "missing gallery entry %q", name)
+			Expect(entry.Tags).To(ContainElements("vllm-cpp", "exl3", "gpu", "cuda"), name)
+			Expect(entry.Overrides).To(HaveKeyWithValue("backend", "vllm-cpp"), name)
+			cfg := config.ModelConfig{}
+			Expect(remarshal(entry.Overrides, &cfg)).To(Succeed(), name)
+			Expect(cfg.Artifacts).ToNot(BeEmpty(), name)
+			Expect(cfg.Artifacts[0].Source.Repo).To(Equal(want.repo), name)
+			Expect(cfg.Artifacts[0].Source.Revision).To(Equal(want.revision), name)
+		}
+
+		plain := byName["qwen3.8-27b-exl3-vllm-cpp"]
+		Expect(plain.Tags).ToNot(ContainElement("dflash"))
+		dflashTags := 0
+		for name := range expected {
+			if contains(byName[name].Tags, "dflash") {
+				dflashTags++
+			}
+		}
+		Expect(dflashTags).To(Equal(1))
+
+		dflash := byName["qwen3.8-27b-dflash2-exl3-vllm-cpp"]
+		Expect(dflash.Tags).To(ContainElement("dflash"))
+		Expect(dflash.Variants).To(ConsistOf(gallery.Variant{Model: "qwen3.8-27b-exl3-vllm-cpp"}))
+		cfg := config.ModelConfig{}
+		Expect(remarshal(dflash.Overrides, &cfg)).To(Succeed())
+		Expect(cfg.ContextSize).To(HaveValue(Equal(8192)))
+		Expect(cfg.EngineArgs).To(HaveKeyWithValue("num_blocks", 2048))
+		Expect(cfg.EngineArgs).To(HaveKeyWithValue("max_num_seqs", 8))
+		Expect(cfg.EngineArgs).To(HaveKeyWithValue("max_num_batched_tokens", 16384))
+		Expect(cfg.EngineArgs).To(HaveKeyWithValue("enable_prefix_caching", false))
+		spec, ok := cfg.EngineArgs["speculative_config"].(map[string]any)
+		Expect(ok).To(BeTrue())
+		Expect(spec).To(HaveKeyWithValue("method", "dflash"))
+		Expect(spec).To(HaveKeyWithValue("num_speculative_tokens", 7))
+		Expect(cfg.Artifacts).To(HaveLen(2))
+		Expect(cfg.Artifacts[1].Name).To(Equal("draft_model"))
+		Expect(cfg.Artifacts[1].Target).To(Equal("companion"))
+		Expect(cfg.Artifacts[1].Source.Repo).To(Equal("Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw"))
+		Expect(cfg.Artifacts[1].Source.Revision).To(Equal("4f0436269bca761b071f05319e8e04a87cc633f9"))
+	})
+})
+
+func contains(values []string, target string) bool {
+	for _, value := range values {
+		if value == target {
+			return true
+		}
+	}
+	return false
+}

core/gallery/inference_defaults_install_test.go

@@ -0,0 +1,189 @@
+package gallery_test
+
+import (
+	"context"
+	"fmt"
+	"maps"
+	"os"
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"gopkg.in/yaml.v3"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/pkg/modelartifacts"
+	"github.com/mudler/LocalAI/pkg/system"
+)
+
+// The recommended sampling parameters for a model family are applied at install
+// and persisted into the model YAML. Persisting them is only worth anything if
+// they are written where the loader reads them back: PredictionOptions is nested
+// under "parameters" in ModelConfig, so a top level "temperature" key parses
+// without error and is then ignored for the life of the model.
+//
+// The expected values are read from the family table rather than written out
+// here, so that retuning a family stays a one file change.
+//
+// Nothing here reaches the network.
+var _ = Describe("Inference defaults persisted at install", func() {
+	var tempdir string
+	var galleries []config.Gallery
+	var systemState *system.SystemState
+	// The gallery listing is cached on the name and URL pair, so every spec
+	// needs a gallery of its own or it reads the previous spec's catalog.
+	galleryRevision := 0
+
+	// The name has to contain a pattern from inference_defaults.json, otherwise
+	// no defaults are applied and every assertion below passes vacuously.
+	const modelName = "qwen3.5-install-defaults"
+
+	newGallery := func(entries ...gallery.GalleryModel) {
+		out, err := yaml.Marshal(entries)
+		Expect(err).ToNot(HaveOccurred())
+		name := fmt.Sprintf("inference-defaults-%d", galleryRevision)
+		galleryRevision++
+		galleryPath := filepath.Join(tempdir, name+".yaml")
+		Expect(os.WriteFile(galleryPath, out, 0600)).To(Succeed())
+		galleries = []config.Gallery{{Name: name, URL: "file://" + galleryPath}}
+	}
+
+	install := func(name string) error {
+		return gallery.InstallModelFromGallery(
+			context.TODO(), galleries, []config.Gallery{}, systemState, nil,
+			name, gallery.GalleryModel{}, func(string, string, string, float64) {}, false, false, false)
+	}
+
+	installedConfig := func(name string) map[string]any {
+		dat, err := os.ReadFile(filepath.Join(tempdir, name+".yaml"))
+		Expect(err).ToNot(HaveOccurred())
+		content := map[string]any{}
+		Expect(yaml.Unmarshal(dat, &content)).To(Succeed())
+		return content
+	}
+
+	// Seeding the weights keeps the install off the network: the downloader
+	// treats an already-present destination with no declared sha256 as fetched.
+	// extra goes into parameters:, so a spec can pin a value the defaults would
+	// otherwise supply.
+	seedGallery := func(extra map[string]any) {
+		Expect(os.WriteFile(filepath.Join(tempdir, "weights.gguf"), []byte("weights"), 0600)).To(Succeed())
+
+		params := map[string]any{"model": "weights.gguf"}
+		maps.Copy(params, extra)
+
+		e := gallery.GalleryModel{Overrides: map[string]any{
+			"backend":    "llama-cpp",
+			"parameters": params,
+		}}
+		e.Name = modelName
+		e.AdditionalFiles = []gallery.File{{Filename: "weights.gguf", URI: "https://example.com/weights.gguf"}}
+		newGallery(e)
+	}
+
+	// Guards the fixture itself. If the name stops matching a family the specs
+	// below would still pass while asserting nothing at all.
+	expectedFamily := func() map[string]float64 {
+		family := config.MatchModelFamily(modelName)
+		Expect(family).ToNot(BeEmpty(), "fixture name no longer matches a family in inference_defaults.json")
+		return family
+	}
+
+	BeforeEach(func() {
+		var err error
+		tempdir, err = os.MkdirTemp("", "inference-defaults-install")
+		Expect(err).ToNot(HaveOccurred())
+		DeferCleanup(func() { Expect(os.RemoveAll(tempdir)).To(Succeed()) })
+
+		systemState, err = system.GetSystemState(system.WithModelPath(tempdir))
+		Expect(err).ToNot(HaveOccurred())
+	})
+
+	It("writes them under parameters, where the loader reads them back", func() {
+		family := expectedFamily()
+		seedGallery(nil)
+
+		Expect(install(modelName)).To(Succeed())
+
+		params, ok := installedConfig(modelName)["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue(), "parameters should be a map")
+
+		for key, want := range family {
+			Expect(params).To(HaveKey(key))
+			Expect(params[key]).To(BeNumerically("==", want), "parameters.%s", key)
+		}
+	})
+
+	It("does not leave them at the top level, where they are ignored", func() {
+		family := expectedFamily()
+		seedGallery(nil)
+
+		Expect(install(modelName)).To(Succeed())
+
+		cfg := installedConfig(modelName)
+		for key := range family {
+			Expect(cfg).ToNot(HaveKey(key), "%s at the top level is never read", key)
+		}
+	})
+
+	It("leaves a value the entry already sets alone", func() {
+		family := expectedFamily()
+		Expect(family).To(HaveKey("temperature"))
+		Expect(family["temperature"]).ToNot(BeNumerically("==", 0.05), "pick a value the family does not use")
+
+		seedGallery(map[string]any{"temperature": 0.05})
+
+		Expect(install(modelName)).To(Succeed())
+
+		params, ok := installedConfig(modelName)["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue(), "parameters should be a map")
+		Expect(params["temperature"]).To(BeNumerically("==", 0.05))
+	})
+
+	// An entry that binds a primary artifact carries no files: of its own, so it
+	// takes the other branch of the install and none of the specs above reach it.
+	// It is also the one branch that already re-marshalled, which is why the
+	// defaults did land on disk there, at the top level where nothing reads them.
+	It("writes them under parameters on the artifact binding path too", func() {
+		family := expectedFamily()
+
+		definition := &gallery.ModelConfig{ConfigFile: `
+backend: transformers
+artifacts:
+  - name: model
+    target: model
+    source:
+      type: huggingface
+      repo: owner/repo
+parameters:
+  model: owner/repo
+`}
+		// Standing in for the materializer keeps the install off the network.
+		materializer := &fakeArtifactMaterializer{result: modelartifacts.Result{
+			Spec: modelartifacts.Spec{
+				Name: "model", Target: "model",
+				Source: modelartifacts.Source{Type: "huggingface", Repo: "owner/repo", Revision: "main"},
+				Resolved: &modelartifacts.Resolved{
+					Endpoint: "https://huggingface.co",
+					Revision: "0123456789abcdef0123456789abcdef01234567",
+					CacheKey: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
+				},
+			},
+			RelativePath: ".artifacts/huggingface/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef/snapshot",
+		}}
+
+		_, err := gallery.InstallModel(context.TODO(), systemState, modelName, definition, nil, nil, false,
+			gallery.WithArtifactMaterializer(materializer))
+		Expect(err).ToNot(HaveOccurred())
+
+		cfg := installedConfig(modelName)
+		params, ok := cfg["parameters"].(map[string]any)
+		Expect(ok).To(BeTrue(), "parameters should be a map")
+		for key, want := range family {
+			Expect(params).To(HaveKey(key))
+			Expect(params[key]).To(BeNumerically("==", want), "parameters.%s", key)
+			Expect(cfg).ToNot(HaveKey(key), "%s at the top level is never read", key)
+		}
+	})
+})

core/gallery/models.go

@@ -622,35 +622,51 @@ func InstallModel(ctx context.Context, systemState *system.SystemState, nameOver
 		lconfig.ApplyInferenceDefaults(&modelConfig, name, modelConfig.Model)
 
 		// Merge inference defaults into configMap so they are persisted without losing unknown fields.
-		if modelConfig.Temperature != nil {
-			if _, exists := configMap["temperature"]; !exists {
-				configMap["temperature"] = *modelConfig.Temperature
+		// They belong under "parameters": ModelConfig embeds PredictionOptions with
+		// that yaml key, so a top level "temperature" parses without error and is
+		// then ignored for the life of the model.
+		params, mergeable := configMap["parameters"].(map[string]any)
+		if configMap["parameters"] == nil {
+			params, mergeable = map[string]any{}, true
+		}
+		if mergeable {
+			// An entry that sets one of these keeps its own value. ApplyInferenceDefaults
+			// already skipped those fields; this keeps the write side symmetric.
+			setDefault := func(key string, value any) {
+				if _, exists := params[key]; !exists {
+					params[key] = value
+				}
+			}
+			if modelConfig.Temperature != nil {
+				setDefault("temperature", *modelConfig.Temperature)
+			}
+			if modelConfig.TopP != nil {
+				setDefault("top_p", *modelConfig.TopP)
+			}
+			if modelConfig.TopK != nil {
+				setDefault("top_k", *modelConfig.TopK)
+			}
+			if modelConfig.MinP != nil {
+				setDefault("min_p", *modelConfig.MinP)
+			}
+			if modelConfig.RepeatPenalty != 0 {
+				setDefault("repeat_penalty", modelConfig.RepeatPenalty)
+			}
+			if modelConfig.PresencePenalty != 0 {
+				setDefault("presence_penalty", modelConfig.PresencePenalty)
+			}
+			if len(params) > 0 {
+				configMap["parameters"] = params
 			}
 		}
-		if modelConfig.TopP != nil {
-			if _, exists := configMap["top_p"]; !exists {
-				configMap["top_p"] = *modelConfig.TopP
-			}
-		}
-		if modelConfig.TopK != nil {
-			if _, exists := configMap["top_k"]; !exists {
-				configMap["top_k"] = *modelConfig.TopK
-			}
-		}
-		if modelConfig.MinP != nil {
-			if _, exists := configMap["min_p"]; !exists {
-				configMap["min_p"] = *modelConfig.MinP
-			}
-		}
-		if modelConfig.RepeatPenalty != 0 {
-			if _, exists := configMap["repeat_penalty"]; !exists {
-				configMap["repeat_penalty"] = modelConfig.RepeatPenalty
-			}
-		}
-		if modelConfig.PresencePenalty != 0 {
-			if _, exists := configMap["presence_penalty"]; !exists {
-				configMap["presence_penalty"] = modelConfig.PresencePenalty
-			}
+
+		// The marshal above predates this merge, and the only other re-marshal is
+		// behind the artifact binding below, which an entry carrying files: never
+		// reaches. Without this the defaults are computed and then dropped on the
+		// way to disk.
+		updatedConfigYAML, err = yaml.Marshal(configMap)
+		if err != nil {
+			return nil, fmt.Errorf("failed to marshal config with inference defaults: %v", err)
 		}
 
 		if valid, err := modelConfig.Validate(); !valid {

core/gallery/variants_lint_test.go

@@ -540,6 +540,29 @@ var _ = Describe("gallery/index.yaml Higgs Audio entry", func() {
 	})
 })
 
+var _ = Describe("gallery/index.yaml qwythos-9b-claude-mythos-5-1m mmproj", func() {
+	It("points at the published F16 mmproj artifact", func() {
+		entries, err := loadGalleryIndex()
+		Expect(err).ToNot(HaveOccurred())
+
+		models := make([]*gallery.GalleryModel, 0, len(entries))
+		for i := range entries {
+			models = append(models, &entries[i])
+		}
+		entry := gallery.FindGalleryElement(models, "qwythos-9b-claude-mythos-5-1m")
+		Expect(entry).ToNot(BeNil())
+		Expect(entry.Overrides).To(HaveKeyWithValue(
+			"mmproj",
+			"llama-cpp/mmproj/Qwythos-9B-Claude-Mythos-5-1M-GGUF/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf",
+		))
+		Expect(entry.AdditionalFiles).To(ContainElement(gallery.File{
+			Filename: "llama-cpp/mmproj/Qwythos-9B-Claude-Mythos-5-1M-GGUF/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf",
+			SHA256:   "f977efc337a2ac2ba183eea0c73e25b75fc240d56c05ed4d9b56ab451f64c82c",
+			URI:      "https://huggingface.co/empero-ai/Qwythos-9B-Claude-Mythos-5-1M-GGUF/resolve/main/mmproj-Qwythos-9B-Claude-Mythos-5-1M-F16.gguf",
+		}))
+	})
+})
+
 // The lint rules above check the catalog as text. This drives the real
 // resolution path for the entry a user actually clicked and failed to install,
 // so the fix is proven at the layer that broke and not only at the layer that

core/http/endpoints/localai/face_register.go

@@ -1,6 +1,7 @@
 package localai
 
 import (
+	"errors"
 	"net/http"
 
 	"github.com/labstack/echo/v4"
@@ -33,22 +34,31 @@ func FaceRegisterEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, a
 			return echo.NewHTTPError(http.StatusBadRequest, "name is required")
 		}
 
-		img, err := decodeImageInput(input.Img)
-		if err != nil {
-			return err
+		if (input.Img == "") == (len(input.Embedding) == 0) {
+			return echo.NewHTTPError(http.StatusBadRequest, "provide exactly one of img or embedding")
 		}
-
-		xlog.Debug("FaceRegister", "model", cfg.Name, "name", input.Name)
-		embedding, err := backend.FaceEmbed(c.Request().Context(), img, ml, appConfig, *cfg)
-		if err != nil {
-			return mapBackendError(err)
+		embedding := input.Embedding
+		if len(embedding) == 0 {
+			img, err := decodeImageInput(input.Img)
+			if err != nil {
+				return err
+			}
+			xlog.Debug("FaceRegister", "model", cfg.Name, "name", input.Name)
+			embedding, err = backend.FaceEmbed(c.Request().Context(), img, ml, appConfig, *cfg)
+			if err != nil {
+				return mapBackendError(err)
+			}
 		}
 
 		stored, err := registry.Register(c.Request().Context(), embedding, facerecognition.Metadata{
-			Name:   input.Name,
-			Labels: input.Labels,
+			Name:         input.Name,
+			RegisteredAt: input.RegisteredAt,
+			Labels:       input.Labels,
 		})
 		if err != nil {
+			if errors.Is(err, facerecognition.ErrInvalidEmbedding) || errors.Is(err, facerecognition.ErrDimensionMismatch) {
+				return echo.NewHTTPError(http.StatusBadRequest, err.Error())
+			}
 			return err
 		}
 		return c.JSON(http.StatusOK, schema.FaceRegisterResponse{

core/http/endpoints/localai/face_register_test.go

@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: MIT
+
+package localai_test
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"time"
+
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/config"
+	. "github.com/mudler/LocalAI/core/http/endpoints/localai"
+	"github.com/mudler/LocalAI/core/http/middleware"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/facerecognition"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+type registrationRecorder struct {
+	facerecognition.Registry
+	vector []float32
+	meta   facerecognition.Metadata
+	err    error
+}
+
+func (r *registrationRecorder) Register(_ context.Context, v []float32, m facerecognition.Metadata) (facerecognition.Metadata, error) {
+	r.vector = v
+	r.meta = m
+	m.ID = "saved-id"
+	return m, r.err
+}
+
+var _ = Describe("Face registration replay", func() {
+	var reg *registrationRecorder
+	call := func(in schema.FaceRegisterRequest) (*httptest.ResponseRecorder, error) {
+		e := echo.New()
+		rec := httptest.NewRecorder()
+		c := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/face/register", nil), rec)
+		c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, &in)
+		c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, &config.ModelConfig{})
+		// No model loader: replay must not call the embedding backend.
+		err := FaceRegisterEndpoint(nil, nil, nil, reg)(c)
+		return rec, err
+	}
+	BeforeEach(func() { reg = &registrationRecorder{} })
+	It("accepts the saved vector and timestamp without running inference", func() {
+		at := time.Now().UTC()
+		in := schema.FaceRegisterRequest{Name: "Alice", Embedding: []float32{1, 0}, RegisteredAt: at, Labels: map[string]string{"client_id": "alice"}}
+		in.Model = "faces"
+		rec, err := call(in)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(rec.Code).To(Equal(http.StatusOK))
+		Expect(reg.vector).To(Equal(in.Embedding))
+		Expect(reg.meta.RegisteredAt).To(Equal(at))
+		Expect(reg.meta.Labels).To(Equal(in.Labels))
+		Expect(rec.Body.String()).To(ContainSubstring("saved-id"))
+	})
+	It("rejects ambiguous and missing inputs before inference", func() {
+		for _, in := range []schema.FaceRegisterRequest{
+			{Name: "Alice"},
+			{Name: "Alice", Img: "image", Embedding: []float32{1, 0}},
+		} {
+			in.Model = "faces"
+			_, err := call(in)
+			Expect(err).To(HaveOccurred())
+			Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusBadRequest))
+			Expect(reg.vector).To(BeNil())
+		}
+	})
+	It("reports invalid vectors as a client error", func() {
+		reg.err = facerecognition.ErrInvalidEmbedding
+		in := schema.FaceRegisterRequest{Name: "Alice", Embedding: []float32{0, 0}}
+		in.Model = "faces"
+		_, err := call(in)
+		Expect(err).To(HaveOccurred())
+		Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusBadRequest))
+	})
+})

core/http/endpoints/openai/chat.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"fmt"
 	"net/http"
+	"strings"
 	"time"
 
 	"github.com/google/uuid"
@@ -24,18 +25,106 @@ import (
 	"github.com/mudler/xlog"
 )
 
+// messageText returns the textual content of a message, preferring the
+// middleware-populated StringContent and falling back to a string Content.
+func messageText(m schema.Message) string {
+	if m.StringContent != "" {
+		return m.StringContent
+	}
+	if s, ok := m.Content.(string); ok {
+		return s
+	}
+	return ""
+}
+
 // hasSystemMessage reports whether the message slice already contains a
-// system-role message — used to avoid clobbering a caller-supplied system
-// prompt when the LocalAI Assistant modality is on.
+// non-empty system-role message — used to avoid clobbering a caller-supplied
+// system prompt when the LocalAI Assistant modality is on. Empty / whitespace
+// system turns (historically sent by the web Chat UI) are ignored so they do
+// not suppress the model config system_prompt.
 func hasSystemMessage(messages []schema.Message) bool {
 	for _, m := range messages {
-		if m.Role == "system" {
+		if m.Role == "system" && strings.TrimSpace(messageText(m)) != "" {
 			return true
 		}
 	}
 	return false
 }
 
+// stripEmptySystemMessages drops system-role messages whose content is empty
+// or whitespace-only. An explicit blank system turn would otherwise satisfy
+// tokenizer chat templates' `messages[0].role == "system"` check and suppress
+// both the model's configured system_prompt and any template default.
+func stripEmptySystemMessages(messages []schema.Message) []schema.Message {
+	out := messages[:0:0]
+	for _, m := range messages {
+		if m.Role == "system" && strings.TrimSpace(messageText(m)) == "" {
+			continue
+		}
+		out = append(out, m)
+	}
+	return out
+}
+
+// normalizeLateSystemMessages handles system-role messages that appear after the
+// leading system block, according to template.system_messages_after_first:
+// "merge" folds them into the first system message (created if absent), "user"
+// forwards them as user-role turns at their original position. Any other value
+// returns the messages unchanged. Needed for tokenizer templates that reject
+// late system turns (Qwen3.8: "System message must be at the beginning") while
+// agent frameworks append instructions mid-conversation.
+func normalizeLateSystemMessages(messages []schema.Message, mode string) []schema.Message {
+	if mode != "merge" && mode != "user" {
+		return messages
+	}
+	lead := 0
+	for lead < len(messages) && messages[lead].Role == "system" {
+		lead++
+	}
+	late := false
+	for _, m := range messages[lead:] {
+		if m.Role == "system" {
+			late = true
+			break
+		}
+	}
+	if !late {
+		return messages
+	}
+	out := make([]schema.Message, 0, len(messages)+1)
+	out = append(out, messages[:lead]...)
+	if mode == "merge" && lead == 0 {
+		out = append(out, schema.Message{Role: "system"})
+	}
+	for _, m := range messages[lead:] {
+		if m.Role != "system" {
+			out = append(out, m)
+			continue
+		}
+		text := strings.TrimSpace(messageText(m))
+		if text == "" {
+			continue
+		}
+		switch mode {
+		case "merge":
+			first := &out[0]
+			joined := strings.TrimSpace(messageText(*first))
+			if joined != "" {
+				joined += "\n\n"
+			}
+			joined += text
+			first.Content = joined
+			first.StringContent = joined
+		case "user":
+			m.Role = "user"
+			m.Content = text
+			m.StringContent = text
+			out = append(out, m)
+		}
+	}
+	return out
+}
+
 // mergeToolCallDeltas merges streaming tool call deltas into complete tool calls.
 // In SSE streaming, a single tool call arrives as multiple chunks sharing the same Index:
 // the first chunk carries the ID, Type, and Name; subsequent chunks append to Arguments.
@@ -149,6 +238,19 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
 
 		xlog.Debug("Chat endpoint configuration read", "config", config)
 
+		// Drop blank system turns from the web UI (and similar clients) so they
+		// cannot suppress the model YAML system_prompt / tokenizer defaults.
+		input.Messages = stripEmptySystemMessages(input.Messages)
+		input.Messages = normalizeLateSystemMessages(input.Messages, config.TemplateConfig.SystemMessagesAfterFirst)
+
+		// Tokenizer-template models pass messages through to the backend as-is,
+		// so apply the configured system_prompt when the request did not supply
+		// one. Go-template models already receive SystemPrompt via PromptTemplateData.
+		if config.TemplateConfig.UseTokenizerTemplate && config.SystemPrompt != "" && !hasSystemMessage(input.Messages) {
+			prompt := config.SystemPrompt
+			input.Messages = append([]schema.Message{{Role: "system", Content: prompt, StringContent: prompt}}, input.Messages...)
+		}
+
 		// Cloud-proxy bail. Bypasses the local pipeline (templating,
 		// MCP injection, gRPC backend) and forwards via the cloud-
 		// proxy backend, which does the outbound HTTP. Request-side PII

core/http/endpoints/openai/chat_test.go

@@ -357,3 +357,91 @@ var _ = Describe("mergeToolCallDeltas", func() {
 		})
 	})
 })
+
+var _ = Describe("system message helpers", func() {
+	Describe("hasSystemMessage", func() {
+		It("ignores empty and whitespace-only system turns", func() {
+			Expect(hasSystemMessage([]schema.Message{
+				{Role: "system", Content: "", StringContent: ""},
+				{Role: "user", Content: "hi", StringContent: "hi"},
+			})).To(BeFalse())
+			Expect(hasSystemMessage([]schema.Message{
+				{Role: "system", Content: "   ", StringContent: "   "},
+			})).To(BeFalse())
+		})
+
+		It("detects a real system prompt", func() {
+			Expect(hasSystemMessage([]schema.Message{
+				{Role: "system", Content: "You are helpful.", StringContent: "You are helpful."},
+				{Role: "user", Content: "hi", StringContent: "hi"},
+			})).To(BeTrue())
+		})
+	})
+
+	Describe("normalizeLateSystemMessages", func() {
+		msgs := func() []schema.Message {
+			return []schema.Message{
+				{Role: "system", Content: "lead", StringContent: "lead"},
+				{Role: "user", Content: "q", StringContent: "q"},
+				{Role: "assistant", Content: "a", StringContent: "a"},
+				{Role: "system", Content: "late", StringContent: "late"},
+				{Role: "user", Content: "q2", StringContent: "q2"},
+			}
+		}
+		It("leaves messages untouched by default", func() {
+			out := normalizeLateSystemMessages(msgs(), "")
+			Expect(out).To(HaveLen(5))
+			Expect(out[3].Role).To(Equal("system"))
+		})
+		It("merge folds late system turns into the leading one", func() {
+			out := normalizeLateSystemMessages(msgs(), "merge")
+			Expect(out).To(HaveLen(4))
+			Expect(out[0].Role).To(Equal("system"))
+			Expect(out[0].StringContent).To(Equal("lead\n\nlate"))
+			for _, m := range out[1:] {
+				Expect(m.Role).NotTo(Equal("system"))
+			}
+		})
+		It("merge creates a leading system message when none exists", func() {
+			in := msgs()[1:]
+			out := normalizeLateSystemMessages(in, "merge")
+			Expect(out[0].Role).To(Equal("system"))
+			Expect(out[0].StringContent).To(Equal("late"))
+			Expect(out).To(HaveLen(4))
+		})
+		It("user forwards late system turns as user turns in place", func() {
+			out := normalizeLateSystemMessages(msgs(), "user")
+			Expect(out).To(HaveLen(5))
+			Expect(out[3].Role).To(Equal("user"))
+			Expect(out[3].StringContent).To(Equal("late"))
+			Expect(out[0].Role).To(Equal("system"))
+		})
+		It("does nothing when no late system turn exists", func() {
+			in := msgs()[:3]
+			Expect(normalizeLateSystemMessages(in, "user")).To(HaveLen(3))
+		})
+	})
+
+	Describe("stripEmptySystemMessages", func() {
+		It("removes blank system turns and keeps the rest", func() {
+			in := []schema.Message{
+				{Role: "system", Content: "", StringContent: ""},
+				{Role: "system", Content: "  ", StringContent: "  "},
+				{Role: "user", Content: "Explain how this works", StringContent: "Explain how this works"},
+			}
+			out := stripEmptySystemMessages(in)
+			Expect(out).To(HaveLen(1))
+			Expect(out[0].Role).To(Equal("user"))
+		})
+
+		It("keeps a non-empty system turn", func() {
+			in := []schema.Message{
+				{Role: "system", Content: "You are LocalAI.", StringContent: "You are LocalAI."},
+				{Role: "user", Content: "hi", StringContent: "hi"},
+			}
+			out := stripEmptySystemMessages(in)
+			Expect(out).To(HaveLen(2))
+			Expect(out[0].StringContent).To(Equal("You are LocalAI."))
+		})
+	})
+})

core/http/endpoints/openai/realtime.go

@@ -9,9 +9,11 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"maps"
 	"math"
 	"os"
 	"strconv"
+	"strings"
 	"sync"
 	"time"
 
@@ -31,6 +33,7 @@ import (
 	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
 	"github.com/mudler/LocalAI/core/schema"
 	"github.com/mudler/LocalAI/core/services/routing/router"
+	"github.com/mudler/LocalAI/core/services/voiceprofile"
 	"github.com/mudler/LocalAI/core/templates"
 	laudio "github.com/mudler/LocalAI/pkg/audio"
 	"github.com/mudler/LocalAI/pkg/functions"
@@ -135,6 +138,8 @@ type Session struct {
 	Instructions           string
 	DefaultConversationID  string
 	ModelInterface         Model
+	ttsParams              map[string]string
+	voiceRelease           func()
 	// The pipeline model config or the config for an any-to-any model
 	ModelConfig      *config.ModelConfig
 	InputSampleRate  int
@@ -198,6 +203,22 @@ type Session struct {
 	respSink *responseSink
 }
 
+func (s *Session) installVoiceBinding(voice string, params map[string]string, release func()) {
+	if release == nil {
+		release = func() {}
+	}
+	var once sync.Once
+	s.Voice = voice
+	s.ttsParams = maps.Clone(params)
+	s.voiceRelease = func() { once.Do(release) }
+}
+
+func (s *Session) releaseVoiceBinding() {
+	if s.voiceRelease != nil {
+		s.voiceRelease()
+	}
+}
+
 func (s *Session) FromClient(session *types.SessionUnion) {
 }
 
@@ -633,6 +654,17 @@ func runRealtimeSession(application *application.Application, t Transport, model
 		sendError(t, "model_load_error", "Failed to load model", "", "")
 		return
 	}
+	if wrapped, ok := m.(*wrappedModel); ok {
+		resolvedVoice, params, release, resolveErr := resolveRealtimeVoice(context.Background(), wrapped.TTSConfig.TTSConfig.Voice, wrapped.TTSConfig, application.VoiceProfileStore())
+		if resolveErr != nil {
+			xlog.Error("failed to resolve realtime voice", "error", resolveErr)
+			sendError(t, "voice_profile_error", resolveErr.Error(), "", "")
+			return
+		}
+		session.installVoiceBinding(resolvedVoice, params, release)
+		defer session.releaseVoiceBinding()
+		wrapped.setTTSParams(params)
+	}
 	session.ModelInterface = m
 	// A pipeline-seeded option list gets its scoring prompt prewarmed
 	// alongside the model warm-up below, so the session's first turn
@@ -826,6 +858,7 @@ func runRealtimeSession(application *application.Application, t Transport, model
 					application.ApplicationConfig(),
 					evaluator,
 					buildRealtimeRoutingContext(application, session.ID),
+					application.VoiceProfileStore(),
 				); err != nil {
 					xlog.Error("failed to update session", "error", err)
 					sendError(t, "session_update_error", fmt.Sprintf("Failed to update session: %v", err), "", "")
@@ -1164,7 +1197,7 @@ func updateTransSession(session *Session, update *types.SessionUnion, cl *config
 	return nil
 }
 
-func updateSession(session *Session, update *types.SessionUnion, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig, evaluator *templates.Evaluator, routing *RealtimeRoutingContext) error {
+func updateSession(session *Session, update *types.SessionUnion, cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig, evaluator *templates.Evaluator, routing *RealtimeRoutingContext, profiles *voiceprofile.Store) error {
 	sessionLock.Lock()
 	defer sessionLock.Unlock()
 
@@ -1172,8 +1205,13 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 		return nil
 	}
 
-	session.TranscriptionOnly = false
 	rt := update.Realtime
+	explicitVoice := rt.Audio != nil && rt.Audio.Output != nil && rt.Audio.Output.Voice != ""
+	rebuild := rt.Model != "" || explicitVoice || (rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.Transcription != nil)
+
+	candidateModelName := session.Model
+	candidateConfig := session.ModelConfig
+	candidateTranscription := session.InputAudioTranscription
 
 	if rt.Model != "" {
 		cfg, err := cl.LoadModelConfigFileByNameDefaultOptions(rt.Model, appConfig)
@@ -1184,40 +1222,78 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 			return fmt.Errorf("model is not a valid pipeline model: %s", rt.Model)
 		}
 
-		if session.InputAudioTranscription == nil {
-			session.InputAudioTranscription = &types.AudioTranscription{}
-		}
-		session.InputAudioTranscription.Model = cfg.Pipeline.Transcription
-		session.Voice = cfg.TTSConfig.Voice
-		session.Model = rt.Model
-		session.ModelConfig = cfg
-	}
-
-	if rt.Audio != nil && rt.Audio.Output != nil && rt.Audio.Output.Voice != "" {
-		session.Voice = string(rt.Audio.Output.Voice)
+		candidateModelName = rt.Model
+		candidateConfig = cfg
+		candidateTranscription = &types.AudioTranscription{Model: cfg.Pipeline.Transcription}
 	}
 
 	if rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.Transcription != nil {
-		trUpd := rt.Audio.Input.Transcription
+		trUpd := *rt.Audio.Input.Transcription
 		// A language-only update (e.g. a client forcing the STT language) carries
 		// an empty Model. Preserve the pipeline's configured transcription backend
 		// instead of blanking it — otherwise the next utterance transcribes against
 		// an empty model and the backend RPC fails with "unimplemented".
-		if trUpd.Model == "" && session.InputAudioTranscription != nil {
-			trUpd.Model = session.InputAudioTranscription.Model
+		if trUpd.Model == "" && candidateTranscription != nil {
+			trUpd.Model = candidateTranscription.Model
 		}
-		session.InputAudioTranscription = trUpd
+		candidateTranscription = &trUpd
 		if trUpd.Model != "" {
-			session.ModelConfig.Pipeline.Transcription = trUpd.Model
+			cfgCopy := *candidateConfig
+			candidateConfig = &cfgCopy
+			candidateConfig.Pipeline.Transcription = trUpd.Model
 		}
 	}
 
-	if rt.Model != "" || (rt.Audio != nil && rt.Audio.Output != nil && rt.Audio.Output.Voice != "") || (rt.Audio != nil && rt.Audio.Input != nil && rt.Audio.Input.Transcription != nil) {
-		m, err := newModel(&session.ModelConfig.Pipeline, cl, ml, appConfig, evaluator, routing)
+	candidateModel := session.ModelInterface
+	candidateVoice := session.Voice
+	candidateParams := maps.Clone(session.ttsParams)
+	var candidateRelease func()
+	selectVoice := rt.Model != "" || explicitVoice
+	if rebuild {
+		m, err := newModel(&candidateConfig.Pipeline, cl, ml, appConfig, evaluator, routing)
 		if err != nil {
 			return err
 		}
-		session.ModelInterface = m
+		candidateModel = m
+		wrapped := m.(*wrappedModel)
+		if selectVoice {
+			configuredVoice := wrapped.TTSConfig.TTSConfig.Voice
+			if explicitVoice {
+				configuredVoice = string(rt.Audio.Output.Voice)
+			}
+			candidateVoice, candidateParams, candidateRelease, err = resolveRealtimeVoice(
+				context.Background(), configuredVoice, wrapped.TTSConfig, profiles,
+			)
+			if err != nil {
+				return err
+			}
+		}
+		wrapped.setTTSParams(candidateParams)
+	}
+
+	if rt.LocalAIClassifier != nil {
+		if err := validateClassifierActivation(candidateModel, rt.LocalAIClassifier); err != nil {
+			if candidateRelease != nil {
+				candidateRelease()
+			}
+			return err
+		}
+	}
+
+	oldRelease := session.voiceRelease
+	session.TranscriptionOnly = false
+	session.Model = candidateModelName
+	session.ModelConfig = candidateConfig
+	session.ModelInterface = candidateModel
+	session.InputAudioTranscription = candidateTranscription
+	if selectVoice {
+		session.installVoiceBinding(candidateVoice, candidateParams, candidateRelease)
+		if oldRelease != nil {
+			oldRelease()
+		}
+	}
+
+	if rebuild {
 		// A session.update that swaps the model/voice rebuilds the pipeline, so
 		// warm the new backends too (unless opted out) — otherwise the next turn
 		// pays the cold-start load the original session warm-up already avoided.
@@ -1226,9 +1302,9 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 		// stall every other session. Load errors are logged (and still surface on
 		// first use); per-stage failures are already warned inside
 		// backend.PreloadStages.
-		if !session.ModelConfig.Pipeline.DisableWarmup {
+		if !candidateConfig.Pipeline.DisableWarmup {
 			go func() {
-				if err := m.Warmup(context.Background()); err != nil {
+				if err := candidateModel.Warmup(context.Background()); err != nil {
 					xlog.Error("realtime warmup failed after session.update", "error", err)
 				}
 			}()
@@ -1287,9 +1363,6 @@ func updateSession(session *Session, update *types.SessionUnion, cl *config.Mode
 		// Replace-not-merge, like tools: the client owns the whole option
 		// list. Invalid configs reject the update without touching the
 		// session's current classifier.
-		if err := validateClassifierActivation(session.ModelInterface, rt.LocalAIClassifier); err != nil {
-			return err
-		}
 		session.Classifier = rt.LocalAIClassifier
 		prewarmClassifier(session)
 	}
@@ -1923,7 +1996,7 @@ func commitUtteranceWithTranscript(ctx context.Context, utt []byte, live *liveUt
 	// Generate an LLM response only when there is a transcript to feed it. A
 	// sound-detection-only session (no transcription) has no LLM stage, so it
 	// stops here after emitting the sound-detection event.
-	if session.InputAudioTranscription != nil && !session.TranscriptionOnly {
+	if session.InputAudioTranscription != nil && !session.TranscriptionOnly && strings.TrimSpace(transcript) != "" {
 		generateResponse(ctx, session, utt, transcript, speaker, conv, t)
 	}
 }

core/http/endpoints/openai/realtime_model.go

@@ -6,7 +6,9 @@ import (
 	"encoding/binary"
 	"encoding/hex"
 	"encoding/json"
+	"errors"
 	"fmt"
+	"maps"
 	"strings"
 	"sync"
 	"time"
@@ -18,6 +20,7 @@ import (
 	"github.com/mudler/LocalAI/core/http/middleware"
 	"github.com/mudler/LocalAI/core/schema"
 	"github.com/mudler/LocalAI/core/services/routing/router"
+	"github.com/mudler/LocalAI/core/services/voiceprofile"
 	"github.com/mudler/LocalAI/core/templates"
 	"github.com/mudler/LocalAI/pkg/functions"
 	"github.com/mudler/LocalAI/pkg/grpc/proto"
@@ -35,6 +38,7 @@ var (
 // which are for Any-To-Any models, but instead we will call a pipeline (for e.g STT->LLM->TTS)
 type wrappedModel struct {
 	TTSConfig            *config.ModelConfig
+	ttsParams            map[string]string
 	TranscriptionConfig  *config.ModelConfig
 	LLMConfig            *config.ModelConfig
 	VADConfig            *config.ModelConfig
@@ -391,11 +395,39 @@ func newRealtimeDecisionID() string {
 }
 
 func (m *wrappedModel) TTS(ctx context.Context, text, voice, language string) (string, *proto.Result, error) {
-	return backend.ModelTTS(ctx, text, voice, language, "", nil, m.modelLoader, m.appConfig, *m.TTSConfig)
+	return backend.ModelTTS(ctx, text, voice, language, "", maps.Clone(m.ttsParams), m.modelLoader, m.appConfig, *m.TTSConfig)
+}
+
+func (m *wrappedModel) setTTSParams(params map[string]string) {
+	m.ttsParams = maps.Clone(params)
 }
 
 func (m *wrappedModel) TTSStream(ctx context.Context, text, voice, language string, onAudio func(pcm []byte, sampleRate int) error) error {
-	return ttsStream(ctx, m.modelLoader, m.appConfig, *m.TTSConfig, text, voice, language, onAudio)
+	return ttsStream(ctx, m.modelLoader, m.appConfig, *m.TTSConfig, text, voice, language, maps.Clone(m.ttsParams), onAudio)
+}
+
+func resolveRealtimeVoice(ctx context.Context, configuredVoice string, ttsConfig *config.ModelConfig, profiles *voiceprofile.Store) (string, map[string]string, func(), error) {
+	if !voiceprofile.IsReference(configuredVoice) {
+		return configuredVoice, nil, func() {}, nil
+	}
+	profileID, valid := voiceprofile.ParseReference(configuredVoice)
+	if !valid {
+		return "", nil, nil, fmt.Errorf("invalid voice profile reference %q", configuredVoice)
+	}
+	if config.VoiceCloningForModel(ttsConfig) == nil {
+		return "", nil, nil, fmt.Errorf("selected TTS model does not support reference-audio voice cloning")
+	}
+	if profiles == nil {
+		return "", nil, nil, fmt.Errorf("voice profile store is unavailable")
+	}
+	profile, referencePath, release, err := profiles.LeaseAudio(ctx, profileID)
+	if err != nil {
+		if errors.Is(err, voiceprofile.ErrNotFound) {
+			return "", nil, nil, fmt.Errorf("voice profile not found: %w", err)
+		}
+		return "", nil, nil, fmt.Errorf("resolve voice profile: %w", err)
+	}
+	return referencePath, map[string]string{"ref_text": profile.Transcript}, release, nil
 }
 
 func (m *wrappedModel) TranscribeStream(ctx context.Context, audio, language string, translate, diarize bool, prompt string, onDelta func(text string)) (*schema.TranscriptionResult, error) {
@@ -674,11 +706,11 @@ const wavStreamHeaderBytes = 44
 // callback, which wants raw PCM plus the sample rate. The header is buffered
 // until complete, the sample rate is read from it, and subsequent bytes are
 // forwarded as PCM.
-func ttsStream(ctx context.Context, ml *model.ModelLoader, appConfig *config.ApplicationConfig, ttsConfig config.ModelConfig, text, voice, language string, onAudio func(pcm []byte, sampleRate int) error) error {
+func ttsStream(ctx context.Context, ml *model.ModelLoader, appConfig *config.ApplicationConfig, ttsConfig config.ModelConfig, text, voice, language string, params map[string]string, onAudio func(pcm []byte, sampleRate int) error) error {
 	var header []byte
 	headerDone := false
 	sampleRate := 0
-	return backend.ModelTTSStream(ctx, text, voice, language, "", nil, ml, appConfig, ttsConfig, func(b []byte) error {
+	return backend.ModelTTSStream(ctx, text, voice, language, "", params, ml, appConfig, ttsConfig, func(b []byte) error {
 		if headerDone {
 			if len(b) == 0 {
 				return nil

core/http/endpoints/openai/realtime_semantic_vad_test.go

@@ -355,6 +355,19 @@ var _ = Describe("commitUtteranceWithTranscript", func() {
 
 		Expect(tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted)).To(Equal(1))
 	})
+
+	It("does not generate a response for a blank transcript", func() {
+		session, model := itSession(nil)
+		model.transcribeFinal = &schema.TranscriptionResult{Text: " \t\n"}
+		tr := &fakeTransport{}
+		conv := &Conversation{}
+
+		commitUtterance(context.Background(), []byte{1, 2}, session, conv, tr)
+
+		Expect(tr.countEvents(types.ServerEventTypeConversationItemInputAudioTranscriptionCompleted)).To(Equal(1))
+		Expect(conv.Items).To(BeEmpty())
+		Expect(tr.countEvents(types.ServerEventTypeResponseCreated)).To(Equal(0))
+	})
 })
 
 // transcribeUtterance is the retranscribe gate's offline decode of the

core/http/endpoints/openai/realtime_voice_profile_test.go

@@ -0,0 +1,354 @@
+package openai
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/mudler/LocalAI/core/http/endpoints/openai/types"
+	grpcPkg "github.com/mudler/LocalAI/pkg/grpc"
+	"github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/model"
+	"github.com/mudler/LocalAI/pkg/system"
+	"google.golang.org/grpc"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/services/voiceprofile"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func realtimeProfileWAV(duration time.Duration) []byte {
+	const (
+		sampleRate    = 16000
+		channels      = 1
+		bitsPerSample = 16
+	)
+	dataSize := int(duration.Seconds() * sampleRate * channels * bitsPerSample / 8)
+	buf := bytes.NewBuffer(nil)
+	buf.WriteString("RIFF")
+	_ = binary.Write(buf, binary.LittleEndian, uint32(36+dataSize))
+	buf.WriteString("WAVEfmt ")
+	_ = binary.Write(buf, binary.LittleEndian, uint32(16))
+	_ = binary.Write(buf, binary.LittleEndian, uint16(1))
+	_ = binary.Write(buf, binary.LittleEndian, uint16(channels))
+	_ = binary.Write(buf, binary.LittleEndian, uint32(sampleRate))
+	_ = binary.Write(buf, binary.LittleEndian, uint32(sampleRate*channels*bitsPerSample/8))
+	_ = binary.Write(buf, binary.LittleEndian, uint16(channels*bitsPerSample/8))
+	_ = binary.Write(buf, binary.LittleEndian, uint16(bitsPerSample))
+	buf.WriteString("data")
+	_ = binary.Write(buf, binary.LittleEndian, uint32(dataSize))
+	buf.Write(make([]byte, dataSize))
+	return buf.Bytes()
+}
+
+var _ = Describe("realtime pipeline voice profiles", func() {
+	It("resolves a saved profile to an immutable lease and transcript", func(ctx SpecContext) {
+		store := voiceprofile.NewStore(GinkgoT().TempDir())
+		DeferCleanup(func() { Expect(store.Close()).To(Succeed()) })
+		profile, err := store.Create(ctx, voiceprofile.CreateInput{
+			Name:             "Narrator",
+			Language:         "en-US",
+			Transcript:       "The reference transcript.",
+			ConsentConfirmed: true,
+		}, bytes.NewReader(realtimeProfileWAV(time.Second)))
+		Expect(err).NotTo(HaveOccurred())
+
+		voice, params, release, err := resolveRealtimeVoice(ctx, profile.Voice, &config.ModelConfig{
+			Name:      "clone-base",
+			Backend:   "qwen3-tts-cpp",
+			TTSConfig: config.TTSConfig{VoiceCloning: ptrTo(true)},
+		}, store)
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(voice).To(BeAnExistingFile())
+		Expect(params).To(Equal(map[string]string{"ref_text": "The reference transcript."}))
+		release()
+		release()
+		Expect(voice).NotTo(BeAnExistingFile())
+	})
+
+	It("leaves an ordinary backend voice unchanged with no parameters", func() {
+		voice, params, release, err := resolveRealtimeVoice(context.Background(), "speaker-7", &config.ModelConfig{}, nil)
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(voice).To(Equal("speaker-7"))
+		Expect(params).To(BeNil())
+		Expect(release).NotTo(BeNil())
+		Expect(func() { release(); release() }).NotTo(Panic())
+	})
+
+	DescribeTable("returns actionable reference errors",
+		func(configuredVoice string, cfg *config.ModelConfig, store *voiceprofile.Store, expected string) {
+			_, _, release, err := resolveRealtimeVoice(context.Background(), configuredVoice, cfg, store)
+			Expect(err).To(MatchError(ContainSubstring(expected)))
+			Expect(release).To(BeNil())
+		},
+		Entry("malformed reference", "localai://voice-profiles/not-a-uuid", &config.ModelConfig{}, nil, "invalid voice profile reference"),
+		Entry("unsupported model", "localai://voice-profiles/00000000-0000-0000-0000-000000000001", &config.ModelConfig{Backend: "piper"}, nil, "does not support reference-audio voice cloning"),
+		Entry("unavailable store", "localai://voice-profiles/00000000-0000-0000-0000-000000000001", &config.ModelConfig{Name: "clone-base", Backend: "qwen3-tts-cpp", TTSConfig: config.TTSConfig{VoiceCloning: ptrTo(true)}}, nil, "voice profile store is unavailable"),
+	)
+
+	It("reports a missing profile", func() {
+		store := voiceprofile.NewStore(GinkgoT().TempDir())
+		DeferCleanup(func() { Expect(store.Close()).To(Succeed()) })
+		_, _, release, err := resolveRealtimeVoice(context.Background(), "localai://voice-profiles/00000000-0000-0000-0000-000000000001", &config.ModelConfig{
+			Name: "clone-base", Backend: "qwen3-tts-cpp", TTSConfig: config.TTSConfig{VoiceCloning: ptrTo(true)},
+		}, store)
+		Expect(errors.Is(err, voiceprofile.ErrNotFound)).To(BeTrue())
+		Expect(err.Error()).To(ContainSubstring("voice profile not found"))
+		Expect(release).To(BeNil())
+	})
+})
+
+type recordingTTSBackend struct {
+	grpcPkg.Backend
+	requests []*proto.TTSRequest
+}
+
+func (b *recordingTTSBackend) HealthCheck(context.Context) (bool, error) { return true, nil }
+func (b *recordingTTSBackend) IsBusy() bool                              { return false }
+
+func (b *recordingTTSBackend) record(req *proto.TTSRequest) {
+	b.requests = append(b.requests, req)
+	req.Params["ref_text"] = "backend mutation"
+}
+
+func (b *recordingTTSBackend) TTS(_ context.Context, req *proto.TTSRequest, _ ...grpc.CallOption) (*proto.Result, error) {
+	b.record(req)
+	return &proto.Result{Success: true}, nil
+}
+
+func (b *recordingTTSBackend) TTSStream(_ context.Context, req *proto.TTSRequest, callback func(*proto.Reply), _ ...grpc.CallOption) error {
+	b.record(req)
+	header := make([]byte, wavStreamHeaderBytes)
+	binary.LittleEndian.PutUint32(header[24:28], 24000)
+	callback(&proto.Reply{Audio: header})
+	return nil
+}
+
+var _ = Describe("wrappedModel voice profile parameters", func() {
+	var (
+		wrapped         *wrappedModel
+		backendRecorder *recordingTTSBackend
+	)
+
+	BeforeEach(func() {
+		state, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
+		Expect(err).NotTo(HaveOccurred())
+		appConfig := config.NewApplicationConfig(config.WithSystemState(state))
+		appConfig.GeneratedContentDir = GinkgoT().TempDir()
+		loader := model.NewModelLoader(state)
+		backendRecorder = &recordingTTSBackend{}
+		cfg := &config.ModelConfig{Name: "tts-test", Backend: "test"}
+		cfg.Model = "weights"
+		loaded := model.NewModelWithClient(cfg.ModelID(), "in-process", backendRecorder)
+		loaded.MarkHealthy()
+		_, err = loader.LoadModel(cfg.ModelID(), cfg.Model, func(_, _, _ string) (*model.Model, error) { return loaded, nil })
+		Expect(err).NotTo(HaveOccurred())
+		wrapped = &wrappedModel{
+			TTSConfig:   cfg,
+			ttsParams:   map[string]string{"ref_text": "Original transcript"},
+			modelLoader: loader,
+			appConfig:   appConfig,
+		}
+	})
+
+	It("forwards a fresh transcript parameter map to every unary request", func() {
+		_, _, err := wrapped.TTS(context.Background(), "one", "voice.wav", "en")
+		Expect(err).NotTo(HaveOccurred())
+		_, _, err = wrapped.TTS(context.Background(), "two", "voice.wav", "en")
+		Expect(err).NotTo(HaveOccurred())
+
+		Expect(backendRecorder.requests).To(HaveLen(2))
+		Expect(backendRecorder.requests[0].Params).To(HaveKeyWithValue("ref_text", "backend mutation"))
+		Expect(backendRecorder.requests[1].Params).To(HaveKeyWithValue("ref_text", "backend mutation"))
+		Expect(wrapped.ttsParams).To(HaveKeyWithValue("ref_text", "Original transcript"))
+	})
+
+	It("forwards a copied transcript parameter map to streaming requests", func() {
+		err := wrapped.TTSStream(context.Background(), "one", "voice.wav", "en", func([]byte, int) error { return nil })
+		Expect(err).NotTo(HaveOccurred())
+
+		Expect(backendRecorder.requests).To(HaveLen(1))
+		Expect(backendRecorder.requests[0].Params).To(HaveKeyWithValue("ref_text", "backend mutation"))
+		Expect(wrapped.ttsParams).To(HaveKeyWithValue("ref_text", "Original transcript"))
+	})
+})
+
+var _ = Describe("realtime session voice switching", func() {
+	type fixture struct {
+		store     *voiceprofile.Store
+		voiceDir  string
+		loader    *config.ModelConfigLoader
+		models    *model.ModelLoader
+		appConfig *config.ApplicationConfig
+		profileA  voiceprofile.Profile
+		profileB  voiceprofile.Profile
+	}
+
+	newFixture := func(ctx SpecContext) *fixture {
+		modelDir := GinkgoT().TempDir()
+		voiceDir := GinkgoT().TempDir()
+		store := voiceprofile.NewStore(voiceDir)
+		DeferCleanup(func() { Expect(store.Close()).To(Succeed()) })
+		profileA, err := store.Create(ctx, voiceprofile.CreateInput{
+			Name: "Alpha", Language: "en", Transcript: "Alpha transcript", ConsentConfirmed: true,
+		}, bytes.NewReader(realtimeProfileWAV(time.Second)))
+		Expect(err).NotTo(HaveOccurred())
+		profileB, err := store.Create(ctx, voiceprofile.CreateInput{
+			Name: "Beta", Language: "it", Transcript: "Beta transcript", ConsentConfirmed: true,
+		}, bytes.NewReader(realtimeProfileWAV(time.Second)))
+		Expect(err).NotTo(HaveOccurred())
+
+		configs := map[string]string{
+			"vad":    "name: vad\nbackend: test\nparameters:\n  model: vad.bin\n",
+			"stt":    "name: stt\nbackend: test\nparameters:\n  model: stt.bin\n",
+			"llm":    "name: llm\nbackend: test\nparameters:\n  model: llm.bin\n",
+			"tts-a":  fmt.Sprintf("name: tts-a\nbackend: qwen3-tts-cpp\nparameters:\n  model: tts-a.bin\ntts:\n  voice: %s\n  voice_cloning: true\n", profileA.Voice),
+			"tts-b":  fmt.Sprintf("name: tts-b\nbackend: qwen3-tts-cpp\nparameters:\n  model: tts-b.bin\ntts:\n  voice: %s\n  voice_cloning: true\n", profileB.Voice),
+			"pipe-a": "name: pipe-a\npipeline:\n  vad: vad\n  transcription: stt\n  llm: llm\n  tts: tts-a\n  disable_warmup: true\n",
+			"pipe-b": "name: pipe-b\npipeline:\n  vad: vad\n  transcription: stt\n  llm: llm\n  tts: tts-b\n  disable_warmup: true\n",
+		}
+		for name, body := range configs {
+			Expect(os.WriteFile(filepath.Join(modelDir, name+".yaml"), []byte(body), 0o644)).To(Succeed())
+		}
+		loader := config.NewModelConfigLoader(modelDir)
+		Expect(loader.LoadModelConfigsFromPath(modelDir)).To(Succeed())
+		state, err := system.GetSystemState(system.WithModelPath(modelDir))
+		Expect(err).NotTo(HaveOccurred())
+		return &fixture{
+			store: store, voiceDir: voiceDir, loader: loader, models: model.NewModelLoader(state),
+			appConfig: config.NewApplicationConfig(config.WithSystemState(state)),
+			profileA:  profileA, profileB: profileB,
+		}
+	}
+
+	newSession := func(f *fixture, voice string) *Session {
+		cfg, err := f.loader.LoadModelConfigFileByNameDefaultOptions("pipe-a", f.appConfig)
+		Expect(err).NotTo(HaveOccurred())
+		m, err := newModel(&cfg.Pipeline, f.loader, f.models, f.appConfig, nil, nil)
+		Expect(err).NotTo(HaveOccurred())
+		session := &Session{
+			Model: "pipe-a", Voice: voice, ModelConfig: cfg, ModelInterface: m,
+			InputAudioTranscription: &types.AudioTranscription{Model: "stt"},
+		}
+		resolved, params, release, err := resolveRealtimeVoice(context.Background(), voice, m.(*wrappedModel).TTSConfig, f.store)
+		Expect(err).NotTo(HaveOccurred())
+		session.installVoiceBinding(resolved, params, release)
+		return session
+	}
+
+	update := func(f *fixture, session *Session, rt *types.RealtimeSession) error {
+		return updateSession(session, &types.SessionUnion{Realtime: rt}, f.loader, f.models, f.appConfig, nil, nil, f.store)
+	}
+
+	It("switches ordinary voices to profiles and clears the lease at final cleanup", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, "speaker-1")
+		Expect(update(f, session, &types.RealtimeSession{Audio: &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: types.Voice(f.profileA.Voice)}}})).To(Succeed())
+
+		Expect(session.Voice).To(BeAnExistingFile())
+		Expect(session.ttsParams).To(Equal(map[string]string{"ref_text": "Alpha transcript"}))
+		leased := session.Voice
+		session.releaseVoiceBinding()
+		session.releaseVoiceBinding()
+		Expect(leased).NotTo(BeAnExistingFile())
+	})
+
+	It("replaces one profile lease with another and then an ordinary voice", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, f.profileA.Voice)
+		firstLease := session.Voice
+
+		Expect(update(f, session, &types.RealtimeSession{Audio: &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: types.Voice(f.profileB.Voice)}}})).To(Succeed())
+		Expect(firstLease).NotTo(BeAnExistingFile())
+		secondLease := session.Voice
+		Expect(secondLease).To(BeAnExistingFile())
+		Expect(session.ttsParams).To(HaveKeyWithValue("ref_text", "Beta transcript"))
+
+		Expect(update(f, session, &types.RealtimeSession{Audio: &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: "speaker-2"}}})).To(Succeed())
+		Expect(secondLease).NotTo(BeAnExistingFile())
+		Expect(session.Voice).To(Equal("speaker-2"))
+		Expect(session.ttsParams).To(BeNil())
+		Expect(session.ModelInterface.(*wrappedModel).ttsParams).To(BeNil())
+	})
+
+	It("uses a new model default profile unless an explicit voice takes precedence", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, "speaker-1")
+		Expect(update(f, session, &types.RealtimeSession{Model: "pipe-b"})).To(Succeed())
+		Expect(session.ttsParams).To(HaveKeyWithValue("ref_text", "Beta transcript"))
+		defaultLease := session.Voice
+
+		Expect(update(f, session, &types.RealtimeSession{
+			Model: "pipe-a",
+			Audio: &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: "speaker-explicit"}},
+		})).To(Succeed())
+		Expect(defaultLease).NotTo(BeAnExistingFile())
+		Expect(session.Voice).To(Equal("speaker-explicit"))
+		Expect(session.ttsParams).To(BeNil())
+	})
+
+	It("preserves a profile binding across a language-only rebuild", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, f.profileA.Voice)
+		lease := session.Voice
+		Expect(update(f, session, &types.RealtimeSession{Audio: &types.RealtimeSessionAudio{Input: &types.SessionAudioInput{
+			Transcription: &types.AudioTranscription{Language: "fr"},
+		}}})).To(Succeed())
+
+		Expect(session.Voice).To(Equal(lease))
+		Expect(session.InputAudioTranscription.Model).To(Equal("stt"))
+		Expect(session.InputAudioTranscription.Language).To(Equal("fr"))
+		wrapped := session.ModelInterface.(*wrappedModel)
+		Expect(wrapped.ttsParams).To(Equal(map[string]string{"ref_text": "Alpha transcript"}))
+		wrapped.ttsParams["ref_text"] = "wrapper mutation"
+		Expect(session.ttsParams).To(HaveKeyWithValue("ref_text", "Alpha transcript"))
+	})
+
+	It("rolls back the model, wrapper, voice, and lease when preparation fails", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, f.profileA.Voice)
+		oldModel, oldConfig, oldVoice := session.ModelInterface, session.ModelConfig, session.Voice
+		err := update(f, session, &types.RealtimeSession{
+			Model: "pipe-b",
+			Audio: &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: "localai://voice-profiles/00000000-0000-0000-0000-000000000001"}},
+		})
+
+		Expect(err).To(HaveOccurred())
+		Expect(session.Model).To(Equal("pipe-a"))
+		Expect(session.ModelConfig).To(BeIdenticalTo(oldConfig))
+		Expect(session.ModelInterface).To(BeIdenticalTo(oldModel))
+		Expect(session.Voice).To(Equal(oldVoice))
+		Expect(oldVoice).To(BeAnExistingFile())
+		Expect(session.ttsParams).To(HaveKeyWithValue("ref_text", "Alpha transcript"))
+	})
+
+	It("releases a candidate profile lease when later validation fails", func(ctx SpecContext) {
+		f := newFixture(ctx)
+		session := newSession(f, "speaker-1")
+		leases := func() []string {
+			matches, err := filepath.Glob(filepath.Join(f.voiceDir, voiceprofile.DirectoryName, ".leases", "*", "*.wav"))
+			Expect(err).NotTo(HaveOccurred())
+			return matches
+		}
+		Expect(leases()).To(BeEmpty())
+
+		err := update(f, session, &types.RealtimeSession{
+			Audio:             &types.RealtimeSessionAudio{Output: &types.SessionAudioOutput{Voice: types.Voice(f.profileA.Voice)}},
+			LocalAIClassifier: classifierTestConfig(0, nil),
+		})
+
+		Expect(err).To(HaveOccurred())
+		Expect(session.Voice).To(Equal("speaker-1"))
+		Expect(leases()).To(BeEmpty())
+	})
+})
+
+func ptrTo[T any](value T) *T { return &value }

core/http/react-ui/bun.lock

@@ -53,6 +53,7 @@
   "overrides": {
     "hono": "4.12.34",
     "ip-address": "10.3.1",
+    "path-to-regexp": "^8.4.0",
   },
   "packages": {
     "@babel/code-frame": ["@babel/code-frame@7.29.0", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.28.5", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw=="],
@@ -807,7 +808,7 @@
 
     "path-scurry": ["path-scurry@2.0.2", "", { "dependencies": { "lru-cache": "^11.0.0", "minipass": "^7.1.2" } }, "sha512-3O/iVVsJAPsOnpwWIeD+d6z/7PmqApyQePUtCndjatj/9I5LylHvt5qluFaBT3I5h3r1ejfR056c+FCv+NnNXg=="],
 
-    "path-to-regexp": ["path-to-regexp@8.3.0", "", {}, "sha512-7jdwVIRtsP8MYpdXSwOS0YdD0Du+qOoF/AEPIt88PcCFrZCzx41oxku1jD88hZBwbNUIEfpqvuhjFaMAqMTWnA=="],
+    "path-to-regexp": ["path-to-regexp@8.4.0", "", {}, "sha512-PuseHIvAnz3bjrM2rGJtSgo1zjgxapTLZ7x2pjhzWwlp4SJQgK3f3iZIQwkpEnBaKz6seKBADpM4B4ySkuYypg=="],
 
     "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="],
 

core/http/react-ui/e2e/collections.spec.js

@@ -19,4 +19,33 @@ test.describe('Collections page', () => {
     await input.fill('my-kb')
     await expect(input).toHaveValue('my-kb')
   })
+
+  test('posts the source update interval as a JSON number', async ({ page }) => {
+    const collectionName = 'interval-regression'
+    const collectionPath = encodeURIComponent(collectionName)
+    let postedBody
+
+    await page.route(`**/api/agents/collections/${collectionPath}/entries`, route =>
+      route.fulfill({ contentType: 'application/json', body: JSON.stringify({ entries: [] }) }))
+    await page.route(`**/api/agents/collections/${collectionPath}/sources`, async route => {
+      if (route.request().method() === 'POST') {
+        postedBody = route.request().postDataJSON()
+        await route.fulfill({ contentType: 'application/json', body: JSON.stringify({ status: 'ok' }) })
+      } else {
+        await route.fulfill({ contentType: 'application/json', body: JSON.stringify({ sources: [] }) })
+      }
+    })
+
+    await page.goto(`/app/collections/${collectionPath}`)
+    await page.getByRole('button', { name: 'Sources' }).click()
+    await page.locator('#source-url').fill('https://example.com/feed')
+    await page.locator('#source-interval').fill('3600')
+    await page.getByRole('button', { name: 'Add Source' }).click()
+
+    await expect.poll(() => postedBody).toEqual({
+      url: 'https://example.com/feed',
+      update_interval: 3600,
+    })
+    expect(typeof postedBody.update_interval).toBe('number')
+  })
 })

core/http/react-ui/e2e/models-recommended-panel.spec.js

@@ -103,3 +103,48 @@ test.describe("Models gallery - recommended panel prominence", () => {
     await expect(grid(page).locator(".lane__tag--evidence")).toHaveCount(1);
   });
 });
+
+// Start with a fitting model so absence assertions cannot pass during loading.
+// Then change the polled hardware budget while keeping the same gallery.
+for (const view of ["models", "home"]) {
+  test(`${view} removes GPU recommendations when no candidate fits`, async ({ page }) => {
+    await mockGallery(page, 0);
+    await page.route("**/v1/models", (route) =>
+      route.fulfill({ json: { data: [] } }),
+    );
+    const gib = 1024 ** 3;
+    let budget = 24 * gib;
+    await page.route("**/api/resources", (route) =>
+      route.fulfill({ json: {
+        type: "gpu",
+        aggregate: { total_memory: budget, gpu_count: 1 },
+        gpus: [{ vendor: "nvidia", total_memory: budget }],
+      } }),
+    );
+    await page.route("**/api/models/estimate/*", (route) =>
+      route.fulfill({ json: {
+        sizeBytes: 17.4 * gib,
+        sizeDisplay: "17.4 GB",
+        estimates: { 4096: { vramBytes: 18.4 * gib, vramDisplay: "18.4 GB" } },
+      } }),
+    );
+    await page.goto(view === "models" ? "/app/models" : "/app/");
+    const section = view === "models" ? panel(page) : page.locator(".home-starters");
+    await expect(section).toBeVisible();
+    await expect(section).toContainText("tiny-chat");
+
+    // Wait for BOTH recommendation estimates, not the hook's loading render
+    // or the gallery rail's separate context-size requests.
+    const estimatesFinished = REC_MODELS.map(model => page.waitForResponse(response => {
+      const url = new URL(response.url());
+      return url.pathname.endsWith('/api/models/estimate/' + model.name) &&
+        url.searchParams.get('contexts') === '4096' && response.status() === 200;
+    }).then(response => response.finished()));
+    budget = 12 * gib;
+    await Promise.all(estimatesFinished);
+    await page.evaluate(() => new Promise(resolve =>
+      requestAnimationFrame(() => requestAnimationFrame(resolve)),
+    ));
+    await expect(section).toHaveCount(0, { timeout: 15_000 });
+  });
+}

core/http/react-ui/e2e/traces-pagination.spec.js

@@ -79,4 +79,30 @@ test.describe('Traces - bounded list and on-demand detail', () => {
     await expect(page.locator('text=hello from the response body')).toBeVisible()
     await expect(page.locator('text=203.0.113.9').first()).toBeVisible()
   })
+
+  test('keeps the expanded trace open when a refresh prepends a new row', async ({ page }) => {
+    await page.locator('tr', { hasText: '/v1/chat/completions' }).first().click()
+    await expect(page.locator('text=hello from the request body')).toBeVisible()
+
+    await page.route('**/api/traces?*', (route) => {
+      route.fulfill({
+        contentType: 'application/json',
+        headers: { 'X-Total-Count': '843' },
+        body: JSON.stringify([
+          {
+            id: '8',
+            request: { method: 'GET', path: '/v1/models', body: null },
+            response: { status: 200, body: null },
+          },
+          ...LIST_BODY,
+        ]),
+      })
+    })
+
+    await page.getByRole('button', { name: 'Refresh' }).click()
+
+    await expect(page.locator('text=hello from the request body')).toBeVisible()
+    const originalRow = page.locator('tr', { hasText: '/v1/chat/completions' }).first()
+    await expect(originalRow.locator('i.fa-chevron-down')).toBeVisible()
+  })
 })

core/http/react-ui/package-lock.json

@@ -24,7 +24,7 @@
         "@modelcontextprotocol/sdk": "^1.30.0",
         "dompurify": "^3.4.13",
         "highlight.js": "^11.11.1",
-        "hono": "4.12.34",
+        "hono": "4.13.5",
         "i18next": "^26.0.8",
         "i18next-browser-languagedetector": "^8.2.1",
         "i18next-http-backend": "^3.0.6",
@@ -32,7 +32,8 @@
         "react": "^19.1.0",
         "react-dom": "^19.1.0",
         "react-i18next": "^17.0.6",
-        "react-router-dom": "^7.18.1",
+        "react-router": "8.3.1",
+        "react-router-dom": "7.18.2",
         "yaml": "^2.8.3"
       },
       "devDependencies": {
@@ -648,27 +649,43 @@
       }
     },
     "node_modules/@humanfs/core": {
-      "version": "0.19.1",
-      "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.1.tgz",
-      "integrity": "sha512-5DyQ4+1JEUzejeK1JGICcideyfUbGixgS9jNgex5nqkW+cY7WZhxBigmieN5Qnw9ZosSNVC9KQKyb+GUaGyKUA==",
+      "version": "0.19.2",
+      "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.2.tgz",
+      "integrity": "sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA==",
       "dev": true,
+      "license": "Apache-2.0",
+      "dependencies": {
+        "@humanfs/types": "^0.15.0"
+      },
       "engines": {
         "node": ">=18.18.0"
       }
     },
     "node_modules/@humanfs/node": {
-      "version": "0.16.7",
-      "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.7.tgz",
-      "integrity": "sha512-/zUx+yOsIrG4Y43Eh2peDeKCxlRt/gET6aHfaKpuq267qXdYDFViVHfMaLyygZOnl0kGWxFIgsBy8QFuTLUXEQ==",
+      "version": "0.16.8",
+      "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.8.tgz",
+      "integrity": "sha512-gE1eQNZ3R++kTzFUpdGlpmy8kDZD/MLyHqDwqjkVQI0JMdI1D51sy1H958PNXYkM2rAac7e5/CnIKZrHtPh3BQ==",
       "dev": true,
+      "license": "Apache-2.0",
       "dependencies": {
-        "@humanfs/core": "^0.19.1",
+        "@humanfs/core": "^0.19.2",
+        "@humanfs/types": "^0.15.0",
         "@humanwhocodes/retry": "^0.4.0"
       },
       "engines": {
         "node": ">=18.18.0"
       }
     },
+    "node_modules/@humanfs/types": {
+      "version": "0.15.0",
+      "resolved": "https://registry.npmjs.org/@humanfs/types/-/types-0.15.0.tgz",
+      "integrity": "sha512-ZZ1w0aoQkwuUuC7Yf+7sdeaNfqQiiLcSRbfI08oAxqLtpXQr9AIVX7Ay7HLDuiLYAaFPu8oBYNq/QIi9URHJ3Q==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "engines": {
+        "node": ">=18.18.0"
+      }
+    },
     "node_modules/@humanwhocodes/module-importer": {
       "version": "1.0.1",
       "resolved": "https://registry.npmjs.org/@humanwhocodes/module-importer/-/module-importer-1.0.1.tgz",
@@ -754,9 +771,9 @@
       }
     },
     "node_modules/@istanbuljs/load-nyc-config/node_modules/js-yaml": {
-      "version": "3.14.2",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.14.2.tgz",
-      "integrity": "sha512-PMSmkqxr106Xa156c2M265Z+FTrPl+oxd/rgOQy2tijQeK5TxQ43psO1ZCwhVOSdnn+RzkzlRz/eY4BgJBYVpg==",
+      "version": "3.15.2",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.15.2.tgz",
+      "integrity": "sha512-6EuL879VkRA+1Cz578mKMiKvjPNEuk6+r1JaFzoSWejZmtf7xWbIyw1e3KkxlkzTIt9Taw6JBhEppG7utc1P+w==",
       "dev": true,
       "license": "MIT",
       "dependencies": {
@@ -1639,9 +1656,9 @@
       ]
     },
     "node_modules/baseline-browser-mapping": {
-      "version": "2.10.38",
-      "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.38.tgz",
-      "integrity": "sha512-31/02mVB4yuQU6adKk5SlY6m+mxDwUq5KZkyYgnLrrKl7TEm1+3PyDtDBz2kOv/wxZz41GHsvV1A/u6RmiyBvw==",
+      "version": "2.11.20",
+      "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.20.tgz",
+      "integrity": "sha512-H0ulySigv6icDJ1F7SjtdCD6PrhTpdYCmP0CactWy1+ekh0AFd0o1Wn5T8b+hnTmdBx19u9yhL6wvCylXMY7zw==",
       "dev": true,
       "license": "Apache-2.0",
       "bin": {
@@ -1810,9 +1827,9 @@
       }
     },
     "node_modules/browserslist": {
-      "version": "4.28.2",
-      "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.2.tgz",
-      "integrity": "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg==",
+      "version": "4.28.8",
+      "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.8.tgz",
+      "integrity": "sha512-V2NpofLblG64mfOtSgDhOJESZEGogzDMBv/q+W6oc4LXWP/q75eOXoOaaOu1EOadB9U4Bwx/e0yzbvwKH8zalA==",
       "dev": true,
       "funding": [
         {
@@ -1830,11 +1847,11 @@
       ],
       "license": "MIT",
       "dependencies": {
-        "baseline-browser-mapping": "^2.10.12",
-        "caniuse-lite": "^1.0.30001782",
-        "electron-to-chromium": "^1.5.328",
-        "node-releases": "^2.0.36",
-        "update-browserslist-db": "^1.2.3"
+        "baseline-browser-mapping": "^2.11.12",
+        "caniuse-lite": "^1.0.30001809",
+        "electron-to-chromium": "^1.5.402",
+        "node-releases": "^2.0.53",
+        "update-browserslist-db": "^1.3.0"
       },
       "bin": {
         "browserslist": "cli.js"
@@ -1940,9 +1957,9 @@
       }
     },
     "node_modules/caniuse-lite": {
-      "version": "1.0.30001799",
-      "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001799.tgz",
-      "integrity": "sha512-hG1bReV+OUU+MOqK4t/ZWI0tZOyz3rqS9XuhOUz1cIcbwBKjOyJEJuw9ER5JuNyqxNk8u/JUVbGibBOL1yrjFw==",
+      "version": "1.0.30001810",
+      "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001810.tgz",
+      "integrity": "sha512-TITQPUkaz+aVk5GL6NhOdwk1aEaNTSDPsGFWrTuhKGtjTF70jL/Oht2W4c6rXUe5fu7Ie19VIahAXHIIiWWNeg==",
       "dev": true,
       "funding": [
         {
@@ -2197,6 +2214,12 @@
         "url": "https://opencollective.com/express"
       }
     },
+    "node_modules/cookie-es": {
+      "version": "3.1.1",
+      "resolved": "https://registry.npmjs.org/cookie-es/-/cookie-es-3.1.1.tgz",
+      "integrity": "sha512-UaXxwISYJPTr9hwQxMFYZ7kNhSXboMXP+Z3TRX6f1/NyaGPfuNUZOWP1pUEb75B2HjfklIYLVRfWiFZJyC6Npg==",
+      "license": "MIT"
+    },
     "node_modules/cookie-signature": {
       "version": "1.2.2",
       "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz",
@@ -2437,9 +2460,9 @@
       "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow=="
     },
     "node_modules/electron-to-chromium": {
-      "version": "1.5.375",
-      "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.375.tgz",
-      "integrity": "sha512-ZWP5eB4BVPW/ZYo9252hQZHZ5XavtsTgpbhcmMmRwymavC5AsLWQWBPaKMeNd2LW0KGby5HPXvj7+sr4ta5j/Q==",
+      "version": "1.5.420",
+      "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.420.tgz",
+      "integrity": "sha512-2yD6XreGusOfNV+dUcvipJEXc3n/n7fgr7996aszTG+YY5E4mqM4tOq/3uhP129cazL9YHbVWSpc79ePotWtPA==",
       "dev": true,
       "license": "ISC"
     },
@@ -2847,6 +2870,15 @@
         "express": ">= 4.11"
       }
     },
+    "node_modules/express-rate-limit/node_modules/ip-address": {
+      "version": "10.3.1",
+      "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.3.1.tgz",
+      "integrity": "sha512-1e9d3kb97NHJTIJDZW9rKqW2h6+dFa50Dy0fpPSMQp2ADje5gvKsXmdiK6dwY5t76TaTt5+P5N1Y/LoToIxP6g==",
+      "license": "MIT",
+      "engines": {
+        "node": ">= 12"
+      }
+    },
     "node_modules/express/node_modules/cookie": {
       "version": "0.7.2",
       "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
@@ -2879,9 +2911,9 @@
       "dev": true
     },
     "node_modules/fast-uri": {
-      "version": "3.1.5",
-      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.5.tgz",
-      "integrity": "sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==",
+      "version": "3.1.7",
+      "resolved": "https://registry.npmjs.org/fast-uri/-/fast-uri-3.1.7.tgz",
+      "integrity": "sha512-dOvZVzjdZdz7phd9v6jCbwxrBW3fK6n8Rc0CtdmM4bumzMnxywBYhuph6J819RRw/ku+rLbelwfMunktuzVVHg==",
       "funding": [
         {
           "type": "github",
@@ -3435,9 +3467,9 @@
       }
     },
     "node_modules/hono": {
-      "version": "4.12.34",
-      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.34.tgz",
-      "integrity": "sha512-GqXJqY/xJkJmuloTrnV1ZEXG3fqte+VjkUqoRNZXcrUidiUOP4fMSIHHY4tsqZBK++kVyWmt/AAfSUuy57/eSA==",
+      "version": "4.13.5",
+      "resolved": "https://registry.npmjs.org/hono/-/hono-4.13.5.tgz",
+      "integrity": "sha512-O6+/eCYRkzzzy0rPWwKLiGBR1nFuUPZynnwjxN1MBA62NNqbT0wQEzQyK2gSO5yDIDB336sXQleAhOHrzlYyKw==",
       "license": "MIT",
       "engines": {
         "node": ">=16.9.0"
@@ -4195,15 +4227,6 @@
       "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
       "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="
     },
-    "node_modules/ip-address": {
-      "version": "10.4.0",
-      "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.4.0.tgz",
-      "integrity": "sha512-oSK96Grm3aP6OrS263xVxbNDGVL7rzBtYdpGqlDG8iQdoenDoTs/nkki+DflYbAEE8Xl6o5YxhxlrKvI3nqKXQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 12"
-      }
-    },
     "node_modules/ipaddr.js": {
       "version": "1.9.1",
       "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
@@ -4570,10 +4593,21 @@
       "license": "MIT"
     },
     "node_modules/js-yaml": {
-      "version": "4.1.1",
-      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz",
-      "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==",
+      "version": "4.3.2",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.3.2.tgz",
+      "integrity": "sha512-SFNOvSJ+Dgf/9An904Yx+CgSlIPCkIpao4qo51lpee25TIRejdH3rhR4EZMGoNx3/TP3O+wzWuiTFl4sqbltzA==",
       "dev": true,
+      "funding": [
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/puzrin"
+        },
+        {
+          "type": "github",
+          "url": "https://github.com/sponsors/nodeca"
+        }
+      ],
+      "license": "MIT",
       "dependencies": {
         "argparse": "^2.0.1"
       },
@@ -5186,9 +5220,9 @@
       }
     },
     "node_modules/node-releases": {
-      "version": "2.0.48",
-      "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.48.tgz",
-      "integrity": "sha512-1uz8041X6LoI6ZSdZacM9lVY28vuzDlSKitnpbSNK0RfKoIJkX29NBPVEFXhnuSuEOA9Ww0xnPJ+ILWbGAv8DA==",
+      "version": "2.0.54",
+      "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.54.tgz",
+      "integrity": "sha512-YHs7BmmcsdAI5Ozuf8JZo6PT0mv2GIWC9vMfvUC3dp65M8hn7Ux8CPL+2oBI7juNuj9d0ndhTcznq2ODBps9cQ==",
       "dev": true,
       "license": "MIT",
       "engines": {
@@ -5950,9 +5984,9 @@
       }
     },
     "node_modules/qs": {
-      "version": "6.15.3",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
-      "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
+      "version": "6.16.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
+      "integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",
       "license": "BSD-3-Clause",
       "dependencies": {
         "es-define-property": "^1.0.1",
@@ -6060,22 +6094,24 @@
       }
     },
     "node_modules/react": {
-      "version": "19.2.4",
-      "resolved": "https://registry.npmjs.org/react/-/react-19.2.4.tgz",
-      "integrity": "sha512-9nfp2hYpCwOjAN+8TZFGhtWEwgvWHXqESH8qT89AT/lWklpLON22Lc8pEtnpsZz7VmawabSU0gCjnj8aC0euHQ==",
+      "version": "19.2.8",
+      "resolved": "https://registry.npmjs.org/react/-/react-19.2.8.tgz",
+      "integrity": "sha512-PWaYA1L/q9u2u7xYQi+Y3L3Yfnie7XyLeaJICV1MGD6LprsBxcAqGjYyr0eY3p+QdsA+x/Irkt4Qif8D63+Sbw==",
+      "license": "MIT",
       "engines": {
         "node": ">=0.10.0"
       }
     },
     "node_modules/react-dom": {
-      "version": "19.2.4",
-      "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.4.tgz",
-      "integrity": "sha512-AXJdLo8kgMbimY95O2aKQqsz2iWi9jMgKJhRBAxECE4IFxfcazB2LmzloIoibJI3C12IlY20+KFaLv+71bUJeQ==",
+      "version": "19.2.8",
+      "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.8.tgz",
+      "integrity": "sha512-rVprimfGBG3DR+Tq0IQG2DT5PxKth1WIGDmj5yPmlzr4YBe7uyE+Du4oVqTDXZSHGGGXRtTJEGSSePyQCMBglQ==",
+      "license": "MIT",
       "dependencies": {
         "scheduler": "^0.27.0"
       },
       "peerDependencies": {
-        "react": "^19.2.4"
+        "react": "^19.2.8"
       }
     },
     "node_modules/react-i18next": {
@@ -6105,9 +6141,46 @@
       }
     },
     "node_modules/react-router": {
-      "version": "7.18.1",
-      "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.18.1.tgz",
-      "integrity": "sha512-GDLgg3i3uM0aeJO3Fm+TCS+sDQ7gu12T6x0qdTEzcwqEfleci7JwugVNIF3U//0FWKnJT7ptG+20B2jfDqnZAg==",
+      "version": "8.3.1",
+      "resolved": "https://registry.npmjs.org/react-router/-/react-router-8.3.1.tgz",
+      "integrity": "sha512-TEOpiO2g0TJHEOJeRVv4amUFun9v1npCKszvcquNvzETUtJ8udV86ah5eFoHT7g26bsBvT6EiIhqulR8eDF++A==",
+      "license": "MIT",
+      "dependencies": {
+        "cookie-es": "^3.1.1"
+      },
+      "engines": {
+        "node": ">=22.22.0"
+      },
+      "peerDependencies": {
+        "react": ">=19.2.7",
+        "react-dom": ">=19.2.7"
+      },
+      "peerDependenciesMeta": {
+        "react-dom": {
+          "optional": true
+        }
+      }
+    },
+    "node_modules/react-router-dom": {
+      "version": "7.18.2",
+      "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.18.2.tgz",
+      "integrity": "sha512-AIKJ/jgGlFb3EbfCXk5Gzshiwt+l3mqbCrNjmEWMMjqQxNJ3svBa6bgzFyCC2Sw3RA0VWF1kg3uQf2OFhxb8hw==",
+      "license": "MIT",
+      "dependencies": {
+        "react-router": "7.18.2"
+      },
+      "engines": {
+        "node": ">=20.0.0"
+      },
+      "peerDependencies": {
+        "react": ">=18",
+        "react-dom": ">=18"
+      }
+    },
+    "node_modules/react-router-dom/node_modules/react-router": {
+      "version": "7.18.2",
+      "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.18.2.tgz",
+      "integrity": "sha512-aUVMjFm3GAPTTZL7oYr5E7ETiqfQCHRLH+B+5afnICvf0r7kkK4eR6SMuwbSTJw/7t+12khT/Kahij49fqOCIg==",
       "license": "MIT",
       "dependencies": {
         "cookie": "^1.0.1",
@@ -6126,22 +6199,6 @@
         }
       }
     },
-    "node_modules/react-router-dom": {
-      "version": "7.18.1",
-      "resolved": "https://registry.npmjs.org/react-router-dom/-/react-router-dom-7.18.1.tgz",
-      "integrity": "sha512-KaZh+X/6UtEp28x51AUYZDMg9NGoz2ja3dNHa+ta/tk40vCzKhQ/RypCWBMLbmDr6//E24Vv5uPsrqXFozdkAg==",
-      "license": "MIT",
-      "dependencies": {
-        "react-router": "7.18.1"
-      },
-      "engines": {
-        "node": ">=20.0.0"
-      },
-      "peerDependencies": {
-        "react": ">=18",
-        "react-dom": ">=18"
-      }
-    },
     "node_modules/readable-stream": {
       "version": "2.3.8",
       "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz",
@@ -7165,9 +7222,9 @@
       }
     },
     "node_modules/update-browserslist-db": {
-      "version": "1.2.3",
-      "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz",
-      "integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==",
+      "version": "1.3.2",
+      "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.3.2.tgz",
+      "integrity": "sha512-UQ+MSxlhRm1bzjhU+DcuXfjFO1FzNtqhK5+9Yvlp90ItDLk5vT932A0rFu619nf7RVS+Y/VeaUW1jaRDqZ8VJw==",
       "dev": true,
       "funding": [
         {