feat(gallery): add GLM-5.3 Flash variants

Add verified Q4 and Q8 GGUF builds for the new multimodal GLM-5.3 Flash model. Group the quantizations so LocalAI can select the highest-quality build that fits.

Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/ai-coding-assistants.md

@@ -49,40 +49,6 @@ AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
 A human reviewer owns the contribution; the AI's involvement is recorded
 via `Assisted-by` (see below).
 
-### Exception: automation operated by a maintainer
-
-The rule above addresses the common case, an AI assistant helping a human
-contributor who then signs off. It does not fit automation that a
-maintainer runs themselves, which opens pull requests with no human
-submitter to sign. Applied literally there, nothing ever signs and the
-DCO check blocks the pull request permanently.
-
-A maintainer-operated bot MUST therefore add a `Signed-off-by` trailer
-naming **the maintainer who operates it**, not the bot and not the model:
-
-```
-Assisted-by: Codex:gpt-5
-Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
-```
-
-This is not the AI certifying the DCO. The maintainer is, exactly as they
-do for a commit they typed by hand: they configured the automation, they
-own its output, and they take responsibility for it when they merge it.
-The `Assisted-by` trailer still records that a model produced the code, so
-the provenance trail is unchanged.
-
-The exception is narrow and does not widen the rule for anyone else:
-
-- It applies only to automation a LocalAI maintainer operates and whose
-  output that maintainer reviews before merge.
-- The sign-off names a real person who accepts DCO responsibility.
-- An AI assistant helping an outside contributor still MUST NOT sign off.
-  That contributor adds their own trailer.
-- A bot MUST NOT sign off on behalf of anyone other than its operator, and
-  MUST NOT add a trailer for a contributor whose branch it pushes to. If
-  automation contributes to someone else's branch, it leaves the sign-off
-  to that contributor.
-
 ## Attribution
 
 When AI tools contribute to LocalAI development, proper attribution helps

.agents/api-endpoints-and-auth.md

@@ -236,58 +236,6 @@ Use these HTTP status codes:
 
 If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
 
-## Control-plane database health metrics
-
-In distributed mode the frontend registers three OpenTelemetry gauges over the
-PostgreSQL control-plane database (`core/services/monitoring/control_plane_db.go`,
-wired in `core/application/distributed.go`). They reach `/metrics` through the
-same Prometheus exporter as the rest of the API metrics.
-
-| Metric | Meaning | Page when |
-|--------|---------|-----------|
-| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
-| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
-| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
-
-A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
-While it grows, autovacuum can reclaim nothing anywhere in the database no
-matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
-registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
-The fix is to find the transaction holding the horizon open and clear it:
-
-```sql
-SELECT pid, state, age(backend_xmin) AS xmin_age, now() - xact_start AS xact_age, query
-FROM pg_stat_activity
-WHERE backend_xmin IS NOT NULL
-ORDER BY age(backend_xmin) DESC;
-```
-
-Then `pg_terminate_backend(pid)` on the offenders, and `VACUUM (VERBOSE)` the
-bloated tables once the horizon has moved.
-
-**A healthy-looking xmin age does not on its own prove the horizon is free.**
-The gauge reads `pg_stat_activity`, which only sees live backends. Two other
-things pin the very same horizon and are invisible there, so either one can hold
-vacuum back while the gauge reads 0:
-
-```sql
-SELECT gid, prepared, database, transaction FROM pg_prepared_xacts;
-SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;
-```
-
-An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
-and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
-before concluding that a bloated table has some other cause.
-
-Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
-translate into database load. Failed and timed-out samples cost the same interval
-as successful ones, so a database that is already struggling is not retried on
-every scrape. A failed sample reports the last good values rather than failing the
-scrape, because these gauges matter most when the database is struggling. Before
-the first successful sample the gauges are absent rather than zero, since a zero
-xmin age would read as a healthy horizon: alert on `absent()` too if you need to
-distinguish "healthy" from "never sampled".
-
 ## Advertising surfaces — where to register a new capability
 
 Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.

.agents/ds4-backend.md

@@ -77,56 +77,6 @@ spectrum. **Metal (Darwin) only** - it is a no-op on CUDA/CPU. Enable with
 budget). Gallery entries built on this: `deepseek-v4-flash-q4-ssd` (153 GB Flash
 on a 128 GB Mac) and `deepseek-v4-pro-q2-ssd` (433 GB Pro, experimental).
 
-## CUDA architecture (do not build without one)
-
-`backend/cpp/ds4/Makefile` drives upstream's **object targets** directly
-(`$(MAKE) -C ds4 ds4.o ds4_cuda.o ...`), which bypasses upstream's own guard:
-its `cuda` target refuses to build unless `CUDA_ARCH` is set, and offers
-`cuda-spark` (sm_121, DGX Spark / GB10) and `cuda-generic` (native) instead.
-Built with no `-arch`, nvcc targets its default architecture and the kernels run
-as JIT'd PTX. On GB10 that silently corrupted every prefill batch of >=128
-tokens - the model emitted text unrelated to the prompt and never closed its
-thinking block, so `content` came back empty - and cost close to two orders of
-magnitude of prefill throughput (4.21 t/s vs 325.70 t/s, same box, same model).
-Short prompts stayed correct, which is why it went unnoticed.
-
-The Makefile therefore picks a gencode list from `CUDA_MAJOR_VERSION` (a build
-arg the backend matrix already declares, forwarded by `Dockerfile.ds4`) and
-`uname -m`, and passes it as `NVCC_ARCH_FLAGS` to the sub-make. Upstream's
-`CUDA_ARCH` accepts a single value, so it cannot express the fat binary the
-shipped images need; a command-line assignment beats its `:=`. An empty
-`CUDA_MAJOR_VERSION` falls back to upstream's `native` for local developer
-builds, and an unrecognised one is a hard error - no CI runner has a GPU, so a
-silent `native` there is exactly the failure mode this guards against.
-
-`DS4_CUDA_HAVE_MXF4` is deliberately unset: upstream defines it only for
-single-arch sm_120/sm_121 builds and guards it with a plain `#ifdef` rather than
-`__CUDA_ARCH__`, so it cannot be combined with older archs. It gates an optional
-MXFP4 indexer fast path whose `#ifndef` branch returns 0, so omitting it costs
-speed, not correctness.
-
-### Verifying a build
-
-Check which flags a configuration resolves to, without compiling anything:
-
-```
-make -C backend/cpp/ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 NATIVE=false \
-  --eval='show: ; @echo [$(DS4_ARCH_MAKEVARS)]' show
-```
-
-Do not use `make -n` for this: the recipe is `+$(MAKE) ...`, and the `+` prefix
-makes it run even under `-n`.
-
-Then exercise the failure mode itself against a built backend. It only appears
-above one prefill batch, so the ordinary `predict` spec cannot catch it:
-
-```
-BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
-BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
-BACKEND_TEST_CAPS=health,load,predict,long_prefill \
-go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
-```
-
 ## Build matrix
 
 | Build | Where | Notes |

.github/PULL_REQUEST_TEMPLATE.md

@@ -5,7 +5,7 @@ This PR fixes #
 **Notes for Reviewers**
 
 
-**[Signed commits](../CONTRIBUTING.md#commit-messages)**
+**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
 - [ ] Yes, I signed my commits.
 - [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
  

.github/backend-matrix.yml

@@ -166,19 +166,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-whisper-medusa'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'true'
-    backend: "whisper-medusa"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -367,19 +354,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "8"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-funasr'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -652,19 +626,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "12"
-    cuda-minor-version: "1"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-12-whisper-medusa'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "whisper-medusa"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1108,19 +1069,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'cublas'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-nvidia-cuda-13-funasr'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1456,19 +1404,6 @@ include:
     backend: "qwen-asr"
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
-  - build-type: 'l4t'
-    cuda-major-version: "13"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-cuda-13-arm64-funasr'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    ubuntu-version: '2404'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
   - build-type: 'l4t'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2453,19 +2388,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-funasr'
-    runs-on: 'ubuntu-latest'
-    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'hipblas'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2746,19 +2668,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2204'
-  - build-type: 'l4t'
-    cuda-major-version: "12"
-    cuda-minor-version: "0"
-    platforms: 'linux/arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-nvidia-l4t-funasr'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
-    skip-drivers: 'true'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2204'
   - build-type: 'l4t'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -2968,19 +2877,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'intel'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-intel-funasr'
-    runs-on: 'ubuntu-latest'
-    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: 'intel'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3312,10 +3208,9 @@ include:
   # consumed. Same reason CUDA needs its toolkit in base-image rather than in a
   # builder image: this is the ds4 shape, not the llama-cpp one.
   #
-  # ROCm uses upstream's HIP backend and the project-wide ROCm 7.2.1 base. No
-  # CUDA arm64 or L4T entry: upstream documents and validates CUDA on x86 only.
-  # Darwin/Metal is in the includeDarwin matrix below, built by
-  # scripts/build/audio-cpp-darwin.sh.
+  # No ROCm entry: upstream has no HIP configuration. No CUDA arm64 or L4T
+  # entry: upstream documents and validates CUDA on x86 only. Darwin/Metal is in
+  # the includeDarwin matrix below, built by scripts/build/audio-cpp-darwin.sh.
   #
   # No vulkan entry either, though Dockerfile.audio-cpp and the backend Makefile
   # both handle BUILD_TYPE=vulkan for local builds. Every other vulkan backend
@@ -3383,19 +3278,6 @@ include:
     dockerfile: "./backend/Dockerfile.audio-cpp"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: 'hipblas'
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-gpu-rocm-hipblas-audio-cpp'
-    runs-on: 'ubuntu-latest'
-    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
-    skip-drivers: 'false'
-    backend: "audio-cpp"
-    dockerfile: "./backend/Dockerfile.audio-cpp"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -6189,20 +6071,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/amd64'
-    platform-tag: 'amd64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-funasr'
-    runs-on: 'ubuntu-latest'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -6217,20 +6085,6 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
-  - build-type: ''
-    cuda-major-version: ""
-    cuda-minor-version: ""
-    platforms: 'linux/arm64'
-    platform-tag: 'arm64'
-    tag-latest: 'auto'
-    tag-suffix: '-cpu-funasr'
-    runs-on: 'ubuntu-24.04-arm'
-    base-image: "ubuntu:24.04"
-    skip-drivers: 'false'
-    backend: "funasr"
-    dockerfile: "./backend/Dockerfile.python"
-    context: "./"
-    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -6470,9 +6324,6 @@ includeDarwin:
   - backend: "mlx"
     tag-suffix: "-metal-darwin-arm64-mlx"
     build-type: "mps"
-  - backend: "mlx-video"
-    tag-suffix: "-metal-darwin-arm64-mlx-video"
-    build-type: "mps"
   - backend: "chatterbox"
     tag-suffix: "-metal-darwin-arm64-chatterbox"
     build-type: "mps"
@@ -6609,9 +6460,6 @@ includeDarwin:
   - backend: "qwen-asr"
     tag-suffix: "-metal-darwin-arm64-qwen-asr"
     build-type: "mps"
-  - backend: "funasr"
-    tag-suffix: "-metal-darwin-arm64-funasr"
-    build-type: "mps"
   - backend: "nemo"
     tag-suffix: "-metal-darwin-arm64-nemo"
     build-type: "mps"

.github/bump_ctranslate2_rocm_wheel.sh

@@ -1,49 +0,0 @@
-#!/bin/bash
-# Bump the CTranslate2 ROCm Python wheel release pin used by faster-whisper.
-set -xe
-
-REPO=$1   # OpenNMT/CTranslate2
-FILE=$2   # backend/python/faster-whisper/install.sh
-VAR=$3    # CTRANSLATE2_VERSION (used for output file names so the workflow can read them)
-
-if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
-    echo "usage: $0 <repo> <install-script> <var-name>" >&2
-    exit 1
-fi
-
-LATEST_RELEASE=$(curl -sS -H "Accept: application/vnd.github+json" \
-    "https://api.github.com/repos/$REPO/releases/latest")
-LATEST_TAG=$(python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])" <<< "$LATEST_RELEASE")
-
-set +e
-CURRENT_VERSION=$(grep -m1 "^${VAR}=" "$FILE" | cut -d= -f2 | sed -E 's/^\$\{[^:]+:-([^}]+)\}$/\1/')
-CTRANSLATE2_ROCM_WHEEL_OS=$(grep -m1 '^CTRANSLATE2_ROCM_WHEEL_OS=' "$FILE" | cut -d= -f2 | sed -E 's/^\$\{[^:]+:-([^}]+)\}$/\1/')
-set -e
-
-if [ -z "$CURRENT_VERSION" ]; then
-    echo "Could not find $VAR in $FILE."
-    exit 0
-fi
-
-if [ -z "$CTRANSLATE2_ROCM_WHEEL_OS" ]; then
-    echo "Could not find CTRANSLATE2_ROCM_WHEEL_OS in $FILE."
-    exit 0
-fi
-
-ASSET_NAME="rocm-python-wheels-${CTRANSLATE2_ROCM_WHEEL_OS}.zip"
-LATEST_RELEASE="$LATEST_RELEASE" python3 - "$ASSET_NAME" <<'PY'
-import json
-import os
-import sys
-
-asset_name = sys.argv[1]
-release = json.loads(os.environ["LATEST_RELEASE"])
-assets = {asset.get("name") for asset in release.get("assets", [])}
-if asset_name not in assets:
-    raise SystemExit(f"Could not find release asset {asset_name!r}")
-PY
-
-sed -i "$FILE" -e "s/^${VAR}=.*/${VAR}=\${${VAR}:-${LATEST_TAG}}/"
-
-echo "Changes: https://github.com/$REPO/compare/${CURRENT_VERSION}...${LATEST_TAG}" >> "${VAR}_message.txt"
-echo "${LATEST_TAG}" >> "${VAR}_commit.txt"

.github/bump_vllm_metal.sh

@@ -3,9 +3,9 @@
 # darwin (Apple Silicon) install path. The macOS/Metal build
 # (backend/python/vllm/install.sh, Darwin branch) installs vllm-metal, which is
 # version-locked to a specific vLLM source release. install.sh derives that vLLM
-# version, and the wheel asset name, at build time from the pinned tag, so there
-# is only ONE value to bump here -- mirroring bump_vllm_wheel.sh, which bumps the
-# Linux cu130 wheel pin.
+# version at build time from vllm-metal's own installer at the pinned
+# tag, so there is only ONE value to bump here -- mirroring bump_vllm_wheel.sh,
+# which bumps the Linux cu130 wheel pin.
 #
 # This deliberately tracks vllm-project/vllm-metal, NOT vllm-project/vllm: the
 # darwin build can only use the exact vLLM version vllm-metal supports, so it may
@@ -23,20 +23,15 @@ if [ -z "$FILE" ] || [ -z "$REPO" ] || [ -z "$VAR" ]; then
     exit 1
 fi
 
-# vllm-metal ships frequent .dev releases, flagged as prereleases, alongside the
-# stable ones. /releases/latest skips the prereleases and returns the newest
-# stable tag, which is what darwin should pin: upstream deletes and re-cuts .dev
-# tags, and a pin to a deleted tag 404s the whole build.
+# vllm-metal ships frequent dev releases, all flagged as non-prerelease, so
+# /releases/latest returns the newest one (with its cp312 wheel asset).
 LATEST_TAG=$(gh_curl -H "Accept: application/vnd.github+json" \
     "https://api.github.com/repos/$REPO/releases/latest" \
     | python3 -c "import json,sys; print(json.load(sys.stdin)['tag_name'])")
 
-# The coupled vLLM release lives in .github/vllm-release-tag.commit at that tag
-# (since vllm-metal 0.28); releases predating that file pinned it inline in their
-# own install.sh. The extractor reads both forms.
-NEW_VLLM_VERSION=$( { gh_curl \
-    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/.github/vllm-release-tag.commit" \
-    || gh_curl "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh"; } \
+# The coupled vLLM source version lives in vllm-metal's installer at that tag.
+NEW_VLLM_VERSION=$(gh_curl \
+    "https://raw.githubusercontent.com/$REPO/$LATEST_TAG/install.sh" \
     | "$(dirname "${BASH_SOURCE[0]}")/../scripts/lib/extract-vllm-metal-version.sh")
 
 if [ -z "$LATEST_TAG" ] || [ -z "$NEW_VLLM_VERSION" ]; then

.github/workflows/bump_deps.yaml

@@ -166,40 +166,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update ${{ matrix.repository }}'
           title: 'chore: :arrow_up: Update ${{ matrix.repository }} to `${{ steps.bump.outputs.commit }}`'
-          branch: "bump/${{ matrix.variable }}"
-          body: ${{ steps.bump.outputs.message }}
-          signoff: true
-
-  bump-ctranslate2-rocm-wheel:
-    # CTranslate2's ROCm wheels are published as release assets, so the
-    # faster-whisper hipblas install path pins the release tag used in the URL.
-    if: github.repository == 'mudler/LocalAI'
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v6
-      - name: Bump CTranslate2 ROCm wheel pin 🔧
-        id: bump
-        run: |
-          bash .github/bump_ctranslate2_rocm_wheel.sh OpenNMT/CTranslate2 backend/python/faster-whisper/install.sh CTRANSLATE2_VERSION
-          {
-            echo 'message<<EOF'
-            cat "CTRANSLATE2_VERSION_message.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          {
-            echo 'commit<<EOF'
-            cat "CTRANSLATE2_VERSION_commit.txt"
-            echo EOF
-          } >> "$GITHUB_OUTPUT"
-          rm -rfv CTRANSLATE2_VERSION_message.txt CTRANSLATE2_VERSION_commit.txt
-      - name: Create Pull Request
-        uses: peter-evans/create-pull-request@v8
-        with:
-          token: ${{ secrets.UPDATE_BOT_TOKEN }}
-          push-to-fork: ci-forks/LocalAI
-          commit-message: ':arrow_up: Update OpenNMT/CTranslate2 ROCm wheel'
-          title: 'chore: :arrow_up: Update OpenNMT/CTranslate2 ROCm wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "update/CTRANSLATE2_VERSION"
+          branch: "update/${{ matrix.variable }}"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -236,7 +203,7 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm cu130 wheel'
           title: 'chore: :arrow_up: Update vllm-project/vllm cu130 wheel to `${{ steps.bump.outputs.commit }}`'
-          branch: "bump/VLLM_VERSION"
+          branch: "update/VLLM_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true
 
@@ -274,6 +241,6 @@ jobs:
           push-to-fork: ci-forks/LocalAI
           commit-message: ':arrow_up: Update vllm-project/vllm-metal (darwin)'
           title: 'chore: :arrow_up: Update vllm-metal (darwin) to `${{ steps.bump.outputs.commit }}`'
-          branch: "bump/VLLM_METAL_VERSION"
+          branch: "update/VLLM_METAL_VERSION"
           body: ${{ steps.bump.outputs.message }}
           signoff: true

.github/workflows/secscan.yaml

@@ -31,7 +31,7 @@ jobs:
         if: ${{ !github.repository.fork && github.actor != 'dependabot[bot]' }}
       - name: Run Gosec Security Scanner
         if: ${{ !github.repository.fork && github.actor != 'dependabot[bot]' }}
-        uses: securego/gosec@v2.29.0
+        uses: securego/gosec@v2.27.1
         with:
           # we let the report trigger content trigger a failure using the GitHub Security features.
           # backend/go/supertonic is excluded: it vendors upstream supertone-inc/supertonic

.github/workflows/test.yml

@@ -80,13 +80,8 @@ jobs:
             coverage/coverage.out
             coverage/coverage.html
           if-no-files-found: ignore
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true
@@ -130,13 +125,8 @@ jobs:
           export PATH="/opt/homebrew/opt/make/libexec/gnubin:$PATH"
           PATH="$PATH:$HOME/go/bin" make protogen-go
           PATH="$PATH:$HOME/go/bin" BUILD_TYPE="GITHUB_CI_HAS_BROKEN_METAL" CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make --jobs 4 --output-sync=target test
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-aio.yml

@@ -77,13 +77,8 @@ jobs:
       - name: Test
         run: |
             PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-e2e.yml

@@ -63,13 +63,8 @@ jobs:
       - name: Test Backend E2E
         run: |
           PATH="$PATH:$HOME/go/bin" make build-mock-backend test-e2e
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-pii-ner-e2e.yml

@@ -88,13 +88,8 @@ jobs:
       # CPU and runs the token_classify capability spec (byte-offset contract).
       - name: Run live PII NER backend E2E
         run: PATH="$PATH:$HOME/go/bin" make test-extra-backend-privacy-filter
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

.github/workflows/tests-ui-e2e.yml

@@ -75,13 +75,8 @@ jobs:
           path: core/http/react-ui/coverage/
           if-no-files-found: ignore
           retention-days: 7
-      # tmate keeps the runner busy until the 6 hour job limit, so a single
-      # failure costs a whole runner slot. Only open a session when someone
-      # asked for one by labelling the pull request `ci-debug`, and cap the
-      # session so a forgotten label cannot idle a runner either.
       - name: Setup tmate session if tests fail
-        if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
-        timeout-minutes: 30
+        if: ${{ failure() }}
         uses: mxschmitt/action-tmate@v3.23
         with:
           detached: true

AGENTS.md

@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w
 
 LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
 
-- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
+- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
 - **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
 - **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
 - **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.

CONTRIBUTING.md

@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis
 
 The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
 
-- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
+- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
 - **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
 - **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
 

Makefile

@@ -1,7 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/mlx-video backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
-.NOTPARALLEL: backends/whisper-medusa
-.NOTPARALLEL: backends/funasr
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/nemo-speech-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -36,11 +34,6 @@ TEST_FLAKES?=5
 RANDOM := $(shell bash -c 'echo $$RANDOM')
 
 VERSION?=$(shell git describe --always --tags || echo "dev" )
-# fyne package only accepts numeric x[.y[.z]] app versions, so reduce git
-# describe output (v4.9.0, v4.9.0-14-gabc1234, or a bare sha on untagged
-# checkouts) to its numeric core; anything non-numeric falls back to 0.0.0.
-# Without this the packaged launcher reports itself as version 0.0.0 (#11673).
-LAUNCHER_APP_VERSION?=$(shell v=$$(echo "$(VERSION)" | sed -E 's/^v//; s/[+-].*$$//'); echo "$$v" | grep -qE '^[0-9]+(\.[0-9]+){0,2}$$' && echo "$$v" || echo "0.0.0")
 # go tool nm ./local-ai | grep Commit
 LD_FLAGS?=-s -w
 override LD_FLAGS += -X "github.com/mudler/LocalAI/internal.Version=$(VERSION)"
@@ -242,7 +235,7 @@ test-ci-scripts:
 ## pure stdlib on purpose so they run without any backend venv; the list is
 ## explicit because their siblings (model_identity_test) import grpc and the
 ## generated protobufs, which only exist inside a built backend.
-PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test temp_utils_test
+PYTHON_HELPER_TESTS?=python_utils_test vllm_utils_test model_utils_test mlx_utils_test parent_watch_test
 test-python-helpers:
 	cd backend/python/common && python3 -m unittest $(PYTHON_HELPER_TESTS)
 
@@ -395,17 +388,9 @@ test-e2e: build-mock-backend build-cloud-proxy-backend prepare-e2e run-e2e-image
 	$(MAKE) teardown-e2e
 	docker rmi localai-tests
 
-# `docker stop` returns as soon as the container exits, but Docker reaps a
-# `--rm` container asynchronously after that. The `docker rmi localai-tests` in
-# test-e2e then loses the race against the reaper and fails on a still
-# referenced image, turning a green suite red. Removing the container ourselves
-# is synchronous, so the image reference is gone before we return. It also
-# covers the case where nothing is running, which `docker stop` could not
-# because it rejects an empty argument list.
 teardown-e2e:
 	rm -rf $(TEST_DIR) || true
-	@CONTAINERS=$$(docker ps -aq --filter ancestor=localai-tests 2>/dev/null); \
-	if [ -n "$$CONTAINERS" ]; then docker rm -f $$CONTAINERS || true; fi
+	docker stop $$(docker ps -q --filter ancestor=localai-tests)
 
 ########################################################
 ## Integration and unit tests
@@ -614,7 +599,6 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/vllm
 	$(MAKE) -C backend/python/vllm-omni
 	$(MAKE) -C backend/python/longcat-video
-	$(MAKE) -C backend/python/mlx-video
 	$(MAKE) -C backend/python/sglang
 	$(MAKE) -C backend/python/vibevoice
 	$(MAKE) -C backend/python/liquid-audio
@@ -624,11 +608,9 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/fish-speech
 	$(MAKE) -C backend/python/faster-qwen3-tts
 	$(MAKE) -C backend/python/qwen-asr
-	$(MAKE) -C backend/python/funasr
 	$(MAKE) -C backend/python/nemo
 	$(MAKE) -C backend/python/voxcpm
 	$(MAKE) -C backend/python/faster-whisper
-	$(MAKE) -C backend/python/whisper-medusa
 	$(MAKE) -C backend/python/whisperx
 	$(MAKE) -C backend/python/ace-step
 	$(MAKE) -C backend/python/trl
@@ -649,7 +631,6 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/vllm test
 	$(MAKE) -C backend/python/vllm-omni test
 	$(MAKE) -C backend/python/longcat-video test
-	$(MAKE) -C backend/python/mlx-video test
 	$(MAKE) -C backend/python/vibevoice test
 	$(MAKE) -C backend/python/liquid-audio test
 	$(MAKE) -C backend/python/moonshine test
@@ -658,11 +639,9 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/fish-speech test
 	$(MAKE) -C backend/python/faster-qwen3-tts test
 	$(MAKE) -C backend/python/qwen-asr test
-	$(MAKE) -C backend/python/funasr test
 	$(MAKE) -C backend/python/nemo test
 	$(MAKE) -C backend/python/voxcpm test
 	$(MAKE) -C backend/python/faster-whisper test
-	$(MAKE) -C backend/python/whisper-medusa test
 	$(MAKE) -C backend/python/whisperx test
 	$(MAKE) -C backend/python/ace-step test
 	$(MAKE) -C backend/python/trl test
@@ -1266,10 +1245,6 @@ backends/mlx:
 	BACKEND=mlx $(MAKE) build-darwin-python-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx.tar)"
 
-backends/mlx-video:
-	BACKEND=mlx-video $(MAKE) build-darwin-python-backend
-	./local-ai backends install "ocifile://$(abspath ./backend-images/mlx-video.tar)"
-
 backends/diffuser-darwin:
 	BACKEND=diffusers $(MAKE) build-darwin-python-backend
 	./local-ai backends install "ocifile://$(abspath ./backend-images/diffusers.tar)"
@@ -1356,7 +1331,6 @@ BACKEND_RERANKERS = rerankers|python|.|false|true
 BACKEND_TRANSFORMERS = transformers|python|.|false|true
 BACKEND_OUTETTS = outetts|python|.|false|true
 BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
-BACKEND_WHISPER_MEDUSA = whisper-medusa|python|.|false|true
 BACKEND_COQUI = coqui|python|.|false|true
 BACKEND_RFDETR = rfdetr|python|.|false|true
 BACKEND_INSIGHTFACE = insightface|python|.|false|true
@@ -1378,7 +1352,6 @@ BACKEND_QWEN_TTS = qwen-tts|python|.|false|true
 BACKEND_FISH_SPEECH = fish-speech|python|.|false|true
 BACKEND_FASTER_QWEN3_TTS = faster-qwen3-tts|python|.|false|true
 BACKEND_QWEN_ASR = qwen-asr|python|.|false|true
-BACKEND_FUNASR = funasr|python|.|false|true
 BACKEND_NEMO = nemo|python|.|false|true
 BACKEND_VOXCPM = voxcpm|python|.|false|true
 BACKEND_WHISPERX = whisperx|python|.|false|true
@@ -1449,7 +1422,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_RERANKERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_TRANSFORMERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OUTETTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
-$(eval $(call generate-docker-build-target,$(BACKEND_WHISPER_MEDUSA)))
 $(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
@@ -1471,7 +1443,6 @@ $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FISH_SPEECH)))
 $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_QWEN3_TTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_QWEN_ASR)))
-$(eval $(call generate-docker-build-target,$(BACKEND_FUNASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_NEMO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXCPM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
@@ -1501,8 +1472,6 @@ docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
 docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-nemo-speech-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
-docker-build-backends: docker-build-whisper-medusa
-docker-build-backends: docker-build-funasr
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1653,7 +1622,7 @@ site-serve: site
 build-launcher-darwin:
 	rm -rf dist/LocalAI.app cmd/launcher/LocalAI.app
 	mkdir -p dist
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
 	mv cmd/launcher/LocalAI.app dist/LocalAI.app
 	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
 
@@ -1680,4 +1649,4 @@ release-launcher-darwin: notarize-launcher-darwin
 	@echo "dist/LocalAI.dmg is ready"
 
 build-launcher-linux:
-	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION) && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
+	cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz

backend/Dockerfile.audio-cpp

@@ -6,14 +6,13 @@ ARG APT_PORTS_MIRROR=""
 # ASR, VAD, diarization, source separation and music generation, wrapped as a
 # LocalAI gRPC backend.
 #
-# BASE_IMAGE is ubuntu:24.04 for cpu and vulkan builds,
-# nvidia/cuda:<ver>-devel-ubuntu24.04 for cublas builds, or
-# rocm/dev-ubuntu-24.04:<ver> for hipblas builds. All ship apt and Ubuntu Noble
-# packages; the GPU bases also provide their toolkits. BUILD_TYPE selects the
-# engine backend in the Makefile: "" = portable CPU with all ggml CPU variants,
-# "cublas" -> -DENGINE_ENABLE_CUDA=ON, "hipblas" -> -DENGINE_ENABLE_HIP=ON,
-# and "vulkan" -> -DENGINE_ENABLE_VULKAN=ON. Darwin (Metal) builds bypass this
-# Dockerfile entirely.
+# BASE_IMAGE is ubuntu:24.04 for cpu and vulkan builds, or
+# nvidia/cuda:<ver>-devel-ubuntu24.04 for cublas builds; both ship apt and
+# Ubuntu Noble packages, and the CUDA base additionally provides
+# /usr/local/cuda. BUILD_TYPE selects the engine backend in the Makefile:
+# "" = portable CPU with all ggml CPU variants, "cublas" ->
+# -DENGINE_ENABLE_CUDA=ON, "vulkan" -> -DENGINE_ENABLE_VULKAN=ON. Darwin
+# (Metal) builds bypass this Dockerfile entirely.
 #
 # Upstream needs GCC 13 or newer, which ubuntu:24.04 and the CUDA 12/13
 # devel-ubuntu24.04 images all provide.
@@ -63,7 +62,7 @@ ENV BUILD_TYPE=${BUILD_TYPE} \
     APT_MIRROR=${APT_MIRROR} \
     APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
     DEBIAN_FRONTEND=noninteractive \
-    PATH=/opt/rocm/bin:/usr/local/cuda/bin:${PATH}
+    PATH=/usr/local/cuda/bin:${PATH}
 
 WORKDIR /build
 
@@ -74,7 +73,7 @@ WORKDIR /build
 # fallback of its own.
 #
 # BUILD_TYPE=vulkan additionally needs the loader headers and glslc; both are in
-# Noble. The CUDA and ROCm toolkits come from their matching BASE_IMAGE.
+# Noble. The CUDA toolkit for BUILD_TYPE=cublas comes from BASE_IMAGE.
 RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
     sh /usr/local/sbin/apt-mirror && \
     apt-get update && \
@@ -84,9 +83,6 @@ RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mi
     if [ "${BUILD_TYPE}" = "vulkan" ]; then \
         apt-get install -y --no-install-recommends libvulkan-dev glslc; \
     fi && \
-    if [ "${BUILD_TYPE}" = "hipblas" ]; then \
-        apt-get install -y --no-install-recommends hipblas-dev rocblas-dev; \
-    fi && \
     if [ "${TARGETARCH}" = "arm64" ]; then \
         apt-get install -y --no-install-recommends gcc-14 g++-14; \
     fi && \

backend/Dockerfile.ds4

@@ -10,7 +10,6 @@ FROM ${BASE_IMAGE} AS builder
 ARG BUILD_TYPE
 ARG TARGETARCH
 ARG TARGETVARIANT
-ARG CUDA_MAJOR_VERSION
 
 ENV BUILD_TYPE=${BUILD_TYPE} \
     DEBIAN_FRONTEND=noninteractive \
@@ -36,8 +35,7 @@ RUN apt-get update && \
 COPY . /LocalAI
 
 RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
-    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} \
-        CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} NATIVE=false grpc-server package
+    make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
 
 FROM scratch
 COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./

backend/README.md

@@ -47,7 +47,7 @@ The backend system provides language-specific Dockerfiles that handle the build
 - **mlx**: Apple Silicon optimization
 - **diffusers**: Stable Diffusion models
 - **longcat-video**: CUDA text/image-to-video and speech-driven avatar generation
-- **Audio**: coqui, faster-whisper, funasr, kitten-tts
+- **Audio**: coqui, faster-whisper, kitten-tts
 - **Vision**: mlx-vlm, rfdetr
 - **Specialized**: rerankers, chatterbox, kokoro
 

backend/backend.proto

@@ -499,10 +499,6 @@ message ModelOptions {
   // applied verbatim to the backend's engine constructor (e.g. vLLM AsyncEngineArgs).
   // Unknown keys produce an error at LoadModel time.
   string EngineArgs = 73;
-  string OriginalConfigFile = 77;
-
-  // EnvVars carries environment variables to be passed to the backend process.
-  map<string, string> EnvVars = 76;
 
   // Proxy carries the cloud-proxy backend's per-model configuration.
   // Empty for non-proxy backends.

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=efb04233dab73aeee4b2912042a90e7b36329061
+AUDIO_CPP_VERSION?=89a0e9803380880305e9e1b83c93614f9df2c893
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -77,16 +77,6 @@ endif
 
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DENGINE_ENABLE_CUDA=ON "-DCMAKE_CUDA_ARCHITECTURES=$(CUDA_ARCHITECTURES)"
-else ifeq ($(BUILD_TYPE),hipblas)
-    ROCM_HOME ?= /opt/rocm
-    ROCM_PATH ?= /opt/rocm
-    export CXX=$(ROCM_HOME)/llvm/bin/clang++
-    export CC=$(ROCM_HOME)/llvm/bin/clang
-    AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
-    # audio.cpp forwards GPU_TARGETS to CMake's semicolon-delimited HIP list.
-    comma := ,
-    HIP_TARGETS := $(subst $(comma),;,$(AMDGPU_TARGETS))
-    CMAKE_ARGS += -DENGINE_ENABLE_HIP=ON "-DGPU_TARGETS=$(HIP_TARGETS)"
 else ifeq ($(BUILD_TYPE),vulkan)
     CMAKE_ARGS += -DENGINE_ENABLE_VULKAN=ON
 else ifeq ($(UNAME_S),Darwin)

backend/cpp/audio-cpp/generation_request.cpp

@@ -22,8 +22,7 @@ bool voice_is_reference_file(const std::string &voice) {
 
 RequestShape build_tts_shape(const backend::TTSRequest &request) {
     RequestShape shape;
-    shape.has_voice_reference = voice_is_reference_file(request.voice()) ||
-                                request.params().find("multi_reference_cond") != request.params().end();
+    shape.has_voice_reference = voice_is_reference_file(request.voice());
     // !empty() as well as has_instructions(), and it must match the guard in
     // build_tts_request: a request whose instructions are an empty string
     // carries no style condition, so telling routing to prefer VoiceDesign for

backend/cpp/audio-cpp/generation_request_ctest.cpp

@@ -128,14 +128,6 @@ static void test_tts_shape() {
         check(!shape.has_voice_reference,
               "shape: a preset name is not a voice reference");
     }
-    {
-        backend::TTSRequest request;
-        (*request.mutable_params())["multi_reference_cond"] =
-            R"([{"audio":"one.wav","text":"one"}])";
-        const auto shape = build_tts_shape(request);
-        check(shape.has_voice_reference,
-              "shape: multi-reference conditioning is a voice reference");
-    }
     {
         backend::TTSRequest request;
         request.set_voice(dir.string());

backend/cpp/audio-cpp/loaded_model.cpp

@@ -103,9 +103,6 @@ engine::core::BackendType parse_backend_type(const std::string &value) {
     if (value == "cuda") {
         return engine::core::BackendType::Cuda;
     }
-    if (value == "hip" || value == "rocm") {
-        return engine::core::BackendType::Hip;
-    }
     if (value == "vulkan") {
         return engine::core::BackendType::Vulkan;
     }
@@ -119,7 +116,7 @@ engine::core::BackendType parse_backend_type(const std::string &value) {
         return engine::core::BackendType::Cpu;
     }
     throw ConfigError("audio-cpp: unknown backend option '" + value +
-                      "'. Known backends: cpu, cuda, hip, rocm, vulkan, metal, best");
+                      "'. Known backends: cpu, cuda, vulkan, metal, best");
 }
 
 std::filesystem::path executable_directory() {

backend/cpp/audio-cpp/model_options.cpp

@@ -59,12 +59,6 @@ bool starts_with(const std::string &value, const std::string &prefix) {
            value.compare(0, prefix.size(), prefix) == 0;
 }
 
-bool is_known_backend(const std::string &value) {
-    return value == "cpu" || value == "cuda" || value == "hip" ||
-           value == "rocm" || value == "vulkan" || value == "metal" ||
-           value == "best";
-}
-
 } // namespace
 
 ParsedOptions parse_model_options(const std::vector<std::string> &entries) {
@@ -113,12 +107,6 @@ ParsedOptions parse_model_options(const std::vector<std::string> &entries) {
         } else if (key == "task") {
             parsed.options.task = value;
         } else if (key == "backend") {
-            if (!is_known_backend(value)) {
-                parsed.error = "audio-cpp: unknown backend option '" + value +
-                               "'. Known backends: cpu, cuda, hip, rocm, "
-                               "vulkan, metal, best";
-                return parsed;
-            }
             parsed.options.backend = value;
         } else if (key == "model_spec_override") {
             parsed.options.model_spec_override = value;

backend/cpp/audio-cpp/model_options.h

@@ -17,7 +17,7 @@ struct ModelOptions {
     std::string family;
     // Pins the audio.cpp task, overriding RPC-based routing. Empty means route.
     std::string task;
-    // ggml backend: cpu, cuda, hip (or rocm), vulkan, metal, best.
+    // ggml backend: cpu, cuda, vulkan, metal, best.
     std::string backend = "cpu";
     int device = 0;
     // True once a `device:` entry has been seen. 0 is both the default and a

backend/cpp/audio-cpp/model_options_test.cpp

@@ -75,11 +75,6 @@ static void test_scalar_options() {
     check(parse_model_options({"live_idle_timeout_ms:0"}).options.live_idle_timeout_ms == 0,
           "an explicit 0 turns the live idle limit off rather than reverting to "
           "the default");
-
-    check(parse_model_options({"backend:hip"}).error.empty(),
-          "HIP backend option is accepted");
-    check(parse_model_options({"backend:rocm"}).error.empty(),
-          "ROCm backend alias is accepted");
 }
 
 // Values containing colons must survive: split on the FIRST colon only.
@@ -129,8 +124,6 @@ static void test_errors() {
           "negative device is rejected");
     check(!parse_model_options({"threads:x"}).error.empty(),
           "non-numeric threads is rejected");
-    check(!parse_model_options({"backend:unknown"}).error.empty(),
-          "unknown compute backend is rejected before model loading");
 
     // Values too large for int must be rejected, not silently wrapped into a
     // negative device index that then reaches the ggml backend selector.

backend/cpp/ds4/CMakeLists.txt

@@ -84,10 +84,9 @@ elseif(DS4_GPU STREQUAL "cpu")
     set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
 endif()
 
-# Upstream splits image preprocessing, distributed inference, tensor-parallel
-# transport, the SSD expert cache, and layer placement into GPU-agnostic
-# translation units. Link them regardless of DS4_GPU.
-list(APPEND DS4_OBJS "${DS4_DIR}/ds4_image.o")
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. Link
+# them regardless of DS4_GPU.
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_distributed.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_tp.o")
 list(APPEND DS4_OBJS "${DS4_DIR}/ds4_ssd.o")

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
+# Upstream pin lives below as DS4_VERSION?=8db89fe083ae4d17c9a2428ccd29803d3ae8f577
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=6289c516273979173abbc062209a81dd3706b804
+DS4_VERSION?=8db89fe083ae4d17c9a2428ccd29803d3ae8f577
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
@@ -18,83 +18,21 @@ UNAME_S := $(shell uname -s)
 
 CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
 
-# nvcc must be told the target architecture explicitly for a cublas build, and
-# this is not a tuning knob. Upstream's Makefile leaves CUDA_ARCH empty and its
-# `cuda` target REFUSES to build without one, offering `cuda-spark`
-# (CUDA_ARCH=sm_121) and `cuda-generic` (CUDA_ARCH=native) instead. We drive its
-# object targets directly, which bypasses that guard: nvcc then compiles with no
-# -arch at all, and the kernels run as JIT'd PTX for its default architecture.
-# On GB10 (sm_121) that silently produced corrupt inference output above a
-# ~128-token prefill batch and ~77x slower prefill (4.21 t/s vs 325.70 t/s,
-# measured on the same box with the same model). No CI runner has a GPU, so
-# `native` has nothing to enumerate there.
-#
-# Upstream's CUDA_ARCH takes a SINGLE value (see its sm_120/sm_121 special cases
-# and the `-arch=$(CUDA_ARCH)` fallback), so it cannot express the fat binary
-# these images need. NVCC_ARCH_FLAGS is overridden instead: a command-line
-# assignment wins over the `:=` in upstream's Makefile, and its NVCCFLAGS
-# expands whatever we pass.
-#
-# The architecture lists are copied from backend/go/vllm-cpp/Makefile rather
-# than invented, so the two CUDA images cover the same GPUs: amd64 datacenter +
-# consumer, and l4t/arm64 covering Orin (87), Thor (110) and GB10 (121a).
-#
-# -DDS4_CUDA_HAVE_MXF4=1 is deliberately NOT set. Upstream only defines it for
-# single-arch sm_120/sm_121 builds and guards the code with a plain #ifdef
-# rather than __CUDA_ARCH__, so it cannot be combined with older archs in one
-# fat binary. It gates an optional MXFP4 indexer fast path whose #ifndef branch
-# returns 0 and falls back to the generic path, so omitting it costs some speed
-# on GB10, not correctness. Revisit if upstream adds __CUDA_ARCH__ guards.
-#
-# An EMPTY CUDA_MAJOR_VERSION means a local developer build, not CI: fall back
-# to upstream's own `native` handling, which needs a GPU present but is what a
-# developer building on their own machine wants. Both variables are `?=` so an
-# explicit value on the command line always wins.
-UNAME_M := $(shell uname -m)
-CUDA_MAJOR_VERSION ?=
-ifeq ($(BUILD_TYPE),cublas)
-ifeq ($(CUDA_MAJOR_VERSION),13)
-ifeq ($(UNAME_M),aarch64)
-    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_87,code=sm_87 \
-                           -gencode arch=compute_90a,code=sm_90a \
-                           -gencode arch=compute_100a,code=sm_100a \
-                           -gencode arch=compute_110,code=sm_110 \
-                           -gencode arch=compute_121a,code=sm_121a
-else
-    DS4_NVCC_ARCH_FLAGS ?= -gencode arch=compute_80,code=sm_80 \
-                           -gencode arch=compute_86,code=sm_86 \
-                           -gencode arch=compute_89,code=sm_89 \
-                           -gencode arch=compute_90a,code=sm_90a \
-                           -gencode arch=compute_100a,code=sm_100a \
-                           -gencode arch=compute_103a,code=sm_103a \
-                           -gencode arch=compute_120a,code=sm_120a \
-                           -gencode arch=compute_121a,code=sm_121a
-endif
-    DS4_ARCH_MAKEVARS := NVCC_ARCH_FLAGS="$(DS4_NVCC_ARCH_FLAGS)"
-else ifeq ($(CUDA_MAJOR_VERSION),)
-    # Local build: let upstream resolve the host GPU.
-    DS4_ARCH_MAKEVARS := CUDA_ARCH=native
-else
-    $(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (13 does). Leave it empty for a native build, or pass DS4_NVCC_ARCH_FLAGS explicitly.)
-endif
-endif
-
-# Upstream splits image preprocessing, distributed inference, tensor-parallel
-# transport, the SSD expert cache, and layer placement into GPU-agnostic
-# translation units. They are shared by every GPU mode, so append them
-# unconditionally below.
+# Upstream splits distributed inference, tensor-parallel transport, the SSD
+# expert cache, and layer placement into GPU-agnostic translation units. They
+# are shared by every GPU mode, so append them unconditionally below.
 ifeq ($(BUILD_TYPE),cublas)
     CMAKE_ARGS += -DDS4_GPU=cuda
-    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
+    DS4_OBJ_TARGET := ds4.o ds4_cuda.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o \
         cuda/mmq/ds4_ggml_stubs.o cuda/mmq/ds4_mmq.o cuda/mmq/ds4_mmq_d2r.o \
         cuda/mmq/quantize.o cuda/mmq/mmid.o cuda/mmq/mmvq.o cuda/mmq/ds4_repack.o
 else ifeq ($(UNAME_S),Darwin)
     CMAKE_ARGS += -DDS4_GPU=metal
-    DS4_OBJ_TARGET := ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
     # CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
     CMAKE_ARGS += -DDS4_GPU=cpu
-    DS4_OBJ_TARGET := ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+    DS4_OBJ_TARGET := ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 ifneq ($(NATIVE),true)
@@ -119,11 +57,11 @@ ds4:
 # the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
 ds4/ds4.o: ds4
 ifeq ($(BUILD_TYPE),cublas)
-	+$(MAKE) -C ds4 $(DS4_ARCH_MAKEVARS) $(DS4_OBJ_TARGET)
+	+$(MAKE) -C ds4 $(DS4_OBJ_TARGET)
 else ifeq ($(UNAME_S),Darwin)
-	+$(MAKE) -C ds4 ds4.o ds4_image.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4.o ds4_metal.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 else
-	+$(MAKE) -C ds4 ds4_cpu.o ds4_image.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
+	+$(MAKE) -C ds4 ds4_cpu.o ds4_distributed.o ds4_tp.o ds4_ssd.o ds4_layer_pack.o
 endif
 
 grpc-server: ds4/ds4.o

backend/cpp/ds4/dsml_parser.cpp

@@ -92,8 +92,7 @@ std::string json_escape(const std::string &in) {
 
 } // namespace
 
-DsmlParser::DsmlParser(bool starts_in_thinking)
-    : state_(starts_in_thinking ? State::THINK : State::TEXT) {}
+DsmlParser::DsmlParser() = default;
 
 bool DsmlParser::IsInDsmlStructural() const {
     switch (state_) {

backend/cpp/ds4/dsml_parser.h

@@ -17,9 +17,7 @@ struct ParserEvent {
 // Streaming parser. Stateless across instances; one per Predict call.
 class DsmlParser {
 public:
-    // The chat prompt may already contain the opening thinking marker, so the
-    // generated text can begin directly with reasoning bytes.
-    explicit DsmlParser(bool starts_in_thinking = false);
+    DsmlParser();
 
     // Feed a chunk of raw model-emitted text. Appends classified events to
     // `out`. May buffer the tail of `chunk` internally if it looks like a
@@ -45,7 +43,7 @@ public:
 
 private:
     enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
-    State state_;
+    State state_ = State::TEXT;
     std::string buf_;
     std::string current_tool_name_;
     int tool_index_ = -1;

backend/cpp/ds4/dsml_parser_test.cpp

@@ -1,133 +0,0 @@
-// SPDX-License-Identifier: MIT
-// Standalone regression tests for the DSML streaming parser.
-//
-// The repository's backend/cpp/run-unit-tests.sh harness compiles each
-// *_test.cpp as a single translation unit, so include the implementation here.
-
-#include "dsml_parser.cpp"
-
-#include <cstdio>
-#include <string>
-#include <type_traits>
-#include <vector>
-
-namespace {
-
-struct ParsedText {
-    std::string content;
-    std::string reasoning;
-};
-
-int failures = 0;
-
-void check_equal(const std::string &got, const std::string &want,
-                 const char *name) {
-    if (got == want) return;
-    std::fprintf(stderr, "FAIL %s: got \"%s\", want \"%s\"\n",
-                 name, got.c_str(), want.c_str());
-    failures++;
-}
-
-void collect_text(const std::vector<ds4cpp::ParserEvent> &events,
-                  ParsedText *parsed) {
-    for (const auto &event : events) {
-        if (event.type == ds4cpp::ParserEvent::CONTENT) {
-            parsed->content += event.text;
-        } else if (event.type == ds4cpp::ParserEvent::REASONING) {
-            parsed->reasoning += event.text;
-        }
-    }
-}
-
-ParsedText parse_chunks(ds4cpp::DsmlParser *parser,
-                        const std::vector<std::string> &chunks) {
-    ParsedText parsed;
-    for (const auto &chunk : chunks) {
-        std::vector<ds4cpp::ParserEvent> events;
-        parser->Feed(chunk, events);
-        collect_text(events, &parsed);
-    }
-    std::vector<ds4cpp::ParserEvent> events;
-    parser->Flush(events);
-    collect_text(events, &parsed);
-    return parsed;
-}
-
-template <typename Parser>
-void test_reasoning_opened_by_prompt() {
-    if constexpr (!std::is_constructible_v<Parser, bool>) {
-        std::fprintf(stderr,
-                     "FAIL reasoning_opened_by_prompt: parser cannot start in thinking state\n");
-        failures++;
-    } else {
-        Parser parser(true);
-        ParsedText parsed = parse_chunks(
-            &parser,
-            {"We need to calculate factorial recursively.</think>Here is the answer."});
-        check_equal(parsed.reasoning,
-                    "We need to calculate factorial recursively.",
-                    "reasoning_opened_by_prompt:reasoning");
-        check_equal(parsed.content, "Here is the answer.",
-                    "reasoning_opened_by_prompt:content");
-    }
-}
-
-template <typename Parser>
-Parser text_parser() {
-    if constexpr (std::is_constructible_v<Parser, bool>) {
-        return Parser(false);
-    } else {
-        return Parser();
-    }
-}
-
-void test_reasoning_disabled() {
-    auto parser = text_parser<ds4cpp::DsmlParser>();
-    ParsedText parsed = parse_chunks(&parser, {"Here is the answer."});
-    check_equal(parsed.reasoning, "", "reasoning_disabled:reasoning");
-    check_equal(parsed.content, "Here is the answer.",
-                "reasoning_disabled:content");
-}
-
-void test_explicit_think_tag() {
-    auto parser = text_parser<ds4cpp::DsmlParser>();
-    ParsedText parsed = parse_chunks(
-        &parser, {"<think>reasoning</think>answer"});
-    check_equal(parsed.reasoning, "reasoning", "explicit_think_tag:reasoning");
-    check_equal(parsed.content, "answer", "explicit_think_tag:content");
-}
-
-template <typename Parser>
-void test_split_think_close_marker() {
-    if constexpr (!std::is_constructible_v<Parser, bool>) {
-        std::fprintf(stderr,
-                     "FAIL split_think_close_marker: parser cannot start in thinking state\n");
-        failures++;
-    } else {
-        Parser parser(true);
-        ParsedText parsed = parse_chunks(
-            &parser,
-            {"We need ", "to calculate ", "factorial", "</thi", "nk>",
-             "Here is ", "the answer."});
-        check_equal(parsed.reasoning, "We need to calculate factorial",
-                    "split_think_close_marker:reasoning");
-        check_equal(parsed.content, "Here is the answer.",
-                    "split_think_close_marker:content");
-    }
-}
-
-} // namespace
-
-int main() {
-    test_reasoning_opened_by_prompt<ds4cpp::DsmlParser>();
-    test_reasoning_disabled();
-    test_explicit_think_tag();
-    test_split_think_close_marker<ds4cpp::DsmlParser>();
-
-    if (failures == 0) {
-        std::fprintf(stderr, "all dsml_parser checks passed\n");
-        return 0;
-    }
-    std::fprintf(stderr, "%d check(s) failed\n", failures);
-    return 1;
-}

backend/cpp/ds4/generation_limits.h

@@ -1,27 +0,0 @@
-// SPDX-License-Identifier: MIT
-#pragma once
-
-#include <algorithm>
-
-namespace ds4cpp {
-
-inline int EffectiveGenerationLimit(int requested, int context_size,
-                                    int session_position) {
-  const int limit = requested > 0 ? requested : 256;
-  const int room = context_size - session_position;
-  if (room <= 1) return 0;
-  return std::min(limit, room - 1);
-}
-
-inline int RemainingGenerationBudget(int effective_limit, int produced) {
-  if (effective_limit <= produced) return 0;
-  return effective_limit - produced;
-}
-
-inline int SpeculativeAcceptedCapacity(int remaining, int draft_allowance,
-                                       int buffer_capacity) {
-  if (remaining <= 0 || draft_allowance < 0 || buffer_capacity <= 0) return 0;
-  return std::min({remaining, draft_allowance + 1, buffer_capacity});
-}
-
-} // namespace ds4cpp

backend/cpp/ds4/generation_limits_test.cpp

@@ -1,92 +0,0 @@
-// SPDX-License-Identifier: MIT
-
-#include "generation_limits.h"
-
-#include <cstdio>
-
-namespace {
-
-int failures = 0;
-
-void check_equal(int got, int want, const char *name) {
-  if (got == want) return;
-  std::fprintf(stderr, "FAIL %s: got %d, want %d\n", name, got, want);
-  failures++;
-}
-
-// Mutation caught: treating omitted or negative max_tokens as unlimited instead
-// of preserving DS4's legacy 256-token default.
-void test_nonpositive_uses_legacy_default_when_space_permits() {
-  check_equal(ds4cpp::EffectiveGenerationLimit(0, 4096, 100), 256,
-              "zero max_tokens uses legacy default");
-  check_equal(ds4cpp::EffectiveGenerationLimit(-1, 4096, 100), 256,
-              "negative max_tokens uses legacy default");
-}
-
-// Mutation caught: applying the legacy default without clamping it to the
-// post-prefill context room and reserved slot.
-void test_legacy_default_is_clamped_by_context() {
-  check_equal(ds4cpp::EffectiveGenerationLimit(0, 300, 100), 199,
-              "legacy default is context-clamped");
-}
-
-// Mutation caught: allowing an explicitly large request to overrun the
-// post-prefill context boundary.
-void test_large_positive_limit_is_clamped_to_context() {
-  check_equal(ds4cpp::EffectiveGenerationLimit(32768, 32768, 100), 32667,
-              "large positive is context-clamped");
-}
-
-// Mutation caught: replacing every positive request with the legacy default
-// rather than preserving a smaller configured limit.
-void test_smaller_positive_limit_is_preserved() {
-  check_equal(ds4cpp::EffectiveGenerationLimit(64, 4096, 100), 64,
-              "smaller positive is preserved");
-}
-
-// Mutation caught: consuming the final context slot instead of reserving it as
-// required by DS4's generation loop.
-void test_no_usable_room_returns_zero() {
-  check_equal(ds4cpp::EffectiveGenerationLimit(32, 100, 99), 0,
-              "one remaining context slot is not usable");
-}
-
-// Mutation caught: sending the original generation limit to a later
-// speculative cycle instead of subtracting tokens already produced.
-void test_remaining_budget_accounts_for_produced_tokens() {
-  check_equal(ds4cpp::RemainingGenerationBudget(10, 4), 6,
-              "remaining budget subtracts produced tokens");
-  check_equal(ds4cpp::RemainingGenerationBudget(10, 12), 0,
-              "remaining budget never becomes negative");
-}
-
-// Mutation caught: giving speculative evaluation capacity beyond either the
-// output budget, the draft allowance plus its first target token, or the fixed
-// accepted-token buffer.
-void test_speculative_capacity_obeys_all_bounds() {
-  check_equal(ds4cpp::SpeculativeAcceptedCapacity(3, 8, 8), 3,
-              "capacity respects remaining output budget");
-  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 4, 8), 5,
-              "capacity includes one target token beyond draft allowance");
-  check_equal(ds4cpp::SpeculativeAcceptedCapacity(20, 8, 6), 6,
-              "capacity respects fixed buffer");
-}
-
-} // namespace
-
-int main() {
-  test_nonpositive_uses_legacy_default_when_space_permits();
-  test_legacy_default_is_clamped_by_context();
-  test_large_positive_limit_is_clamped_to_context();
-  test_smaller_positive_limit_is_preserved();
-  test_no_usable_room_returns_zero();
-  test_remaining_budget_accounts_for_produced_tokens();
-  test_speculative_capacity_obeys_all_bounds();
-
-  if (failures == 0) {
-    std::fprintf(stderr, "all generation limit checks passed\n");
-    return 0;
-  }
-  std::fprintf(stderr, "%d check(s) failed\n", failures);
-  return 1;
-}

backend/cpp/ds4/grpc-server.cpp

@@ -10,9 +10,7 @@
 
 #include "dsml_parser.h"   // populated in Task 12
 #include "dsml_renderer.h" // populated in Task 16
-#include "generation_limits.h"
 #include "kv_cache.h"      // populated in Task 17
-#include "request_lifecycle.h"
 
 extern "C" {
 #include "ds4.h"
@@ -37,7 +35,6 @@ extern "C" {
 #include <mutex>
 #include <string>
 #include <thread>
-#include <utility>
 #include <vector>
 
 using grpc::Server;
@@ -72,21 +69,6 @@ int g_route_timeout_sec = 60;
 
 std::atomic<Server *> g_server{nullptr};
 
-static bool server_context_cancelled(void *ud) {
-    return static_cast<ServerContext *>(ud)->IsCancelled();
-}
-
-static void set_session_cancel(void *target, ds4cpp::CancelCallback callback,
-                               void *userdata) noexcept {
-    ds4_session_set_cancel(static_cast<ds4_session *>(target), callback, userdata);
-}
-
-static bool request_should_continue(ds4cpp::RequestLifecycle *request,
-                                    ServerContext *context) {
-    request->ObserveContextCancellation(context->IsCancelled());
-    return request->ShouldContinue();
-}
-
 // Parse a "key:value" option string. Returns empty when no colon.
 static std::pair<std::string, std::string> split_option(const std::string &opt) {
     auto colon = opt.find(':');
@@ -256,58 +238,37 @@ static bool apply_engine_option(ds4_engine_options *opt, const std::string &key,
 
 // When acting as a distributed coordinator, block until the worker route
 // covers all layers (ds4_session_distributed_route_ready == 1) or the timeout
-// elapses. No-op when not distributed.
+// elapses. Returns an empty string on success, or an error message to return
+// to the client. No-op when not distributed.
 //
 // Takes the g_engine_mu lock by reference and RELEASES it during each poll
 // sleep. The wait can span up to g_route_timeout_sec seconds while workers
 // connect; holding g_engine_mu the whole time would block the Status/Health
 // readiness probes (they also lock g_engine_mu), making LocalAI's loader treat
 // a still-starting worker as hung.
-struct RouteWaitResult {
-    ds4cpp::RouteWaitDecision decision;
-    std::string error;
-};
-
-static RouteWaitResult wait_route_ready(std::unique_lock<std::mutex> &lock,
-                                        ServerContext *context) {
-    if (!g_distributed) return {ds4cpp::RouteWaitDecision::Ready, ""};
+static std::string wait_route_ready(std::unique_lock<std::mutex> &lock) {
+    if (!g_distributed) return "";
     char err[256] = {0};
     const int deadline_polls = g_route_timeout_sec * 10; // 100ms per poll
     for (int i = 0; i <= deadline_polls; ++i) {
         int ready = ds4_session_distributed_route_ready(g_session, err, sizeof(err));
-        switch (ds4cpp::DecideRouteWait(ready, context->IsCancelled())) {
-        case ds4cpp::RouteWaitDecision::Ready:
-            return {ds4cpp::RouteWaitDecision::Ready, ""};
-        case ds4cpp::RouteWaitDecision::Error:
-            return {ds4cpp::RouteWaitDecision::Error,
-                    std::string("ds4 distributed route error: ") +
-                        (err[0] ? err : "unknown")};
-        case ds4cpp::RouteWaitDecision::Cancelled:
-            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
-        case ds4cpp::RouteWaitDecision::Pending:
-            break;
+        if (ready == 1) return "";
+        if (ready < 0) {
+            return std::string("ds4 distributed route error: ") +
+                   (err[0] ? err : "unknown");
         }
-        if (i == deadline_polls) break;
         // Release the lock while sleeping so Status/Health and other RPCs can
         // interleave during worker startup.
         lock.unlock();
         struct timespec ts = {0, 100L * 1000L * 1000L}; // 100ms
         nanosleep(&ts, nullptr);
         lock.lock();
-        if (context->IsCancelled()) {
-            return {ds4cpp::RouteWaitDecision::Cancelled, ""};
-        }
         // A concurrent Free() may have torn down the engine while we slept.
         if (!g_engine || !g_session) {
-            return {ds4cpp::RouteWaitDecision::Error,
-                    "ds4: model unloaded while waiting for distributed route"};
+            return "ds4: model unloaded while waiting for distributed route";
         }
     }
-    if (context->IsCancelled()) {
-        return {ds4cpp::RouteWaitDecision::Cancelled, ""};
-    }
-    return {ds4cpp::RouteWaitDecision::Error,
-            "ds4 distributed route incomplete: workers not connected (layers uncovered)"};
+    return "ds4 distributed route incomplete: workers not connected (layers uncovered)";
 }
 
 static void append_token_text(ds4_engine *engine, int token, std::string &out) {
@@ -380,9 +341,9 @@ static void collect_done(void *) {}
 struct StreamCtx {
     ds4_engine *engine;
     ServerWriter<backend::Reply> *writer;
-    ds4cpp::RequestLifecycle *request;
     ds4cpp::DsmlParser parser;
     int tokens;
+    bool aborted;
     // Track which tool indices we've seen TOOL_START for, so subsequent
     // ARGS deltas can elide the redundant id/name fields.
     std::vector<bool> tool_started;
@@ -390,7 +351,7 @@ struct StreamCtx {
 
 static void stream_emit(void *ud, int token) {
     auto *s = static_cast<StreamCtx *>(ud);
-    if (!s->request->ShouldContinue()) return;
+    if (s->aborted) return;
     if (token == ds4_token_eos(s->engine)) return;
     size_t len = 0;
     const char *text = ds4_token_text(s->engine, token, &len);
@@ -440,7 +401,7 @@ static void stream_emit(void *ud, int token) {
     reply.set_message(chunk);
     reply.set_tokens(1);
     if (any_field) {
-        s->request->ObserveStreamWrite(s->writer->Write(reply));
+        if (!s->writer->Write(reply)) s->aborted = true;
     }
     s->tokens++;
 }
@@ -796,30 +757,21 @@ public:
         return GStatus::OK;
     }
 
-    GStatus Predict(ServerContext *context, const backend::PredictOptions *request,
+    GStatus Predict(ServerContext *, const backend::PredictOptions *request,
                    backend::Reply *reply) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        RouteWaitResult route = wait_route_ready(lock, context);
-        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
-            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
-        }
-        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
-            return GStatus(StatusCode::UNAVAILABLE, route.error);
+        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
+            return GStatus(StatusCode::UNAVAILABLE, route_err);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
+        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
-        const bool starts_in_thinking = think_enabled &&
-            request->usetokenizertemplate() && request->messages_size() > 0;
-        CollectCtx collect = {
-            g_engine, "", ds4cpp::DsmlParser(starts_in_thinking),
-            reply, 0, {}, "", ""};
-        ds4cpp::RequestLifecycle lifecycle;
+        CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit; // future: skip prompt prefix if hit covers full prompt
@@ -831,27 +783,15 @@ public:
         // Either way g_session advances so the disk KV cache picks up a
         // real checkpoint after the call (see maybe_save_cache below).
         char err[256] = {0};
-        int rc;
-        {
-            ds4cpp::CancelCallbackScope cancel_scope(
-                g_session, set_session_cancel, server_context_cancelled, context);
-            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
-        }
+        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
         int prompt_len = prompt.len;
         ds4_tokens_free(&prompt);
-        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
-            lifecycle.ObserveContextCancellation(true);
-        }
-        const bool generation_started = rc == 0;
-        if (generation_started) {
-            const int n_predict = ds4cpp::EffectiveGenerationLimit(
-                request->tokens(), ds4_session_ctx(g_session),
-                ds4_session_pos(g_session));
+        if (rc == 0) {
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
+            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
             while (produced < n_predict) {
-                if (!request_should_continue(&lifecycle, context)) break;
                 SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -866,20 +806,13 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    const int remaining = ds4cpp::RemainingGenerationBudget(
-                        n_predict, produced);
-                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
-                        remaining, draft_max, kAcceptedMax);
+                    int cap = std::min(kAcceptedMax, draft_max + 1);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, remaining, eos,
+                        g_session, first, draft_max, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
-                        if (!request_should_continue(&lifecycle, context)) {
-                            stop = true;
-                            break;
-                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         collect_emit(&collect, accepted[j]);
                         if (++produced >= n_predict) { stop = true; break; }
@@ -888,26 +821,12 @@ public:
                 } else {
                     collect_emit(&collect, first);
                     if (++produced >= n_predict) break;
-                    if (!request_should_continue(&lifecycle, context)) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
+            collect_done(&collect);
         }
-
-        request_should_continue(&lifecycle, context);
-        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
-            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
-            !lifecycle.ShouldFinalize());
-        if (!terminal.should_finalize) {
-            if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
-                return GStatus(StatusCode::INTERNAL,
-                              std::string("ds4 generation failed: ") + err);
-            }
-            return GStatus(StatusCode::CANCELLED,
-                          "ds4 request cancelled");
-        }
-        if (generation_started) collect_done(&collect);
         maybe_save_cache(cache_key);
 
         // Flush any buffered parser state.
@@ -915,7 +834,7 @@ public:
         collect.parser.Flush(events);
         apply_events(&collect, events);
 
-        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
+        if (rc != 0) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
@@ -938,30 +857,21 @@ public:
         return GStatus::OK;
     }
 
-    GStatus PredictStream(ServerContext *context, const backend::PredictOptions *request,
+    GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
                          ServerWriter<backend::Reply> *writer) override {
         std::unique_lock<std::mutex> lock(g_engine_mu);
         if (!g_engine || !g_session) {
             return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
         }
         if (GStatus id = check_model_identity(request); !id.ok()) return id;
-        RouteWaitResult route = wait_route_ready(lock, context);
-        if (route.decision == ds4cpp::RouteWaitDecision::Cancelled) {
-            return GStatus(StatusCode::CANCELLED, "ds4 request cancelled");
-        }
-        if (route.decision == ds4cpp::RouteWaitDecision::Error) {
-            return GStatus(StatusCode::UNAVAILABLE, route.error);
+        if (std::string route_err = wait_route_ready(lock); !route_err.empty()) {
+            return GStatus(StatusCode::UNAVAILABLE, route_err);
         }
         ds4_tokens prompt = {};
         build_prompt(g_engine, request, &prompt);
+        int n_predict = request->tokens() > 0 ? request->tokens() : 256;
 
-        const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
-        const bool starts_in_thinking = think_enabled &&
-            request->usetokenizertemplate() && request->messages_size() > 0;
-        ds4cpp::RequestLifecycle lifecycle;
-        StreamCtx s = {
-            g_engine, writer, &lifecycle,
-            ds4cpp::DsmlParser(starts_in_thinking), 0, {}};
+        StreamCtx s = {g_engine, writer, {}, 0, false, {}};
         std::string cache_key = render_prompt_text(request);
         size_t cache_hit = maybe_load_cache(cache_key);
         (void)cache_hit;
@@ -969,26 +879,14 @@ public:
         // Manual loop on g_session - see Predict() above for the rationale.
         // MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
         char err[256] = {0};
-        int rc;
-        {
-            ds4cpp::CancelCallbackScope cancel_scope(
-                g_session, set_session_cancel, server_context_cancelled, context);
-            rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
-        }
+        int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
         ds4_tokens_free(&prompt);
-        if (rc == DS4_SESSION_SYNC_INTERRUPTED) {
-            lifecycle.ObserveContextCancellation(true);
-        }
-        const bool generation_started = rc == 0;
-        if (generation_started) {
-            const int n_predict = ds4cpp::EffectiveGenerationLimit(
-                request->tokens(), ds4_session_ctx(g_session),
-                ds4_session_pos(g_session));
+        if (rc == 0) {
             const int eos = ds4_token_eos(g_engine);
             const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
+            const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
             int produced = 0;
-            while (produced < n_predict) {
-                if (!request_should_continue(&lifecycle, context)) break;
+            while (produced < n_predict && !s.aborted) {
                 SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
                 int first;
                 if (sp.temperature <= 0.0f) {
@@ -1002,77 +900,50 @@ public:
                 if (draft_max > 0 && sp.temperature <= 0.0f) {
                     constexpr int kAcceptedMax = 8;
                     int accepted[kAcceptedMax];
-                    const int remaining = ds4cpp::RemainingGenerationBudget(
-                        n_predict, produced);
-                    const int cap = ds4cpp::SpeculativeAcceptedCapacity(
-                        remaining, draft_max, kAcceptedMax);
+                    int cap = std::min(kAcceptedMax, draft_max + 1);
                     int n = ds4_session_eval_speculative_argmax(
-                        g_session, first, remaining, eos,
+                        g_session, first, draft_max, eos,
                         accepted, cap, err, sizeof(err));
                     if (n < 0) { rc = -1; break; }
                     bool stop = false;
                     for (int j = 0; j < n; ++j) {
-                        if (!request_should_continue(&lifecycle, context)) {
-                            stop = true;
-                            break;
-                        }
                         if (accepted[j] == eos) { stop = true; break; }
                         stream_emit(&s, accepted[j]);
-                        if (!lifecycle.ShouldContinue()) { stop = true; break; }
+                        if (s.aborted) { stop = true; break; }
                         if (++produced >= n_predict) { stop = true; break; }
                     }
                     if (stop) break;
                 } else {
                     stream_emit(&s, first);
-                    if (!lifecycle.ShouldContinue() || ++produced >= n_predict) break;
-                    if (!request_should_continue(&lifecycle, context)) break;
+                    if (s.aborted || ++produced >= n_predict) break;
                     rc = ds4_session_eval(g_session, first, err, sizeof(err));
                     if (rc != 0) break;
                 }
             }
+            stream_done(&s);
+        }
+        maybe_save_cache(cache_key);
+
+        // Flush parser state.
+        std::vector<ds4cpp::ParserEvent> events;
+        s.parser.Flush(events);
+        if (!events.empty() && !s.aborted) {
+            backend::Reply reply;
+            auto *delta = reply.add_chat_deltas();
+            for (const auto &e : events) {
+                if (e.type == ds4cpp::ParserEvent::CONTENT) {
+                    delta->set_content(delta->content() + e.text);
+                } else if (e.type == ds4cpp::ParserEvent::REASONING) {
+                    delta->set_reasoning_content(delta->reasoning_content() + e.text);
+                }
+            }
+            s.writer->Write(reply);
         }
 
-        request_should_continue(&lifecycle, context);
-        ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
-            rc == DS4_SESSION_SYNC_INTERRUPTED, rc != 0,
-            !lifecycle.ShouldFinalize());
-        terminal = ds4cpp::RunPostlude(
-            terminal,
-            [&]() {
-                ds4cpp::DsmlParser staged_parser = s.parser;
-                std::vector<ds4cpp::ParserEvent> events;
-                staged_parser.Flush(events);
-                bool write_succeeded = true;
-                if (!events.empty()) {
-                    backend::Reply reply;
-                    auto *delta = reply.add_chat_deltas();
-                    for (const auto &e : events) {
-                        if (e.type == ds4cpp::ParserEvent::CONTENT) {
-                            delta->set_content(delta->content() + e.text);
-                        } else if (e.type == ds4cpp::ParserEvent::REASONING) {
-                            delta->set_reasoning_content(
-                                delta->reasoning_content() + e.text);
-                        }
-                    }
-                    write_succeeded = s.writer->Write(reply);
-                }
-                lifecycle.ObserveStreamWrite(write_succeeded);
-                request_should_continue(&lifecycle, context);
-                if (!lifecycle.ShouldFinalize()) return false;
-                s.parser = std::move(staged_parser);
-                if (generation_started) stream_done(&s);
-                return true;
-            },
-            [&]() { maybe_save_cache(cache_key); });
-
-        if (terminal.cause == ds4cpp::TerminalCause::EngineError) {
+        if (rc != 0 && !s.aborted) {
             return GStatus(StatusCode::INTERNAL,
                           std::string("ds4 generation failed: ") + err);
         }
-        if (terminal.cause == ds4cpp::TerminalCause::Cancelled) {
-            return GStatus(StatusCode::CANCELLED,
-                          "ds4 request cancelled");
-        }
         return GStatus::OK;
     }
 

backend/cpp/ds4/request_lifecycle.h

@@ -1,111 +0,0 @@
-// SPDX-License-Identifier: MIT
-#pragma once
-
-namespace ds4cpp {
-
-using CancelCallback = bool (*)(void *);
-using CancelSetter = void (*)(void *, CancelCallback, void *) noexcept;
-
-class CancelCallbackScope {
-public:
-    CancelCallbackScope(void *target, CancelSetter setter,
-                        CancelCallback callback, void *userdata) noexcept
-        : target_(target), setter_(setter) {
-        setter_(target_, callback, userdata);
-    }
-
-    ~CancelCallbackScope() noexcept {
-        setter_(target_, nullptr, nullptr);
-    }
-
-    CancelCallbackScope(const CancelCallbackScope &) = delete;
-    CancelCallbackScope &operator=(const CancelCallbackScope &) = delete;
-
-private:
-    void *target_;
-    CancelSetter setter_;
-};
-
-enum class RouteWaitDecision {
-    Pending,
-    Ready,
-    Error,
-    Cancelled,
-};
-
-inline RouteWaitDecision DecideRouteWait(int route_status, bool cancelled) {
-    if (cancelled) return RouteWaitDecision::Cancelled;
-    if (route_status > 0) return RouteWaitDecision::Ready;
-    if (route_status < 0) return RouteWaitDecision::Error;
-    return RouteWaitDecision::Pending;
-}
-
-enum class TerminalCause {
-    Success,
-    Cancelled,
-    EngineError,
-};
-
-inline TerminalCause DecideTerminalCause(bool sync_interrupted,
-                                         bool engine_error,
-                                         bool abandoned) {
-    if (sync_interrupted) return TerminalCause::Cancelled;
-    if (engine_error) return TerminalCause::EngineError;
-    if (abandoned) return TerminalCause::Cancelled;
-    return TerminalCause::Success;
-}
-
-struct TerminalDecision {
-    TerminalCause cause;
-    bool should_finalize;
-};
-
-inline TerminalDecision ResolveTerminalDecision(bool sync_interrupted,
-                                                 bool engine_error,
-                                                 bool abandoned) {
-    return {
-        DecideTerminalCause(sync_interrupted, engine_error, abandoned),
-        !sync_interrupted && !abandoned,
-    };
-}
-
-template <typename Finalize, typename Persist>
-TerminalDecision RunPostlude(TerminalDecision terminal,
-                             Finalize transactional_finalize,
-                             Persist persist) {
-    if (!terminal.should_finalize) return terminal;
-    if (!transactional_finalize()) {
-        terminal.should_finalize = false;
-        if (terminal.cause != TerminalCause::EngineError) {
-            terminal.cause = TerminalCause::Cancelled;
-        }
-        return terminal;
-    }
-    persist();
-    return terminal;
-}
-
-class RequestLifecycle {
-public:
-    void ObserveContextCancellation(bool cancelled) {
-        context_cancelled_ = context_cancelled_ || cancelled;
-    }
-
-    void ObserveStreamWrite(bool succeeded) {
-        stream_write_aborted_ = stream_write_aborted_ || !succeeded;
-    }
-
-    bool ShouldContinue() const {
-        return !context_cancelled_ && !stream_write_aborted_;
-    }
-
-    bool ShouldFinalize() const {
-        return ShouldContinue();
-    }
-
-private:
-    bool context_cancelled_ = false;
-    bool stream_write_aborted_ = false;
-};
-
-} // namespace ds4cpp

backend/cpp/ds4/request_lifecycle_test.cpp

@@ -1,414 +0,0 @@
-// SPDX-License-Identifier: MIT
-// Standalone regression tests for DS4 request cancellation policy.
-
-#include "request_lifecycle.h"
-
-#include <cstdio>
-
-namespace {
-
-int failures = 0;
-
-struct FakeCancelTarget {
-    ds4cpp::CancelCallback callback = nullptr;
-    void *userdata = nullptr;
-    int installs = 0;
-    int clears = 0;
-};
-
-struct PostludeCounts {
-    int finalize_attempts = 0;
-    int finalize_commits = 0;
-    int cache_persists = 0;
-    bool cache_followed_commit = true;
-};
-
-ds4cpp::TerminalDecision run_fake_postlude(
-        ds4cpp::TerminalDecision terminal, bool finalize_succeeds,
-        PostludeCounts *counts) {
-    return ds4cpp::RunPostlude(
-        terminal,
-        [=]() {
-            counts->finalize_attempts++;
-            if (!finalize_succeeds) return false;
-            counts->finalize_commits++;
-            return true;
-        },
-        [=]() {
-            counts->cache_followed_commit = counts->finalize_commits == 1;
-            counts->cache_persists++;
-        });
-}
-
-bool fake_cancel(void *) {
-    return false;
-}
-
-void fake_set_cancel(void *target, ds4cpp::CancelCallback callback,
-                     void *userdata) noexcept {
-    auto *fake = static_cast<FakeCancelTarget *>(target);
-    fake->callback = callback;
-    fake->userdata = userdata;
-    if (callback) {
-        fake->installs++;
-    } else {
-        fake->clears++;
-    }
-}
-
-void check(bool condition, const char *name) {
-    if (condition) return;
-    std::fprintf(stderr, "FAIL %s\n", name);
-    failures++;
-}
-
-// Production mutation caught: treating an active request as abandoned would
-// skip its parser finalization and cache save.
-void test_active_request_continues_and_finalizes() {
-    ds4cpp::RequestLifecycle request;
-
-    check(request.ShouldContinue(), "active:continue");
-    check(request.ShouldFinalize(), "active:finalize");
-}
-
-// Production mutation caught: omitting the ServerContext cancellation branch
-// would continue decoding and finalize a partial response.
-void test_context_cancellation_stops_without_finalizing() {
-    ds4cpp::RequestLifecycle request;
-
-    request.ObserveContextCancellation(true);
-
-    check(!request.ShouldContinue(), "context_cancelled:stop");
-    check(!request.ShouldFinalize(), "context_cancelled:no_finalize");
-}
-
-// Production mutation caught: ignoring ServerWriter::Write failure would keep
-// streaming and finalize a response whose client has gone away.
-void test_stream_write_abort_stops_without_finalizing() {
-    ds4cpp::RequestLifecycle request;
-
-    request.ObserveStreamWrite(false);
-
-    check(!request.ShouldContinue(), "write_abort:stop");
-    check(!request.ShouldFinalize(), "write_abort:no_finalize");
-}
-
-// Production mutation caught: combining cancellation and write failure with
-// AND would fail to stop when either signal occurs on its own.
-void test_cancellation_and_write_abort_are_independent_or_conditions() {
-    ds4cpp::RequestLifecycle cancelled;
-    cancelled.ObserveContextCancellation(true);
-    cancelled.ObserveStreamWrite(true);
-
-    ds4cpp::RequestLifecycle write_aborted;
-    write_aborted.ObserveContextCancellation(false);
-    write_aborted.ObserveStreamWrite(false);
-
-    check(!cancelled.ShouldContinue(), "or:context_only");
-    check(!write_aborted.ShouldContinue(), "or:write_only");
-}
-
-// Production mutation caught: treating an incomplete distributed route as an
-// error would return before workers have time to connect.
-void test_route_wait_pending() {
-    check(ds4cpp::DecideRouteWait(0, false) ==
-              ds4cpp::RouteWaitDecision::Pending,
-          "route_wait:pending");
-}
-
-// Production mutation caught: failing to recognize a complete route would
-// keep a ready inference request in the polling loop.
-void test_route_wait_ready() {
-    check(ds4cpp::DecideRouteWait(1, false) ==
-              ds4cpp::RouteWaitDecision::Ready,
-          "route_wait:ready");
-}
-
-// Production mutation caught: ignoring a route probe error would poll until a
-// misleading timeout instead of returning UNAVAILABLE promptly.
-void test_route_wait_error() {
-    check(ds4cpp::DecideRouteWait(-1, false) ==
-              ds4cpp::RouteWaitDecision::Error,
-          "route_wait:error");
-}
-
-// Production mutation caught: omitting cancellation from route waiting would
-// leave an abandoned request blocked until the distributed timeout.
-void test_route_wait_cancellation() {
-    check(ds4cpp::DecideRouteWait(0, true) ==
-              ds4cpp::RouteWaitDecision::Cancelled,
-          "route_wait:cancelled");
-}
-
-// Production mutation caught: checking route errors before cancellation would
-// report UNAVAILABLE for a request the client already abandoned.
-void test_route_wait_cancellation_precedes_error() {
-    check(ds4cpp::DecideRouteWait(-1, true) ==
-              ds4cpp::RouteWaitDecision::Cancelled,
-          "route_wait:cancellation_precedence");
-}
-
-// Production mutation caught: classifying a successful active request as a
-// terminal failure would suppress its normal response finalization.
-void test_terminal_success() {
-    check(ds4cpp::DecideTerminalCause(false, false, false) ==
-              ds4cpp::TerminalCause::Success,
-          "terminal:success");
-}
-
-// Production mutation caught: treating DS4's cooperative sync interruption
-// as an ordinary engine error would return INTERNAL instead of CANCELLED.
-void test_terminal_sync_interruption_is_cancelled() {
-    check(ds4cpp::DecideTerminalCause(true, true, true) ==
-              ds4cpp::TerminalCause::Cancelled,
-          "terminal:sync_interrupted");
-}
-
-// Production mutation caught: treating every nonzero engine result as client
-// abandonment would hide genuine DS4 failures behind CANCELLED.
-void test_terminal_engine_error() {
-    check(ds4cpp::DecideTerminalCause(false, true, false) ==
-              ds4cpp::TerminalCause::EngineError,
-          "terminal:engine_error");
-}
-
-// Production mutation caught: ignoring an rc==0 context cancellation would
-// finalize and cache an abandoned request.
-void test_terminal_context_abandonment() {
-    ds4cpp::RequestLifecycle request;
-    request.ObserveContextCancellation(true);
-
-    check(ds4cpp::DecideTerminalCause(
-              false, false, !request.ShouldFinalize()) ==
-              ds4cpp::TerminalCause::Cancelled,
-          "terminal:context_abandonment");
-}
-
-// Production mutation caught: ignoring an rc==0 stream write failure would
-// finalize and cache an abandoned streaming request.
-void test_terminal_write_abandonment() {
-    ds4cpp::RequestLifecycle request;
-    request.ObserveStreamWrite(false);
-
-    check(ds4cpp::DecideTerminalCause(
-              false, false, !request.ShouldFinalize()) ==
-              ds4cpp::TerminalCause::Cancelled,
-          "terminal:write_abandonment");
-}
-
-// Production mutation caught: checking late cancellation or write failure
-// before a determined ordinary DS4 error would replace INTERNAL with CANCELLED.
-void test_terminal_engine_error_precedes_late_abandonment() {
-    ds4cpp::RequestLifecycle cancelled;
-    cancelled.ObserveContextCancellation(true);
-    ds4cpp::RequestLifecycle write_aborted;
-    write_aborted.ObserveStreamWrite(false);
-
-    check(ds4cpp::DecideTerminalCause(
-              false, true, !cancelled.ShouldFinalize()) ==
-              ds4cpp::TerminalCause::EngineError,
-          "terminal:engine_error_precedes_cancellation");
-    check(ds4cpp::DecideTerminalCause(
-              false, true, !write_aborted.ShouldFinalize()) ==
-              ds4cpp::TerminalCause::EngineError,
-          "terminal:engine_error_precedes_write_abort");
-}
-
-// Production mutation caught: using status precedence alone to gate side
-// effects would finalize and persist an engine-error request abandoned later.
-void test_abandoned_engine_error_keeps_internal_without_finalizing() {
-    ds4cpp::RequestLifecycle request;
-    request.ObserveContextCancellation(true);
-
-    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
-        false, true, !request.ShouldFinalize());
-
-    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
-          "terminal_decision:abandoned_engine_error_status");
-    check(!terminal.should_finalize,
-          "terminal_decision:abandoned_engine_error_no_finalize");
-}
-
-// Production mutation caught: suppressing side effects for every engine error
-// would change the existing finalization and cache behavior of active failures.
-void test_active_engine_error_still_finalizes() {
-    ds4cpp::RequestLifecycle request;
-
-    ds4cpp::TerminalDecision terminal = ds4cpp::ResolveTerminalDecision(
-        false, true, !request.ShouldFinalize());
-
-    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
-          "terminal_decision:active_engine_error_status");
-    check(terminal.should_finalize,
-          "terminal_decision:active_engine_error_finalize");
-}
-
-// Production mutation caught: persisting before committed finalization would
-// cache a state whose final buffered stream reply was never completed.
-void test_postlude_active_success_commits_then_persists() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::Success, true}, true, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::Success,
-          "postlude:success_outcome");
-    check(terminal.should_finalize, "postlude:success_committed");
-    check(counts.finalize_attempts == 1, "postlude:success_attempts");
-    check(counts.finalize_commits == 1, "postlude:success_commits");
-    check(counts.cache_persists == 1, "postlude:success_cache");
-    check(counts.cache_followed_commit, "postlude:success_cache_order");
-}
-
-// Production mutation caught: starting the postlude for an already-cancelled
-// request would flush buffered parser state or persist an abandoned session.
-void test_postlude_cancellation_skips_all_side_effects() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::Cancelled, false}, true, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
-          "postlude:cancelled_outcome");
-    check(counts.finalize_attempts == 0, "postlude:cancelled_attempts");
-    check(counts.finalize_commits == 0, "postlude:cancelled_commits");
-    check(counts.cache_persists == 0, "postlude:cancelled_cache");
-}
-
-// Production mutation caught: committing the live parser or cache after a
-// failed final Write would publish an abandoned streaming postlude.
-void test_postlude_finalize_failure_cancels_without_commit_or_cache() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::Success, true}, false, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::Cancelled,
-          "postlude:write_failure_outcome");
-    check(!terminal.should_finalize, "postlude:write_failure_not_committed");
-    check(counts.finalize_attempts == 1, "postlude:write_failure_attempts");
-    check(counts.finalize_commits == 0, "postlude:write_failure_commits");
-    check(counts.cache_persists == 0, "postlude:write_failure_cache");
-}
-
-// Production mutation caught: skipping the postlude for every engine error
-// would change active internal-error finalization and cache behavior.
-void test_postlude_active_engine_error_finalizes_and_persists() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::EngineError, true}, true, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
-          "postlude:engine_error_outcome");
-    check(counts.finalize_attempts == 1, "postlude:engine_error_attempts");
-    check(counts.finalize_commits == 1, "postlude:engine_error_commits");
-    check(counts.cache_persists == 1, "postlude:engine_error_cache");
-    check(counts.cache_followed_commit, "postlude:engine_error_cache_order");
-}
-
-// Production mutation caught: replacing every failed transactional finalize
-// with cancellation would hide an already-determined engine error.
-void test_postlude_engine_error_finalize_failure_preserves_internal() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::EngineError, true}, false, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
-          "postlude:engine_error_write_failure_outcome");
-    check(!terminal.should_finalize,
-          "postlude:engine_error_write_failure_not_committed");
-    check(counts.finalize_attempts == 1,
-          "postlude:engine_error_write_failure_attempts");
-    check(counts.finalize_commits == 0,
-          "postlude:engine_error_write_failure_commits");
-    check(counts.cache_persists == 0,
-          "postlude:engine_error_write_failure_cache");
-}
-
-// Production mutation caught: status precedence must not grant side-effect
-// permission to an engine-error request that was also abandoned.
-void test_postlude_abandoned_engine_error_skips_all_side_effects() {
-    PostludeCounts counts;
-
-    ds4cpp::TerminalDecision terminal = run_fake_postlude(
-        {ds4cpp::TerminalCause::EngineError, false}, true, &counts);
-
-    check(terminal.cause == ds4cpp::TerminalCause::EngineError,
-          "postlude:abandoned_engine_error_outcome");
-    check(counts.finalize_attempts == 0,
-          "postlude:abandoned_engine_error_attempts");
-    check(counts.finalize_commits == 0,
-          "postlude:abandoned_engine_error_commits");
-    check(counts.cache_persists == 0,
-          "postlude:abandoned_engine_error_cache");
-}
-
-// Production mutation caught: failing to install the request callback would
-// make DS4 prompt synchronization unable to observe client cancellation.
-void test_cancel_callback_scope_installs_callback() {
-    FakeCancelTarget target;
-    int request_context = 42;
-
-    {
-        ds4cpp::CancelCallbackScope scope(
-            &target, fake_set_cancel, fake_cancel, &request_context);
-        check(target.callback == fake_cancel, "cancel_scope:callback_installed");
-        check(target.userdata == &request_context, "cancel_scope:userdata_installed");
-        check(target.installs == 1, "cancel_scope:installed_once");
-    }
-}
-
-// Production mutation caught: failing to clear the callback at every scope
-// exit would leave DS4 pointing at a destroyed stack-owned ServerContext.
-void test_cancel_callback_scope_clears_callback() {
-    FakeCancelTarget target;
-    int request_context = 42;
-
-    {
-        ds4cpp::CancelCallbackScope scope(
-            &target, fake_set_cancel, fake_cancel, &request_context);
-    }
-
-    check(target.callback == nullptr, "cancel_scope:callback_cleared");
-    check(target.userdata == nullptr, "cancel_scope:userdata_cleared");
-    check(target.clears == 1, "cancel_scope:cleared_once");
-}
-
-} // namespace
-
-int main() {
-    test_active_request_continues_and_finalizes();
-    test_context_cancellation_stops_without_finalizing();
-    test_stream_write_abort_stops_without_finalizing();
-    test_cancellation_and_write_abort_are_independent_or_conditions();
-    test_route_wait_pending();
-    test_route_wait_ready();
-    test_route_wait_error();
-    test_route_wait_cancellation();
-    test_route_wait_cancellation_precedes_error();
-    test_terminal_success();
-    test_terminal_sync_interruption_is_cancelled();
-    test_terminal_engine_error();
-    test_terminal_context_abandonment();
-    test_terminal_write_abandonment();
-    test_terminal_engine_error_precedes_late_abandonment();
-    test_abandoned_engine_error_keeps_internal_without_finalizing();
-    test_active_engine_error_still_finalizes();
-    test_postlude_active_success_commits_then_persists();
-    test_postlude_cancellation_skips_all_side_effects();
-    test_postlude_finalize_failure_cancels_without_commit_or_cache();
-    test_postlude_active_engine_error_finalizes_and_persists();
-    test_postlude_engine_error_finalize_failure_preserves_internal();
-    test_postlude_abandoned_engine_error_skips_all_side_effects();
-    test_cancel_callback_scope_installs_callback();
-    test_cancel_callback_scope_clears_callback();
-
-    if (failures == 0) {
-        std::fprintf(stderr, "all request_lifecycle checks passed\n");
-        return 0;
-    }
-    std::fprintf(stderr, "%d check(s) failed\n", failures);
-    return 1;
-}

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=3bb386eb68ffee0a5dc7db21da0735d594929eeb
+IK_LLAMA_VERSION?=15dddc60b3fc937a9e2a210359ecce392ccdf446
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/CMakeLists.txt

@@ -125,15 +125,4 @@ if(LLAMA_GRPC_BUILD_TESTS)
     target_include_directories(thread_params_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
     target_compile_features(thread_params_test PRIVATE cxx_std_17)
     add_test(NAME thread_params_test COMMAND thread_params_test)
-
-    add_executable(model_load_error_test model_load_error_test.cpp model_load_error.h)
-    target_include_directories(model_load_error_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
-    target_compile_features(model_load_error_test PRIVATE cxx_std_17)
-    add_test(NAME model_load_error_test COMMAND model_load_error_test)
-
-    # Dead-stream tracker (standard library only).
-    add_executable(stream_peer_test stream_peer_test.cpp stream_peer.h)
-    target_include_directories(stream_peer_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
-    target_compile_features(stream_peer_test PRIVATE cxx_std_17)
-    add_test(NAME stream_peer_test COMMAND stream_peer_test)
 endif()

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=df03399b885831b2a1603b3abb0d8c156808e363
+LLAMA_VERSION?=d7bd3bfcad3e29c7e49fd26f38c79ee3e9a3fd6b
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/grpc-server.cpp

@@ -53,11 +53,9 @@
 #include "arg.h"
 #include "chat-auto-parser.h"
 #include "llama_compat.h" // fork-skew switches, generated by prepare.sh
-#include "model_load_error.h"
 #include "thread_params.h"
 #include "message_content.h"
 #include "passthrough_options.h"
-#include "stream_peer.h"
 #include "tts_request_options.h"
 #include <getopt.h>
 #include <grpcpp/ext/proto_server_reflection_plugin.h>
@@ -1630,8 +1628,7 @@ public:
             {
                 std::lock_guard<std::mutex> lock(error_capture_data.error_mutex);
                 if (!error_capture_data.captured_error.empty()) {
-                    error_msg += ". Error: " +
-                                 localai::model_load_error_with_hint(error_capture_data.captured_error);
+                    error_msg += ". Error: " + error_capture_data.captured_error;
                 } else {
                     error_msg += ". Model file may not exist or be invalid.";
                 }
@@ -2269,11 +2266,6 @@ public:
         // such concept, so there is nothing to emit — the real tokens arrive in
         // the loop below. Feeding this null into build_reply_from_json would
         // throw (uncaught) and surface as a generic RPC error.
-        // A write that returns false means the peer is gone for good. Track it
-        // so the loop below stops decoding instead of feeding a dead stream —
-        // see stream_peer.h for why that matters to everyone else's requests.
-        llama_grpc::StreamPeer peer;
-
         if (first_res_json.is_null()) {
             // skip the begin-of-stream marker
         } else if (first_res_json.is_array()) {
@@ -2286,21 +2278,17 @@ public:
                 if (!is_role_init) {
                     attach_chat_deltas(reply, first_result.get());
                 }
-                peer.observe_write(writer->Write(reply));
-                if (peer.gone()) {
-                    break;
-                }
+                writer->Write(reply);
             }
         } else {
             auto reply = build_reply_from_json(first_res_json, first_result.get());
             attach_chat_deltas(reply, first_result.get());
-            peer.observe_write(writer->Write(reply));
+            writer->Write(reply);
         }
 
         // Process subsequent results
         while (rd.has_next()) {
-            peer.observe_cancelled(context->IsCancelled());
-            if (peer.gone()) {
+            if (context->IsCancelled()) {
                 break;
             }
 
@@ -2321,22 +2309,17 @@ public:
                     if (!is_role_init) {
                         attach_chat_deltas(reply, result.get());
                     }
-                    peer.observe_write(writer->Write(reply));
-                    if (peer.gone()) {
-                        break;
-                    }
+                    writer->Write(reply);
                 }
             } else {
                 auto reply = build_reply_from_json(res_json, result.get());
                 attach_chat_deltas(reply, result.get());
-                peer.observe_write(writer->Write(reply));
+                writer->Write(reply);
             }
         }
 
-        // Returning here is what releases the slot: ~server_response_reader()
-        // posts SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
-        peer.observe_cancelled(context->IsCancelled());
-        if (peer.gone()) {
+        // Check if context was cancelled during processing
+        if (context->IsCancelled()) {
             return grpc::Status(grpc::StatusCode::CANCELLED, "Request cancelled by client");
         }
 

backend/cpp/llama-cpp/model_load_error.h

@@ -1,24 +0,0 @@
-// SPDX-License-Identifier: MIT
-
-#pragma once
-
-#include <string>
-
-namespace localai {
-
-inline std::string model_load_error_with_hint(const std::string& error) {
-  const std::string mismatch = "wrong number of tensors; expected ";
-  const std::string got = ", got ";
-  const std::string::size_type mismatch_pos = error.find(mismatch);
-  if (mismatch_pos == std::string::npos ||
-      error.find(got, mismatch_pos + mismatch.size()) == std::string::npos) {
-    return error;
-  }
-
-  return error +
-         " Hint: the model may be incompatible with this llama.cpp backend "
-         "or the GGUF file may be corrupt. Try a newer compatible backend "
-         "and verify or re-download the model file.";
-}
-
-}  // namespace localai

backend/cpp/llama-cpp/model_load_error_test.cpp

@@ -1,25 +0,0 @@
-// SPDX-License-Identifier: MIT
-
-#include "model_load_error.h"
-
-#include <cassert>
-#include <string>
-
-int main() {
-  const std::string issue_error =
-      "llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 2131, got 720; "
-      "llama_model_load_from_file_impl: failed to load model";
-  const std::string issue_result = localai::model_load_error_with_hint(issue_error);
-  assert(issue_result.compare(0, issue_error.size(), issue_error) == 0);
-  assert(issue_result.find("incompatible") != std::string::npos);
-  assert(issue_result.find("corrupt") != std::string::npos);
-
-  const std::string generic_error =
-      "wrong number of tensors; expected 42, got 17";
-  const std::string generic_result = localai::model_load_error_with_hint(generic_error);
-  assert(generic_result.compare(0, generic_error.size(), generic_error) == 0);
-  assert(generic_result.size() > generic_error.size());
-
-  const std::string unrelated_error = "failed to open GGUF file";
-  assert(localai::model_load_error_with_hint(unrelated_error) == unrelated_error);
-}

backend/cpp/llama-cpp/prepare.sh

@@ -15,40 +15,12 @@ if [ -d "patches" ]; then
     done
 fi
 
-## Apple RDMA link fixup.
-
-## ggml-rpc hands Apple's librdma to the linker with
-## target_link_options(ggml-rpc PRIVATE "LINKER:-weak_library,..."). Link options are not
-## a usage requirement of a static library, so in our BUILD_SHARED_LIBS=OFF build the flag
-## dies with libggml-rpc.a and every ibv_* symbol transport-apple.cpp reaches for comes out
-## undefined when grpc-server and ggml-rpc-server link. Re-declare the same weak link as
-## INTERFACE so it travels to whoever links the static library.
-##
-## Guarded on the marker so a second prepare.sh over the same checkout is a no-op, and on
-## GGML_RPC_RDMA_APPLE so forks that branched before the Apple RDMA transport (turboquant,
-## bonsai) are left alone.
-RPC_CMAKE=llama.cpp/ggml/src/ggml-rpc/CMakeLists.txt
-if [ -f "$RPC_CMAKE" ] && grep -q "GGML_RPC_RDMA_APPLE" "$RPC_CMAKE" && ! grep -q "LOCALAI_RDMA_IFACE" "$RPC_CMAKE"; then
-    echo "==> ggml-rpc carries the Apple RDMA transport, re-declaring its weak librdma link as INTERFACE"
-    cat >> "$RPC_CMAKE" <<'EOF'
-
-# LOCALAI_RDMA_IFACE: added by backend/cpp/llama-cpp/prepare.sh
-if (GGML_RPC_RDMA AND APPLE AND NOT BUILD_SHARED_LIBS)
-    target_link_options(ggml-rpc INTERFACE "LINKER:-weak_library,${RDMA_LIB}")
-endif()
-EOF
-fi
-
 for file in $(ls llama.cpp/tools/server/); do
     cp -rfv llama.cpp/tools/server/$file llama.cpp/tools/grpc-server/
 done
 
 cp -r CMakeLists.txt llama.cpp/tools/grpc-server/
 cp -r grpc-server.cpp llama.cpp/tools/grpc-server/
-# Model-load diagnostics (included by grpc-server.cpp) and their standalone
-# regression test.
-cp -r model_load_error.h llama.cpp/tools/grpc-server/
-cp -r model_load_error_test.cpp llama.cpp/tools/grpc-server/
 # Shared message-reconstruction helpers (included by grpc-server.cpp) and their
 # unit test (compiled only when -DLLAMA_GRPC_BUILD_TESTS=ON).
 cp -r message_content.h llama.cpp/tools/grpc-server/
@@ -67,10 +39,6 @@ cp -r thread_params_test.cpp llama.cpp/tools/grpc-server/
 # test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
 cp -r parent_watch.h llama.cpp/tools/grpc-server/
 cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
-# Dead-stream tracker (included by grpc-server.cpp) and its standalone unit
-# test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
-cp -r stream_peer.h llama.cpp/tools/grpc-server/
-cp -r stream_peer_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 

backend/cpp/llama-cpp/stream_peer.h

@@ -1,44 +0,0 @@
-// SPDX-License-Identifier: MIT
-#pragma once
-
-namespace llama_grpc {
-
-// Tracks whether a server-streaming RPC still has somewhere to send tokens.
-//
-// grpc::ServerWriter::Write() returns false once the peer is gone, and a
-// stream never recovers afterwards. Ignoring that result is not harmless: the
-// handler goes on draining decoded tokens into a dead stream, so the llama.cpp
-// slot stays busy for the rest of the request's token budget. A model config
-// with no max_tokens and a large context turns that into tens of minutes per
-// abandoned request, and the slots are exactly what every other request queues
-// behind.
-//
-// Returning as soon as the peer is gone is what frees the slot: the handler's
-// server_response_reader then goes out of scope and its destructor posts
-// SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
-class StreamPeer {
-public:
-    // Records the outcome of a Write(). Once a write has failed the peer stays
-    // gone -- a later write cannot succeed on a broken stream.
-    void observe_write(bool ok) noexcept {
-        if (!ok) {
-            gone_ = true;
-        }
-    }
-
-    // Folds in the RPC's own cancellation flag, so callers have a single
-    // predicate to test rather than two that can disagree.
-    void observe_cancelled(bool cancelled) noexcept {
-        if (cancelled) {
-            gone_ = true;
-        }
-    }
-
-    bool gone() const noexcept { return gone_; }
-    bool alive() const noexcept { return !gone_; }
-
-private:
-    bool gone_ = false;
-};
-
-} // namespace llama_grpc

backend/cpp/llama-cpp/stream_peer_test.cpp

@@ -1,67 +0,0 @@
-#include "stream_peer.h"
-
-#include <cstdio>
-
-namespace {
-
-int failures = 0;
-
-void check(bool condition, const char *what) {
-    if (!condition) {
-        std::fprintf(stderr, "FAIL: %s\n", what);
-        ++failures;
-    }
-}
-
-} // namespace
-
-int main() {
-    {
-        llama_grpc::StreamPeer peer;
-        check(peer.alive(), "a fresh peer is alive");
-        check(!peer.gone(), "a fresh peer is not gone");
-    }
-
-    {
-        llama_grpc::StreamPeer peer;
-        peer.observe_write(true);
-        peer.observe_write(true);
-        check(peer.alive(), "successful writes keep the peer alive");
-    }
-
-    {
-        llama_grpc::StreamPeer peer;
-        peer.observe_write(false);
-        check(peer.gone(), "a failed write marks the peer gone");
-    }
-
-    {
-        // The whole point of the guard: a stream never comes back, so a later
-        // success must not resurrect a peer an earlier failure retired.
-        llama_grpc::StreamPeer peer;
-        peer.observe_write(false);
-        peer.observe_write(true);
-        check(peer.gone(), "a failed write is sticky across later writes");
-    }
-
-    {
-        llama_grpc::StreamPeer peer;
-        peer.observe_cancelled(false);
-        check(peer.alive(), "an uncancelled RPC keeps the peer alive");
-        peer.observe_cancelled(true);
-        check(peer.gone(), "cancellation marks the peer gone");
-    }
-
-    {
-        llama_grpc::StreamPeer peer;
-        peer.observe_cancelled(true);
-        peer.observe_cancelled(false);
-        check(peer.gone(), "cancellation is sticky across later checks");
-    }
-
-    if (failures != 0) {
-        std::fprintf(stderr, "%d check(s) failed\n", failures);
-        return 1;
-    }
-    return 0;
-}

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=301acd87b036764973b8bfba71e0a21818036d33
+CRISPASR_VERSION?=ae4474dd8306384a0e697183d863dfc52e69a2fb
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/crispasr/gocrispasr.go

@@ -615,10 +615,10 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 		return fmt.Errorf("crispasr: tempfile: %w", err)
 	}
 	dst := tmp.Name()
-	defer func() { _ = os.Remove(dst) }()
 	if err := tmp.Close(); err != nil {
 		return fmt.Errorf("crispasr: close tempfile: %w", err)
 	}
+	defer func() { _ = os.Remove(dst) }()
 
 	if err := writeWAV(dst, pcm, w.sampleRate); err != nil {
 		return err

backend/go/depth-anything-cpp/Makefile

@@ -14,7 +14,7 @@ JOBS?=$(shell nproc --ignore=1)
 # It is kept alive by the upstream tag da2-support (survives a squash-merge);
 # repoint to the master merge commit once mudler/depth-anything.cpp PR #1 lands.
 DEPTHANYTHING_REPO?=https://github.com/mudler/depth-anything.cpp.git
-DEPTHANYTHING_VERSION?=14f7461d1f704761a038ac9f50dbde8fdb7275e2
+DEPTHANYTHING_VERSION?=739992d10bf9472c46dcd4622b14d2b20766c58d
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/locate-anything-cpp/golocateanythingcpp.go

@@ -11,7 +11,6 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
-	"runtime"
 	"unsafe"
 
 	"github.com/mudler/LocalAI/pkg/grpc/base"
@@ -110,25 +109,30 @@ func (r *LocateAnythingCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, e
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: a text prompt is required (open-vocabulary detection)")
 	}
 
+	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to decode base64 image: %w", err)
 	}
-	if len(imgData) == 0 {
-		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: decoded image is empty")
+
+	tmpFile, err := os.CreateTemp("", "locate-anything-*.img")
+	if err != nil {
+		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to create temp file: %w", err)
+	}
+	defer func() { _ = os.Remove(tmpFile.Name()) }()
+
+	if _, err := tmpFile.Write(imgData); err != nil {
+		_ = tmpFile.Close()
+		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to write temp file: %w", err)
+	}
+	if err := tmpFile.Close(); err != nil {
+		return pb.DetectResponse{}, fmt.Errorf("locate-anything-cpp: failed to close temp file: %w", err)
 	}
 
 	// mode 0 = hybrid (Parallel Box Decoding). The JSON return value is unused:
 	// structured detections are read via the accessor functions. Still must
 	// free the returned string.
-	jsonPtr := CapiLocateBuffer(
-		r.handle,
-		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
-		uintptr(len(imgData)),
-		prompt,
-		0,
-	)
-	runtime.KeepAlive(imgData)
+	jsonPtr := CapiLocatePath(r.handle, tmpFile.Name(), prompt, 0)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}

backend/go/locate-anything-cpp/golocateanythingcpp_unit_test.go

@@ -1,54 +0,0 @@
-package main
-
-import (
-	"encoding/base64"
-	"path/filepath"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("LocateAnythingCpp detection input", func() {
-	It("detects from memory when the temporary directory is unavailable", func() {
-		originalLocateBuffer := CapiLocateBuffer
-		originalLocatePath := CapiLocatePath
-		originalGetNDetections := CapiGetNDetections
-		defer func() {
-			CapiLocateBuffer = originalLocateBuffer
-			CapiLocatePath = originalLocatePath
-			CapiGetNDetections = originalGetNDetections
-		}()
-
-		image := []byte("encoded-image")
-		var receivedData uintptr
-		var receivedLength uintptr
-		CapiLocateBuffer = func(_ uintptr, data uintptr, length uintptr, _ string, _ int32) uintptr {
-			receivedData = data
-			receivedLength = length
-			return 0
-		}
-		CapiLocatePath = func(_ uintptr, _ string, _ string, _ int32) uintptr {
-			Fail("path-based detection must not be called")
-			return 0
-		}
-		CapiGetNDetections = func(uintptr) int32 { return 0 }
-		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
-
-		result, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{
-			Src:    base64.StdEncoding.EncodeToString(image),
-			Prompt: "the object",
-		})
-
-		Expect(err).NotTo(HaveOccurred())
-		Expect(result.Detections).To(BeEmpty())
-		Expect(receivedData).NotTo(BeZero())
-		Expect(receivedLength).To(Equal(uintptr(len(image))))
-	})
-
-	It("rejects an empty decoded image", func() {
-		_, err := (&LocateAnythingCpp{handle: 1}).Detect(&pb.DetectOptions{Prompt: "the object"})
-
-		Expect(err).To(MatchError("locate-anything-cpp: decoded image is empty"))
-	})
-})

backend/go/nemo-speech-cpp/Makefile

@@ -12,7 +12,7 @@
 # runs 'make -C backend/go/$(BACKEND) build' and then copies package/), so it
 # has to produce the binary and the package, not just the shared libraries.
 
-NEMO_SPEECH_VERSION?=a5b6953c4a579a2bbd1c0913ad8a85c2a4d99953
+NEMO_SPEECH_VERSION?=4f9676226f667d14608487df744f375db87127f8
 NEMO_SPEECH_REPO?=https://github.com/NVIDIA/NeMo-Speech.cpp
 
 GOCMD?=go

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=040c8b344d8c670ce1475194751d119b5ef82c78
+OMNIVOICE_VERSION?=4f33af825d66e6ef1cb185e87b4589cacf747291
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/rfdetr-cpp/gorfdetrcpp.go

@@ -10,7 +10,6 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
-	"runtime"
 	"strconv"
 	"unsafe"
 
@@ -103,12 +102,24 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: model not loaded")
 	}
 
+	// Decode base64 image and write to temp file.
 	imgData, err := base64.StdEncoding.DecodeString(opts.Src)
 	if err != nil {
 		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to decode base64 image: %w", err)
 	}
-	if len(imgData) == 0 {
-		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: decoded image is empty")
+
+	tmpFile, err := os.CreateTemp("", "rfdetr-*.img")
+	if err != nil {
+		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to create temp file: %w", err)
+	}
+	defer func() { _ = os.Remove(tmpFile.Name()) }()
+
+	if _, err := tmpFile.Write(imgData); err != nil {
+		_ = tmpFile.Close()
+		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to write temp file: %w", err)
+	}
+	if err := tmpFile.Close(); err != nil {
+		return pb.DetectResponse{}, fmt.Errorf("rfdetr-cpp: failed to close temp file: %w", err)
 	}
 
 	threshold := opts.Threshold
@@ -116,18 +127,10 @@ func (r *RFDetrCpp) Detect(opts *pb.DetectOptions) (pb.DetectResponse, error) {
 		threshold = 0.5
 	}
 
-	// JSON output from the detection ABI is unused: we read structured detections via
+	// JSON output from detect_path is unused: we read structured detections via
 	// the accessor functions. Still must free the returned string.
 	var jsonPtr uintptr
-	rc := CapiDetectBuffer(
-		r.handle,
-		uintptr(unsafe.Pointer(unsafe.SliceData(imgData))),
-		uintptr(len(imgData)),
-		threshold,
-		uint32(defaultTopK),
-		&jsonPtr,
-	)
-	runtime.KeepAlive(imgData)
+	rc := CapiDetectPath(r.handle, tmpFile.Name(), threshold, uint32(defaultTopK), &jsonPtr)
 	if jsonPtr != 0 {
 		CapiFreeString(jsonPtr)
 	}

backend/go/rfdetr-cpp/gorfdetrcpp_unit_test.go

@@ -1,56 +0,0 @@
-package main
-
-import (
-	"encoding/base64"
-	"path/filepath"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("RFDetrCpp detection input", func() {
-	It("detects from memory when the temporary directory is unavailable", func() {
-		originalDetectBuffer := CapiDetectBuffer
-		originalDetectPath := CapiDetectPath
-		originalFreeString := CapiFreeString
-		originalGetNDetections := CapiGetNDetections
-		defer func() {
-			CapiDetectBuffer = originalDetectBuffer
-			CapiDetectPath = originalDetectPath
-			CapiFreeString = originalFreeString
-			CapiGetNDetections = originalGetNDetections
-		}()
-
-		image := []byte("encoded-image")
-		var receivedData uintptr
-		var receivedLength uintptr
-		CapiDetectBuffer = func(_ uintptr, data uintptr, length uintptr, _ float32, _ uint32, _ *uintptr) int32 {
-			receivedData = data
-			receivedLength = length
-			return 0
-		}
-		CapiDetectPath = func(_ uintptr, _ string, _ float32, _ uint32, _ *uintptr) int32 {
-			Fail("path-based detection must not be called")
-			return -1
-		}
-		CapiFreeString = func(uintptr) {}
-		CapiGetNDetections = func(uintptr) int32 { return 0 }
-		GinkgoT().Setenv("TMPDIR", filepath.Join(GinkgoT().TempDir(), "missing"))
-
-		result, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{
-			Src: base64.StdEncoding.EncodeToString(image),
-		})
-
-		Expect(err).NotTo(HaveOccurred())
-		Expect(result.Detections).To(BeEmpty())
-		Expect(receivedData).NotTo(BeZero())
-		Expect(receivedLength).To(Equal(uintptr(len(image))))
-	})
-
-	It("rejects an empty decoded image", func() {
-		_, err := (&RFDetrCpp{handle: 1}).Detect(&pb.DetectOptions{})
-
-		Expect(err).To(MatchError("rfdetr-cpp: decoded image is empty"))
-	})
-})

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=b68d58624d227682eb4b95ef8bcf569cd1311eb5
+STABLEDIFFUSION_GGML_VERSION?=be0e34480dada95f8ce9a021bbb95c5de85d67c7
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/stablediffusion-ggml/cpp/gosd.cpp

@@ -401,6 +401,7 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     const char *params_backend_arg = "";
     const char *rpc_servers_arg = "";
     const char *max_vram_arg = "";
+    bool stream_layers = false;
 
     int n_threads = threads;
     enum sd_type_t wtype = SD_TYPE_COUNT;
@@ -509,10 +510,7 @@ int load_model(const char *model, char *model_path, char* options[], int threads
         if (!strcmp(optname, "params_backend")) params_backend_arg = strdup(optval);
         if (!strcmp(optname, "rpc_servers")) rpc_servers_arg = strdup(optval);
         if (!strcmp(optname, "max_vram")) max_vram_arg = strdup(optval);
-        if (!strcmp(optname, "stream_layers")) {
-            // Retained as a no-op for existing configurations. Upstream now
-            // selects segmented weight streaming automatically.
-        }
+        if (!strcmp(optname, "stream_layers")) stream_layers = (strcmp(optval, "true") == 0 || strcmp(optval, "1") == 0);
 
         // vae_decode_only is still accepted for backwards compatibility with
         // existing gallery configs, but upstream dropped the option (the model
@@ -652,9 +650,11 @@ int load_model(const char *model, char *model_path, char* options[], int threads
             ctx_params.rpc_servers = env_rpc_servers;
         }
     }
-    // max_vram is an optional GiB budget or per-backend spec for automatic
-    // graph-cut execution. A zero value uses the live free-VRAM budget.
+    // max_vram: GiB budget or per-backend spec for graph-cut segmented param
+    // offload ("0" = disabled, "-1" = auto). stream_layers only has effect when
+    // max_vram is set.
     if (strlen(max_vram_arg) > 0) ctx_params.max_vram = max_vram_arg;
+    ctx_params.stream_layers = stream_layers;
     ctx_params.diffusion_flash_attn = diffusion_flash_attn;
     ctx_params.tae_preview_only = tae_preview_only;
     ctx_params.diffusion_conv_direct = diffusion_conv_direct;
@@ -1144,25 +1144,17 @@ static uint8_t* load_and_resize_image(const char* path, int target_width, int ta
 // Write sd.cpp's audio buffer to a temp WAV file (IEEE float, interleaved).
 // sd_audio_t.data is planar (all channel 0 samples, then channel 1, etc.) — we
 // interleave on the fly so ffmpeg's standard wav demuxer can read it directly.
-// Returns 0 on success and fills wav_path.
+// Returns 0 on success and fills wav_path (must be at least 64 bytes).
 static int write_planar_float_wav(const sd_audio_t* a, char* wav_path, size_t wav_path_sz) {
     if (!a || !a->data || a->sample_count == 0 || a->channels == 0 || a->sample_rate == 0) {
         return -1;
     }
 
-    const char* temp_dir = getenv("TMPDIR");
-    if (!temp_dir || temp_dir[0] == '\0') {
-        temp_dir = "/tmp";
-    }
-    int path_len = snprintf(wav_path, wav_path_sz, "%s/gosd-audio-XXXXXX.wav", temp_dir);
-    if (path_len < 0 || (size_t)path_len >= wav_path_sz) {
-        fprintf(stderr, "temporary directory path is too long\n");
-        return -1;
-    }
+    snprintf(wav_path, wav_path_sz, "/tmp/gosd-audio-XXXXXX.wav");
     int fd = mkstemps(wav_path, 4);
     if (fd < 0) { perror("mkstemps wav"); return -1; }
     FILE* f = fdopen(fd, "wb");
-    if (!f) { perror("fdopen wav"); close(fd); unlink(wav_path); return -1; }
+    if (!f) { perror("fdopen wav"); close(fd); return -1; }
 
     uint64_t frames = a->sample_count;
     uint32_t channels = a->channels;
@@ -1229,7 +1221,7 @@ static int ffmpeg_mux_raw_to_mp4(sd_image_t* frames, int num_frames, int fps,
     snprintf(fps_str, sizeof(fps_str), "%d", fps);
 
     // Optional audio: write a temp WAV file if the model produced audio.
-    char wav_path[4096] = {0};
+    char wav_path[64] = {0};
     bool have_audio = false;
     if (audio && audio->data && audio->sample_count > 0 && audio->channels > 0 && audio->sample_rate > 0) {
         if (write_planar_float_wav(audio, wav_path, sizeof(wav_path)) == 0) {
@@ -1446,3 +1438,4 @@ int unload() {
     free_sd_ctx(sd_c);
     return 0;
 }
+

backend/go/vllm-cpp/Makefile

@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e
 
 # vllm.cpp version
 VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
-VLLM_CPP_VERSION?=6bf3abb580982f4fd2e4525ef37802ee0ce28981
+VLLM_CPP_VERSION?=150b37852c123f7855fb219b37347572ca9427e7
 
 # MLX GEMM provider (darwin/metal only; see the metal branch below for why).
 # Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun

backend/go/vllm-cpp/options.go

@@ -128,40 +128,9 @@ func parseOptions(opts *pb.ModelOptions) loadOptions {
 	lo := loadOptions{}
 	applyOptionsList(&lo, opts.GetOptions())
 	applyEngineArgs(&lo, opts.GetEngineArgs())
-	applyDraftModelOption(&lo, opts.GetOptions())
 	return lo
 }
 
-// applyDraftModelOption binds a managed companion snapshot after engine_args
-// has supplied the speculative document. Companion paths do not exist until
-// LocalAI materializes the artifact, so they must replace the gallery's static
-// repository reference without disturbing the method or token budget.
-func applyDraftModelOption(lo *loadOptions, options []string) {
-	if strings.TrimSpace(lo.speculativeConfig) == "" {
-		return
-	}
-	var draftModel string
-	for _, option := range options {
-		key, value, found := strings.Cut(option, ":")
-		if found && strings.TrimSpace(key) == "draft_model" {
-			draftModel = strings.TrimSpace(value)
-		}
-	}
-	if draftModel == "" {
-		return
-	}
-
-	var spec map[string]any
-	if err := json.Unmarshal([]byte(lo.speculativeConfig), &spec); err != nil {
-		return
-	}
-	spec["model"] = draftModel
-	encoded, err := json.Marshal(spec)
-	if err == nil {
-		lo.speculativeConfig = string(encoded)
-	}
-}
-
 // applyOptionsList reads the legacy free-form "key:value" list. strings.Cut
 // splits on the FIRST colon only, so a JSON object value survives intact.
 func applyOptionsList(lo *loadOptions, options []string) {

backend/go/vllm-cpp/options_test.go

@@ -1,38 +0,0 @@
-package main
-
-import (
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-
-	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
-)
-
-var _ = Describe("managed DFlash companion options", func() {
-	It("replaces only the draft model in an existing speculative configuration", func() {
-		managedPath := ".artifacts/huggingface/0123456789abcdef/snapshot"
-		lo := parseOptions(&pb.ModelOptions{
-			Options: []string{"draft_model:" + managedPath},
-			EngineArgs: `{
-				"speculative_config": {
-					"method": "dflash",
-					"model": "Mia-AiLab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw",
-					"num_speculative_tokens": 7
-				}
-			}`,
-		})
-
-		Expect(lo.speculativeConfig).To(MatchJSON(`{
-			"method": "dflash",
-			"model": ".artifacts/huggingface/0123456789abcdef/snapshot",
-			"num_speculative_tokens": 7
-		}`))
-	})
-
-	It("ignores a draft companion when speculative decoding is not configured", func() {
-		lo := parseOptions(&pb.ModelOptions{
-			Options: []string{"draft_model:.artifacts/huggingface/0123456789abcdef/snapshot"},
-		})
-
-		Expect(lo.speculativeConfig).To(BeEmpty())
-	})
-})

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=c44b60b8053bbf2a5c1e014f11323fb3f2485177
+WHISPER_CPP_VERSION?=978113305b2ead22249b881deafa131dc8884911
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/whisper/addr_test.go

@@ -1,35 +0,0 @@
-package main
-
-import (
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-// Specs for resolveAddr run under the suite bootstrap in gowhisper_test.go
-// (TestWhisper); they need no native library, so they never skip.
-var _ = Describe("resolveAddr", func() {
-	It("prefers an explicitly set -addr over a positional argument", func() {
-		Expect(resolveAddr("127.0.0.1:12345", true, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:12345"))
-	})
-
-	It("keeps an explicit -addr equal to the default over a positional argument", func() {
-		Expect(resolveAddr(defaultAddr, true, []string{"127.0.0.1:59999"})).To(Equal(defaultAddr))
-	})
-
-	It("falls back to the positional argument when -addr is unset", func() {
-		Expect(resolveAddr(defaultAddr, false, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:59999"))
-	})
-
-	It("keeps the default when -addr is unset and no positional argument is given", func() {
-		Expect(resolveAddr(defaultAddr, false, nil)).To(Equal(defaultAddr))
-	})
-
-	It("uses the first positional argument among several", func() {
-		Expect(resolveAddr(defaultAddr, false, []string{"127.0.0.1:59999", "extra"})).To(Equal("127.0.0.1:59999"))
-	})
-
-	It("treats an explicitly empty -addr as unset", func() {
-		Expect(resolveAddr("", true, []string{"127.0.0.1:59999"})).To(Equal("127.0.0.1:59999"))
-		Expect(resolveAddr("", true, nil)).To(Equal(defaultAddr))
-	})
-})

backend/go/whisper/main.go

@@ -10,31 +10,10 @@ import (
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 )
 
-const (
-	defaultAddr = "localhost:50051"
-)
-
 var (
-	addr = flag.String("addr", defaultAddr, "the address to listen on")
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
 )
 
-// resolveAddr picks the address the gRPC server binds to. An explicitly set
-// -addr always wins. Launchers may hand us the listen address as a bare
-// positional argument, which Go's flag package silently drops — honour it
-// next so the server binds the port its caller actually allocated instead of
-// the default one (#11623). An explicitly empty -addr counts as unset:
-// binding the empty address would listen on an OS-chosen port on every
-// interface instead of the one the caller allocated.
-func resolveAddr(flagAddr string, addrSet bool, args []string) string {
-	if addrSet && flagAddr != "" {
-		return flagAddr
-	}
-	if len(args) > 0 {
-		return args[0]
-	}
-	return defaultAddr
-}
-
 type LibFuncs struct {
 	FuncPtr any
 	Name    string
@@ -82,16 +61,7 @@ func main() {
 
 	flag.Parse()
 
-	// flag.Visit reports only flags that were explicitly set, so an -addr
-	// equal to the default is still distinguished from an untouched one.
-	addrSet := false
-	flag.Visit(func(f *flag.Flag) {
-		if f.Name == "addr" {
-			addrSet = true
-		}
-	})
-
-	if err := grpc.StartServer(resolveAddr(*addr, addrSet, flag.Args()), &Whisper{}); err != nil {
+	if err := grpc.StartServer(*addr, &Whisper{}); err != nil {
 		panic(err)
 	}
 }

backend/index.yaml

@@ -150,7 +150,6 @@
     - audio-transcription
     - CPU
     - CUDA
-    - HIP
     - Metal
   # No vulkan key: the vulkan image would carry a Vulkan loader with no Mesa ICD
   # (see the audio-cpp block in .github/backend-matrix.yml). Pointing a
@@ -162,7 +161,6 @@
     nvidia: "cuda12-audio-cpp"
     nvidia-cuda-12: "cuda12-audio-cpp"
     nvidia-cuda-13: "cuda13-audio-cpp"
-    amd: "rocm-audio-cpp"
     metal: "metal-audio-cpp"
     metal-darwin-arm64: "metal-audio-cpp"
 - &whispercpp
@@ -1062,21 +1060,6 @@
     nvidia-l4t: "nvidia-l4t-mlx"
     nvidia-l4t-cuda-12: "nvidia-l4t-mlx"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-mlx"
-- &mlx-video
-  name: "mlx-video"
-  icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
-  urls:
-    - https://github.com/Blaizzy/mlx-video
-  license: MIT
-  description: |
-      Generate videos with LTX-2 and converted Wan2.1/Wan2.2 checkpoints using
-      MLX on Apple Silicon.
-  tags:
-    - text-to-video
-    - image-to-video
-    - MLX
-  capabilities:
-    metal: "metal-mlx-video"
 - &mlx-vlm
   name: "mlx-vlm"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
@@ -1211,7 +1194,6 @@
   tags:
     - image-generation
     - video-generation
-    - sound-generation
     - diffusion-models
   license: apache-2.0
   alias: "diffusers"
@@ -1536,25 +1518,6 @@
     nvidia-cuda-12: "cuda12-faster-whisper"
     nvidia-l4t: "nvidia-l4t-arm64-faster-whisper"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-faster-whisper"
-- &whisper-medusa
-  description: |
-    Whisper-Medusa accelerates Whisper transcription by predicting multiple tokens per decoding step.
-    The upstream checkpoints use custom Transformers generation code and accept audio clips up to 30 seconds.
-  urls:
-    - https://github.com/aiola-lab/whisper-medusa
-    - https://huggingface.co/collections/aiola/whisper-medusa
-  tags:
-    - speech-to-text
-    - transcription
-    - Whisper
-    - Medusa
-  license: MIT
-  name: "whisper-medusa"
-  alias: "whisper-medusa"
-  capabilities:
-    default: "cpu-whisper-medusa"
-    nvidia: "cuda12-whisper-medusa"
-    nvidia-cuda-12: "cuda12-whisper-medusa"
 - &moonshine
   description: |
     Moonshine is a fast, accurate, and efficient speech-to-text transcription model using ONNX Runtime.
@@ -1822,32 +1785,6 @@
     nvidia-l4t-cuda-12: "nvidia-l4t-faster-qwen3-tts"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-faster-qwen3-tts"
   icon: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/-s1gyJfvbE1RgO5iBeNOi.png
-- &funasr
-  urls:
-    - https://github.com/modelscope/FunASR
-  description: |
-    FunASR is an industrial-grade speech recognition toolkit supporting 50+ languages.
-    Includes SenseVoice (170x realtime, emotion detection), Paraformer (highest Chinese
-    accuracy), and built-in VAD, punctuation restoration, and speaker diarization.
-  tags:
-    - speech-recognition
-    - ASR
-    - multilingual
-  license: mit
-  name: "funasr"
-  alias: "funasr"
-  capabilities:
-    nvidia: "cuda12-funasr"
-    intel: "intel-funasr"
-    amd: "rocm-funasr"
-    metal: "metal-funasr"
-    default: "cpu-funasr"
-    nvidia-cuda-13: "cuda13-funasr"
-    nvidia-cuda-12: "cuda12-funasr"
-    nvidia-l4t: "nvidia-l4t-funasr"
-    nvidia-l4t-cuda-12: "nvidia-l4t-funasr"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-funasr"
-  icon: https://avatars.githubusercontent.com/u/109454077
 - &qwen-asr
   urls:
     - https://github.com/QwenLM/Qwen3-ASR
@@ -2164,7 +2101,6 @@
     nvidia: "cuda12-audio-cpp-development"
     nvidia-cuda-12: "cuda12-audio-cpp-development"
     nvidia-cuda-13: "cuda13-audio-cpp-development"
-    amd: "rocm-audio-cpp-development"
     metal: "metal-audio-cpp-development"
     metal-darwin-arm64: "metal-audio-cpp-development"
 - !!merge <<: *stablediffusionggml
@@ -2228,20 +2164,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx"
   mirrors:
     - localai/localai-backends:latest-metal-darwin-arm64-mlx
-- !!merge <<: *mlx-video
-  name: "metal-mlx-video"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-mlx-video"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-mlx-video
-- !!merge <<: *mlx-video
-  name: "metal-mlx-video-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx-video"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-mlx-video
-- !!merge <<: *mlx-video
-  name: "mlx-video-development"
-  capabilities:
-    metal: "metal-mlx-video-development"
 - !!merge <<: *mlx
   name: "metal-mlx-development"
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-mlx"
@@ -6446,34 +6368,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-faster-qwen3-tts"
   mirrors:
     - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-faster-qwen3-tts
-## whisper-medusa
-- !!merge <<: *whisper-medusa
-  name: "whisper-medusa-development"
-  capabilities:
-    default: "cpu-whisper-medusa-development"
-    nvidia: "cuda12-whisper-medusa-development"
-    nvidia-cuda-12: "cuda12-whisper-medusa-development"
-- !!merge <<: *whisper-medusa
-  name: "cpu-whisper-medusa"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-whisper-medusa"
-  mirrors:
-    - localai/localai-backends:latest-cpu-whisper-medusa
-- !!merge <<: *whisper-medusa
-  name: "cpu-whisper-medusa-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-whisper-medusa"
-  mirrors:
-    - localai/localai-backends:master-cpu-whisper-medusa
-- !!merge <<: *whisper-medusa
-  name: "cuda12-whisper-medusa"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-whisper-medusa"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-whisper-medusa
-- !!merge <<: *whisper-medusa
-  name: "cuda12-whisper-medusa-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-whisper-medusa"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-whisper-medusa
-
 ## qwen-asr
 - !!merge <<: *qwen-asr
   name: "qwen-asr-development"
@@ -7160,16 +7054,6 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-audio-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-audio-cpp
-- !!merge <<: *audiocpp
-  name: "rocm-audio-cpp"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-audio-cpp"
-  mirrors:
-    - localai/localai-backends:latest-gpu-rocm-hipblas-audio-cpp
-- !!merge <<: *audiocpp
-  name: "rocm-audio-cpp-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-audio-cpp"
-  mirrors:
-    - localai/localai-backends:master-gpu-rocm-hipblas-audio-cpp
 - !!merge <<: *audiocpp
   name: "metal-audio-cpp"
   uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-audio-cpp"
@@ -7180,97 +7064,3 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-audio-cpp"
   mirrors:
     - localai/localai-backends:master-metal-darwin-arm64-audio-cpp
-## funasr
-- !!merge <<: *funasr
-  name: "funasr-development"
-  capabilities:
-    nvidia: "cuda12-funasr-development"
-    intel: "intel-funasr-development"
-    amd: "rocm-funasr-development"
-    nvidia-l4t: "nvidia-l4t-funasr-development"
-    metal: "metal-funasr-development"
-    default: "cpu-funasr-development"
-    nvidia-cuda-13: "cuda13-funasr-development"
-    nvidia-cuda-12: "cuda12-funasr-development"
-    nvidia-l4t-cuda-12: "nvidia-l4t-funasr-development"
-    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-funasr-development"
-- !!merge <<: *funasr
-  name: "cpu-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-funasr"
-  mirrors:
-    - localai/localai-backends:latest-cpu-funasr
-- !!merge <<: *funasr
-  name: "cpu-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-funasr"
-  mirrors:
-    - localai/localai-backends:master-cpu-funasr
-- !!merge <<: *funasr
-  name: "cuda12-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-funasr"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-12-funasr
-- !!merge <<: *funasr
-  name: "cuda12-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-funasr"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-12-funasr
-- !!merge <<: *funasr
-  name: "cuda13-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-funasr"
-  mirrors:
-    - localai/localai-backends:latest-gpu-nvidia-cuda-13-funasr
-- !!merge <<: *funasr
-  name: "cuda13-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-funasr"
-  mirrors:
-    - localai/localai-backends:master-gpu-nvidia-cuda-13-funasr
-- !!merge <<: *funasr
-  name: "intel-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-funasr"
-  mirrors:
-    - localai/localai-backends:latest-gpu-intel-funasr
-- !!merge <<: *funasr
-  name: "intel-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-funasr"
-  mirrors:
-    - localai/localai-backends:master-gpu-intel-funasr
-- !!merge <<: *funasr
-  name: "rocm-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-funasr"
-  mirrors:
-    - localai/localai-backends:latest-gpu-rocm-hipblas-funasr
-- !!merge <<: *funasr
-  name: "rocm-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-funasr"
-  mirrors:
-    - localai/localai-backends:master-gpu-rocm-hipblas-funasr
-- !!merge <<: *funasr
-  name: "nvidia-l4t-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-funasr"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-funasr
-- !!merge <<: *funasr
-  name: "nvidia-l4t-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-funasr"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-funasr
-- !!merge <<: *funasr
-  name: "cuda13-nvidia-l4t-arm64-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-funasr"
-  mirrors:
-    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-funasr
-- !!merge <<: *funasr
-  name: "cuda13-nvidia-l4t-arm64-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-funasr"
-  mirrors:
-    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-funasr
-- !!merge <<: *funasr
-  name: "metal-funasr"
-  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-funasr"
-  mirrors:
-    - localai/localai-backends:latest-metal-darwin-arm64-funasr
-- !!merge <<: *funasr
-  name: "metal-funasr-development"
-  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-funasr"
-  mirrors:
-    - localai/localai-backends:master-metal-darwin-arm64-funasr

backend/python/README.md

@@ -20,7 +20,6 @@ The Python backends use a unified build system based on `libbackend.sh` that pro
 ### Audio & Speech
 - **coqui** - Coqui TTS models
 - **faster-whisper** - Fast Whisper speech recognition
-- **funasr** - Local multilingual transcription with FunASR and SenseVoice
 - **kitten-tts** - Lightweight TTS
 - **mlx-audio** - Apple Silicon audio processing
 - **chatterbox** - TTS model

backend/python/chatterbox/backend.py

@@ -19,7 +19,6 @@ import grpc
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
-from temp_utils import cleanup_paths
 
 import tempfile
 
@@ -116,6 +115,11 @@ def merge_audio_files(audio_files, output_path, sample_rate):
     # Save the merged audio
     ta.save(output_path, merged_waveform, sample_rate)
     
+    # Clean up temporary files
+    for audio_file in audio_files:
+        if os.path.exists(audio_file):
+            os.remove(audio_file)
+
 _ONE_DAY_IN_SECONDS = 60 * 60 * 24
 
 # If MAX_WORKERS are specified in the environment use it, otherwise default to 1
@@ -222,20 +226,19 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 text_chunks = split_text_at_word_boundary(request.text, max_length=250)
                 print(f"Splitting text into chunks of 250 characters: {len(text_chunks)}", file=sys.stderr)
                 # Generate audio for each chunk
-                with cleanup_paths() as temp_audio_files:
-                    for i, chunk in enumerate(text_chunks):
-                        # Generate audio for this chunk
-                        wav = self.model.generate(chunk, **kwargs)
-
-                        # Register ownership before saving so a partial write is
-                        # removed too when generation or encoding fails.
-                        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
-                        temp_file.close()
-                        temp_audio_files.append(temp_file.name)
-                        ta.save(temp_file.name, wav, self.model.sr)
-
-                    # Merge all audio files
-                    merge_audio_files(temp_audio_files, request.dst, self.model.sr)
+                temp_audio_files = []
+                for i, chunk in enumerate(text_chunks):
+                    # Generate audio for this chunk
+                    wav = self.model.generate(chunk, **kwargs)
+                    
+                    # Create temporary file for this chunk
+                    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+                    temp_file.close()
+                    ta.save(temp_file.name, wav, self.model.sr)
+                    temp_audio_files.append(temp_file.name)
+                
+                # Merge all audio files
+                merge_audio_files(temp_audio_files, request.dst, self.model.sr)
             else:
                 # Generate audio using ChatterboxTTS for short text
                 wav = self.model.generate(request.text, **kwargs)

backend/python/common/libbackend.sh

@@ -563,18 +563,6 @@ function startBackend() {
         echo "Added ${EDIR}/lib to LD_LIBRARY_PATH for GPU libraries"
     fi
 
-    # XPU wheels carry a matching SYCL/Unified Runtime set. Prefer it over
-    # packaged oneAPI or host libraries, which may expose an older loader ABI.
-    if [ "$(uname -s)" = "Linux" ]; then
-        local sycl_runtime
-        for sycl_runtime in "${EDIR}/venv/lib"/libsycl.so*; do
-            if [ -f "${sycl_runtime}" ]; then
-                export LD_LIBRARY_PATH="${EDIR}/venv/lib${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"
-                break
-            fi
-        done
-    fi
-
     if [ ! -z "${BACKEND_FILE:-}" ]; then
         exec "${EDIR}/venv/bin/python" "${BACKEND_FILE}" "$@"
     elif [ -e "${MY_DIR}/server.py" ]; then

backend/python/common/python_utils.py

@@ -37,46 +37,6 @@ def parse_options(options_list):
     return opts
 
 
-def attach_media_parts(messages_dicts, n_images=0, n_videos=0):
-    """Rebuild the last user message as content *parts* carrying media markers.
-
-    Backends that let the tokenizer do the templating hand plain string content
-    to ``apply_chat_template``, but a chat template only emits the model's own
-    media tokens (``<|vision_start|><|image_pad|><|vision_end|>`` for the
-    Qwen-VL family, and the equivalents elsewhere) when the content is a list
-    of parts. Without those markers the engine's multimodal processor finds
-    nothing to substitute and silently discards the pixels, even though they
-    were forwarded correctly out of band.
-
-    Returns a new list whose last user message has
-    ``[{"type": "image"} * n_images, {"type": "video"} * n_videos, text]`` as
-    its content, or ``None`` when there is nothing to attach - no media, no
-    user turn, or content that is already a list of parts - so the caller can
-    keep using the original string-content list.
-    """
-    if not n_images and not n_videos:
-        return None
-    idx = next(
-        (
-            i
-            for i in reversed(range(len(messages_dicts)))
-            if messages_dicts[i].get("role") == "user"
-        ),
-        None,
-    )
-    if idx is None:
-        return None
-    text = messages_dicts[idx].get("content") or ""
-    if not isinstance(text, str):
-        return None
-    parts = [{"type": "image"}] * n_images + [{"type": "video"}] * n_videos
-    if text:
-        parts.append({"type": "text", "text": text})
-    patched = list(messages_dicts)
-    patched[idx] = dict(patched[idx], content=parts)
-    return patched
-
-
 def messages_to_dicts(proto_messages):
     """Convert proto ``Message`` objects to dicts suitable for ``apply_chat_template``.
 

backend/python/common/python_utils_test.py

@@ -14,7 +14,7 @@ import json
 import types
 import unittest
 
-from python_utils import attach_media_parts, messages_to_dicts, parse_options
+from python_utils import messages_to_dicts, parse_options
 
 
 def _msg(**fields):
@@ -118,63 +118,5 @@ class TestMessagesToDicts(unittest.TestCase):
         self.assertNotIn("tool_calls", out[0])
 
 
-class TestAttachMediaParts(unittest.TestCase):
-    def test_image_marker_added_to_last_user_turn(self):
-        messages = [
-            {"role": "system", "content": "be brief"},
-            {"role": "user", "content": "first"},
-            {"role": "assistant", "content": "ok"},
-            {"role": "user", "content": "how high is the water?"},
-        ]
-        out = attach_media_parts(messages, n_images=1)
-        self.assertEqual(
-            out[3]["content"],
-            [{"type": "image"}, {"type": "text", "text": "how high is the water?"}],
-        )
-        # Earlier turns and the input list itself are untouched.
-        self.assertEqual(out[:3], messages[:3])
-        self.assertEqual(messages[3]["content"], "how high is the water?")
-
-    def test_counts_and_order_images_then_videos(self):
-        out = attach_media_parts(
-            [{"role": "user", "content": "describe"}], n_images=2, n_videos=1
-        )
-        self.assertEqual(
-            out[0]["content"],
-            [
-                {"type": "image"},
-                {"type": "image"},
-                {"type": "video"},
-                {"type": "text", "text": "describe"},
-            ],
-        )
-
-    def test_empty_text_yields_media_only_parts(self):
-        out = attach_media_parts([{"role": "user", "content": ""}], n_images=1)
-        self.assertEqual(out[0]["content"], [{"type": "image"}])
-
-    def test_other_message_keys_are_preserved(self):
-        out = attach_media_parts(
-            [{"role": "user", "content": "hi", "name": "bob"}], n_images=1
-        )
-        self.assertEqual(out[0]["name"], "bob")
-
-    def test_no_media_is_a_no_op(self):
-        self.assertIsNone(attach_media_parts([{"role": "user", "content": "hi"}]))
-
-    def test_no_user_turn_is_a_no_op(self):
-        self.assertIsNone(
-            attach_media_parts([{"role": "system", "content": "hi"}], n_images=1)
-        )
-
-    def test_content_already_parts_is_a_no_op(self):
-        self.assertIsNone(
-            attach_media_parts(
-                [{"role": "user", "content": [{"type": "text", "text": "hi"}]}],
-                n_images=1,
-            )
-        )
-
-
 if __name__ == "__main__":
     unittest.main()

backend/python/common/temp_utils.py

@@ -1,36 +0,0 @@
-import base64
-import contextlib
-import os
-import tempfile
-
-
-@contextlib.contextmanager
-def materialize_base64(data, suffix=""):
-    """Materialize base64 data for a path-only library and always remove it."""
-    descriptor, path = tempfile.mkstemp(prefix="localai-media-", suffix=suffix)
-    try:
-        with os.fdopen(descriptor, "wb") as output:
-            descriptor = None
-            output.write(base64.b64decode(data))
-        yield path
-    finally:
-        if descriptor is not None:
-            os.close(descriptor)
-        try:
-            os.remove(path)
-        except OSError:
-            pass
-
-
-@contextlib.contextmanager
-def cleanup_paths():
-    """Collect temporary paths and remove them on success or failure."""
-    paths = []
-    try:
-        yield paths
-    finally:
-        for path in paths:
-            try:
-                os.remove(path)
-            except OSError:
-                pass

backend/python/common/temp_utils_test.py

@@ -1,41 +0,0 @@
-import os
-import tempfile
-import unittest
-from unittest import mock
-
-from temp_utils import cleanup_paths, materialize_base64
-
-
-class MaterializeBase64Test(unittest.TestCase):
-    def test_removes_materialized_file_after_success(self):
-        with tempfile.TemporaryDirectory() as directory:
-            with mock.patch.object(tempfile, "tempdir", directory):
-                with materialize_base64("aGVsbG8=", suffix=".data") as path:
-                    with open(path, "rb") as materialized:
-                        self.assertEqual(materialized.read(), b"hello")
-                self.assertFalse(os.path.exists(path))
-
-    def test_removes_materialized_file_when_consumer_fails(self):
-        with tempfile.TemporaryDirectory() as directory:
-            with mock.patch.object(tempfile, "tempdir", directory):
-                with self.assertRaisesRegex(RuntimeError, "decode failed"):
-                    with materialize_base64("aGVsbG8="):
-                        raise RuntimeError("decode failed")
-                self.assertEqual(os.listdir(directory), [])
-
-
-class CleanupPathsTest(unittest.TestCase):
-    def test_removes_every_registered_path_after_failure(self):
-        with tempfile.TemporaryDirectory() as directory:
-            paths = [os.path.join(directory, name) for name in ("one.wav", "two.wav")]
-            with self.assertRaisesRegex(RuntimeError, "merge failed"):
-                with cleanup_paths() as registered:
-                    for path in paths:
-                        open(path, "wb").close()
-                        registered.append(path)
-                    raise RuntimeError("merge failed")
-            self.assertEqual(os.listdir(directory), [])
-
-
-if __name__ == "__main__":
-    unittest.main()

backend/python/common/template/requirements-intel.txt

@@ -1,4 +1,4 @@
 --extra-index-url https://download.pytorch.org/whl/xpu
-torch==2.14.0+xpu
+torch==2.13.0+xpu
 oneccl_bind_pt==2.8.0+xpu
 optimum[openvino]

backend/python/common/template/requirements.txt

@@ -1,3 +1,3 @@
-grpcio==1.83.1
+grpcio==1.82.1
 protobuf
 grpcio-tools

backend/python/coqui/requirements.txt

@@ -1,4 +1,4 @@
-grpcio==1.83.1
+grpcio==1.83.0
 protobuf
 certifi
 packaging==26.3

backend/python/diffusers/audio_utils.py

@@ -1,24 +0,0 @@
-import array
-import sys
-import wave
-
-
-def write_pcm_wav(destination, samples, sampling_rate):
-    """Write normalized floating-point audio samples as mono 16-bit PCM."""
-    pcm = array.array(
-        "h",
-        (
-            max(-32768, min(32767, round(float(sample) * 32768)))
-            for sample in samples
-        ),
-    )
-    if pcm.itemsize != 2:
-        raise RuntimeError("16-bit PCM requires two-byte signed integers")
-    if sys.byteorder != "little":
-        pcm.byteswap()
-
-    with wave.open(destination, "wb") as output:
-        output.setnchannels(1)
-        output.setsampwidth(2)
-        output.setframerate(sampling_rate)
-        output.writeframes(pcm.tobytes())

backend/python/diffusers/backend.py

@@ -26,7 +26,6 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
 sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
 from grpc_auth import get_auth_interceptors
 from model_utils import resolve_model_reference
-from audio_utils import write_pcm_wav
 
 
 # Import dynamic loader for pipeline discovery
@@ -36,7 +35,6 @@ from diffusers_dynamic_loader import (
     get_available_pipelines,
     load_diffusers_pipeline,
 )
-from load_options import single_file_load_kwargs
 
 # Import specific items still needed for special cases and safety checker
 from diffusers import DiffusionPipeline, ControlNetModel
@@ -124,21 +122,6 @@ from diffusers.schedulers import (
     UniPCMultistepScheduler,
 )
 
-def select_device(request_cuda, device_option, cuda_available, xpu, mps_available):
-    """Pick the pipeline device. An explicit `device:` model option wins;
-    otherwise CUDA is used whenever torch reports it available (ROCm
-    builds included) or the model config forces it with `cuda: true`,
-    keeping the pre-existing XPU/MPS overrides. CPU is the fallback, not
-    the default."""
-    if device_option:
-        return device_option
-    device = "cuda" if (request_cuda or cuda_available) else "cpu"
-    if xpu:
-        device = "xpu"
-    if mps_available:
-        device = "mps"
-    return device
-
 def is_float(s):
     """Check if a string can be converted to float."""
     try:
@@ -481,9 +464,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
         # Build kwargs for dynamic loading
         load_kwargs = {"torch_dtype": torchType}
-        load_kwargs.update(
-            single_file_load_kwargs(request.OriginalConfigFile, from_single_file)
-        )
 
         # Add variant if not loading from single file
         if not from_single_file and variant:
@@ -647,13 +627,12 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 # modify LoraAdapter to be relative to modelFileBase
                 request.LoraAdapter = os.path.join(request.ModelPath, request.LoraAdapter)
 
-            device = select_device(
-                request.CUDA,
-                self.options.pop("device", None),
-                torch.cuda.is_available(),
-                XPU,
-                hasattr(torch.backends, "mps") and torch.backends.mps.is_available(),
-            )
+            device = "cpu" if not request.CUDA else "cuda"
+            if XPU:
+                device = "xpu"
+            mps_available = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+            if mps_available:
+                device = "mps"
             self.device = device
             if request.LoraAdapter:
                 # Check if its a local file and not a directory ( we load lora differently for a safetensor file )
@@ -821,12 +800,12 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             image = image.resize((1024, 576))
 
             generator = torch.manual_seed(request.seed)
-            frames = self.pipe(image=image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0]
+            frames = self.pipe(image, guidance_scale=self.cfg_scale, decode_chunk_size=CHUNK_SIZE, generator=generator).frames[0]
             export_to_video(frames, request.dst, fps=FPS)
             return backend_pb2.Result(message="Media generated successfully", success=True)
 
         if self.txt2vid:
-            video_frames = self.pipe(prompt=prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames
+            video_frames = self.pipe(prompt, guidance_scale=self.cfg_scale, num_inference_steps=steps, num_frames=int(FRAMES)).frames
             export_to_video(video_frames, request.dst)
             return backend_pb2.Result(message="Media generated successfully", success=True)
 
@@ -889,7 +868,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         else:
             # pass the kwargs dictionary to the self.pipe method
             image = self.pipe(
-                prompt=prompt,
+                prompt,
                 guidance_scale=self.cfg_scale,
                 **kwargs
             ).images[0]
@@ -904,49 +883,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
         return backend_pb2.Result(message="Media generated", success=True)
 
-    def SoundGeneration(self, request, context):
-        if not request.dst:
-            return backend_pb2.Result(success=False, message="request.dst is required")
-
-        prompt = request.text or request.caption
-        if not prompt:
-            return backend_pb2.Result(success=False, message="request.text is required")
-
-        try:
-            generation_options = dict(self.options)
-            if "num_inference_steps" in generation_options:
-                generation_options["num_inference_steps"] = int(
-                    generation_options["num_inference_steps"]
-                )
-            generation_options["prompt"] = prompt
-            if request.HasField("duration"):
-                generation_options["audio_length_in_s"] = request.duration
-            if request.HasField("temperature"):
-                generation_options["guidance_scale"] = request.temperature
-
-            generated = self.pipe(**generation_options)
-            if not hasattr(generated, "audios") or len(generated.audios) == 0:
-                return backend_pb2.Result(
-                    success=False,
-                    message="The diffusers pipeline returned no audio",
-                )
-
-            samples = generated.audios[0]
-            if hasattr(samples, "reshape"):
-                samples = samples.reshape(-1)
-            if hasattr(samples, "tolist"):
-                samples = samples.tolist()
-
-            sampling_rate = getattr(
-                getattr(getattr(self.pipe, "vae", None), "config", None),
-                "sampling_rate",
-                16000,
-            )
-            write_pcm_wav(request.dst, samples, sampling_rate)
-            return backend_pb2.Result(success=True, message="Sound generated successfully")
-        except Exception as err:
-            return backend_pb2.Result(success=False, message=f"SoundGeneration error: {err}")
-
     def UpscaleImage(self, request, context):
         try:
             if not request.src:

backend/python/diffusers/load_options.py

@@ -1,7 +0,0 @@
-# SPDX-License-Identifier: MIT
-
-
-def single_file_load_kwargs(original_config_file: str, from_single_file: bool) -> dict:
-    if from_single_file and original_config_file:
-        return {"original_config_file": original_config_file}
-    return {}

backend/python/diffusers/test.py

@@ -4,13 +4,9 @@ A test script to test the gRPC service and dynamic loader
 import unittest
 import subprocess
 import time
-import os
-import tempfile
-import wave
 from unittest.mock import patch, MagicMock
 
 # Import dynamic loader for testing (these don't need gRPC)
-import backend
 import diffusers_dynamic_loader as loader
 from diffusers import DiffusionPipeline, StableDiffusionPipeline
 
@@ -377,135 +373,3 @@ class TestGenerateImageOptionsKwargsMerge(unittest.TestCase):
         finally:
             os.unlink(src_file.name)
             os.unlink(dst_file.name)
-
-    def test_text_to_image_prompt_is_passed_by_keyword(self):
-        """Test compatibility with pipelines that take image before prompt."""
-        import os
-        import tempfile
-
-        from PIL import Image
-
-        from backend import BackendServicer
-
-        class Flux2CompatiblePipeline:
-            """Model the FLUX.2 call signature: image is before prompt."""
-
-            def __call__(self, image=None, prompt=None, **kwargs):
-                if prompt is None:
-                    raise ValueError("prompt was not passed by keyword")
-                self.prompt = prompt
-                self.kwargs = kwargs
-                return MagicMock(images=[Image.new("RGB", (4, 4))])
-
-        pipeline = Flux2CompatiblePipeline()
-        svc = BackendServicer.__new__(BackendServicer)
-        svc.pipe = pipeline
-        svc.cfg_scale = 7.5
-        svc.controlnet = None
-        svc.img2vid = False
-        svc.txt2vid = False
-        svc.clip_skip = 0
-        svc.PipelineType = "Flux2KleinPipeline"
-        svc.options = {}
-
-        with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as dst_file:
-            dst_path = dst_file.name
-
-        try:
-            request = MagicMock()
-            request.positive_prompt = "a red apple on a wooden table"
-            request.negative_prompt = ""
-            request.step = 4
-            request.seed = 0
-            request.width = 0
-            request.height = 0
-            request.src = ""
-            request.ref_images = []
-            request.dst = dst_path
-
-            svc.GenerateImage(request, context=None)
-
-            self.assertEqual(pipeline.prompt, request.positive_prompt)
-            self.assertEqual(pipeline.kwargs["num_inference_steps"], 4)
-        finally:
-            os.unlink(dst_path)
-
-
-class TestDeviceSelection(unittest.TestCase):
-    """Unit tests for backend.select_device (no GPU required)."""
-
-    def test_autodetect_cuda(self):
-        self.assertEqual(backend.select_device(False, None, True, False, False), "cuda")
-
-    def test_cpu_fallback(self):
-        self.assertEqual(backend.select_device(False, None, False, False, False), "cpu")
-
-    def test_forced_cuda(self):
-        self.assertEqual(backend.select_device(True, None, False, False, False), "cuda")
-
-    def test_device_option_wins(self):
-        self.assertEqual(backend.select_device(True, "cpu", True, True, True), "cpu")
-
-    def test_mps_overrides(self):
-        self.assertEqual(backend.select_device(False, None, True, False, True), "mps")
-
-
-class TestWritePcmWav(unittest.TestCase):
-    def test_writes_clipped_float_samples_as_mono_pcm(self):
-        from audio_utils import write_pcm_wav
-
-        destination = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
-        destination.close()
-
-        try:
-            write_pcm_wav(destination.name, [0.0, 0.5, -0.5, 2.0], 16000)
-
-            with wave.open(destination.name, "rb") as generated:
-                self.assertEqual(generated.getframerate(), 16000)
-                self.assertEqual(generated.getnchannels(), 1)
-                self.assertEqual(generated.getsampwidth(), 2)
-                self.assertEqual(generated.getnframes(), 4)
-                self.assertEqual(
-                    generated.readframes(4),
-                    b"\x00\x00\x00@\x00\xc0\xff\x7f",
-                )
-        finally:
-            os.unlink(destination.name)
-
-
-@unittest.skipUnless(GRPC_AVAILABLE, "gRPC modules not available")
-class TestSoundGeneration(unittest.TestCase):
-    def test_maps_request_options_and_writes_pipeline_audio(self):
-        from backend import BackendServicer
-
-        service = BackendServicer.__new__(BackendServicer)
-        service.options = {"num_inference_steps": 200.0}
-        service.pipe = MagicMock()
-        service.pipe.return_value.audios = [[0.0, 0.5, -0.5]]
-        service.pipe.vae.config.sampling_rate = 16000
-
-        destination = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
-        destination.close()
-
-        try:
-            request = backend_pb2.SoundGenerationRequest(
-                text="ocean waves",
-                dst=destination.name,
-                duration=2.5,
-                temperature=0,
-            )
-
-            result = service.SoundGeneration(request, context=None)
-
-            self.assertTrue(result.success, result.message)
-            service.pipe.assert_called_once_with(
-                num_inference_steps=200,
-                prompt="ocean waves",
-                audio_length_in_s=2.5,
-                guidance_scale=0,
-            )
-            with wave.open(destination.name, "rb") as generated:
-                self.assertEqual(generated.getframerate(), 16000)
-                self.assertEqual(generated.getnframes(), 3)
-        finally:
-            os.unlink(destination.name)

backend/python/diffusers/test_load_options.py

@@ -1,26 +0,0 @@
-# SPDX-License-Identifier: MIT
-
-import unittest
-
-from backend.python.diffusers.load_options import single_file_load_kwargs
-
-
-class SingleFileLoadKwargsTest(unittest.TestCase):
-    def test_includes_original_config_for_single_file(self):
-        self.assertEqual(
-            single_file_load_kwargs("configs/v1-inference.yaml", True),
-            {"original_config_file": "configs/v1-inference.yaml"},
-        )
-
-    def test_omits_original_config_for_pretrained_model(self):
-        self.assertEqual(
-            single_file_load_kwargs("configs/v1-inference.yaml", False),
-            {},
-        )
-
-    def test_omits_empty_original_config(self):
-        self.assertEqual(single_file_load_kwargs("", True), {})
-
-
-if __name__ == "__main__":
-    unittest.main()

backend/python/faster-whisper/install.sh

@@ -26,17 +26,4 @@ if [ "x${BUILD_PROFILE}" == "xl4t12" ]; then
     USE_PIP=true
 fi
 
-CTRANSLATE2_VERSION=${CTRANSLATE2_VERSION:-v4.7.1}
-CTRANSLATE2_ROCM_WHEEL_OS=${CTRANSLATE2_ROCM_WHEEL_OS:-Linux}
-CTRANSLATE2_ROCM_WHEEL_ARCHIVE="rocm-python-wheels-${CTRANSLATE2_ROCM_WHEEL_OS}.zip"
-
-if [ "x${BUILD_PROFILE}" == "xhipblas" ]; then
-  ensureVenv
-  mkdir /tmp/ctranslate2-rocm
-  wget -O "/tmp/ctranslate2-rocm/${CTRANSLATE2_ROCM_WHEEL_ARCHIVE}" "https://github.com/OpenNMT/CTranslate2/releases/download/${CTRANSLATE2_VERSION}/${CTRANSLATE2_ROCM_WHEEL_ARCHIVE}"
-  unzip "/tmp/ctranslate2-rocm/${CTRANSLATE2_ROCM_WHEEL_ARCHIVE}" -d /tmp/ctranslate2-rocm/
-  python3 -m ensurepip
-  python3 -m pip install --no-dependencies --no-index --find-links=/tmp/ctranslate2-rocm/temp-linux/ ctranslate2
-fi
-
 installRequirements

backend/python/fish-speech/Makefile

@@ -1,26 +1,18 @@
-.PHONY: fish-speech test-source-preparation test-runtime-import
-fish-speech: test-source-preparation test-runtime-import
+.PHONY: fish-speech test-source-preparation
+fish-speech: test-source-preparation
 	bash install.sh
 
 test-source-preparation:
 	bash prepare-source_test.sh
 
-test-runtime-import:
-	bash run_test.sh
-
 .PHONY: run
 run: fish-speech
 	@echo "Running fish-speech..."
 	bash run.sh
 	@echo "fish-speech run."
 
-.PHONY: test-unit
-test-unit:
-	python3 -m unittest -v prepare_upstream_test.py
-	bash run_test.sh
-
 .PHONY: test
-test: fish-speech test-unit
+test: fish-speech
 	@echo "Testing fish-speech..."
 	bash test.sh
 	@echo "fish-speech tested."

backend/python/fish-speech/backend.py

@@ -326,29 +326,11 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             max_new_tokens = self.options.get("max_new_tokens", 1024)
             chunk_length = self.options.get("chunk_length", 200)
 
-            # Build references list for voice cloning. Saved LocalAI
-            # personalities use the same ordered JSON shape as audio.cpp.
+            # Build references list for voice cloning
             references = []
             voice_name = request.voice if request.voice else None
 
-            multi_reference_cond = request.params.get("multi_reference_cond", "") if hasattr(request, "params") else ""
-            if multi_reference_cond:
-                reference_entries = json.loads(multi_reference_cond)
-                if not isinstance(reference_entries, list) or not reference_entries:
-                    raise ValueError("multi_reference_cond must be a non-empty JSON array")
-                for entry in reference_entries:
-                    if not isinstance(entry, dict) or not entry.get("audio") or not entry.get("text"):
-                        raise ValueError("multi_reference_cond entries require audio and text")
-                    ref_audio_path = self._get_ref_audio_path(entry["audio"])
-                    with open(ref_audio_path, "rb") as f:
-                        audio_bytes = f.read()
-                    references.append(ServeReferenceAudio(audio=audio_bytes, text=entry["text"]))
-                print(
-                    f"[INFO] Using {len(references)} per-request reference audios",
-                    file=sys.stderr,
-                )
-
-            elif voice_name and os.path.isfile(voice_name):
+            if voice_name and os.path.isfile(voice_name):
                 ref_audio_path = self._get_ref_audio_path(voice_name)
                 with open(ref_audio_path, "rb") as f:
                     audio_bytes = f.read()

backend/python/fish-speech/install.sh

@@ -44,13 +44,6 @@ fi
 # editable install. pyaudio is only used by the upstream playback client.
 bash "${backend_dir}/prepare-source.sh" "${BUILD_TYPE:-}" "${FISH_SPEECH_DIR}/pyproject.toml"
 
-# CUDA 13 has no torch 2.8 wheels, so fish-speech's exact upstream pin would
-# make pip select the CPU-only aarch64 wheel from PyPI. Prepare the cloned tree
-# before resolving it, and use soundfile for reference audio because torchcodec
-# does not publish Linux aarch64 wheels.
-python3 "${backend_dir}/prepare_upstream.py" "${FISH_SPEECH_DIR}" \
-    --cuda-major "${CUDA_MAJOR_VERSION:-}"
-
 # Install fish-speech deps from source (without the package itself since we use PYTHONPATH)
 ensureVenv
 if [ "x${USE_PIP}" == "xtrue" ]; then

backend/python/fish-speech/prepare_upstream.py

@@ -1,70 +0,0 @@
-#!/usr/bin/env python3
-# SPDX-License-Identifier: MIT
-
-import argparse
-from pathlib import Path
-
-
-TORCH_28 = '"torch==2.8.0"'
-TORCH_29 = '"torch==2.9.1"'
-TORCHAUDIO_28 = '"torchaudio==2.8.0"'
-TORCHAUDIO_29 = '"torchaudio==2.9.1"'
-TORCHAUDIO_LOAD = (
-    "        waveform, original_sr = "
-    "torchaudio.load(reference_audio, backend=self.backend)"
-)
-SOUNDFILE_LOAD = "\n".join(
-    (
-        "        import soundfile as _sf",
-        "        import torch as _torch",
-        "",
-        "        data, original_sr = _sf.read(",
-        '            reference_audio, dtype="float32", always_2d=True',
-        "        )",
-        "        waveform = _torch.from_numpy(data.T.copy())",
-    )
-)
-
-
-def patch_cuda13_dependencies(pyproject: Path) -> None:
-    content = pyproject.read_text()
-    if (
-        TORCH_28 not in content
-        and TORCHAUDIO_28 not in content
-        and TORCH_29 in content
-        and TORCHAUDIO_29 in content
-    ):
-        return
-    if TORCH_28 not in content or TORCHAUDIO_28 not in content:
-        raise RuntimeError("fish-speech's torch 2.8 dependency pins have changed")
-
-    content = content.replace(TORCH_28, TORCH_29)
-    content = content.replace(TORCHAUDIO_28, TORCHAUDIO_29)
-    pyproject.write_text(content)
-
-
-def patch_reference_loader(loader: Path) -> None:
-    content = loader.read_text()
-    if TORCHAUDIO_LOAD not in content and content.count(SOUNDFILE_LOAD) == 1:
-        return
-    if content.count(TORCHAUDIO_LOAD) != 1:
-        raise RuntimeError("fish-speech's torchaudio.load call has changed")
-
-    loader.write_text(content.replace(TORCHAUDIO_LOAD, SOUNDFILE_LOAD))
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser()
-    parser.add_argument("source", type=Path)
-    parser.add_argument("--cuda-major")
-    args = parser.parse_args()
-
-    if args.cuda_major == "13":
-        patch_cuda13_dependencies(args.source / "pyproject.toml")
-    patch_reference_loader(
-        args.source / "fish_speech/inference_engine/reference_loader.py"
-    )
-
-
-if __name__ == "__main__":
-    main()

backend/python/fish-speech/prepare_upstream_test.py

@@ -1,136 +0,0 @@
-# SPDX-License-Identifier: MIT
-
-import importlib.util
-import sys
-import tempfile
-import types
-import unittest
-from pathlib import Path
-
-
-MODULE_PATH = Path(__file__).with_name("prepare_upstream.py")
-
-
-def load_prepare_upstream():
-    if not MODULE_PATH.exists():
-        raise AssertionError("prepare_upstream.py is missing")
-    spec = importlib.util.spec_from_file_location("prepare_upstream", MODULE_PATH)
-    module = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(module)
-    return module
-
-
-class FakeAudioData:
-    @property
-    def T(self):
-        return self
-
-    def copy(self):
-        return "channels-first"
-
-
-class PrepareUpstreamTests(unittest.TestCase):
-    def test_cuda13_dependencies_follow_available_pytorch_wheels(self):
-        prepare_upstream = load_prepare_upstream()
-
-        with tempfile.TemporaryDirectory() as tmp:
-            pyproject = Path(tmp) / "pyproject.toml"
-            pyproject.write_text(
-                'dependencies = [\n    "torch==2.8.0",\n    "torchaudio==2.8.0",\n]\n'
-                'stable = [\n    "torch==2.8.0",\n    "torchaudio",\n]\n'
-            )
-
-            prepare_upstream.patch_cuda13_dependencies(pyproject)
-
-            self.assertEqual(
-                pyproject.read_text(),
-                'dependencies = [\n    "torch==2.9.1",\n    "torchaudio==2.9.1",\n]\n'
-                'stable = [\n    "torch==2.9.1",\n    "torchaudio",\n]\n',
-            )
-
-    def test_reference_audio_uses_soundfile_without_torchcodec(self):
-        prepare_upstream = load_prepare_upstream()
-
-        with tempfile.TemporaryDirectory() as tmp:
-            loader = Path(tmp) / "reference_loader.py"
-            loader.write_text(
-                "class ReferenceLoader:\n"
-                "    def load_audio(self, reference_audio):\n"
-                "        waveform, original_sr = torchaudio.load(reference_audio, backend=self.backend)\n"
-                "        return waveform, original_sr\n"
-            )
-            prepare_upstream.patch_reference_loader(loader)
-
-            calls = []
-            fake_soundfile = types.SimpleNamespace(
-                read=lambda source, **kwargs: (
-                    calls.append((source, kwargs)) or FakeAudioData(),
-                    24000,
-                )
-            )
-            fake_torch = types.SimpleNamespace(
-                from_numpy=lambda data: ("tensor", data),
-            )
-            previous_soundfile = sys.modules.get("soundfile")
-            previous_torch = sys.modules.get("torch")
-            sys.modules["soundfile"] = fake_soundfile
-            sys.modules["torch"] = fake_torch
-            try:
-                namespace = {"torchaudio": None}
-                exec(compile(loader.read_text(), str(loader), "exec"), namespace)
-                instance = namespace["ReferenceLoader"]()
-                instance.backend = "soundfile"
-
-                waveform, sample_rate = instance.load_audio("voice.wav")
-            finally:
-                if previous_soundfile is None:
-                    del sys.modules["soundfile"]
-                else:
-                    sys.modules["soundfile"] = previous_soundfile
-                if previous_torch is None:
-                    del sys.modules["torch"]
-                else:
-                    sys.modules["torch"] = previous_torch
-
-            self.assertEqual(waveform, ("tensor", "channels-first"))
-            self.assertEqual(sample_rate, 24000)
-            self.assertEqual(
-                calls,
-                [("voice.wav", {"dtype": "float32", "always_2d": True})],
-            )
-
-    def test_reference_loader_drift_fails_the_build(self):
-        prepare_upstream = load_prepare_upstream()
-
-        with tempfile.TemporaryDirectory() as tmp:
-            loader = Path(tmp) / "reference_loader.py"
-            loader.write_text("def load_audio():\n    pass\n")
-
-            with self.assertRaisesRegex(RuntimeError, "torchaudio.load call"):
-                prepare_upstream.patch_reference_loader(loader)
-
-    def test_preparation_can_be_repeated(self):
-        prepare_upstream = load_prepare_upstream()
-
-        with tempfile.TemporaryDirectory() as tmp:
-            pyproject = Path(tmp) / "pyproject.toml"
-            pyproject.write_text(
-                'dependencies = ["torch==2.8.0", "torchaudio==2.8.0"]\n'
-            )
-            loader = Path(tmp) / "reference_loader.py"
-            loader.write_text(
-                "def load_audio(reference_audio):\n"
-                "        waveform, original_sr = torchaudio.load(reference_audio, backend=self.backend)\n"
-            )
-
-            prepare_upstream.patch_cuda13_dependencies(pyproject)
-            prepare_upstream.patch_reference_loader(loader)
-            try:
-                prepare_upstream.patch_cuda13_dependencies(pyproject)
-                prepare_upstream.patch_reference_loader(loader)
-            except RuntimeError as err:
-                self.fail(f"preparation is not idempotent: {err}")
-
-
-if __name__ == "__main__":
-    unittest.main()

backend/python/fish-speech/requirements-cublas13.txt

@@ -1,3 +1,3 @@
 --extra-index-url https://download.pytorch.org/whl/cu130
-torch==2.9.1+cu130
-torchaudio==2.9.1
+torch
+torchaudio

backend/python/fish-speech/requirements-l4t13.txt

@@ -1,3 +1,3 @@
 --extra-index-url https://download.pytorch.org/whl/cu130
-torch==2.9.1+cu130
-torchaudio==2.9.1
+torch
+torchaudio

backend/python/fish-speech/run.sh

@@ -1,18 +1,9 @@
 #!/bin/bash
-backend_dir=$(dirname "$(realpath "$0")")
+backend_dir=$(dirname $0)
 if [ -d $backend_dir/common ]; then
     source $backend_dir/common/libbackend.sh
 else
     source $backend_dir/../common/libbackend.sh
 fi
 
-# Editable installs record their build-time absolute source path, which becomes
-# stale when the backend is relocated under /backends at install time.
-export PYTHONPATH="${backend_dir}/fish-speech-src${PYTHONPATH:+:${PYTHONPATH}}"
-
-cuda_home=${CUDA_HOME:-/usr/local/cuda}
-if [ -z "${TRITON_PTXAS_PATH:-}" ] && [ -x "$cuda_home/bin/ptxas" ]; then
-    export TRITON_PTXAS_PATH="$cuda_home/bin/ptxas"
-fi
-
-startBackend "$@"
+startBackend $@

backend/python/fish-speech/run_test.sh

@@ -1,23 +0,0 @@
-#!/bin/bash
-set -euo pipefail
-
-SCRIPT_DIR=$(dirname "$(realpath "$0")")
-WORK_DIR=$(mktemp -d)
-trap 'rm -rf "$WORK_DIR"' EXIT
-
-mkdir -p "$WORK_DIR/runtime/common" "$WORK_DIR/runtime/fish-speech-src/fish_speech"
-cp "$SCRIPT_DIR/run.sh" "$WORK_DIR/runtime/run.sh"
-
-cat > "$WORK_DIR/runtime/fish-speech-src/fish_speech/inference_engine.py" <<'PY'
-IMPORT_OK = True
-PY
-
-cat > "$WORK_DIR/runtime/common/libbackend.sh" <<'SH'
-startBackend() {
-    python3 -c 'from fish_speech.inference_engine import IMPORT_OK; assert IMPORT_OK'
-}
-SH
-
-bash "$WORK_DIR/runtime/run.sh"
-
-echo "PASS: runtime launcher imports relocated fish-speech source"

backend/python/funasr/Makefile

@@ -1,23 +0,0 @@
-.PHONY: funasr
-funasr:
-	bash install.sh
-
-.PHONY: run
-run: funasr
-	@echo "Running funasr..."
-	bash run.sh
-	@echo "funasr run."
-
-.PHONY: test
-test: funasr
-	@echo "Testing funasr..."
-	bash test.sh
-	@echo "funasr tested."
-
-.PHONY: protogen-clean
-protogen-clean:
-	$(RM) backend_pb2_grpc.py backend_pb2.py
-
-.PHONY: clean
-clean: protogen-clean
-	rm -rf venv __pycache__

backend/python/funasr/backend.py

@@ -1,152 +0,0 @@
-#!/usr/bin/env python3
-"""
-gRPC backend for LocalAI wrapping FunASR (SenseVoice / Paraformer).
-"""
-from concurrent import futures
-import time
-import argparse
-import signal
-import sys
-import os
-import backend_pb2
-import backend_pb2_grpc
-import torch
-
-import grpc
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'common'))
-sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'common'))
-from grpc_auth import get_auth_interceptors
-
-
-_ONE_DAY_IN_SECONDS = 60 * 60 * 24
-MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1'))
-
-
-class BackendServicer(backend_pb2_grpc.BackendServicer):
-    def Health(self, request, context):
-        return backend_pb2.Reply(message=bytes("OK", 'utf-8'))
-
-    def LoadModel(self, request, context):
-        from funasr import AutoModel
-
-        device = "cpu"
-        if request.CUDA and torch.cuda.is_available():
-            device = "cuda"
-        elif hasattr(torch, "xpu") and torch.xpu.is_available():
-            device = "xpu"
-        elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
-            device = "mps"
-
-        model_id = request.Model or "iic/SenseVoiceSmall"
-        candidate_devices = [device]
-        if device in ("xpu", "mps"):
-            candidate_devices.append("cpu")
-
-        for candidate_device in candidate_devices:
-            try:
-                print(
-                    f"Loading FunASR model: {model_id} on {candidate_device}",
-                    file=sys.stderr,
-                )
-                self.model = AutoModel(
-                    model=model_id,
-                    vad_model="fsmn-vad",
-                    device=candidate_device,
-                    disable_update=True,
-                )
-                print("FunASR model loaded successfully", file=sys.stderr)
-                break
-            except Exception as err:
-                if candidate_device != candidate_devices[-1]:
-                    print(
-                        f"[WARN] FunASR {candidate_device} initialization failed: "
-                        f"{err}; retrying on cpu",
-                        file=sys.stderr,
-                    )
-                    continue
-                print(f"[ERROR] LoadModel failed: {err}", file=sys.stderr)
-                import traceback
-                traceback.print_exc(file=sys.stderr)
-                return backend_pb2.Result(success=False, message=str(err))
-
-        return backend_pb2.Result(message="Model loaded successfully", success=True)
-
-    def AudioTranscription(self, request, context):
-        from funasr.utils.postprocess_utils import rich_transcription_postprocess
-
-        result_segments = []
-        text = ""
-        try:
-            audio_path = request.dst
-            if not audio_path or not os.path.exists(audio_path):
-                print(f"Error: Audio file not found: {audio_path}", file=sys.stderr)
-                return backend_pb2.TranscriptResult(segments=[], text="")
-
-            language = None
-            if request.language and request.language.strip():
-                language = request.language.strip()
-
-            kwargs = {}
-            if language:
-                kwargs["language"] = language
-
-            results = self.model.generate(input=audio_path, **kwargs)
-
-            if not results:
-                return backend_pb2.TranscriptResult(segments=[], text="")
-
-            for idx, r in enumerate(results):
-                seg_text = r.get("text", "") if isinstance(r, dict) else str(r)
-                seg_text = rich_transcription_postprocess(seg_text)
-                text += seg_text
-                result_segments.append(backend_pb2.TranscriptSegment(
-                    id=idx,
-                    start=0,
-                    end=0,
-                    text=seg_text,
-                ))
-
-        except Exception as err:
-            print(f"Error in AudioTranscription: {err}", file=sys.stderr)
-            import traceback
-            traceback.print_exc(file=sys.stderr)
-            return backend_pb2.TranscriptResult(segments=[], text="")
-
-        return backend_pb2.TranscriptResult(segments=result_segments, text=text)
-
-
-def serve(address):
-    server = grpc.server(
-        futures.ThreadPoolExecutor(max_workers=MAX_WORKERS),
-        options=[
-            ('grpc.max_message_length', 50 * 1024 * 1024),
-            ('grpc.max_send_message_length', 50 * 1024 * 1024),
-            ('grpc.max_receive_message_length', 50 * 1024 * 1024),
-        ],
-        interceptors=get_auth_interceptors(),
-    )
-    backend_pb2_grpc.add_BackendServicer_to_server(BackendServicer(), server)
-    server.add_insecure_port(address)
-    server.start()
-    print("Server started. Listening on: " + address, file=sys.stderr)
-
-    def signal_handler(sig, frame):
-        print("Received termination signal. Shutting down...")
-        server.stop(0)
-        sys.exit(0)
-
-    signal.signal(signal.SIGINT, signal_handler)
-    signal.signal(signal.SIGTERM, signal_handler)
-
-    try:
-        while True:
-            time.sleep(_ONE_DAY_IN_SECONDS)
-    except KeyboardInterrupt:
-        server.stop(0)
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description="Run the gRPC server.")
-    parser.add_argument("--addr", default="localhost:50051", help="The address to bind the server to.")
-    args = parser.parse_args()
-    serve(args.addr)

backend/python/funasr/install.sh

@@ -1,21 +0,0 @@
-#!/bin/bash
-set -e
-
-EXTRA_PIP_INSTALL_FLAGS="--no-build-isolation"
-
-backend_dir=$(dirname $0)
-if [ -d $backend_dir/common ]; then
-    source $backend_dir/common/libbackend.sh
-else
-    source $backend_dir/../common/libbackend.sh
-fi
-
-if [ "x${BUILD_PROFILE}" == "xintel" ]; then
-    EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
-fi
-
-PYTHON_VERSION="3.12"
-PYTHON_PATCH="12"
-PY_STANDALONE_TAG="20251120"
-
-installRequirements

backend/python/funasr/requirements-cpu.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/cpu
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-cublas12.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/cu121
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-cublas13.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/cu130
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-hipblas.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/rocm7.0
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-intel.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/xpu
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-l4t12.txt

@@ -1,4 +0,0 @@
---extra-index-url https://pypi.jetson-ai-lab.io/jp6/cu129/
-torch
-torchaudio
-funasr

backend/python/funasr/requirements-l4t13.txt

@@ -1,4 +0,0 @@
---extra-index-url https://download.pytorch.org/whl/cu130
-torch
-torchaudio
-funasr