fix(config): only inject llama.cpp serving options on the llama.cpp path (#10822)

SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".

Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.

Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.


Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

.agents/adding-backends.md

@@ -251,6 +251,7 @@ After adding a new backend, verify:
 - [ ] No YAML syntax errors (check with linter)
 - [ ] No Makefile syntax errors (check with linter)
 - [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
+- [ ] **`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
 - [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
 - [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`
 

.githooks/pre-commit

@@ -7,8 +7,11 @@
 # Runs only the checks relevant to what's staged:
 #   - Go files          -> make lint + make test-coverage-check
 #   - core/http/react-ui -> make test-ui-coverage-check (Playwright e2e + gate)
-# A commit touching neither is skipped entirely (docs/YAML/etc. can't change
-# lint findings, Go coverage, or the UI).
+#   - realtime state machines / specs -> make test-realtime-conformance
+#       (respcoord/**, turncoord/**, or formal-verification/** -- a pure .fizz
+#        spec edit must still re-verify the design, detected separately from Go)
+# A commit touching none of these is skipped entirely (other docs/YAML can't
+# change lint findings, Go coverage, the UI, or the realtime conformance gate).
 #
 # To bypass for a single commit (e.g. a WIP checkpoint): git commit --no-verify
 set -eu
@@ -20,11 +23,13 @@ staged="$(git diff --cached --name-only --diff-filter=ACMRD)"
 
 go_changed=0
 ui_changed=0
+rt_changed=0
 if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
 if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
+if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
 
-if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ]; then
-	echo "pre-commit: no Go or React UI changes staged — skipping."
+if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
+	echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
 	exit 0
 fi
 
@@ -57,4 +62,11 @@ if [ "$ui_changed" -eq 1 ]; then
 	make test-ui-coverage-check
 fi
 
+if [ "$rt_changed" -eq 1 ]; then
+	echo "pre-commit ▶ realtime state-machine conformance (make test-realtime-conformance) —"
+	echo "             Go transition/rapid tests under -race + FizzBee model check of the"
+	echo "             authoritative specs. Fail-closed: needs FizzBee (make install-fizzbee)."
+	make test-realtime-conformance
+fi
+
 echo "pre-commit ✓ all relevant checks passed"

.github/backend-matrix.yml

@@ -478,6 +478,19 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-longcat-video'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "longcat-video"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -790,6 +803,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1136,6 +1162,19 @@ include:
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-longcat-video'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "longcat-video"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1344,6 +1383,19 @@ include:
     backend: "vllm-omni"
     dockerfile: "./backend/Dockerfile.python"
     context: "./"
+  - build-type: 'l4t'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-longcat-video'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    ubuntu-version: '2404'
+    backend: "longcat-video"
+    dockerfile: "./backend/Dockerfile.python"
+    context: "./"
   - build-type: 'l4t'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1721,6 +1773,19 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1760,6 +1825,19 @@ include:
     backend: "parakeet-cpp"
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp'
+    base-image: "ubuntu:24.04"
+    ubuntu-version: '2404'
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -3597,6 +3675,115 @@ include:
     dockerfile: "./backend/Dockerfile.golang"
     context: "./"
     ubuntu-version: '2404'
+  # moss-transcribe-cpp
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-moss-transcribe-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-moss-transcribe-cpp'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-moss-transcribe-cpp'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-moss-transcribe-cpp'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2204'
+  - build-type: 'hipblas'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-rocm-hipblas-moss-transcribe-cpp'
+    base-image: "rocm/dev-ubuntu-24.04:7.2.1"
+    runs-on: 'ubuntu-latest'
+    skip-drivers: 'false'
+    backend: "moss-transcribe-cpp"
+    dockerfile: "./backend/Dockerfile.golang"
+    context: "./"
+    ubuntu-version: '2404'
   # ced
   - build-type: 'cublas'
     cuda-major-version: "12"
@@ -5220,6 +5407,10 @@ includeDarwin:
     tag-suffix: "-metal-darwin-arm64-parakeet-cpp"
     build-type: "metal"
     lang: "go"
+  - backend: "moss-transcribe-cpp"
+    tag-suffix: "-metal-darwin-arm64-moss-transcribe-cpp"
+    build-type: "metal"
+    lang: "go"
   - backend: "ced"
     tag-suffix: "-metal-darwin-arm64-ced"
     build-type: "metal"

.github/workflows/backend.yml

@@ -32,16 +32,30 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     outputs:
-      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
       matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
       matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
       merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
-      merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
-      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
       has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
       has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
       has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
-      has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
+      # Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
+      # stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
+      matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
+      merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
+      has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
+      has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
+      matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
+      merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
+      has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
+      has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
+      matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
+      merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
+      has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
+      has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
+      matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
+      merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
+      has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
+      has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v7
@@ -109,9 +123,9 @@ jobs:
   # take their full ~6h cold without blocking manifest assembly for the
   # multi-arch backends whose per-arch digests would otherwise sit untagged
   # on quay long enough to be GC'd.
-  backend-jobs-singlearch:
+  backend-jobs-singlearch-1:
     needs: generate-matrix
-    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
+    if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
     uses: ./.github/workflows/backend_build.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -138,7 +152,100 @@ jobs:
     strategy:
       fail-fast: false
       max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
+
+  backend-jobs-singlearch-2:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
+
+  backend-jobs-singlearch-3:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
+
+  backend-jobs-singlearch-4:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
 
   # Apply tags to per-arch digests via `imagetools create`. Split into two
   # jobs that mirror the build split so each merge waits ONLY on its
@@ -174,10 +281,12 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
 
-  backend-merge-jobs-singlearch:
-    needs: [generate-matrix, backend-jobs-singlearch]
-    # See note on backend-merge-jobs-multiarch above for !cancelled().
-    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true' }}
+  # One merge shard per build shard: backend-merge-jobs-singlearch-<n> needs only
+  # backend-jobs-singlearch-<n>, preserving the "merge waits only on its own
+  # build" property while staying under the 256-jobs-per-matrix limit.
+  backend-merge-jobs-singlearch-1:
+    needs: [generate-matrix, backend-jobs-singlearch-1]
+    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
     uses: ./.github/workflows/backend_merge.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -189,7 +298,55 @@ jobs:
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
+
+  backend-merge-jobs-singlearch-2:
+    needs: [generate-matrix, backend-jobs-singlearch-2]
+    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
+
+  backend-merge-jobs-singlearch-3:
+    needs: [generate-matrix, backend-jobs-singlearch-3]
+    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
+
+  backend-merge-jobs-singlearch-4:
+    needs: [generate-matrix, backend-jobs-singlearch-4]
+    if: ${{ !cancelled() && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
+      dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
 
   backend-jobs-darwin:
     needs: generate-matrix

.github/workflows/backend_build_darwin.yml

@@ -82,7 +82,7 @@ jobs:
       # as the Linux registry cache.
       - name: Restore Homebrew cache
         id: brew-cache
-        uses: actions/cache/restore@v4
+        uses: actions/cache/restore@v6
         with:
           path: |
             ~/Library/Caches/Homebrew/downloads
@@ -142,7 +142,7 @@ jobs:
 
       - name: Save Homebrew cache
         if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
+        uses: actions/cache/save@v6
         with:
           path: |
             ~/Library/Caches/Homebrew/downloads
@@ -178,7 +178,7 @@ jobs:
       - name: Restore ccache
         if: inputs.backend == 'llama-cpp'
         id: ccache-cache
-        uses: actions/cache/restore@v4
+        uses: actions/cache/restore@v6
         with:
           path: ~/Library/Caches/ccache
           key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
@@ -211,7 +211,7 @@ jobs:
       - name: Restore Python wheel cache
         if: inputs.lang == 'python'
         id: pyenv-cache
-        uses: actions/cache/restore@v4
+        uses: actions/cache/restore@v6
         with:
           path: |
             ~/Library/Caches/pip
@@ -256,14 +256,14 @@ jobs:
 
       - name: Save ccache
         if: inputs.backend == 'llama-cpp' && github.event_name != 'pull_request'
-        uses: actions/cache/save@v4
+        uses: actions/cache/save@v6
         with:
           path: ~/Library/Caches/ccache
           key: ccache-llama-${{ runner.arch }}-${{ steps.llama-version.outputs.version }}-${{ github.run_id }}
 
       - name: Save Python wheel cache
         if: inputs.lang == 'python' && github.event_name != 'pull_request' && steps.pyenv-cache.outputs.cache-hit != 'true'
-        uses: actions/cache/save@v4
+        uses: actions/cache/save@v6
         with:
           path: |
             ~/Library/Caches/pip

.github/workflows/backend_pr.yml

@@ -11,16 +11,30 @@ jobs:
   generate-matrix:
     runs-on: ubuntu-latest
     outputs:
-      matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
       matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
       matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
       merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
-      merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
-      has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
       has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
       has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
       has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
-      has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
+      # Single-arch backends are sharded across SINGLEARCH_SHARDS matrix jobs to
+      # stay under GitHub's 256-jobs-per-matrix limit (see changed-backends.js).
+      matrix-singlearch-1: ${{ steps.set-matrix.outputs['matrix-singlearch-1'] }}
+      merge-matrix-singlearch-1: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-1'] }}
+      has-backends-singlearch-1: ${{ steps.set-matrix.outputs['has-backends-singlearch-1'] }}
+      has-merges-singlearch-1: ${{ steps.set-matrix.outputs['has-merges-singlearch-1'] }}
+      matrix-singlearch-2: ${{ steps.set-matrix.outputs['matrix-singlearch-2'] }}
+      merge-matrix-singlearch-2: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-2'] }}
+      has-backends-singlearch-2: ${{ steps.set-matrix.outputs['has-backends-singlearch-2'] }}
+      has-merges-singlearch-2: ${{ steps.set-matrix.outputs['has-merges-singlearch-2'] }}
+      matrix-singlearch-3: ${{ steps.set-matrix.outputs['matrix-singlearch-3'] }}
+      merge-matrix-singlearch-3: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-3'] }}
+      has-backends-singlearch-3: ${{ steps.set-matrix.outputs['has-backends-singlearch-3'] }}
+      has-merges-singlearch-3: ${{ steps.set-matrix.outputs['has-merges-singlearch-3'] }}
+      matrix-singlearch-4: ${{ steps.set-matrix.outputs['matrix-singlearch-4'] }}
+      merge-matrix-singlearch-4: ${{ steps.set-matrix.outputs['merge-matrix-singlearch-4'] }}
+      has-backends-singlearch-4: ${{ steps.set-matrix.outputs['has-backends-singlearch-4'] }}
+      has-merges-singlearch-4: ${{ steps.set-matrix.outputs['has-merges-singlearch-4'] }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@v7
@@ -71,10 +85,10 @@ jobs:
       fail-fast: true
       max-parallel: 8
       matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
-  backend-jobs-singlearch:
+  backend-jobs-singlearch-1:
     needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-1'] == 'true'
     uses: ./.github/workflows/backend_build.yml
-    if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
     with:
       tag-latest: ${{ matrix.tag-latest }}
       tag-suffix: ${{ matrix.tag-suffix }}
@@ -98,7 +112,94 @@ jobs:
     strategy:
       fail-fast: true
       max-parallel: 8
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-1']) }}
+
+  backend-jobs-singlearch-2:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-2'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: true
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-2']) }}
+
+  backend-jobs-singlearch-3:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-3'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: true
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-3']) }}
+
+  backend-jobs-singlearch-4:
+    needs: generate-matrix
+    if: needs.generate-matrix.outputs['has-backends-singlearch-4'] == 'true'
+    uses: ./.github/workflows/backend_build.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+      build-type: ${{ matrix.build-type }}
+      cuda-major-version: ${{ matrix.cuda-major-version }}
+      cuda-minor-version: ${{ matrix.cuda-minor-version }}
+      platforms: ${{ matrix.platforms }}
+      platform-tag: ${{ matrix.platform-tag || '' }}
+      runs-on: ${{ matrix.runs-on }}
+      builder-base-image: ${{ matrix.builder-base-image || '' }}
+      base-image: ${{ matrix.base-image }}
+      backend: ${{ matrix.backend }}
+      dockerfile: ${{ matrix.dockerfile }}
+      skip-drivers: ${{ matrix.skip-drivers }}
+      context: ${{ matrix.context }}
+      ubuntu-version: ${{ matrix.ubuntu-version }}
+      amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: true
+      max-parallel: 8
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch-4']) }}
   backend-merge-jobs-multiarch:
     needs: [generate-matrix, backend-jobs-multiarch]
     # backend_merge.yml's push-side steps are all gated on
@@ -118,9 +219,9 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
 
-  backend-merge-jobs-singlearch:
-    needs: [generate-matrix, backend-jobs-singlearch]
-    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true' }}
+  backend-merge-jobs-singlearch-1:
+    needs: [generate-matrix, backend-jobs-singlearch-1]
+    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-1'] == 'true' }}
     uses: ./.github/workflows/backend_merge.yml
     with:
       tag-latest: ${{ matrix.tag-latest }}
@@ -130,7 +231,49 @@ jobs:
       quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
     strategy:
       fail-fast: false
-      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-1']) }}
+
+  backend-merge-jobs-singlearch-2:
+    needs: [generate-matrix, backend-jobs-singlearch-2]
+    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-2'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-2']) }}
+
+  backend-merge-jobs-singlearch-3:
+    needs: [generate-matrix, backend-jobs-singlearch-3]
+    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-3'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-3']) }}
+
+  backend-merge-jobs-singlearch-4:
+    needs: [generate-matrix, backend-jobs-singlearch-4]
+    if: ${{ !cancelled() && github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch-4'] == 'true' }}
+    uses: ./.github/workflows/backend_merge.yml
+    with:
+      tag-latest: ${{ matrix.tag-latest }}
+      tag-suffix: ${{ matrix.tag-suffix }}
+    secrets:
+      quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
+      quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch-4']) }}
   backend-jobs-darwin:
     needs: generate-matrix
     uses: ./.github/workflows/backend_build_darwin.yml

.github/workflows/bump_deps.yaml

@@ -26,6 +26,10 @@ jobs:
             variable: "DS4_VERSION"
             branch: "main"
             file: "backend/cpp/ds4/Makefile"
+          - repository: "meituan-longcat/LongCat-Video"
+            variable: "LONGCAT_VIDEO_VERSION"
+            branch: "main"
+            file: "backend/python/longcat-video/Makefile"
           - repository: "localai-org/privacy-filter.cpp"
             variable: "PRIVACY_FILTER_VERSION"
             branch: "master"
@@ -42,6 +46,10 @@ jobs:
             variable: "PARAKEET_VERSION"
             branch: "master"
             file: "backend/go/parakeet-cpp/Makefile"
+          - repository: "mudler/moss-transcribe.cpp"
+            variable: "MOSS_VERSION"
+            branch: "master"
+            file: "backend/go/moss-transcribe-cpp/Makefile"
           - repository: "mudler/ced.cpp"
             variable: "CED_VERSION"
             branch: "master"

.github/workflows/realtime-conformance.yml

@@ -0,0 +1,69 @@
+---
+name: 'realtime-conformance'
+
+# Verifies the realtime state-machine implementations conform to their formal
+# designs (docs/design/realtime-state-machines.md, formal-verification/). BOTH
+# layers are enforced and the gate is fail-closed: the Go conformance layer
+# (respcoord + turncoord transition/rapid tests under -race) AND the FizzBee model check of
+# the authoritative specs. FizzBee is pinned + checksum-verified
+# (formal-verification/fizzbee.sha256), so a failed install fails the job rather
+# than silently skipping verification.
+
+on:
+  pull_request:
+    paths:
+      - 'core/http/endpoints/openai/coordinator/**'
+      - 'core/http/endpoints/openai/respcoord/**'
+      - 'core/http/endpoints/openai/turncoord/**'
+      - 'core/http/endpoints/openai/conncoord/**'
+      - 'core/http/endpoints/openai/compactcoord/**'
+      - 'core/http/endpoints/openai/ttscoord/**'
+      - 'formal-verification/**'
+      - 'scripts/realtime-conformance.sh'
+      - 'scripts/install-fizzbee.sh'
+      - '.github/workflows/realtime-conformance.yml'
+  push:
+    branches:
+      - master
+    paths:
+      - 'core/http/endpoints/openai/coordinator/**'
+      - 'core/http/endpoints/openai/respcoord/**'
+      - 'core/http/endpoints/openai/turncoord/**'
+      - 'core/http/endpoints/openai/conncoord/**'
+      - 'core/http/endpoints/openai/compactcoord/**'
+      - 'core/http/endpoints/openai/ttscoord/**'
+      - 'formal-verification/**'
+      - 'scripts/realtime-conformance.sh'
+
+concurrency:
+  group: realtime-conformance-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  conformance:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        go-version: ['1.26.x']
+    steps:
+      - name: Clone
+        uses: actions/checkout@v7
+      - name: Setup Go ${{ matrix.go-version }}
+        uses: actions/setup-go@v5
+        with:
+          go-version: ${{ matrix.go-version }}
+          cache: false
+      - name: Cache FizzBee
+        uses: actions/cache@v6
+        with:
+          path: .tools/fizzbee
+          key: fizzbee-v0.5.2-${{ runner.os }}-${{ hashFiles('formal-verification/fizzbee.sha256') }}
+      - name: Install FizzBee (pinned, checksum-verified)
+        # No `|| true`: a failed/forged download must fail the job, not silently
+        # drop the design verification. install-fizzbee.sh is a no-op if the
+        # cached binary is already present and valid.
+        run: ./scripts/install-fizzbee.sh
+      - name: Run conformance gate (fail-closed)
+        # No skip env: both the Go conformance and the FizzBee model check are
+        # required. The gate auto-detects .tools/fizzbee/fizz.
+        run: make test-realtime-conformance

.github/workflows/stalebot.yml

@@ -11,7 +11,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@eb5cf3af3ac0a1aa4c9c45633dd1ae542a27a899 # v9
+      - uses: actions/stale@1e223db275d687790206a7acac4d1a11bd6fe629 # v9
         with:
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
           stale-pr-message: 'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'

.gitignore

@@ -97,3 +97,12 @@ core/http/react-ui/test-results/
 
 # Local Apple signing material (never commit)
 .certs/
+
+# Pinned dev tools (e.g. FizzBee for the realtime-conformance gate)
+.tools/
+
+# FizzBee model-check artifacts: the parser emits <spec>.json next to each
+# .fizz and the checker writes run dirs under out/. Both are regenerated by
+# the realtime-conformance gate; only the .fizz sources are authoritative.
+formal-verification/*.json
+formal-verification/out/

Dockerfile

@@ -171,6 +171,17 @@ RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
     ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
     ; fi
 
+# ROCm's bundled libdrm_amdgpu is built with a hardcoded fallback lookup path
+# for the ASIC ID table (/opt/amdgpu/share/libdrm/amdgpu.ids), which only exists
+# if AMD's full amdgpu graphics/DKMS stack is installed. This compute-only image
+# doesn't have it, so hipblas/rocBLAS log "No such file or directory" on every
+# model load and can fail to identify the GPU. Point it at the equivalent file
+# Ubuntu's libdrm-common package already ships.
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ -f /usr/share/libdrm/amdgpu.ids ] && [ ! -e /opt/amdgpu/share/libdrm/amdgpu.ids ]; then \
+    mkdir -p /opt/amdgpu/share/libdrm && \
+    ln -s /usr/share/libdrm/amdgpu.ids /opt/amdgpu/share/libdrm/amdgpu.ids \
+    ; fi
+
 RUN expr "${BUILD_TYPE}" = intel && echo "intel" > /run/localai/capability || echo "not intel"
 
 # Cuda

Makefile

@@ -1,5 +1,5 @@
 # Disable parallel execution for backend builds
-.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
+.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin
 
 GOCMD=go
 GOTEST=$(GOCMD) test
@@ -405,6 +405,18 @@ test-realtime: build-mock-backend
 	@echo 'Running realtime e2e tests (mock backend)'
 	$(GOCMD) run github.com/onsi/ginkgo/v2/ginkgo --label-filter="Realtime && !real-models" --flake-attempts $(TEST_FLAKES) -v -r ./tests/e2e
 
+# Verify the realtime state-machine implementations conform to their formal
+# designs (Go transition/rapid tests under -race + FizzBee model check of the
+# authoritative specs). See docs/design/realtime-state-machines.md (Part 6) and
+# docs/design/specs/README.md.
+test-realtime-conformance:
+	GOCMD=$(GOCMD) ./scripts/realtime-conformance.sh
+
+# Install the pinned, checksum-verified FizzBee model checker (into .tools/,
+# gitignored) used by test-realtime-conformance. Idempotent; no-op if present.
+install-fizzbee:
+	./scripts/install-fizzbee.sh
+
 # Container-based real-model realtime testing. Build env vars / pipeline
 # definition kept here so test-realtime-models-docker can drive a fully wired
 # pipeline (VAD + STT + LLM + TTS) from inside a containerised runner.
@@ -553,6 +565,7 @@ prepare-test-extra: protogen-python
 	$(MAKE) -C backend/python/chatterbox
 	$(MAKE) -C backend/python/vllm
 	$(MAKE) -C backend/python/vllm-omni
+	$(MAKE) -C backend/python/longcat-video
 	$(MAKE) -C backend/python/sglang
 	$(MAKE) -C backend/python/vibevoice
 	$(MAKE) -C backend/python/liquid-audio
@@ -582,6 +595,7 @@ test-extra: prepare-test-extra
 	$(MAKE) -C backend/python/chatterbox test
 	$(MAKE) -C backend/python/vllm test
 	$(MAKE) -C backend/python/vllm-omni test
+	$(MAKE) -C backend/python/longcat-video test
 	$(MAKE) -C backend/python/vibevoice test
 	$(MAKE) -C backend/python/liquid-audio test
 	$(MAKE) -C backend/python/moonshine test
@@ -633,6 +647,9 @@ test-extra: prepare-test-extra
 ## suite against it.
 ##
 BACKEND_TEST_MODEL_URL?=https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/main/Qwen3-0.6B-Q8_0.gguf
+## Suite timeout for `go test`. Wrappers whose model download alone can eat
+## most of the default (multi-GB models on a slow HF CDN day) override this.
+BACKEND_TEST_TIMEOUT?=30m
 
 ## Generic target — runs the suite against whatever BACKEND_IMAGE points at.
 ## Depends on protogen-go so pkg/grpc/proto is generated before `go test`.
@@ -660,7 +677,7 @@ test-extra-backend: protogen-go
 	BACKEND_TEST_FACE_IMAGE_3_URL="$$BACKEND_TEST_FACE_IMAGE_3_URL" \
 	BACKEND_TEST_FACE_IMAGE_3_FILE="$$BACKEND_TEST_FACE_IMAGE_3_FILE" \
 	BACKEND_TEST_VERIFY_DISTANCE_CEILING="$$BACKEND_TEST_VERIFY_DISTANCE_CEILING" \
-	go test -v -timeout 30m ./tests/e2e-backends/...
+	go test -v -timeout $(BACKEND_TEST_TIMEOUT) ./tests/e2e-backends/...
 
 ## Convenience wrappers: build the image, then exercise it.
 test-extra-backend-llama-cpp: docker-build-llama-cpp
@@ -1000,6 +1017,7 @@ test-extra-backend-vibevoice-cpp-tts: docker-build-vibevoice-cpp
 ## post-image disk budget.
 test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
 	BACKEND_IMAGE=local-ai-backend:vibevoice-cpp \
+	BACKEND_TEST_TIMEOUT=120m \
 	BACKEND_TEST_MODEL_URL='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/vibevoice-asr-q4_k.gguf#vibevoice-asr-q4_k.gguf' \
 	BACKEND_TEST_EXTRA_FILES='https://huggingface.co/mudler/vibevoice.cpp-models/resolve/main/tokenizer.gguf#tokenizer.gguf' \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
@@ -1027,7 +1045,19 @@ test-extra-backend-whisper-transcription: docker-build-whisper
 ## is reachable.
 test-extra-backend-parakeet-cpp-transcription: docker-build-parakeet-cpp
 	BACKEND_IMAGE=local-ai-backend:parakeet-cpp \
-	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/tdt_ctc-110m-f16.gguf \
+	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/realtime_eou_120m-v1-f16.gguf \
+	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
+	BACKEND_TEST_CAPS=health,load,transcription \
+	$(MAKE) test-extra-backend
+
+## Audio transcription wrapper for the moss-transcribe-cpp (moss-transcribe.cpp
+## ggml port) backend. Mirrors test-extra-backend-parakeet-cpp-transcription:
+## drives the AudioTranscription RPC against a published MOSS GGUF using the JFK
+## 11s clip from whisper.cpp's CI samples. Not part of the default test suite -
+## run explicitly once the pinned model URL is reachable.
+test-extra-backend-moss-transcribe-cpp-transcription: docker-build-moss-transcribe-cpp
+	BACKEND_IMAGE=local-ai-backend:moss-transcribe-cpp \
+	BACKEND_TEST_MODEL_URL=https://huggingface.co/mudler/moss-transcribe.cpp-gguf/resolve/main/moss-transcribe-q5_k.gguf \
 	BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
 	BACKEND_TEST_CAPS=health,load,transcription \
 	$(MAKE) test-extra-backend
@@ -1200,6 +1230,7 @@ BACKEND_STABLEDIFFUSION_GGML = stablediffusion-ggml|golang|.|--progress=plain|tr
 BACKEND_WHISPER = whisper|golang|.|false|true
 BACKEND_CRISPASR = crispasr|golang|.|false|true
 BACKEND_PARAKEET_CPP = parakeet-cpp|golang|.|false|true
+BACKEND_MOSS_TRANSCRIBE_CPP = moss-transcribe-cpp|golang|.|false|true
 BACKEND_DEPTH_ANYTHING_CPP = depth-anything-cpp|golang|.|false|true
 BACKEND_VOXTRAL = voxtral|golang|.|false|true
 BACKEND_ACESTEP_CPP = acestep-cpp|golang|.|false|true
@@ -1225,6 +1256,7 @@ BACKEND_NEUTTS = neutts|python|.|false|true
 BACKEND_KOKORO = kokoro|python|.|false|true
 BACKEND_VLLM = vllm|python|.|false|true
 BACKEND_VLLM_OMNI = vllm-omni|python|.|false|true
+BACKEND_LONGCAT_VIDEO = longcat-video|python|.|--progress=plain|true
 BACKEND_SGLANG = sglang|python|.|false|true
 BACKEND_DIFFUSERS = diffusers|python|.|--progress=plain|true
 BACKEND_CHATTERBOX = chatterbox|python|.|false|true
@@ -1293,6 +1325,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_STABLEDIFFUSION_GGML)))
 $(eval $(call generate-docker-build-target,$(BACKEND_WHISPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CRISPASR)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PARAKEET_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_MOSS_TRANSCRIBE_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DEPTH_ANYTHING_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VOXTRAL)))
 $(eval $(call generate-docker-build-target,$(BACKEND_OPUS)))
@@ -1309,6 +1342,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VLLM_OMNI)))
+$(eval $(call generate-docker-build-target,$(BACKEND_LONGCAT_VIDEO)))
 $(eval $(call generate-docker-build-target,$(BACKEND_SGLANG)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DIFFUSERS)))
 $(eval $(call generate-docker-build-target,$(BACKEND_CHATTERBOX)))
@@ -1345,7 +1379,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-privacy-filter
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1470,8 +1504,13 @@ build-launcher-darwin:
 	mv cmd/launcher/LocalAI.app dist/LocalAI.app
 	bash contrib/macos/sign-and-notarize.sh sign dist/LocalAI.app
 
-# Wrap the (signed) app into a drag-to-Applications DMG via hdiutil, then sign the DMG.
+# Notarize + staple the .app itself, then wrap it into a drag-to-Applications
+# DMG via hdiutil and sign the DMG. The app is stapled BEFORE packaging so the
+# bundle carries its own ticket and verifies offline (a dmg-only staple leaves
+# the app relying on an online Gatekeeper check, which fails offline / once the
+# app is copied out of the dmg). No-op without notary secrets.
 dmg-launcher-darwin: build-launcher-darwin
+	bash contrib/macos/sign-and-notarize.sh notarize-app dist/LocalAI.app
 	rm -rf dist/dmg dist/LocalAI.dmg
 	mkdir -p dist/dmg
 	cp -R dist/LocalAI.app dist/dmg/LocalAI.app
@@ -1483,7 +1522,7 @@ dmg-launcher-darwin: build-launcher-darwin
 notarize-launcher-darwin: dmg-launcher-darwin
 	bash contrib/macos/sign-and-notarize.sh notarize dist/LocalAI.dmg
 
-# Single entrypoint for CI: build -> sign app -> dmg -> sign dmg -> notarize -> staple.
+# Single entrypoint for CI: build -> sign app -> notarize+staple app -> dmg -> sign dmg -> notarize+staple dmg.
 release-launcher-darwin: notarize-launcher-darwin
 	@echo "dist/LocalAI.dmg is ready"
 

README.md

@@ -232,12 +232,16 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 | Backend | What it does |
 |---------|-------------|
 | [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
+| [moss-transcribe.cpp](https://github.com/mudler/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
 | [ced.cpp](https://github.com/mudler/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
-| [voxtral.c](https://github.com/mudler/voxtral.c) | Voxtral Realtime 4B speech-to-text in pure C |
+| [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
+| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
 | [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
 | [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
 | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) |
 | [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose estimation |
+| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
+| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
 | [privacy-filter.cpp](https://github.com/localai-org/privacy-filter.cpp) | Standalone GGML PII/NER token-classification engine powering LocalAI's PII redaction tier |
 | [LocalVQE](https://github.com/localai-org/LocalVQE) | Joint acoustic echo cancellation, noise suppression, and dereverberation |
 | [local-store](https://github.com/mudler/LocalAI) | Local-first vector database for embeddings (shipped in-tree) |

backend/README.md

@@ -46,6 +46,7 @@ The backend system provides language-specific Dockerfiles that handle the build
 - **vllm**: High-performance LLM inference
 - **mlx**: Apple Silicon optimization
 - **diffusers**: Stable Diffusion models
+- **longcat-video**: CUDA text/image-to-video and speech-driven avatar generation
 - **Audio**: coqui, faster-whisper, kitten-tts
 - **Vision**: mlx-vlm, rfdetr
 - **Specialized**: rerankers, chatterbox, kokoro

backend/backend.proto

@@ -18,6 +18,18 @@ service Backend {
   rpc GenerateVideo(GenerateVideoRequest) returns (Result) {}
   rpc AudioTranscription(TranscriptRequest) returns (TranscriptResult) {}
   rpc AudioTranscriptionStream(TranscriptRequest) returns (stream TranscriptStreamResponse) {}
+  // AudioTranscriptionLive is the bidirectional live-microphone ASR RPC. The
+  // first message MUST carry a Config; subsequent messages carry Audio frames
+  // (mono float PCM at config.sample_rate, 16 kHz default). After a
+  // successful open the backend replies with a single ready ack
+  // (TranscriptLiveResponse{ready:true}); backends or models without
+  // cache-aware streaming support return UNIMPLEMENTED instead. Newly
+  // finalized text streams back as deltas; eou=true marks the model's
+  // end-of-utterance token. One stream spans many utterances (the decoder
+  // resets itself after each EOU). Closing the send side finalizes: the
+  // backend flushes the decoder tail and emits a terminal message carrying
+  // final_result. A second Config mid-stream resets the decode session.
+  rpc AudioTranscriptionLive(stream TranscriptLiveRequest) returns (stream TranscriptLiveResponse) {}
   rpc TTS(TTSRequest) returns (Result) {}
   rpc TTSStream(TTSRequest) returns (stream Reply) {}
   rpc SoundGeneration(SoundGenerationRequest) returns (Result) {}
@@ -479,6 +491,10 @@ message TranscriptResult {
   string text = 2;
   string language = 3;
   float duration = 4;
+  // True when the decode ended on the model's end-of-utterance special token
+  // (<EOU>/<EOB>, emitted by cache-aware streaming models such as
+  // parakeet_realtime_eou_120m-v1). The marker itself is stripped from text.
+  bool eou = 5;
 }
 
 message TranscriptStreamResponse {
@@ -486,6 +502,34 @@ message TranscriptStreamResponse {
   TranscriptResult final_result = 2;
 }
 
+// === AudioTranscriptionLive messages =====================================
+
+message TranscriptLiveRequest {
+  oneof payload {
+    TranscriptLiveConfig config = 1;
+    TranscriptLiveAudio  audio  = 2;
+  }
+}
+
+message TranscriptLiveConfig {
+  string language = 1;             // "" => model default
+  int32 sample_rate = 2;           // 0 => 16000; backends may reject others
+  map<string, string> params = 3;  // backend-specific tuning
+}
+
+message TranscriptLiveAudio {
+  repeated float pcm = 1;          // mono PCM in [-1,1] at config.sample_rate
+}
+
+message TranscriptLiveResponse {
+  bool ready = 1;                       // open ack: sent once, before any delta
+  string delta = 2;                     // newly-finalized text since previous response
+  bool eou = 3;                         // <EOU> fired during this feed (the user yielded the turn)
+  repeated TranscriptWord words = 4;    // words finalized by this feed (stream-relative ns)
+  TranscriptResult final_result = 5;    // terminal message only, after the send side closes
+  bool eob = 6;                         // <EOB> fired: a backchannel ("uh-huh") ended — NOT a turn boundary
+}
+
 message TranscriptWord {
   int64 start = 1;
   int64 end = 2;
@@ -533,6 +577,10 @@ message GenerateVideoRequest {
   float cfg_scale = 10;    // Classifier-free guidance scale
   int32 step = 11;         // Number of inference steps
   string dst = 12;        // Output path for the generated video
+  string audio = 13;      // Path to staged audio for audio-conditioned video
+  // Backend-specific per-request generation parameters. Values are strings
+  // and are validated/coerced by the selected backend.
+  map<string, string> params = 14;
 }
 
 message TTSRequest {
@@ -1212,4 +1260,3 @@ message ForwardReply {
   repeated ForwardHeader headers = 2;
   bytes body_chunk = 3;
 }
-

backend/cpp/ds4/package.sh

@@ -1,12 +1,14 @@
 #!/bin/bash
-set -e
+set -euo pipefail
 CURDIR=$(dirname "$(realpath "$0")")
 REPO_ROOT="${CURDIR}/../../.."
+PACKAGE_DIR="$CURDIR/package"
 
-mkdir -p "$CURDIR/package/lib"
-cp -avf "$CURDIR/grpc-server" "$CURDIR/package/"
-cp -avf "$CURDIR/ds4-worker"  "$CURDIR/package/"
-cp -rfv "$CURDIR/run.sh"      "$CURDIR/package/"
+rm -rf "$PACKAGE_DIR"
+mkdir -p "$PACKAGE_DIR/lib"
+cp -avf "$CURDIR/grpc-server" "$PACKAGE_DIR/"
+cp -avf "$CURDIR/ds4-worker"  "$PACKAGE_DIR/"
+cp -rfv "$CURDIR/run.sh"      "$PACKAGE_DIR/"
 
 UNAME_S=$(uname -s)
 if [ "$UNAME_S" = "Darwin" ]; then
@@ -16,25 +18,54 @@ if [ "$UNAME_S" = "Darwin" ]; then
 fi
 
 if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
-    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so"
-    LIBDIR=/lib/x86_64-linux-gnu
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$PACKAGE_DIR/lib/ld.so"
 elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
-    cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so"
-    LIBDIR=/lib/aarch64-linux-gnu
+    cp -arfLv /lib/ld-linux-aarch64.so.1 "$PACKAGE_DIR/lib/ld.so"
 else
     echo "package.sh: unknown architecture" >&2; exit 1
 fi
 
-for lib in libc.so.6 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libgomp.so.1 \
-           libdl.so.2 librt.so.1 libpthread.so.0; do
-    cp -arfLv "$LIBDIR/$lib" "$CURDIR/package/lib/$lib"
+# Bundle the complete dependency closure for both executables. In particular,
+# grpc-server links the distro gRPC/protobuf/absl stack; copying only the core
+# C/C++ runtime libraries leaves the scratch image unable to start.
+{
+    ldd "$CURDIR/grpc-server"
+    ldd "$CURDIR/ds4-worker"
+} | awk '$2 == "=>" && $3 ~ /^\// { print $3 }' | sort -u | \
+while read -r so; do
+    cp -arfLv "$so" "$PACKAGE_DIR/lib/"
 done
 
 GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
 if [ -f "$GPU_LIB_SCRIPT" ]; then
-    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    # shellcheck source=/dev/null
+    source "$GPU_LIB_SCRIPT" "$PACKAGE_DIR/lib"
     package_gpu_libs
 fi
 
+# Resolve every dependency through the same loader and library path used by
+# the from-scratch image. The loader can still search host defaults, so reject
+# any absolute dependency path that escapes the package instead of accepting a
+# false-positive validation against a library that scratch will not contain.
+validate_packaged_binary() {
+    local binary="$1"
+    local resolution
+    resolution=$("$PACKAGE_DIR/lib/ld.so" \
+        --library-path "$PACKAGE_DIR/lib" \
+        --list "$PACKAGE_DIR/$binary")
+
+    printf '%s\n' "$resolution" | awk -v prefix="$PACKAGE_DIR/lib/" '
+        $2 == "=>" && $3 ~ /^\// && index($3, prefix) != 1 {
+            print "package.sh: dependency resolved outside package: " $0 > "/dev/stderr"
+            invalid = 1
+        }
+        END { exit invalid }
+    '
+}
+
+for binary in grpc-server ds4-worker; do
+    validate_packaged_binary "$binary"
+done
+
 echo "ds4 package contents:"
-ls -lah "$CURDIR/package/" "$CURDIR/package/lib/"
+ls -lah "$PACKAGE_DIR/" "$PACKAGE_DIR/lib/"

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=f96eaddba8bed6a9a5e628bbf6a566775c70b49c
+IK_LLAMA_VERSION?=6198a356a85ed71534c02a9c1026203389f341e5
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/CMakeLists.txt

@@ -101,4 +101,13 @@ if(LLAMA_GRPC_BUILD_TESTS)
     target_link_libraries(message_content_test PRIVATE ${_LLAMA_COMMON_TARGET})
     target_compile_features(message_content_test PRIVATE cxx_std_17)
     add_test(NAME message_content_test COMMAND message_content_test)
+
+    # Parent-death watcher test (parent_watch.h) — standard library only, but
+    # needs a threading runtime for std::thread.
+    find_package(Threads REQUIRED)
+    add_executable(parent_watch_test parent_watch_test.cpp parent_watch.h)
+    target_include_directories(parent_watch_test PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+    target_link_libraries(parent_watch_test PRIVATE Threads::Threads)
+    target_compile_features(parent_watch_test PRIVATE cxx_std_17)
+    add_test(NAME parent_watch_test COMMAND parent_watch_test)
 endif()

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=0ed235ea2c17a19fc8238668653946721ed136fd
+LLAMA_VERSION?=6b4dc2116a92c5c8f2782bfe51fabe5ee66fb5ef
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/grpc-server.cpp

@@ -75,6 +75,8 @@
 #include <windows.h>
 #endif
 
+#include "parent_watch.h" // best-effort parent-death backstop (see header)
+
 
 using grpc::Server;
 using grpc::ServerBuilder;
@@ -611,6 +613,24 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
     // starts with '-'. Applied once after the loop via common_params_parse.
     std::vector<std::string> extra_argv;
 
+    auto add_device_options = [&](const std::string & devices) {
+        const std::regex regex{ R"([,]+)" };
+        std::sregex_token_iterator it{ devices.begin(), devices.end(), regex, -1 };
+        std::vector<std::string> split_arg{ it, {} };
+
+        for (std::string device : split_arg) {
+            const auto start = device.find_first_not_of(" \t\n\r");
+            if (start == std::string::npos) {
+                continue;
+            }
+            const auto end = device.find_last_not_of(" \t\n\r");
+            device = device.substr(start, end - start + 1);
+
+            extra_argv.push_back("--device");
+            extra_argv.push_back(device);
+        }
+    };
+
      // decode options. Options are in form optname:optvale, or if booleans only optname.
     for (int i = 0; i < request->options_size(); i++) {
         std::string opt = request->options(i);
@@ -742,6 +762,10 @@ static void params_parse(server_context& /*ctx_server*/, const backend::ModelOpt
             } else if (optval_str == "false" || optval_str == "0" || optval_str == "no" || optval_str == "off" || optval_str == "disabled") {
                 params.no_op_offload = false;
             }
+        } else if (!strcmp(optname, "device") || !strcmp(optname, "devices")) {
+            if (optval != NULL) {
+                add_device_options(optval_str);
+            }
         } else if (!strcmp(optname, "split_mode") || !strcmp(optname, "sm")) {
             // Accepts: none | layer | row | tensor (the latter requires a llama.cpp build
             // that includes ggml-org/llama.cpp#19378, FlashAttention enabled, and KV-cache
@@ -3442,6 +3466,10 @@ int main(int argc, char** argv) {
     }
   }
 
+    // Best-effort backstop: self-terminate if the LocalAI process that spawned
+    // us dies without cleaning us up (see parent_watch.h).
+    llama_grpc::start_parent_death_watcher();
+
     server_context ctx_server;
     BackendServiceImpl service(ctx_server);
 

backend/cpp/llama-cpp/parent_watch.h

@@ -0,0 +1,179 @@
+// Parent-death watcher (best-effort backstop) for the llama.cpp gRPC backend.
+//
+// LocalAI spawns this backend as a child process and, on a clean shutdown,
+// tears it down itself (SIGTERM -> grace -> SIGKILL). That graceful path only
+// runs when LocalAI receives a catchable signal and lives long enough to run
+// its handlers. If LocalAI is SIGKILLed (e.g. a supervising process's grace
+// period elapses first), that teardown never runs and this backend would be
+// reparented to init and linger, holding VRAM and its listen port.
+//
+// The watcher here is a best-effort backstop for exactly that case: it does
+// NOT replace the graceful teardown, it only covers the "parent vanished
+// without cleaning up" path. It detects reparenting: when the process that
+// spawned this backend dies, the kernel reparents us to the nearest sub-reaper
+// or to init (PID 1), so getppid() stops matching the value captured at
+// startup. This getppid() approach is portable across Linux/macOS (unlike the
+// Linux-only PR_SET_PDEATHSIG), which is why it is used here, mirroring the Go
+// backends' pkg/grpc/parentwatch.go. It is disabled on Windows, which has no
+// equivalent orphan-reparenting semantics.
+//
+// This header is intentionally dependency-free (C++ standard library only) so
+// it can be exercised by a standalone unit test (parent_watch_test.cpp) without
+// building the full llama.cpp + gRPC backend.
+#ifndef LLAMA_GRPC_PARENT_WATCH_H
+#define LLAMA_GRPC_PARENT_WATCH_H
+
+#include <algorithm>
+#include <cctype>
+#include <chrono>
+#include <cstdio>
+#include <cstdlib>
+#include <functional>
+#include <string>
+#include <thread>
+
+#if !defined(_WIN32)
+#include <unistd.h> // getppid(2), _exit(2)
+#endif
+
+namespace llama_grpc {
+
+// Env var names are shared verbatim with the Go and Python backends for
+// consistency across languages.
+inline const char *kEnvParentWatch()         { return "LOCALAI_BACKEND_PARENT_WATCH"; }
+inline const char *kEnvParentWatchInterval() { return "LOCALAI_BACKEND_PARENT_WATCH_INTERVAL"; }
+
+// Default poll interval in milliseconds. Matches the Go side's 2 * time.Second.
+inline long parent_watch_default_interval_ms() { return 2000; }
+
+namespace detail {
+inline std::string trim_lower(const std::string &in, bool lower) {
+    size_t a = in.find_first_not_of(" \t\r\n");
+    size_t b = in.find_last_not_of(" \t\r\n");
+    if (a == std::string::npos) {
+        return "";
+    }
+    std::string s = in.substr(a, b - a + 1);
+    if (lower) {
+        std::transform(s.begin(), s.end(), s.begin(),
+                       [](unsigned char c) { return std::tolower(c); });
+    }
+    return s;
+}
+} // namespace detail
+
+// parent_watch_enabled reports whether the watcher should run. Enabled by
+// default; a falsey value ("false"/"0"/"no"/"off", case-insensitive) disables
+// it, matching the Go implementation's exact semantics.
+inline bool parent_watch_enabled() {
+#if defined(_WIN32)
+    return false;
+#else
+    const char *v = std::getenv(kEnvParentWatch());
+    if (v == nullptr || v[0] == '\0') {
+        return true;
+    }
+    const std::string s = detail::trim_lower(v, true);
+    return !(s == "false" || s == "0" || s == "no" || s == "off");
+#endif
+}
+
+// parent_watch_interval_ms returns the poll interval in milliseconds. Accepts
+// Go-style duration strings ("500ms", "2s", "1m") for cross-language parity, or
+// a bare number interpreted as seconds. Defaults to
+// parent_watch_default_interval_ms().
+inline long parent_watch_interval_ms() {
+    const long def = parent_watch_default_interval_ms();
+    const char *v = std::getenv(kEnvParentWatchInterval());
+    if (v == nullptr || v[0] == '\0') {
+        return def;
+    }
+    const std::string s = detail::trim_lower(v, false);
+    if (s.empty()) {
+        return def;
+    }
+    size_t i = 0;
+    while (i < s.size() && (std::isdigit((unsigned char)s[i]) || s[i] == '.')) {
+        i++;
+    }
+    if (i == 0) {
+        return def;
+    }
+    double num = 0.0;
+    try {
+        num = std::stod(s.substr(0, i));
+    } catch (...) {
+        return def;
+    }
+    const std::string unit = s.substr(i);
+    long ms;
+    if (unit == "ms") {
+        ms = (long)num;
+    } else if (unit == "s" || unit.empty()) {
+        ms = (long)(num * 1000.0);
+    } else if (unit == "m") {
+        ms = (long)(num * 60000.0);
+    } else {
+        return def; // unrecognized unit
+    }
+    return ms > 0 ? ms : def;
+}
+
+#if !defined(_WIN32)
+// parent_died reports whether this process has been reparented away from the
+// parent it had when the watcher started. Reparenting is the standard POSIX
+// signal that the original parent (here, the LocalAI process that spawned this
+// backend) has exited: the orphan is handed to the nearest sub-reaper or to
+// init (PID 1), so getppid() no longer matches the value captured at startup.
+inline bool parent_died(pid_t orig_ppid) {
+    const pid_t ppid = getppid();
+    return ppid != orig_ppid || ppid == 1;
+}
+
+// watch_parent_death polls until parent_died reports the original parent is
+// gone, then invokes on_death. It blocks, so run it on its own thread.
+inline void watch_parent_death(pid_t orig_ppid, long interval_ms,
+                               const std::function<void()> &on_death) {
+    for (;;) {
+        std::this_thread::sleep_for(std::chrono::milliseconds(interval_ms));
+        if (parent_died(orig_ppid)) {
+            on_death();
+            return;
+        }
+    }
+}
+#endif
+
+// start_parent_death_watcher installs the best-effort safety net described in
+// the file header on the calling backend process. It is a no-op when disabled,
+// on Windows, or when the process is already orphaned at startup
+// (getppid() <= 1). This is a backstop alongside — never a replacement for —
+// LocalAI's graceful teardown.
+inline void start_parent_death_watcher() {
+#if !defined(_WIN32)
+    if (!parent_watch_enabled()) {
+        return;
+    }
+    const pid_t orig_ppid = getppid();
+    // A parent of 1 (or less) at startup means we were already orphaned (or
+    // launched directly under init) — there is no original parent to watch for.
+    if (orig_ppid <= 1) {
+        return;
+    }
+    const long interval_ms = parent_watch_interval_ms();
+    std::thread([orig_ppid, interval_ms]() {
+        watch_parent_death(orig_ppid, interval_ms, [orig_ppid]() {
+            fprintf(stderr,
+                    "backend parent process (pid %d) exited without stopping "
+                    "this backend; self-terminating to avoid orphaning\n",
+                    (int)orig_ppid);
+            fflush(stderr);
+            _exit(1);
+        });
+    }).detach();
+#endif
+}
+
+} // namespace llama_grpc
+
+#endif // LLAMA_GRPC_PARENT_WATCH_H

backend/cpp/llama-cpp/parent_watch_test.cpp

@@ -0,0 +1,197 @@
+// Unit tests for the parent-death watcher (parent_watch.h).
+//
+// Build & run standalone (C++ standard library only, no nlohmann/json needed):
+//   g++ -std=c++17 -pthread parent_watch_test.cpp -o t && ./t
+//
+// The core test (TestDetectsReparent) builds a genuine two-level process tree
+// (test -> middle -> grandchild), lets the middle process die, and asserts the
+// grandchild's watch_parent_death detects the reparenting and self-terminates —
+// mirroring the Go test in pkg/grpc/parentwatch_test.go, but with fork(2).
+//
+// On Windows this file compiles to a no-op success (the watcher is unsupported
+// there), matching parent_watch.h's platform gating.
+
+#include <cstdio>
+#include <cstdlib>
+#include <string>
+
+#include "parent_watch.h"
+
+static int failures = 0;
+
+static void check(bool ok, const std::string &name) {
+    if (!ok) {
+        failures++;
+        fprintf(stderr, "FAIL: %s\n", name.c_str());
+    } else {
+        fprintf(stderr, "ok:   %s\n", name.c_str());
+    }
+}
+
+// Env-parsing tests are platform-independent and always run.
+static void test_env_parsing() {
+    using namespace llama_grpc;
+
+    // Interval: default when unset.
+    unsetenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL");
+    check(parent_watch_interval_ms() == 2000, "interval default 2000ms");
+
+    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "500ms", 1);
+    check(parent_watch_interval_ms() == 500, "interval 500ms");
+
+    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "2s", 1);
+    check(parent_watch_interval_ms() == 2000, "interval 2s");
+
+    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "1m", 1);
+    check(parent_watch_interval_ms() == 60000, "interval 1m");
+
+    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "3", 1); // bare number -> seconds
+    check(parent_watch_interval_ms() == 3000, "interval bare 3 -> 3000ms");
+
+    setenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL", "garbage", 1);
+    check(parent_watch_interval_ms() == 2000, "interval garbage -> default");
+    unsetenv("LOCALAI_BACKEND_PARENT_WATCH_INTERVAL");
+
+#if !defined(_WIN32)
+    // Enabled semantics (POSIX only; always false on Windows).
+    unsetenv("LOCALAI_BACKEND_PARENT_WATCH");
+    check(parent_watch_enabled(), "enabled by default");
+
+    for (const char *falsey : {"false", "0", "no", "off", "OFF", " False "}) {
+        setenv("LOCALAI_BACKEND_PARENT_WATCH", falsey, 1);
+        check(!parent_watch_enabled(), std::string("disabled by '") + falsey + "'");
+    }
+    setenv("LOCALAI_BACKEND_PARENT_WATCH", "true", 1);
+    check(parent_watch_enabled(), "enabled by 'true'");
+    setenv("LOCALAI_BACKEND_PARENT_WATCH", "1", 1);
+    check(parent_watch_enabled(), "enabled by '1'");
+    unsetenv("LOCALAI_BACKEND_PARENT_WATCH");
+#endif
+}
+
+#if !defined(_WIN32)
+
+#include <atomic>
+#include <ctime>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unistd.h>
+
+static bool file_exists(const std::string &p) {
+    struct stat st;
+    return ::stat(p.c_str(), &st) == 0;
+}
+
+static bool wait_for_file(const std::string &p, int timeout_ms) {
+    int waited = 0;
+    while (waited < timeout_ms) {
+        if (file_exists(p)) {
+            return true;
+        }
+        usleep(20 * 1000);
+        waited += 20;
+    }
+    return false;
+}
+
+static void write_file(const std::string &p, const std::string &content) {
+    FILE *f = fopen(p.c_str(), "w");
+    if (f) {
+        fwrite(content.data(), 1, content.size(), f);
+        fclose(f);
+    }
+}
+
+// Builds test -> middle -> grandchild via fork(2). The grandchild arms the REAL
+// watch_parent_death against middle; middle exits, orphaning the grandchild;
+// the watcher must detect the reparenting and self-terminate.
+static void test_detects_reparent() {
+    char tmpl[] = "/tmp/parentwatch_test_XXXXXX";
+    char *dir = mkdtemp(tmpl);
+    if (dir == nullptr) {
+        check(false, "mkdtemp");
+        return;
+    }
+    const std::string ready_file = std::string(dir) + "/ready";
+    const std::string exited_file = std::string(dir) + "/exited";
+
+    pid_t middle = fork();
+    if (middle < 0) {
+        check(false, "fork middle");
+        return;
+    }
+
+    if (middle == 0) {
+        // ---- middle process ----
+        pid_t grandchild = fork();
+        if (grandchild < 0) {
+            _exit(4);
+        }
+        if (grandchild == 0) {
+            // ---- grandchild process ----
+            pid_t orig_ppid = getppid(); // == middle
+            std::thread([&]() {
+                llama_grpc::watch_parent_death(orig_ppid, 50 /*ms*/, [&]() {
+                    write_file(exited_file, "1");
+                    _exit(7);
+                });
+            }).detach();
+
+            // Safety valve: never linger if something goes wrong.
+            std::thread([]() {
+                usleep(30 * 1000 * 1000);
+                _exit(2);
+            }).detach();
+
+            // Signal readiness only after the watcher captured orig_ppid.
+            write_file(ready_file, std::to_string(getpid()));
+            for (;;) {
+                pause();
+            }
+        }
+        // middle: wait until grandchild is ready, then exit to orphan it.
+        if (!wait_for_file(ready_file, 10000)) {
+            _exit(5);
+        }
+        _exit(0);
+    }
+
+    // ---- test (top) process ----
+    int status = 0;
+    waitpid(middle, &status, 0); // reap middle only; grandchild is orphaned
+
+    check(file_exists(ready_file), "grandchild signaled readiness");
+
+    bool detected = wait_for_file(exited_file, 10000);
+    check(detected, "watcher detected parent death and self-terminated");
+
+    // Best-effort cleanup: kill the grandchild if it somehow survived.
+    if (file_exists(ready_file)) {
+        FILE *f = fopen(ready_file.c_str(), "r");
+        if (f) {
+            int pid = 0;
+            if (fscanf(f, "%d", &pid) == 1 && pid > 1) {
+                kill(pid, SIGKILL);
+            }
+            fclose(f);
+        }
+    }
+    unlink(ready_file.c_str());
+    unlink(exited_file.c_str());
+    rmdir(dir);
+}
+
+#endif // !_WIN32
+
+int main() {
+    test_env_parsing();
+#if !defined(_WIN32)
+    test_detects_reparent();
+#endif
+    if (failures == 0) {
+        fprintf(stderr, "\nAll parent_watch tests passed.\n");
+        return 0;
+    }
+    fprintf(stderr, "\n%d parent_watch test(s) failed.\n", failures);
+    return 1;
+}

backend/cpp/llama-cpp/prepare.sh

@@ -22,6 +22,10 @@ cp -r grpc-server.cpp llama.cpp/tools/grpc-server/
 # unit test (compiled only when -DLLAMA_GRPC_BUILD_TESTS=ON).
 cp -r message_content.h llama.cpp/tools/grpc-server/
 cp -r message_content_test.cpp llama.cpp/tools/grpc-server/
+# Parent-death watcher (included by grpc-server.cpp) and its standalone unit
+# test (run via backend/cpp/run-unit-tests.sh; also buildable under ctest).
+cp -r parent_watch.h llama.cpp/tools/grpc-server/
+cp -r parent_watch_test.cpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/nlohmann/json.hpp llama.cpp/tools/grpc-server/
 cp -rfv llama.cpp/vendor/cpp-httplib/httplib.h llama.cpp/tools/grpc-server/
 

backend/cpp/llama-cpp/run.sh

@@ -36,6 +36,12 @@ else
 	if [ -d "$CURDIR/lib/rocblas/library" ]; then
 		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
 	fi
+	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
+	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
+	# the bundled data or it falls back to slow generic kernels (issue #10660).
+	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
+		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
+	fi
 fi
 
 # If there is a lib/ld.so, use it

backend/cpp/privacy-filter/Makefile

@@ -8,7 +8,7 @@
 # Local development: point at a working checkout instead of cloning, e.g.
 #   make PRIVACY_FILTER_SRC=$HOME/c/privacy-filter.cpp grpc-server
 
-PRIVACY_FILTER_VERSION?=98f52c5ef2250f207cc6b9a6aef05393a120cb7c
+PRIVACY_FILTER_VERSION?=735a6c28607ee82afc3a670383f41b55266a3b9a
 PRIVACY_FILTER_REPO?=https://github.com/localai-org/privacy-filter.cpp
 PRIVACY_FILTER_SRC?=
 

backend/cpp/run-unit-tests.sh

@@ -54,7 +54,7 @@ for test_src in "${tests[@]}"; do
     name="$(basename "$test_src" .cpp)"
     bin="$(mktemp -d)/$name"
     echo "==> $test_src"
-    if ! "$CXX" -std=c++17 -Wall -Wextra \
+    if ! "$CXX" -std=c++17 -Wall -Wextra -pthread \
         -I"$JSON_INC" -I"$(dirname "$test_src")" \
         "$test_src" -o "$bin"; then
         echo "COMPILE FAILED: $test_src" >&2

backend/cpp/turboquant/run.sh

@@ -34,6 +34,12 @@ else
 	if [ -d "$CURDIR/lib/rocblas/library" ]; then
 		export ROCBLAS_TENSILE_LIBPATH="$CURDIR"/lib/rocblas/library
 	fi
+	# Same for hipBLASLt (rocblaslt): the bundled libhipblaslt.so resolves its
+	# TensileLibrary_lazy_gfx*.dat kernel data relative to itself, so point it at
+	# the bundled data or it falls back to slow generic kernels (issue #10660).
+	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
+		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
+	fi
 fi
 
 # If there is a lib/ld.so, use it

backend/go/acestep-cpp/CMakeLists.txt

@@ -25,7 +25,7 @@ target_include_directories(goacestepcpp PRIVATE ${ACESTEP_DIR}/src ${ACESTEP_DIR
 target_include_directories(goacestepcpp SYSTEM PRIVATE ${ACESTEP_DIR}/ggml/include)
 
 # Link GPU backends if available (mirrors link_ggml_backends macro)
-foreach(backend blas cuda metal vulkan)
+foreach(backend blas cuda hip metal vulkan)
     if(TARGET ggml-${backend})
         target_link_libraries(goacestepcpp PRIVATE ggml-${backend})
         string(TOUPPER ${backend} BACKEND_UPPER)

backend/go/acestep-cpp/Makefile

@@ -24,7 +24,14 @@ else ifeq ($(BUILD_TYPE),openblas)
 else ifeq ($(BUILD_TYPE),clblas)
 	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
 else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=ON
 else ifeq ($(OS),Darwin)

backend/go/cloud-proxy/provider_anthropic.go

@@ -142,19 +142,12 @@ func buildAnthropicRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream boo
 	if req.MaxTokens <= 0 {
 		req.MaxTokens = anthropicDefaultMaxTokens
 	}
-	// Newer Anthropic models 400 when both temperature and top_p are
-	// set ("`temperature` and `top_p` cannot both be specified for
-	// this model. Please use only one.") even though their docs only
-	// "recommend" picking one. The OpenAI-compatible chat UI almost
-	// always sends both with default values, so prefer temperature
-	// and drop top_p when both are present.
-	if t := opts.GetTemperature(); t != 0 {
-		v := float64(t)
-		req.Temperature = &v
-	} else if t := opts.GetTopP(); t != 0 {
-		v := float64(t)
-		req.TopP = &v
-	}
+	// Do not forward temperature/top_p. Newer Anthropic reasoning models reject
+	// requests that carry temperature ("`temperature` is deprecated for this
+	// model"), and the OpenAI-compatible clients typically send only the
+	// server-side DEFAULT sampling values rather than user intent — dropping
+	// them loses nothing and lets the upstream apply its own defaults.
+	_ = opts
 
 	req.Tools = convertOpenAITools(opts.GetTools())
 	req.ToolChoice = convertOpenAIToolChoice(opts.GetToolChoice())

backend/go/cloud-proxy/provider_anthropic_test.go

@@ -3,7 +3,6 @@ package main
 import (
 	"encoding/json"
 	"io"
-	"math"
 	"net/http"
 	"net/http/httptest"
 	"strings"
@@ -75,15 +74,16 @@ func TestPredict_Anthropic_BasicMessages(t *testing.T) {
 	g.Expect(captured.Messages).To(HaveLen(1))
 	g.Expect(captured.Messages[0].Role).To(Equal("user"))
 	g.Expect(captured.MaxTokens).To(Equal(int32(32)))
-	g.Expect(captured.Temperature).NotTo(BeNil())
-	g.Expect(*captured.Temperature).To(Equal(0.5))
-	// Anthropic 400s when both temperature and top_p are set; the
-	// translator must prefer temperature and drop top_p.
+	// Newer Anthropic reasoning models reject requests carrying temperature
+	// ("`temperature` is deprecated for this model"); clients typically send
+	// only default sampling values, so the translator forwards neither.
+	g.Expect(captured.Temperature).To(BeNil())
 	g.Expect(captured.TopP).To(BeNil())
 	g.Expect(captured.Stream).To(BeFalse())
 }
 
-// When only top_p is set, it should be forwarded.
+// Sampling parameters are not forwarded at all — the upstream applies its
+// own defaults (newest models reject explicit temperature/top_p).
 func TestPredict_Anthropic_TopPOnly(t *testing.T) {
 	g := NewWithT(t)
 	srv, captured := fakeAnthropicUpstream(t, func(_ anthropicRequest) (int, string, string) {
@@ -99,11 +99,7 @@ func TestPredict_Anthropic_TopPOnly(t *testing.T) {
 	})
 	g.Expect(err).NotTo(HaveOccurred())
 	g.Expect(captured.Temperature).To(BeNil())
-	// PredictOptions.TopP is float32 on the wire; the translator widens
-	// to float64 so 0.9 round-trips as 0.8999999761581421… — compare
-	// with a small tolerance rather than exact equality.
-	g.Expect(captured.TopP).NotTo(BeNil())
-	g.Expect(math.Abs(*captured.TopP - 0.9)).To(BeNumerically("<=", 1e-6))
+	g.Expect(captured.TopP).To(BeNil())
 }
 
 func TestPredict_Anthropic_DefaultsMaxTokens(t *testing.T) {

backend/go/cloud-proxy/provider_openai.go

@@ -30,7 +30,7 @@ type openAIRequest struct {
 	Stream           bool            `json:"stream,omitempty"`
 	Temperature      *float64        `json:"temperature,omitempty"`
 	TopP             *float64        `json:"top_p,omitempty"`
-	MaxTokens        *int32          `json:"max_tokens,omitempty"`
+	MaxTokens        *int32          `json:"max_completion_tokens,omitempty"` // newer OpenAI models reject max_tokens ("use max_completion_tokens instead")
 	Stop             []string        `json:"stop,omitempty"`
 	FrequencyPenalty *float64        `json:"frequency_penalty,omitempty"`
 	PresencePenalty  *float64        `json:"presence_penalty,omitempty"`
@@ -107,14 +107,10 @@ func buildOpenAIRequest(opts *pb.PredictOptions, cfg *proxyConfig, stream bool)
 		Tools:      parseRawJSON(opts.GetTools()),
 		ToolChoice: parseRawJSON(opts.GetToolChoice()),
 	}
-	if t := opts.GetTemperature(); t != 0 {
-		v := float64(t)
-		req.Temperature = &v
-	}
-	if t := opts.GetTopP(); t != 0 {
-		v := float64(t)
-		req.TopP = &v
-	}
+	// Do not forward temperature/top_p. Newer OpenAI reasoning models reject
+	// temperature as deprecated, and clients typically send only default
+	// sampling values rather than user intent — let the upstream apply its
+	// own defaults.
 	if n := opts.GetTokens(); n > 0 {
 		req.MaxTokens = &n
 	}

backend/go/cloud-proxy/provider_openai_test.go

@@ -74,8 +74,9 @@ func TestPredict_OpenAI_BasicChat(t *testing.T) {
 	g.Expect(captured.Messages).To(HaveLen(2))
 	g.Expect(captured.Messages[0].Role).To(Equal("system"))
 	g.Expect(captured.Messages[1].Role).To(Equal("user"))
-	g.Expect(captured.Temperature).NotTo(BeNil())
-	g.Expect(*captured.Temperature).To(Equal(0.5))
+	// Sampling parameters are not forwarded (newest models reject explicit
+	// temperature); token limit is serialized as max_completion_tokens.
+	g.Expect(captured.Temperature).To(BeNil())
 	g.Expect(captured.MaxTokens).NotTo(BeNil())
 	g.Expect(*captured.MaxTokens).To(Equal(int32(32)))
 	g.Expect(captured.Stream).To(BeFalse())

backend/go/crispasr/CMakeLists.txt

@@ -17,7 +17,7 @@ target_link_libraries(gocrispasr PRIVATE
     crispasr-lib
     parakeet canary canary_ctc cohere granite_speech granite_nle
     voxtral voxtral4b qwen3_asr qwen3_tts orpheus chatterbox indextts
-    kokoro voxcpm2_tts m2m100 t5_translate wav2vec2-ggml vibevoice
+    kokoro voxcpm2_tts m2m100 t5_translate wav2vec2-ggml vibevoice f5-tts
     silero-lid pyannote-seg funasr paraformer sensevoice
     crisp_audio
     ggml)

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=6514c9da00b03a2f0f1b49a43fae4f3a01a41844
+CRISPASR_VERSION?=d76cce027e3b183fc3d8c72e976e69d11f71bc8b
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/crispasr/gocrispasr.go

@@ -530,11 +530,51 @@ func setVoice(voice string) {
 	}
 }
 
+// applyRequestVoice distinguishes named speakers from per-request reference
+// WAVs. The latter are used by F5-TTS and require the exact transcript under
+// the cross-backend params.ref_text contract.
+func applyRequestVoice(req *pb.TTSRequest) error {
+	voice := strings.TrimSpace(req.Voice)
+	if voice == "" {
+		return nil
+	}
+
+	info, statErr := os.Stat(voice)
+	looksLikeFile := filepath.IsAbs(voice) || strings.EqualFold(filepath.Ext(voice), ".wav")
+	if statErr == nil && info.Mode().IsRegular() {
+		refText := ""
+		if req.Params != nil {
+			refText = strings.TrimSpace(req.Params["ref_text"])
+			if refText == "" {
+				refText = strings.TrimSpace(req.Params["voice_text"])
+			}
+		}
+		if refText == "" {
+			return fmt.Errorf("crispasr: params.ref_text is required with a reference voice WAV")
+		}
+		if rc := CppTTSSetVoiceFile(voice, refText); rc < 0 {
+			return fmt.Errorf("crispasr: failed to apply reference voice %q (rc=%d)", voice, rc)
+		}
+		return nil
+	}
+	if looksLikeFile {
+		if statErr != nil {
+			return fmt.Errorf("crispasr: reference voice %q: %w", voice, statErr)
+		}
+		return fmt.Errorf("crispasr: reference voice %q is not a regular file", voice)
+	}
+
+	setVoice(voice)
+	return nil
+}
+
 func (w *CrispASR) TTS(req *pb.TTSRequest) error {
 	if req.Dst == "" {
 		return fmt.Errorf("crispasr: TTS requires a destination path")
 	}
-	setVoice(req.Voice)
+	if err := applyRequestVoice(req); err != nil {
+		return err
+	}
 	pcm, err := w.synthesize(req.Text)
 	if err != nil {
 		return err
@@ -553,7 +593,9 @@ func (w *CrispASR) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 	if req.Text == "" {
 		return fmt.Errorf("crispasr: TTSStream requires text")
 	}
-	setVoice(req.Voice)
+	if err := applyRequestVoice(req); err != nil {
+		return err
+	}
 	pcm, err := w.synthesize(req.Text)
 	if err != nil {
 		return err

backend/go/crispasr/gocrispasr_test.go

@@ -172,6 +172,32 @@ var _ = Describe("CrispASR", func() {
 	})
 
 	Context("TTS", func() {
+		It("applies a per-request reference WAV and transcript", func() {
+			refWAV := filepath.Join(GinkgoT().TempDir(), "reference.wav")
+			Expect(os.WriteFile(refWAV, []byte("fixture"), 0o600)).To(Succeed())
+
+			original := CppTTSSetVoiceFile
+			DeferCleanup(func() { CppTTSSetVoiceFile = original })
+			var gotPath, gotText string
+			CppTTSSetVoiceFile = func(path, refText string) int {
+				gotPath, gotText = path, refText
+				return 0
+			}
+
+			Expect(applyRequestVoice(&pb.TTSRequest{
+				Voice:  refWAV,
+				Params: map[string]string{"ref_text": "The exact words in the clip."},
+			})).To(Succeed())
+			Expect(gotPath).To(Equal(refWAV))
+			Expect(gotText).To(Equal("The exact words in the clip."))
+		})
+
+		It("rejects a reference WAV without a transcript", func() {
+			refWAV := filepath.Join(GinkgoT().TempDir(), "reference.wav")
+			Expect(os.WriteFile(refWAV, []byte("fixture"), 0o600)).To(Succeed())
+			Expect(applyRequestVoice(&pb.TTSRequest{Voice: refWAV})).To(MatchError(ContainSubstring("params.ref_text")))
+		})
+
 		It("synthesizes a non-empty WAV", func() {
 			ttsModel := ttsModelOrSkip()
 			ensureLibLoaded()
@@ -189,5 +215,32 @@ var _ = Describe("CrispASR", func() {
 			Expect(info.Size()).To(BeNumerically(">", 1024),
 				"expected a non-trivial WAV, got %d bytes", info.Size())
 		})
+
+		It("synthesizes with F5-TTS voice cloning (reference WAV + transcript)", func() {
+			// F5-TTS has no baked speaker: it clones from a reference WAV and
+			// its transcript, supplied via the voice/voice_text options. The
+			// spec skips unless all three fixtures are provided.
+			model := os.Getenv("CRISPASR_F5_MODEL_PATH")
+			refWav := os.Getenv("CRISPASR_F5_REF_WAV")
+			refText := os.Getenv("CRISPASR_F5_REF_TEXT")
+			if model == "" || refWav == "" || refText == "" {
+				Skip("set CRISPASR_F5_MODEL_PATH, CRISPASR_F5_REF_WAV and CRISPASR_F5_REF_TEXT to run this spec")
+			}
+			ensureLibLoaded()
+
+			w := &CrispASR{}
+			Expect(w.Load(&pb.ModelOptions{
+				ModelFile: model,
+				Options:   []string{"voice:" + refWav, "voice_text:" + refText},
+			})).To(Succeed())
+
+			dst := filepath.Join(GinkgoT().TempDir(), "f5.wav")
+			Expect(w.TTS(&pb.TTSRequest{Text: "Hello from LocalAI running F5 text to speech.", Dst: dst})).To(Succeed())
+
+			info, err := os.Stat(dst)
+			Expect(err).ToNot(HaveOccurred(), "synthesized WAV should exist at %q", dst)
+			Expect(info.Size()).To(BeNumerically(">", 1024),
+				"expected a non-trivial WAV, got %d bytes", info.Size())
+		})
 	})
 })

backend/go/face-detect/Makefile

@@ -1,6 +1,6 @@
 # face-detect backend Makefile.
 #
-# Upstream pin lives below as FACEDETECT_VERSION?=06914b0... (.github/bump_deps.sh
+# Upstream pin lives below as FACEDETECT_VERSION?=e22260d5d5490b37b021b7f795079f386d553afd
 # can find and update it - matches the voice-detect / parakeet.cpp / whisper.cpp
 # convention).
 #
@@ -14,7 +14,7 @@
 # The default target below does the proper clone-at-pin + cmake build so CI does
 # not need a side-checkout.
 
-FACEDETECT_VERSION?=06914b077d52f90d5421299138e7be6bdd06b5e8
+FACEDETECT_VERSION?=e22260d5d5490b37b021b7f795079f386d553afd
 FACEDETECT_REPO?=https://github.com/mudler/face-detect.cpp
 
 GOCMD?=go

backend/go/local-store/store.go

@@ -22,6 +22,7 @@ import (
 	"fmt"
 	"math"
 	"slices"
+	"strings"
 
 	"github.com/mudler/LocalAI/pkg/grpc/base"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
@@ -57,12 +58,18 @@ func NewStore() *Store {
 	}
 }
 
-// Load is a no-op — local-store has no on-disk artefact. opts.Model is
-// just a namespace identifier; isolation is already handled upstream
-// (ModelLoader spawns a fresh local-store process per (backend,
-// model) tuple, so each namespace is its own Store{} instance).
+// Load only validates the namespace — local-store has no on-disk
+// artefact. opts.Model is a namespace identifier, which core's
+// StoreBackend always sends with store.NamespacePrefix; anything else
+// is the model loader's greedy autoload probing with a real model name,
+// which must be refused or the LLM binds to the vector store. Isolation
+// is already handled upstream (ModelLoader spawns a fresh local-store
+// process per (backend, model) tuple, so each namespace is its own
+// Store{} instance).
 func (s *Store) Load(opts *pb.ModelOptions) error {
-	_ = opts
+	if !strings.HasPrefix(opts.GetModel(), store.NamespacePrefix) {
+		return fmt.Errorf("local-store: refusing to load %q: not a store namespace (expected %q prefix)", opts.GetModel(), store.NamespacePrefix)
+	}
 	return nil
 }
 

backend/go/local-store/store_test.go

@@ -12,6 +12,7 @@ import (
 	"testing"
 
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/store"
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
@@ -186,8 +187,18 @@ var _ = Describe("StoresFind", func() {
 })
 
 var _ = Describe("StoresLoad", func() {
-	It("is a no-op", func() {
-		Expect(NewStore().Load(&pb.ModelOptions{Model: "any-namespace"})).To(Succeed())
+	It("accepts prefixed store namespaces", func() {
+		Expect(NewStore().Load(&pb.ModelOptions{Model: store.NamespacePrefix + "any-namespace"})).To(Succeed())
+	})
+
+	It("accepts the prefix alone (default store)", func() {
+		Expect(NewStore().Load(&pb.ModelOptions{Model: store.NamespacePrefix})).To(Succeed())
+	})
+
+	It("refuses model names without the namespace prefix", func() {
+		err := NewStore().Load(&pb.ModelOptions{Model: "some-llm.gguf"})
+		Expect(err).To(MatchError(ContainSubstring("not a store namespace")))
+		Expect(NewStore().Load(&pb.ModelOptions{})).NotTo(Succeed())
 	})
 })
 

backend/go/locate-anything-cpp/Makefile

@@ -10,7 +10,7 @@ JOBS?=$(shell nproc --ignore=1)
 # this on `master` always picks up the latest C-API surface (incl. the
 # per-detection accessor functions used by golocateanythingcpp.go).
 LOCATEANYTHING_REPO?=https://github.com/mudler/locate-anything.cpp.git
-LOCATEANYTHING_VERSION?=92c1682da792c1e8a5dec91acc2be4b02c742ded
+LOCATEANYTHING_VERSION?=ade2634f7f79b56121125e5885628744795a478f
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/moss-transcribe-cpp/.gitignore

@@ -0,0 +1,12 @@
+.cache/
+sources/
+build/
+package/
+moss-transcribe-cpp-grpc
+# build artifacts staged in-tree by the Makefile (cp from sources/) or
+# symlinked for local dev; the real sources live in moss-transcribe.cpp upstream.
+*.so
+*.so.*
+*.dylib
+moss_transcribe_capi.h
+compile_commands.json

backend/go/moss-transcribe-cpp/Makefile

@@ -0,0 +1,98 @@
+# moss-transcribe-cpp backend Makefile.
+#
+# Upstream pin lives below as MOSS_VERSION?=92a923dca88a41a34e47a364d55ee25731a9a0a2
+# (.github/bump_deps.sh) can find and update it - matches the
+# whisper.cpp / parakeet-cpp / ds4 convention.
+#
+# Local dev shortcut: if you already have an out-of-tree moss-transcribe.cpp
+# build, you can symlink the .so + header into this directory and skip the
+# clone/cmake steps entirely, e.g.:
+#
+#   ln -sf /path/to/moss-transcribe.cpp/build-shared/libmoss-transcribe.so .
+#   ln -sf /path/to/moss-transcribe.cpp/include/moss_transcribe_capi.h .
+#   go build -o moss-transcribe-cpp-grpc .
+#
+# That's what the L0 smoke test uses. The default target below does the proper
+# clone-at-pin + cmake build so CI doesn't need a side-checkout.
+
+MOSS_VERSION?=92a923dca88a41a34e47a364d55ee25731a9a0a2
+MOSS_REPO?=https://github.com/mudler/moss-transcribe.cpp
+
+GOCMD?=go
+GO_TAGS?=
+JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
+
+BUILD_TYPE?=
+NATIVE?=false
+
+# Build ggml statically into libmoss-transcribe.so (PIC) so the shared lib is
+# self-contained: dlopen needs no libggml*.so alongside it, only system libs
+# (libstdc++/libgomp/libc) that the runtime image already provides. MT_SHARED
+# flips ggml to a static PIC build (see the moss-transcribe.cpp CMakeLists).
+CMAKE_ARGS?=-DCMAKE_BUILD_TYPE=Release -DMT_SHARED=ON -DMT_BUILD_CLI=OFF -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON
+
+ifeq ($(NATIVE),false)
+	CMAKE_ARGS+=-DGGML_NATIVE=OFF
+endif
+
+# moss-transcribe.cpp gates its GGML backends behind MT_GGML_* options and does
+# set(GGML_<be> ${MT_GGML_<be>} CACHE BOOL "" FORCE), so a bare -DGGML_CUDA=ON is
+# overwritten back to OFF and the build silently falls back to CPU. Forward the
+# MT_GGML_* options instead (openblas is not gated, so -DGGML_BLAS passes through).
+ifeq ($(BUILD_TYPE),cublas)
+	CMAKE_ARGS+=-DMT_GGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON
+else ifeq ($(BUILD_TYPE),openblas)
+	CMAKE_ARGS+=-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS
+else ifeq ($(BUILD_TYPE),hipblas)
+	CMAKE_ARGS+=-DMT_GGML_HIP=ON
+else ifeq ($(BUILD_TYPE),vulkan)
+	CMAKE_ARGS+=-DMT_GGML_VULKAN=ON
+else ifeq ($(BUILD_TYPE),metal)
+	CMAKE_ARGS+=-DMT_GGML_METAL=ON
+endif
+
+.PHONY: moss-transcribe-cpp-grpc package build clean purge test all
+
+all: moss-transcribe-cpp-grpc
+
+# Clone the upstream moss-transcribe.cpp source at the pinned commit. Directory
+# acts as the target so make only re-clones when missing. After a MOSS_VERSION
+# bump, run 'make purge && make' to refetch.
+sources/moss-transcribe.cpp:
+	mkdir -p sources/moss-transcribe.cpp
+	cd sources/moss-transcribe.cpp && \
+	git init -q && \
+	git remote add origin $(MOSS_REPO) && \
+	git fetch --depth 1 origin $(MOSS_VERSION) && \
+	git checkout FETCH_HEAD && \
+	git submodule update --init --recursive --depth 1 --single-branch
+
+# Build the shared lib + header out-of-tree, then stage them next to the Go
+# sources so purego.Dlopen("libmoss-transcribe.so") and the cgo-less build both
+# pick them up.
+libmoss-transcribe.so: sources/moss-transcribe.cpp
+	cmake -B sources/moss-transcribe.cpp/build-shared -S sources/moss-transcribe.cpp $(CMAKE_ARGS)
+	cmake --build sources/moss-transcribe.cpp/build-shared --config Release -j$(JOBS)
+	cp -fv sources/moss-transcribe.cpp/build-shared/libmoss-transcribe.so* ./ 2>/dev/null || true
+	cp -fv sources/moss-transcribe.cpp/build-shared/libmoss-transcribe.dylib ./ 2>/dev/null || true
+	cp -fv sources/moss-transcribe.cpp/include/moss_transcribe_capi.h ./
+
+moss-transcribe-cpp-grpc: libmoss-transcribe.so main.go gomosstranscribecpp.go segments.go
+	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o moss-transcribe-cpp-grpc .
+
+package: moss-transcribe-cpp-grpc
+	bash package.sh
+
+build: package
+
+# Test target. The model-backed smoke test is gated on
+# MOSS_BACKEND_TEST_MODEL + MOSS_BACKEND_TEST_WAV; without them that spec
+# auto-skips, leaving the pure-Go parser/unit tests.
+test:
+	LD_LIBRARY_PATH=$(CURDIR):$$LD_LIBRARY_PATH $(GOCMD) test ./... -count=1
+
+clean: purge
+	rm -rf libmoss-transcribe.so* moss_transcribe_capi.h package moss-transcribe-cpp-grpc
+
+purge:
+	rm -rf sources/moss-transcribe.cpp

backend/go/moss-transcribe-cpp/gomosstranscribecpp.go

@@ -0,0 +1,179 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/base"
+	"github.com/mudler/LocalAI/pkg/grpc/grpcerrors"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/LocalAI/pkg/utils"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// purego-bound entry points from libmoss-transcribe.so. Names match
+// moss_transcribe_capi.h exactly so a `nm libmoss-transcribe.so | grep
+// moss_transcribe_capi` is enough to spot drift.
+//
+// The transcribe_* functions return char* declared here as uintptr so we can
+// call moss_transcribe_capi_free_string on the same pointer after copying: the
+// C-API contract is "caller owns and must free the returned buffer".
+var (
+	CppAbiVersion     func() int32
+	CppLoad           func(ggufPath string) uintptr
+	CppFree           func(ctx uintptr)
+	CppTranscribePath func(ctx uintptr, wavPath string, maxNew int32) uintptr
+	CppTranscribePcm  func(ctx uintptr, samples []float32, nSamples int32, sampleRate int32, maxNew int32) uintptr
+	CppFreeString     func(s uintptr)
+	CppLastError      func(ctx uintptr) string
+)
+
+// MossTranscribeCpp owns a single loaded moss_transcribe_ctx. MOSS is an
+// offline transcription + diarization + timestamping engine: one model, one
+// context, no streaming. The C engine holds a single mutable context and is
+// not reentrant, so we embed base.SingleThread — LocalAI serialises every RPC
+// through the server-level lock, and only one transcription touches the engine
+// at a time.
+type MossTranscribeCpp struct {
+	base.SingleThread
+	ctxPtr uintptr
+	maxNew int32
+}
+
+// Load is the LocalAI gRPC entry point for LoadModel: it calls
+// moss_transcribe_capi_load with the GGUF path and stashes the resulting
+// opaque context pointer for AudioTranscription.
+func (m *MossTranscribeCpp) Load(opts *pb.ModelOptions) error {
+	if opts.ModelFile == "" {
+		return errors.New("moss-transcribe-cpp: ModelFile is required")
+	}
+
+	// max_new_tokens caps the generated tokens per transcription; <=0 uses the
+	// GGUF's default_max_new_tokens (the C-API's own default). Exposed as a
+	// model YAML option: (key:value form, like the sibling ggml backends).
+	m.maxNew = int32(optInt(opts, "max_new_tokens", 0))
+
+	ctx := CppLoad(opts.ModelFile)
+	if ctx == 0 {
+		// No ctx to ask for last_error (the C-API's last-error buffer lives on
+		// the ctx that was never returned). Surface the path so the operator at
+		// least knows which load failed.
+		return fmt.Errorf("moss-transcribe-cpp: moss_transcribe_capi_load failed for %q", opts.ModelFile)
+	}
+	m.ctxPtr = ctx
+	return nil
+}
+
+// optInt reads an integer model option (key:value form) from ModelOptions,
+// returning def when absent or unparseable. The options array carries the
+// model YAML's options: entries (see core/config; siblings such as parakeet-cpp
+// parse the same key:value form via strings.Cut on ":").
+func optInt(opts *pb.ModelOptions, key string, def int) int {
+	for _, o := range opts.GetOptions() {
+		k, v, ok := strings.Cut(o, ":")
+		if ok && strings.TrimSpace(k) == key {
+			if n, err := strconv.Atoi(strings.TrimSpace(v)); err == nil {
+				return n
+			}
+		}
+	}
+	return def
+}
+
+// AudioTranscription converts the audio at opts.Dst to a 16 kHz mono WAV and
+// hands the path to moss_transcribe_capi_transcribe_path. The model emits its
+// own speaker-labelled, time-aligned transcript in the compact
+// "[start][Sxx]text[end]..." format (seconds); we parse it into LocalAI
+// TranscriptSegments carrying int64-nanosecond timestamps and the per-segment
+// speaker label.
+//
+// MOSS does joint transcription + diarization + timestamps in one pass, so
+// translate/language/prompt/temperature/threads are not applicable and are
+// ignored. Streaming is not supported (offline model).
+func (m *MossTranscribeCpp) AudioTranscription(ctx context.Context, opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
+	if m.ctxPtr == 0 {
+		return pb.TranscriptResult{}, grpcerrors.ModelNotLoaded("moss-transcribe-cpp")
+	}
+	if opts.Dst == "" {
+		return pb.TranscriptResult{}, errors.New("moss-transcribe-cpp: TranscriptRequest.dst (audio path) is required")
+	}
+	if err := ctx.Err(); err != nil {
+		return pb.TranscriptResult{}, status.Error(codes.Canceled, "transcription cancelled")
+	}
+
+	// The C loader understands WAV; convert any input (MP3, etc.) to 16 kHz
+	// mono WAV first - the same normalisation every other audio backend
+	// (whisper, parakeet-cpp) does via utils.AudioToWav before handing the file
+	// to the engine.
+	converted, cleanup, err := convertToWavMono16k(opts.Dst)
+	if err != nil {
+		return pb.TranscriptResult{}, err
+	}
+	defer cleanup()
+
+	cstr := CppTranscribePath(m.ctxPtr, converted, m.maxNew)
+	if cstr == 0 {
+		return pb.TranscriptResult{}, fmt.Errorf("moss-transcribe-cpp: transcribe_path failed: %s", CppLastError(m.ctxPtr))
+	}
+	raw := goStringFromCPtr(cstr)
+	CppFreeString(cstr)
+
+	return transcriptResultFromRaw(raw), nil
+}
+
+// Free releases the underlying moss_transcribe_ctx. Called by LocalAI when the
+// model is unloaded.
+func (m *MossTranscribeCpp) Free() error {
+	if m.ctxPtr != 0 {
+		CppFree(m.ctxPtr)
+		m.ctxPtr = 0
+	}
+	return nil
+}
+
+// convertToWavMono16k converts an arbitrary audio file to a 16 kHz mono WAV in
+// a fresh temp dir and returns the path together with a cleanup func the caller
+// must defer. WAV inputs already at 16 kHz/mono/16-bit are passed through by
+// utils.AudioToWav (hardlink/copy), everything else is transcoded via ffmpeg.
+func convertToWavMono16k(path string) (string, func(), error) {
+	dir, err := os.MkdirTemp("", "moss-transcribe")
+	if err != nil {
+		return "", func() {}, err
+	}
+	cleanup := func() { _ = os.RemoveAll(dir) }
+
+	converted := filepath.Join(dir, "converted.wav")
+	if err := utils.AudioToWav(path, converted); err != nil {
+		cleanup()
+		return "", func() {}, err
+	}
+	return converted, cleanup, nil
+}
+
+// goStringFromCPtr copies a NUL-terminated C string into Go memory. cptr is the
+// raw pointer returned by purego from the C-API (a malloc'd buffer the caller
+// owns); callers must free it via CppFreeString after the copy lands.
+//
+// The uintptr->unsafe.Pointer conversion below trips go vet's unsafeptr check,
+// which can't distinguish a C-owned heap pointer from Go-managed memory. It is
+// safe here: the pointer addresses a malloc'd C buffer the Go GC neither tracks
+// nor moves, and we dereference it immediately to copy the bytes out (the same
+// pattern the whisper / parakeet-cpp backends use).
+func goStringFromCPtr(cptr uintptr) string {
+	if cptr == 0 {
+		return ""
+	}
+	p := unsafe.Pointer(cptr) //nolint:govet // C-owned malloc'd buffer, not Go-GC memory (see doc above)
+	n := 0
+	for *(*byte)(unsafe.Add(p, n)) != 0 {
+		n++
+	}
+	return string(unsafe.Slice((*byte)(p), n))
+}

backend/go/moss-transcribe-cpp/gomosstranscribecpp_test.go

@@ -0,0 +1,164 @@
+package main
+
+import (
+	"context"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"testing"
+
+	"github.com/ebitengine/purego"
+	"github.com/go-audio/audio"
+	"github.com/go-audio/wav"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestMossTranscribeCpp(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "moss-transcribe-cpp Backend Suite")
+}
+
+var (
+	libLoadOnce sync.Once
+	libLoadErr  error
+)
+
+// ensureLibLoaded mirrors main.go's bootstrap so a Go test can drive the C-API
+// bridge without spinning up the gRPC server. Skips the current spec when
+// libmoss-transcribe.so isn't loadable from cwd ($LD_LIBRARY_PATH or a symlink
+// in ./).
+func ensureLibLoaded() {
+	libLoadOnce.Do(func() {
+		libName := os.Getenv("MOSS_TRANSCRIBE_LIBRARY")
+		if libName == "" {
+			libName = "libmoss-transcribe.so"
+		}
+		lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+		if err != nil {
+			libLoadErr = err
+			return
+		}
+		purego.RegisterLibFunc(&CppAbiVersion, lib, "moss_transcribe_capi_abi_version")
+		purego.RegisterLibFunc(&CppLoad, lib, "moss_transcribe_capi_load")
+		purego.RegisterLibFunc(&CppFree, lib, "moss_transcribe_capi_free")
+		purego.RegisterLibFunc(&CppTranscribePath, lib, "moss_transcribe_capi_transcribe_path")
+		purego.RegisterLibFunc(&CppTranscribePcm, lib, "moss_transcribe_capi_transcribe_pcm")
+		purego.RegisterLibFunc(&CppFreeString, lib, "moss_transcribe_capi_free_string")
+		purego.RegisterLibFunc(&CppLastError, lib, "moss_transcribe_capi_last_error")
+	})
+	if libLoadErr != nil {
+		Skip("libmoss-transcribe.so not loadable: " + libLoadErr.Error())
+	}
+}
+
+// fixturesOrSkip returns the model + audio paths or skips the spec if either
+// env var is unset. The smoke test never runs in default CI; it needs a real
+// MOSS GGUF and a WAV on disk.
+func fixturesOrSkip() (string, string) {
+	modelPath := os.Getenv("MOSS_BACKEND_TEST_MODEL")
+	audioPath := os.Getenv("MOSS_BACKEND_TEST_WAV")
+	if modelPath == "" || audioPath == "" {
+		Skip("set MOSS_BACKEND_TEST_MODEL and MOSS_BACKEND_TEST_WAV to run this spec")
+	}
+	return modelPath, audioPath
+}
+
+// writeMono16kWav writes `samples` frames of 16 kHz mono 16-bit silence to
+// path. The result is already in AudioToWav's target format, so the conversion
+// helper copies it through without invoking ffmpeg.
+func writeMono16kWav(path string, samples int) {
+	GinkgoHelper()
+	f, err := os.Create(path)
+	Expect(err).ToNot(HaveOccurred())
+	enc := wav.NewEncoder(f, 16000, 16, 1, 1)
+	buf := &audio.IntBuffer{
+		Format:         &audio.Format{NumChannels: 1, SampleRate: 16000},
+		SourceBitDepth: 16,
+		Data:           make([]int, samples),
+	}
+	Expect(enc.Write(buf)).To(Succeed())
+	Expect(enc.Close()).To(Succeed())
+	Expect(f.Close()).To(Succeed())
+}
+
+var _ = Describe("MossTranscribeCpp", func() {
+	Context("ABI / load smoke (needs libmoss-transcribe.so)", func() {
+		It("reports a positive ABI version", func() {
+			ensureLibLoaded()
+			Expect(CppAbiVersion()).To(BeNumerically(">=", 1))
+		})
+
+		It("transcribes a WAV into speaker-labelled segments", func() {
+			modelPath, audioPath := fixturesOrSkip()
+			ensureLibLoaded()
+
+			m := &MossTranscribeCpp{}
+			Expect(m.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
+			defer func() { _ = m.Free() }()
+
+			res, err := m.AudioTranscription(context.Background(), &pb.TranscriptRequest{
+				Dst: audioPath,
+			})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(strings.TrimSpace(res.Text)).ToNot(BeEmpty(),
+				"expected non-empty transcript for %s", audioPath)
+			Expect(res.Segments).ToNot(BeEmpty(), "expected at least one segment")
+			var prevEnd int64
+			for i, seg := range res.Segments {
+				Expect(strings.TrimSpace(seg.Text)).ToNot(BeEmpty(),
+					"segment %d must have text", i)
+				Expect(seg.End).To(BeNumerically(">=", seg.Start),
+					"segment %d end must not precede its start", i)
+				Expect(seg.Start).To(BeNumerically(">=", prevEnd),
+					"segments must be in time order")
+				prevEnd = seg.End
+			}
+		})
+	})
+
+	Context("Load validation", func() {
+		It("rejects an empty ModelFile", func() {
+			m := &MossTranscribeCpp{}
+			Expect(m.Load(&pb.ModelOptions{})).To(HaveOccurred())
+		})
+	})
+
+	Context("AudioTranscription guards (no C library required)", func() {
+		It("returns ModelNotLoaded when no context is loaded", func() {
+			m := &MossTranscribeCpp{}
+			_, err := m.AudioTranscription(context.Background(), &pb.TranscriptRequest{Dst: "x.wav"})
+			Expect(err).To(HaveOccurred())
+		})
+
+		It("errors when the audio path is empty", func() {
+			m := &MossTranscribeCpp{ctxPtr: 1}
+			_, err := m.AudioTranscription(context.Background(), &pb.TranscriptRequest{})
+			Expect(err).To(HaveOccurred())
+		})
+	})
+
+	Context("convertToWavMono16k", func() {
+		It("returns a decodable 16kHz mono WAV copy and cleans it up", func() {
+			dir := GinkgoT().TempDir()
+			src := filepath.Join(dir, "input.wav")
+			writeMono16kWav(src, 16000) // 1s of silence at 16 kHz
+
+			converted, cleanup, err := convertToWavMono16k(src)
+			Expect(err).ToNot(HaveOccurred())
+
+			Expect(converted).ToNot(Equal(src))
+			Expect(converted).To(BeAnExistingFile())
+
+			cleanup()
+			Expect(converted).ToNot(BeAnExistingFile(), "cleanup removes the temp dir")
+		})
+
+		It("errors on a non-existent input rather than passing the path through", func() {
+			_, _, err := convertToWavMono16k(filepath.Join(GinkgoT().TempDir(), "missing.mp3"))
+			Expect(err).To(HaveOccurred())
+		})
+	})
+})

backend/go/moss-transcribe-cpp/main.go

@@ -0,0 +1,70 @@
+package main
+
+// Started internally by LocalAI - one gRPC server per loaded model.
+//
+// Loads the moss-transcribe shared library via purego and registers the flat
+// C-API entry points declared in moss_transcribe_capi.h. The library name can
+// be overridden with MOSS_TRANSCRIBE_LIBRARY (mirrors the WHISPER_LIBRARY /
+// PARAKEET_LIBRARY convention in the sibling backends); the default looks next
+// to this binary for libmoss-transcribe.so on Linux and
+// libmoss-transcribe.dylib on macOS.
+import (
+	"flag"
+	"fmt"
+	"os"
+	"runtime"
+
+	"github.com/ebitengine/purego"
+	grpc "github.com/mudler/LocalAI/pkg/grpc"
+)
+
+var (
+	addr = flag.String("addr", "localhost:50051", "the address to connect to")
+)
+
+type LibFuncs struct {
+	FuncPtr any
+	Name    string
+}
+
+func main() {
+	libName := os.Getenv("MOSS_TRANSCRIBE_LIBRARY")
+	if libName == "" {
+		if runtime.GOOS == "darwin" {
+			libName = "libmoss-transcribe.dylib"
+		} else {
+			libName = "libmoss-transcribe.so"
+		}
+	}
+
+	lib, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+	if err != nil {
+		panic(fmt.Errorf("moss-transcribe-cpp: dlopen %q: %w", libName, err))
+	}
+
+	// Bound 1:1 to moss_transcribe_capi.h. The transcribe_* entry points return
+	// a malloc'd char* the caller owns; we register those as uintptr so we get
+	// the raw pointer back and can call moss_transcribe_capi_free_string on it
+	// (purego's string return would copy and forget the original pointer,
+	// leaking it on every call).
+	libFuncs := []LibFuncs{
+		{&CppAbiVersion, "moss_transcribe_capi_abi_version"},
+		{&CppLoad, "moss_transcribe_capi_load"},
+		{&CppFree, "moss_transcribe_capi_free"},
+		{&CppTranscribePath, "moss_transcribe_capi_transcribe_path"},
+		{&CppTranscribePcm, "moss_transcribe_capi_transcribe_pcm"},
+		{&CppFreeString, "moss_transcribe_capi_free_string"},
+		{&CppLastError, "moss_transcribe_capi_last_error"},
+	}
+	for _, lf := range libFuncs {
+		purego.RegisterLibFunc(lf.FuncPtr, lib, lf.Name)
+	}
+
+	fmt.Fprintf(os.Stderr, "[moss-transcribe-cpp] ABI=%d\n", CppAbiVersion())
+
+	flag.Parse()
+
+	if err := grpc.StartServer(*addr, &MossTranscribeCpp{}); err != nil {
+		panic(err)
+	}
+}

backend/go/moss-transcribe-cpp/package.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+#
+# Bundle the moss-transcribe-cpp-grpc binary, libmoss-transcribe.so, the core
+# runtime libs (libc/libstdc++/libgomp + ld.so) and the GPU runtime for the
+# active BUILD_TYPE so the package is self-contained. Mirrors
+# backend/go/parakeet-cpp/package.sh; run.sh routes the (CGO_ENABLED=0) binary
+# through lib/ld.so so the packaged libc is used instead of the host's.
+
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+REPO_ROOT="${CURDIR}/../../.."
+
+mkdir -p "$CURDIR/package/lib"
+
+cp -avf "$CURDIR/moss-transcribe-cpp-grpc" "$CURDIR/package/"
+cp -avf "$CURDIR/run.sh" "$CURDIR/package/"
+
+# libmoss-transcribe shared lib + any soname symlinks. On Linux this is
+# libmoss-transcribe.so[.X.Y]; on macOS it is libmoss-transcribe.dylib.
+# purego.Dlopen resolves it via the *_LIBRARY_PATH that run.sh points at lib/.
+cp -avf "$CURDIR"/libmoss-transcribe.so* "$CURDIR/package/lib/" 2>/dev/null || true
+cp -avf "$CURDIR"/libmoss-transcribe.dylib "$CURDIR/package/lib/" 2>/dev/null || true
+if ! ls "$CURDIR"/package/lib/libmoss-transcribe.* >/dev/null 2>&1; then
+	echo "ERROR: libmoss-transcribe shared library not found in $CURDIR, run 'make' first" >&2
+	exit 1
+fi
+
+# Detect architecture and copy the core runtime libs libmoss-transcribe.so links
+# against, plus the matching dynamic loader as lib/ld.so.
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so"
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6"
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1"
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6"
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6"
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1"
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2"
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1"
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0"
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so"
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 "$CURDIR/package/lib/libc.so.6"
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 "$CURDIR/package/lib/libgcc_s.so.1"
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 "$CURDIR/package/lib/libstdc++.so.6"
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 "$CURDIR/package/lib/libm.so.6"
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 "$CURDIR/package/lib/libgomp.so.1"
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 "$CURDIR/package/lib/libdl.so.2"
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 "$CURDIR/package/lib/librt.so.1"
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 "$CURDIR/package/lib/libpthread.so.0"
+elif [ "$(uname -s)" = "Darwin" ]; then
+    echo "Detected Darwin — system frameworks linked dynamically, no bundled libs needed"
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries (CUDA/ROCm/Intel/Vulkan loader + ICDs + drivers) based
+# on BUILD_TYPE so the backend can reach the GPU without the runtime base image
+# shipping those drivers.
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah "$CURDIR/package/" "$CURDIR/package/lib/"

backend/go/moss-transcribe-cpp/run.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+set -e
+
+CURDIR=$(dirname "$(realpath "$0")")
+
+if [ "$(uname)" = "Darwin" ]; then
+	export DYLD_LIBRARY_PATH="$CURDIR/lib:"$CURDIR":${DYLD_LIBRARY_PATH:-}"
+	export MOSS_TRANSCRIBE_LIBRARY="$CURDIR/lib/libmoss-transcribe.dylib"
+else
+	export LD_LIBRARY_PATH="$CURDIR/lib:"$CURDIR":${LD_LIBRARY_PATH:-}"
+	export MOSS_TRANSCRIBE_LIBRARY="$CURDIR/lib/libmoss-transcribe.so"
+fi
+
+# If a self-contained ld.so was packaged, route through it so the packaged libc
+# / libstdc++ are used instead of the host's (matches the whisper / parakeet-cpp
+# backends' runtime layout). Linux only.
+if [ -f "$CURDIR/lib/ld.so" ]; then
+	echo "Using lib/ld.so"
+	exec "$CURDIR/lib/ld.so" "$CURDIR/moss-transcribe-cpp-grpc" "$@"
+fi
+
+exec "$CURDIR/moss-transcribe-cpp-grpc" "$@"

backend/go/moss-transcribe-cpp/segments.go

@@ -0,0 +1,156 @@
+package main
+
+import (
+	"regexp"
+	"strconv"
+	"strings"
+
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+)
+
+// mossSegment is one parsed unit of the MOSS transcript: a speaker-labelled
+// span with fractional-second start/end times, straight from the model's own
+// output. Timestamps stay in seconds here; secondsToNanos converts them at the
+// TranscriptSegment boundary.
+type mossSegment struct {
+	Start   float64
+	End     float64
+	Speaker string
+	Text    string
+}
+
+// bracketRe matches one "[...]" token (timestamp or speaker tag). The MOSS
+// transcript is a concatenation of "[start][Sxx]text[end]" segments, e.g.
+//
+//	[0.28][S01] And so, my fellow Americans,[7.71][8.12][S02] ask ...[10.59]
+//
+// so the bracketed tokens carry all the structure and the free text lives
+// between a speaker tag and the following (end) timestamp.
+var bracketRe = regexp.MustCompile(`\[([^\]]*)\]`)
+
+// speakerRe matches a speaker tag: an 'S' (any case) followed by one or more
+// digits, e.g. "S01", "S12". Anything else in brackets that isn't a speaker tag
+// is treated as a timestamp candidate.
+var speakerRe = regexp.MustCompile(`^[Ss][0-9]+$`)
+
+// bracketToken is a "[...]" token plus the free text that follows it up to the
+// next token (or end of string). For a speaker tag that trailing text is the
+// segment's transcript.
+type bracketToken struct {
+	content   string
+	textAfter string
+}
+
+// parseTranscript parses the compact "[start][Sxx]text[end]..." MOSS transcript
+// into structured segments. It walks the bracket tokens looking for a
+// start-timestamp, a speaker tag, and an end-timestamp, taking the text that
+// follows the speaker tag as the segment transcript. Tokens that don't fit the
+// grammar are skipped rather than aborting the parse, so a slightly malformed
+// stream still yields the segments it can.
+func parseTranscript(raw string) []mossSegment {
+	locs := bracketRe.FindAllStringSubmatchIndex(raw, -1)
+	toks := make([]bracketToken, len(locs))
+	for i, m := range locs {
+		// m[2]:m[3] is the capture group (the content inside the brackets);
+		// m[1] is the byte just past the closing ']'.
+		nextStart := len(raw)
+		if i+1 < len(locs) {
+			nextStart = locs[i+1][0]
+		}
+		toks[i] = bracketToken{
+			content:   raw[m[2]:m[3]],
+			textAfter: raw[m[1]:nextStart],
+		}
+	}
+
+	var segs []mossSegment
+	i := 0
+	for i < len(toks) {
+		start, ok := parseTimestamp(toks[i].content)
+		if !ok {
+			i++
+			continue
+		}
+		// A start timestamp must be followed by a speaker tag; otherwise this
+		// isn't a segment head, so skip it.
+		if i+1 >= len(toks) || !isSpeakerTag(toks[i+1].content) {
+			i++
+			continue
+		}
+		speaker := strings.ToUpper(strings.TrimSpace(toks[i+1].content))
+		text := strings.TrimSpace(toks[i+1].textAfter)
+
+		// The end timestamp is the next token if present and numeric; a segment
+		// that runs to the end of the stream without a closing timestamp falls
+		// back to start==end.
+		end := start
+		consumed := 2
+		if i+2 < len(toks) {
+			if e, ok := parseTimestamp(toks[i+2].content); ok {
+				end = e
+				consumed = 3
+			}
+		}
+
+		segs = append(segs, mossSegment{Start: start, End: end, Speaker: speaker, Text: text})
+		i += consumed
+	}
+	return segs
+}
+
+// isSpeakerTag reports whether the bracket content is a MOSS speaker tag ("Sxx").
+func isSpeakerTag(content string) bool {
+	return speakerRe.MatchString(strings.TrimSpace(content))
+}
+
+// parseTimestamp parses a bracket content as a fractional-second timestamp. A
+// speaker tag ("S01") never parses as a float, so this cleanly distinguishes
+// the two token kinds.
+func parseTimestamp(content string) (float64, bool) {
+	f, err := strconv.ParseFloat(strings.TrimSpace(content), 64)
+	if err != nil {
+		return 0, false
+	}
+	return f, true
+}
+
+// secondsToNanos converts the transcript's fractional-second timestamps into
+// the int64 nanoseconds LocalAI carries on TranscriptSegment, the same
+// nanosecond convention the whisper / parakeet-cpp backends use.
+func secondsToNanos(sec float64) int64 {
+	return int64(sec * 1e9)
+}
+
+// transcriptResultFromRaw parses the raw MOSS transcript and shapes it into a
+// TranscriptResult. Each parsed segment becomes a TranscriptSegment with
+// nanosecond start/end and the model's own speaker label; Text is the segments
+// joined with single spaces. When nothing parses (no bracket structure) it
+// falls back to a single whole-clip text segment so callers always get a
+// transcript.
+func transcriptResultFromRaw(raw string) pb.TranscriptResult {
+	segs := parseTranscript(raw)
+	if len(segs) == 0 {
+		text := strings.TrimSpace(raw)
+		return pb.TranscriptResult{
+			Text:     text,
+			Segments: []*pb.TranscriptSegment{{Id: 0, Text: text}},
+		}
+	}
+
+	var full strings.Builder
+	pbSegs := make([]*pb.TranscriptSegment, 0, len(segs))
+	for id, s := range segs {
+		if id > 0 && s.Text != "" {
+			full.WriteString(" ")
+		}
+		full.WriteString(s.Text)
+		pbSegs = append(pbSegs, &pb.TranscriptSegment{
+			Id:      int32(id),
+			Start:   secondsToNanos(s.Start),
+			End:     secondsToNanos(s.End),
+			Text:    s.Text,
+			Speaker: s.Speaker,
+		})
+	}
+	return pb.TranscriptResult{Text: strings.TrimSpace(full.String()), Segments: pbSegs}
+}

backend/go/moss-transcribe-cpp/segments_test.go

@@ -0,0 +1,104 @@
+package main
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("parseTranscript ([start][Sxx]text[end] grammar)", func() {
+	It("parses the README two-speaker example into ordered segments", func() {
+		raw := "[0.28][S01] And so, my fellow Americans, ask not what your country can do for you,[7.71][8.12][S02] ask what you can do for your country.[10.59]"
+		segs := parseTranscript(raw)
+		Expect(segs).To(HaveLen(2))
+
+		Expect(segs[0].Speaker).To(Equal("S01"))
+		Expect(segs[0].Start).To(BeNumerically("~", 0.28, 1e-9))
+		Expect(segs[0].End).To(BeNumerically("~", 7.71, 1e-9))
+		Expect(segs[0].Text).To(Equal("And so, my fellow Americans, ask not what your country can do for you,"))
+
+		Expect(segs[1].Speaker).To(Equal("S02"))
+		Expect(segs[1].Start).To(BeNumerically("~", 8.12, 1e-9))
+		Expect(segs[1].End).To(BeNumerically("~", 10.59, 1e-9))
+		Expect(segs[1].Text).To(Equal("ask what you can do for your country."))
+	})
+
+	It("parses a single segment", func() {
+		segs := parseTranscript("[1.50][S01] hello world[3.25]")
+		Expect(segs).To(HaveLen(1))
+		Expect(segs[0].Start).To(BeNumerically("~", 1.50, 1e-9))
+		Expect(segs[0].End).To(BeNumerically("~", 3.25, 1e-9))
+		Expect(segs[0].Speaker).To(Equal("S01"))
+		Expect(segs[0].Text).To(Equal("hello world"))
+	})
+
+	It("uppercases a lowercase speaker tag", func() {
+		segs := parseTranscript("[0.00][s03] hi[0.90]")
+		Expect(segs).To(HaveLen(1))
+		Expect(segs[0].Speaker).To(Equal("S03"))
+	})
+
+	It("falls back to start==end when the closing timestamp is missing", func() {
+		segs := parseTranscript("[2.00][S01] trailing words with no end")
+		Expect(segs).To(HaveLen(1))
+		Expect(segs[0].Start).To(BeNumerically("~", 2.00, 1e-9))
+		Expect(segs[0].End).To(Equal(segs[0].Start))
+		Expect(segs[0].Text).To(Equal("trailing words with no end"))
+	})
+
+	It("handles multi-digit speaker ids and higher speaker counts", func() {
+		segs := parseTranscript("[0.0][S01] a[1.0][1.0][S12] b[2.0]")
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[0].Speaker).To(Equal("S01"))
+		Expect(segs[1].Speaker).To(Equal("S12"))
+	})
+
+	It("returns no segments for text without bracket structure", func() {
+		Expect(parseTranscript("just some plain text")).To(BeEmpty())
+		Expect(parseTranscript("")).To(BeEmpty())
+	})
+
+	It("skips a start timestamp not followed by a speaker tag", func() {
+		// A stray timestamp pair without a speaker tag is not a segment head.
+		segs := parseTranscript("[1.0][2.0][S01] real[3.0]")
+		Expect(segs).To(HaveLen(1))
+		Expect(segs[0].Speaker).To(Equal("S01"))
+		Expect(segs[0].Text).To(Equal("real"))
+	})
+})
+
+var _ = Describe("transcriptResultFromRaw", func() {
+	It("shapes parsed segments into a TranscriptResult with nanosecond times and speakers", func() {
+		raw := "[0.28][S01] first part,[7.71][8.12][S02] second part.[10.59]"
+		res := transcriptResultFromRaw(raw)
+
+		Expect(res.Segments).To(HaveLen(2))
+		Expect(res.Segments[0].Id).To(Equal(int32(0)))
+		Expect(res.Segments[0].Start).To(Equal(secondsToNanos(0.28)))
+		Expect(res.Segments[0].End).To(Equal(secondsToNanos(7.71)))
+		Expect(res.Segments[0].Speaker).To(Equal("S01"))
+		Expect(res.Segments[0].Text).To(Equal("first part,"))
+
+		Expect(res.Segments[1].Id).To(Equal(int32(1)))
+		Expect(res.Segments[1].Speaker).To(Equal("S02"))
+		Expect(res.Segments[1].Start).To(Equal(secondsToNanos(8.12)))
+
+		// Full text is the segments joined with single spaces.
+		Expect(res.Text).To(Equal("first part, second part."))
+	})
+
+	It("falls back to a single whole-clip text segment when nothing parses", func() {
+		res := transcriptResultFromRaw("plain transcript with no markers")
+		Expect(res.Segments).To(HaveLen(1))
+		Expect(res.Segments[0].Text).To(Equal("plain transcript with no markers"))
+		Expect(res.Segments[0].Start).To(Equal(int64(0)))
+		Expect(res.Text).To(Equal("plain transcript with no markers"))
+	})
+})
+
+var _ = Describe("secondsToNanos", func() {
+	It("converts fractional seconds to int64 nanoseconds", func() {
+		Expect(secondsToNanos(0)).To(Equal(int64(0)))
+		Expect(secondsToNanos(1)).To(Equal(int64(1e9)))
+		Expect(secondsToNanos(0.28)).To(Equal(int64(0.28 * 1e9)))
+	})
+})

backend/go/omnivoice-cpp/CMakeLists.txt

@@ -30,7 +30,7 @@ target_include_directories(gomnivoicecpp PRIVATE ${OMNIVOICE_DIR}/src)
 target_include_directories(gomnivoicecpp SYSTEM PRIVATE ${OMNIVOICE_DIR}/ggml/include)
 
 # Link GPU backends if the upstream ggml created them.
-foreach(backend blas cuda metal vulkan sycl)
+foreach(backend blas cuda hip metal vulkan sycl)
     if(TARGET ggml-${backend})
         target_link_libraries(gomnivoicecpp PRIVATE ggml-${backend})
         if(backend STREQUAL "cuda")

backend/go/omnivoice-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # omnivoice.cpp version
 OMNIVOICE_REPO?=https://github.com/ServeurpersoCom/omnivoice.cpp
-OMNIVOICE_VERSION?=0f37401bebe9b20c0160a888e592108fc1d17607
+OMNIVOICE_VERSION?=bbdf4be03aa5bc3c188b5db778b6f3fd63ceff6c
 SO_TARGET?=libgomnivoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -24,7 +24,14 @@ else ifeq ($(BUILD_TYPE),openblas)
 else ifeq ($(BUILD_TYPE),clblas)
 	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
 else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=ON
 else ifeq ($(OS),Darwin)

backend/go/opus/opus.go

@@ -34,7 +34,15 @@ type Opus struct {
 	decoders   map[string]*cachedDecoder
 }
 
+// Load accepts only the codec's own name (what the realtime WebRTC path
+// sends) or no name at all — there is no model artefact here, so without
+// this check the model loader's greedy autoload (which probes every
+// installed backend with real model names) would happily bind an LLM to
+// the audio codec.
 func (o *Opus) Load(opts *pb.ModelOptions) error {
+	if m := opts.GetModel(); m != "" && m != "opus" {
+		return fmt.Errorf("opus: refusing to load %q: opus is an audio codec, not a model backend", m)
+	}
 	o.decoders = make(map[string]*cachedDecoder)
 	go o.evictLoop()
 	return Init()

backend/go/opus/opus_test.go

@@ -304,6 +304,17 @@ func computeTHD(samples []int16, fundamentalHz float64, sampleRate, numHarmonics
 
 // --- Opus specs ---
 
+var _ = Describe("Opus Load", func() {
+	It("accepts its own name (the realtime WebRTC path)", func() {
+		Expect((&Opus{}).Load(&pb.ModelOptions{Model: "opus"})).To(Succeed())
+	})
+
+	It("refuses foreign model names from the greedy autoload", func() {
+		err := (&Opus{}).Load(&pb.ModelOptions{Model: "some-llm.gguf"})
+		Expect(err).To(MatchError(ContainSubstring("audio codec")))
+	})
+})
+
 var _ = Describe("Opus", func() {
 	var o *Opus
 

backend/go/parakeet-cpp/Makefile

@@ -1,6 +1,6 @@
 # parakeet-cpp backend Makefile.
 #
-# Upstream pin lives below as PARAKEET_VERSION?=f469a57270a1cc4554acb15febf60e56619673b9
+# Upstream pin lives below as PARAKEET_VERSION?=e8acc6172a94e20a952cf1843decace5d771a94b
 # (.github/bump_deps.sh) can find and update it - matches the
 # whisper.cpp / ds4 / vibevoice-cpp convention.
 #
@@ -15,7 +15,7 @@
 # That's what the L0 smoke test uses. The default target below does the
 # proper clone-at-pin + cmake build so CI doesn't need a side-checkout.
 
-PARAKEET_VERSION?=f469a57270a1cc4554acb15febf60e56619673b9
+PARAKEET_VERSION?=e8acc6172a94e20a952cf1843decace5d771a94b
 PARAKEET_REPO?=https://github.com/mudler/parakeet.cpp
 
 GOCMD?=go

backend/go/parakeet-cpp/boundary.go

@@ -0,0 +1,81 @@
+package main
+
+// utteranceBoundary is the single definition of a small state machine that was
+// previously open-coded three times — as a bare `finalEou` bool with an ad-hoc
+// toggle — in the live feed (live.go), the file-stream text path, and the
+// file-stream JSON path (goparakeetcpp.go).
+//
+// It answers one running question: does the decode currently rest on an
+// end-of-utterance boundary? That is the value a closing FinalResult reports as
+// .Eou and the realtime turn detector treats as a commit point.
+//
+// parakeet auto-resets its decoder after every <EOU>/<EOB>, so one streaming
+// session is a sequence of utterances and this is a LATCH, not a monotonic
+// flag: it closes on an <EOU> and reopens as soon as the next utterance starts.
+// (Contrast the realtime API's per-turn `eouSeen`, which only ever goes
+// false->true because each turn gets a fresh stream. Here the stream outlives
+// the turn, so the boundary status must be able to reopen.)
+//
+// The only transitions, over the events one streamFeedResult carries — an
+// <EOU>, an <EOB> (backchannel), or plain speech output (text and/or words):
+//
+//	            <EOU>
+//	   open ───────────► closed
+//	    ▲ ▲ │             │ │
+//	    │ └─┘ <EOB>|speech │ │ <EOU>
+//	    │   (stay open)    │ └─┘ (stay closed)
+//	    └──────────────────┘
+//	         <EOB>|speech
+//
+//	open   = NOT on an utterance boundary: mid-utterance, the last boundary was
+//	         a backchannel <EOB>, or the stream just began (the initial state).
+//	closed = the last meaningful event was an <EOU> with no later speech: a real
+//	         turn boundary.
+//
+// A feed that carries nothing (no eou/eob/text/words — e.g. a finalize flush
+// that produced no tail) is a no-op and leaves the state unchanged, matching
+// the legacy "leave finalEou as it was" behaviour.
+//
+// The state carries no data, so it is modelled as a two-valued type (a named
+// bool) rather than an int enum: every inhabitant is legal, so illegal states
+// are unrepresentable — the payload-free analog of the sealed sum types the
+// realtime machines use (those need interfaces because their states carry data,
+// e.g. Active{ID}, where "Active with no ID" is the illegal combination a scalar
+// cannot even express).
+type utteranceBoundary bool
+
+const (
+	// boundaryOpen is the zero value (false), so a fresh decode starts open —
+	// exactly the legacy `var finalEou bool` (false) initial condition.
+	boundaryOpen   utteranceBoundary = false
+	boundaryClosed utteranceBoundary = true
+)
+
+// observe folds one decode increment into the latch and returns the new state.
+//
+// <EOU> takes priority when a single feed carries both an <EOU> and speech
+// (e.g. {"text":"hello","eou":1}): the utterance both produced that text AND
+// ended, so the decode rests on the boundary. This matches the legacy
+// eou-checked-first ordering at every call site.
+func (b utteranceBoundary) observe(r streamFeedResult) utteranceBoundary {
+	switch {
+	case r.Eou:
+		return boundaryClosed
+	case r.Eob || r.Delta != "" || len(r.Words) > 0:
+		return boundaryOpen
+	default:
+		return b
+	}
+}
+
+// ended reports whether the decode currently rests on an end-of-utterance
+// boundary (a real <EOU>, not a backchannel <EOB>). This is what a closing
+// FinalResult carries as .Eou.
+func (b utteranceBoundary) ended() bool { return b == boundaryClosed }
+
+func (b utteranceBoundary) String() string {
+	if b == boundaryClosed {
+		return "closed"
+	}
+	return "open"
+}

backend/go/parakeet-cpp/boundary_test.go

@@ -0,0 +1,92 @@
+package main
+
+import (
+	"math/rand/v2"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("utteranceBoundary (decode end-of-utterance latch)", func() {
+	It("starts open: a fresh decode is not on a boundary", func() {
+		var b utteranceBoundary
+		Expect(b).To(Equal(boundaryOpen))
+		Expect(b.ended()).To(BeFalse())
+	})
+
+	DescribeTable("single feed transition from the open state",
+		func(r streamFeedResult, wantEnded bool) {
+			Expect(boundaryOpen.observe(r).ended()).To(Equal(wantEnded))
+		},
+		Entry("<EOU> closes it", streamFeedResult{Eou: true}, true),
+		Entry("<EOU> with text closes it (eou wins)", streamFeedResult{Delta: "hi", Eou: true}, true),
+		Entry("<EOB> stays open (backchannel is not a turn boundary)", streamFeedResult{Eob: true}, false),
+		Entry("plain text stays open", streamFeedResult{Delta: "hello"}, false),
+		Entry("words-only stays open", streamFeedResult{Words: []transcriptWord{{W: "x"}}}, false),
+		Entry("empty feed is a no-op (stays open)", streamFeedResult{}, false),
+	)
+
+	DescribeTable("single feed transition from the closed state",
+		func(r streamFeedResult, wantEnded bool) {
+			Expect(boundaryClosed.observe(r).ended()).To(Equal(wantEnded))
+		},
+		Entry("another <EOU> stays closed", streamFeedResult{Eou: true}, true),
+		Entry("trailing speech reopens it", streamFeedResult{Delta: "and more"}, false),
+		Entry("words reopen it", streamFeedResult{Words: []transcriptWord{{W: "x"}}}, false),
+		Entry("a backchannel <EOB> reopens it", streamFeedResult{Eob: true}, false),
+		Entry("empty feed is a no-op (stays closed)", streamFeedResult{}, true),
+	)
+
+	It("is a latch: <EOU> then trailing speech reopens, then <EOU> closes again", func() {
+		b := boundaryOpen
+		b = b.observe(streamFeedResult{Delta: "turn one", Eou: true})
+		Expect(b.ended()).To(BeTrue())
+		b = b.observe(streamFeedResult{Delta: " and more"})
+		Expect(b.ended()).To(BeFalse(), "trailing speech without an EOU is an open utterance")
+		b = b.observe(streamFeedResult{Eou: true})
+		Expect(b.ended()).To(BeTrue())
+	})
+
+	It("treats a backchannel before a real EOU correctly", func() {
+		b := boundaryOpen
+		b = b.observe(streamFeedResult{Delta: "uh huh", Eob: true})
+		Expect(b.ended()).To(BeFalse(), "a backchannel must not masquerade as a turn boundary")
+		b = b.observe(streamFeedResult{Delta: "done", Eou: true})
+		Expect(b.ended()).To(BeTrue())
+	})
+
+	It("matches the reference fold over seeded random feed sequences", func() {
+		// The invariant: after any sequence of feeds, ended() is true iff the
+		// last feed that carried ANY event was an <EOU>. <EOU> takes priority
+		// when a feed carries both an EOU and speech; empty feeds are ignored.
+		for seed := uint64(1); seed <= 200; seed++ {
+			rng := rand.New(rand.NewPCG(seed, seed*2654435761))
+			b := boundaryOpen
+			lastWasEou := false // reference: did the last meaningful feed end on EOU?
+			steps := rng.IntN(30)
+			for i := 0; i < steps; i++ {
+				var r streamFeedResult
+				switch rng.IntN(5) {
+				case 0:
+					r = streamFeedResult{Eou: true}
+				case 1:
+					r = streamFeedResult{Eob: true}
+				case 2:
+					r = streamFeedResult{Delta: "w"}
+				case 3:
+					r = streamFeedResult{Delta: "w", Eou: true} // eou + speech, eou wins
+				case 4:
+					r = streamFeedResult{} // empty: no-op
+				}
+				b = b.observe(r)
+				if r.Eou {
+					lastWasEou = true
+				} else if r.Eob || r.Delta != "" || len(r.Words) > 0 {
+					lastWasEou = false
+				}
+			}
+			Expect(b.ended()).To(Equal(lastWasEou),
+				"seed %d: latch disagreed with the reference fold", seed)
+		}
+	})
+})

backend/go/parakeet-cpp/driver.go

@@ -0,0 +1,82 @@
+package main
+
+import (
+	"context"
+
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// streamFeedResult is one decode increment from a cache-aware streaming session:
+// the newly-finalized text plus the model's own per-feed boundary tokens
+// (<EOU>/<EOB>) and word timings. It is the single event type both the live
+// (bidi) and file (server-stream) paths fold over, hiding the ABI v4 JSON vs
+// older text-only entry-point split behind one shape.
+type streamFeedResult struct {
+	Delta string
+	Eou   bool
+	Eob   bool
+	Words []transcriptWord
+}
+
+// feedChunk feeds one PCM chunk to the streaming session (or finalizes it, when
+// finalize is true) and returns the unified decode increment. It prefers the
+// ABI v4 JSON entry points (which also carry per-word timestamps) and falls
+// back to the older text-only entry points against an older libparakeet.so.
+//
+// This is the one place the JSON-vs-text choice is made; every consumer works
+// in terms of streamFeedResult.
+func (p *ParakeetCpp) feedChunk(stream uintptr, pcm []float32, finalize bool) (streamFeedResult, error) {
+	if CppStreamFeedJSON != nil {
+		doc, err := p.streamFeedDoc(stream, pcm, finalize)
+		if err != nil {
+			return streamFeedResult{}, err
+		}
+		return streamFeedResult{Delta: doc.Text, Eou: doc.Eou != 0, Eob: doc.Eob != 0, Words: doc.Words}, nil
+	}
+	delta, eou, eob, err := p.streamFeedText(stream, pcm, finalize)
+	if err != nil {
+		return streamFeedResult{}, err
+	}
+	return streamFeedResult{Delta: delta, Eou: eou, Eob: eob}, nil
+}
+
+// feedSlices feeds pcm through the session in streamChunkSamples slices,
+// invoking onFeed for each decode increment. It does NOT finalize: callers
+// decide when the send side is done. The file path finalizes after the whole
+// file; the live path finalizes only when its request channel closes, never
+// between audio messages. Slicing keeps each per-call engineMu hold short so
+// concurrent unary transcription interleaves fairly (the C session buffers
+// internally).
+//
+// If ctx is non-nil it is checked before each slice so a cancelled file
+// transcription stops promptly; the live path passes nil (it is bounded by its
+// request channel instead of a ctx).
+func (p *ParakeetCpp) feedSlices(ctx context.Context, stream uintptr, pcm []float32, onFeed func(streamFeedResult) error) error {
+	for off := 0; off < len(pcm); off += streamChunkSamples {
+		if ctx != nil {
+			if err := ctx.Err(); err != nil {
+				return status.Error(codes.Canceled, "transcription cancelled")
+			}
+		}
+		end := min(off+streamChunkSamples, len(pcm))
+		res, err := p.feedChunk(stream, pcm[off:end], false)
+		if err != nil {
+			return err
+		}
+		if err := onFeed(res); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// flushTail finalizes the session once and folds the flushed tail (the last
+// ~2 encoder frames of text, which only appear on finalize) through onFeed.
+func (p *ParakeetCpp) flushTail(stream uintptr, onFeed func(streamFeedResult) error) error {
+	res, err := p.feedChunk(stream, nil, true)
+	if err != nil {
+		return err
+	}
+	return onFeed(res)
+}

backend/go/parakeet-cpp/goparakeetcpp.go

@@ -103,12 +103,13 @@ type transcriptJSON struct {
 //	{"text":"...","eou":0,"eob":0,"frame_sec":0.080000,
 //	 "words":[{"w":"...","start":0.480,"end":0.640,"conf":0.9100}, ...]}
 //
-// "text" is the newly-finalized text since the last call; "eou" is 1 when an
-// <EOU> (end of utterance) fired this feed and "eob" is 1 when an <EOB>
-// (backchannel) fired. ABI v4 conflated the two into "eou"; v5 split them, so
-// we read both and treat either as an utterance boundary for segmentation.
-// "words" are the words finalized this call with absolute (stream-relative)
-// start/end seconds.
+// "text" is the newly-finalized text since the last call. Under ABI v5 "eou"
+// is 1 iff an <EOU> fired this feed (the user yielded the turn) and "eob" 1
+// iff an <EOB> fired (a backchannel like "uh-huh" ended — NOT a turn
+// boundary). A v4 library has no "eob" field and its "eou" conflates both
+// tokens: Eob stays 0 and Eou keeps the old any-event meaning. "words" are
+// the words finalized this call with absolute (stream-relative) start/end
+// seconds.
 type streamFeedJSON struct {
 	Text     string           `json:"text"`
 	Eou      int              `json:"eou"`
@@ -364,7 +365,7 @@ var segmentSeparators = []rune{'.', '?', '!'}
 // the caller requested word granularity; token ids populate each segment's
 // Tokens by time-window membership. Shared by the batched and direct paths.
 func transcriptResultFromDoc(doc transcriptJSON, opts *pb.TranscriptRequest, gapFrames int) pb.TranscriptResult {
-	text := strings.TrimSpace(doc.Text)
+	text, eou := stripEouMarker(strings.TrimSpace(doc.Text))
 
 	// Frame-unit gap threshold -> seconds (NeMo segment_gap_threshold). 0 = off.
 	gapSeconds := 0.0
@@ -383,6 +384,7 @@ func transcriptResultFromDoc(doc transcriptJSON, opts *pb.TranscriptRequest, gap
 		return pb.TranscriptResult{
 			Text:     text,
 			Segments: []*pb.TranscriptSegment{{Id: 0, Text: text}},
+			Eou:      eou,
 		}
 	}
 
@@ -409,7 +411,25 @@ func transcriptResultFromDoc(doc transcriptJSON, opts *pb.TranscriptRequest, gap
 		}
 		segments = append(segments, seg)
 	}
-	return pb.TranscriptResult{Text: text, Segments: segments}
+	return pb.TranscriptResult{Text: text, Segments: segments, Eou: eou}
+}
+
+// stripEouMarker removes a trailing literal <EOU>/<EOB> from offline-decode
+// text and reports whether the decode ended on an end-of-UTTERANCE token. The
+// realtime EOU model's offline decode keeps the special token in the
+// detokenized text (the streaming path strips it and surfaces it as flags
+// instead); user-visible transcripts must never carry either marker, but only
+// <EOU> may confirm the semantic_vad retranscribe cross-check — a decode
+// ending on <EOB> means the last thing heard was a backchannel, not the user
+// yielding the turn.
+func stripEouMarker(text string) (string, bool) {
+	if strings.HasSuffix(text, "<EOU>") {
+		return strings.TrimSpace(strings.TrimSuffix(text, "<EOU>")), true
+	}
+	if strings.HasSuffix(text, "<EOB>") {
+		return strings.TrimSpace(strings.TrimSuffix(text, "<EOB>")), false
+	}
+	return text, false
 }
 
 // splitWordsIntoSegments groups words into segments exactly as NeMo's
@@ -476,41 +496,55 @@ func tokensInWindow(tokens []transcriptToken, start, end float64) []int32 {
 	return ids
 }
 
-// streamSegmenter accumulates streaming words into per-utterance segments. EOU
-// is the model's own utterance boundary; each closed segment takes its start/end
-// from its first/last accumulated word.
+// streamSegmenter accumulates streaming decode increments into per-utterance
+// segments. <EOU>/<EOB> are the model's own utterance boundaries; each closes a
+// segment. When the feed carries per-word timings (ABI v4 JSON), a closed
+// segment takes its start/end from its first/last word; against an older
+// text-only library (no words) it falls back to segmenting the delta text, so
+// the same assembler serves both paths.
 type streamSegmenter struct {
-	segs   []*pb.TranscriptSegment
-	cur    []transcriptWord
-	nextID int32
+	segs    []*pb.TranscriptSegment
+	cur     []transcriptWord // words for the open segment (ABI v4 JSON path)
+	curText []string         // delta text for the open segment (text-only path)
+	nextID  int32
 }
 
-func (s *streamSegmenter) add(doc streamFeedJSON) {
-	s.cur = append(s.cur, doc.Words...)
-	// Close the segment on either turn signal: <EOU> (end of utterance) or
-	// <EOB> (backchannel). ABI v4 reported both via "eou"; v5 split them, so we
-	// OR them here to keep the v4 segmentation boundaries.
-	if doc.Eou != 0 || doc.Eob != 0 {
+func (s *streamSegmenter) add(r streamFeedResult) {
+	s.cur = append(s.cur, r.Words...)
+	if len(r.Words) == 0 && r.Delta != "" {
+		// Older libparakeet.so with no per-word timing: segment from the text.
+		s.curText = append(s.curText, r.Delta)
+	}
+	// Both <EOU> and <EOB> reset the decoder, so both close a segment.
+	if r.Eou || r.Eob {
 		s.flush()
 	}
 }
 
 func (s *streamSegmenter) flush() {
-	if len(s.cur) == 0 {
-		return
+	switch {
+	case len(s.cur) > 0:
+		parts := make([]string, len(s.cur))
+		for i, w := range s.cur {
+			parts[i] = w.W
+		}
+		s.segs = append(s.segs, &pb.TranscriptSegment{
+			Id:    s.nextID,
+			Start: secondsToNanos(s.cur[0].Start),
+			End:   secondsToNanos(s.cur[len(s.cur)-1].End),
+			Text:  strings.TrimSpace(strings.Join(parts, " ")),
+		})
+		s.nextID++
+	case len(s.curText) > 0:
+		// No words this segment: emit a text-only segment (no timestamps),
+		// skipping a purely-whitespace one as the legacy text path did.
+		if t := strings.TrimSpace(strings.Join(s.curText, "")); t != "" {
+			s.segs = append(s.segs, &pb.TranscriptSegment{Id: s.nextID, Text: t})
+			s.nextID++
+		}
 	}
-	parts := make([]string, len(s.cur))
-	for i, w := range s.cur {
-		parts[i] = w.W
-	}
-	s.segs = append(s.segs, &pb.TranscriptSegment{
-		Id:    s.nextID,
-		Start: secondsToNanos(s.cur[0].Start),
-		End:   secondsToNanos(s.cur[len(s.cur)-1].End),
-		Text:  strings.TrimSpace(strings.Join(parts, " ")),
-	})
-	s.nextID++
 	s.cur = nil
+	s.curText = nil
 }
 
 func (s *streamSegmenter) segments() []*pb.TranscriptSegment { return s.segs }
@@ -535,18 +569,119 @@ func secondsToNanos(sec float64) int64 {
 	return int64(sec * 1e9)
 }
 
+// Per-C-call engine serialization for the streaming paths.
+//
+// Every individual C call (begin / feed / finalize / free) takes engineMu and
+// re-checks ctxPtr under the lock; the lock is NEVER held across a stream's
+// lifetime. This is safe because each parakeet.cpp call builds its own ggml
+// graph and all streaming caches live in the session object, not the ctx —
+// the only ctx-shared mutable state is last_error, which is why it is read
+// under the same lock as the failing call. Holding the lock per call (rather
+// than per stream, as this file previously did) keeps a long-lived live
+// session from starving batched unary transcription and vice versa.
+//
+// A stream must not outlive its ctx (C-API contract). Free() takes engineMu
+// and zeroes ctxPtr, so a racing per-call helper returns ModelNotLoaded
+// instead of feeding a freed engine; streamFree of an orphaned session only
+// runs the session destructor, which does not touch the ctx.
+
+// streamBegin opens a cache-aware streaming session. A 0 stream with nil
+// error means the loaded model is not a streaming model.
+func (p *ParakeetCpp) streamBegin(lang string) (uintptr, error) {
+	p.engineMu.Lock()
+	defer p.engineMu.Unlock()
+	if p.ctxPtr == 0 {
+		return 0, grpcerrors.ModelNotLoaded("parakeet-cpp")
+	}
+	if CppStreamBeginLang != nil {
+		return CppStreamBeginLang(p.ctxPtr, lang), nil
+	}
+	return CppStreamBegin(p.ctxPtr), nil
+}
+
+func (p *ParakeetCpp) streamFree(stream uintptr) {
+	if stream == 0 {
+		return
+	}
+	p.engineMu.Lock()
+	defer p.engineMu.Unlock()
+	CppStreamFree(stream)
+}
+
+// streamFeedText runs one text-mode feed (or the finalize flush when
+// finalize is true) under engineMu, returning the newly-finalized delta and
+// whether an <EOU>/<EOB> fired during the call.
+func (p *ParakeetCpp) streamFeedText(stream uintptr, pcm []float32, finalize bool) (delta string, eou, eob bool, err error) {
+	p.engineMu.Lock()
+	defer p.engineMu.Unlock()
+	if p.ctxPtr == 0 {
+		return "", false, false, grpcerrors.ModelNotLoaded("parakeet-cpp")
+	}
+	var ret uintptr
+	var events int32
+	if finalize {
+		ret = CppStreamFinalize(stream)
+	} else {
+		ret = CppStreamFeed(stream, pcm, int32(len(pcm)), unsafe.Pointer(&events))
+	}
+	if ret == 0 {
+		// last_error is ctx-shared: read it under the same lock as the call.
+		msg := CppLastError(p.ctxPtr)
+		if msg == "" {
+			msg = "unknown error"
+		}
+		return "", false, false, fmt.Errorf("parakeet-cpp: stream feed/finalize failed: %s", msg)
+	}
+	delta = goStringFromCPtr(ret)
+	CppFreeString(ret)
+	// ABI v5: eou_out is a bitmask (bit 0 = <EOU>, bit 1 = <EOB>). A v4
+	// library sets 0/1 for either token, which the bit-0 test reads as the
+	// old conflated eou — the EOB distinction simply isn't available there.
+	return delta, events&1 != 0, events&2 != 0, nil
+}
+
+// streamFeedDoc runs one ABI v4 JSON feed (or finalize) under engineMu and
+// returns the parsed {text,eou,frame_sec,words} document.
+func (p *ParakeetCpp) streamFeedDoc(stream uintptr, pcm []float32, finalize bool) (streamFeedJSON, error) {
+	p.engineMu.Lock()
+	defer p.engineMu.Unlock()
+	if p.ctxPtr == 0 {
+		return streamFeedJSON{}, grpcerrors.ModelNotLoaded("parakeet-cpp")
+	}
+	var ret uintptr
+	if finalize {
+		ret = CppStreamFinalizeJSON(stream)
+	} else {
+		ret = CppStreamFeedJSON(stream, pcm, int32(len(pcm)))
+	}
+	if ret == 0 {
+		msg := CppLastError(p.ctxPtr)
+		if msg == "" {
+			msg = "unknown error"
+		}
+		return streamFeedJSON{}, fmt.Errorf("parakeet-cpp: stream feed/finalize failed: %s", msg)
+	}
+	raw := goStringFromCPtr(ret)
+	CppFreeString(ret)
+	var doc streamFeedJSON
+	if err := json.Unmarshal([]byte(raw), &doc); err != nil {
+		return streamFeedJSON{}, fmt.Errorf("parakeet-cpp: decode stream json: %w", err)
+	}
+	return doc, nil
+}
+
 // AudioTranscriptionStream drives the cache-aware streaming RNN-T over the
-// audio at opts.Dst: it decodes the file to 16 kHz mono PCM, feeds it in
-// chunks to parakeet_capi_stream_feed, and emits each newly-finalized text
-// run as a TranscriptStreamResponse delta. <EOU>/<EOB> events close the
-// current segment; a closing FinalResult carries the full transcript and the
-// per-utterance segments.
+// audio at opts.Dst: it decodes the file to 16 kHz mono PCM, feeds it through
+// the shared decode driver (feedSlices/flushTail), and emits each
+// newly-finalized text run as a TranscriptStreamResponse delta. <EOU>/<EOB>
+// events close the current segment; a closing FinalResult carries the full
+// transcript, the per-utterance segments, and whether the file ended on an
+// utterance boundary.
 //
 // stream_begin returns 0 for models that are not cache-aware streaming models
-// (only e.g. nvidia/parakeet_realtime_eou_120m-v1 qualifies). For those we fall
-// back to a single offline transcription emitted as one delta plus a closing
-// FinalResult, matching LocalAI's non-streaming streaming contract (and the
-// whisper backend), so the streaming endpoint works for every model.
+// (only e.g. nvidia/parakeet_realtime_eou_120m-v1 qualifies). For those this
+// returns codes.Unimplemented rather than faking a stream from an offline
+// decode — see the stream==0 branch and grpcerrors.StreamTranscriptionUnsupported.
 func (p *ParakeetCpp) AudioTranscriptionStream(ctx context.Context, opts *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
 	defer close(results)
 
@@ -560,185 +695,73 @@ func (p *ParakeetCpp) AudioTranscriptionStream(ctx context.Context, opts *pb.Tra
 		return status.Error(codes.Canceled, "transcription cancelled")
 	}
 
-	var stream uintptr
-	if CppStreamBeginLang != nil {
-		stream = CppStreamBeginLang(p.ctxPtr, opts.GetLanguage())
-	} else {
-		stream = CppStreamBegin(p.ctxPtr)
+	stream, err := p.streamBegin(opts.GetLanguage())
+	if err != nil {
+		return err
 	}
 	if stream == 0 {
-		// Not a cache-aware streaming model: run a normal offline
-		// transcription and emit it as one delta + a closing final result.
-		res, err := p.AudioTranscription(ctx, opts)
-		if err != nil {
-			return err
-		}
-		if t := strings.TrimSpace(res.Text); t != "" {
-			results <- &pb.TranscriptStreamResponse{Delta: t}
-		}
-		results <- &pb.TranscriptStreamResponse{FinalResult: &res}
-		return nil
+		// Not a cache-aware streaming model. Report the missing capability
+		// honestly instead of decoding offline and emitting it as one "delta"
+		// + final: a client that asked for streaming must learn the model
+		// cannot stream, not receive a batch result dressed as a stream (which
+		// is indistinguishable except qualitatively, and silently breaks any
+		// feature that genuinely needs incremental output). Callers wanting a
+		// plain transcript use the unary AudioTranscription path. This mirrors
+		// AudioTranscriptionLive, which already returns Unimplemented here.
+		return grpcerrors.StreamTranscriptionUnsupported("parakeet-cpp",
+			"loaded model is not a cache-aware streaming model")
 	}
-	defer CppStreamFree(stream)
-	// The C engine is a single shared context: a streaming session and a batched
-	// unary dispatch must never touch it at once, so hold engineMu for the whole
-	// stream. This lock is intentionally taken AFTER the non-streaming fallback
-	// above returns: that fallback goes through AudioTranscription -> the batcher
-	// -> runBatch, which itself acquires engineMu, so locking here first would
-	// deadlock. Do not hoist this lock above the fallback.
-	p.engineMu.Lock()
-	defer p.engineMu.Unlock()
+	defer p.streamFree(stream)
 
 	data, duration, err := decodeWavMono16k(opts.Dst)
 	if err != nil {
 		return err
 	}
 
-	// ABI v4: when the streaming JSON entry points are present, drive them so the
-	// per-utterance segments carry per-word start/end timestamps. Falls through to
-	// the text-only loop below against an older libparakeet.so. Runs under the
-	// engineMu already held above.
-	if CppStreamFeedJSON != nil {
-		return p.streamJSON(ctx, stream, data, duration, results)
-	}
-
+	// Fold the shared decode driver's per-feed increments into the streamed
+	// deltas and the closing batch result: words/text accumulate into
+	// per-utterance segments (streamSegmenter), and the utterance-boundary
+	// latch (boundary.go) records whether the file ended on an <EOU>. These
+	// are the offline path's concern — the live RPC carries none of them.
 	var (
 		full     strings.Builder
-		segText  strings.Builder
-		segments []*pb.TranscriptSegment
-		segID    int32
+		seg      streamSegmenter
+		boundary utteranceBoundary
 	)
-
-	flushSegment := func() {
-		t := strings.TrimSpace(segText.String())
-		segText.Reset()
-		if t == "" {
-			return
+	emit := func(r streamFeedResult) error {
+		if r.Delta != "" {
+			full.WriteString(r.Delta)
+			results <- &pb.TranscriptStreamResponse{Delta: r.Delta}
 		}
-		segments = append(segments, &pb.TranscriptSegment{Id: segID, Text: t})
-		segID++
-	}
-
-	// emitDelta consumes the malloc'd char* returned by feed/finalize: frees
-	// it, accumulates the text, and sends a delta when non-empty. A 0 return
-	// is an error (vs the "" empty-but-non-NULL no-new-text case).
-	emitDelta := func(ret uintptr) error {
-		if ret == 0 {
-			msg := CppLastError(p.ctxPtr)
-			if msg == "" {
-				msg = "unknown error"
-			}
-			return fmt.Errorf("parakeet-cpp: stream feed/finalize failed: %s", msg)
-		}
-		delta := goStringFromCPtr(ret)
-		CppFreeString(ret)
-		if delta == "" {
-			return nil
-		}
-		full.WriteString(delta)
-		segText.WriteString(delta)
-		results <- &pb.TranscriptStreamResponse{Delta: delta}
+		seg.add(r)
+		boundary = boundary.observe(r)
 		return nil
 	}
 
-	for off := 0; off < len(data); off += streamChunkSamples {
-		if err := ctx.Err(); err != nil {
-			return status.Error(codes.Canceled, "transcription cancelled")
-		}
-		end := min(off+streamChunkSamples, len(data))
-		chunk := data[off:end]
-
-		var eou int32
-		ret := CppStreamFeed(stream, chunk, int32(len(chunk)), unsafe.Pointer(&eou))
-		if err := emitDelta(ret); err != nil {
-			return err
-		}
-		if eou != 0 {
-			flushSegment()
-		}
-	}
-
-	// Flush the streaming tail (final encoder chunk).
-	if err := emitDelta(CppStreamFinalize(stream)); err != nil {
+	if err := p.feedSlices(ctx, stream, data, emit); err != nil {
 		return err
 	}
-	flushSegment()
-
-	text := strings.TrimSpace(full.String())
-	if len(segments) == 0 && text != "" {
-		segments = append(segments, &pb.TranscriptSegment{Id: 0, Text: text})
-	}
-	results <- &pb.TranscriptStreamResponse{
-		FinalResult: &pb.TranscriptResult{
-			Text:     text,
-			Segments: segments,
-			Duration: duration,
-		},
-	}
-	return nil
-}
-
-// streamJSON drives the streaming JSON entry points (present since ABI v4): each
-// feed/finalize returns a {text,eou,eob,frame_sec,words} document. The
-// newly-finalized text is emitted as a delta (unchanged streaming contract)
-// while words are accumulated into per-utterance segments (closed on <EOU> or
-// <EOB>) so the closing FinalResult carries timestamped segments. Runs under
-// engineMu (already held by the caller).
-func (p *ParakeetCpp) streamJSON(ctx context.Context, stream uintptr, data []float32,
-	duration float32, results chan *pb.TranscriptStreamResponse) error {
-	var (
-		full strings.Builder
-		seg  streamSegmenter
-	)
-	// consume frees the malloc'd char* (a 0 return is an error), parses the JSON,
-	// emits the delta, and routes words through the segmenter.
-	consume := func(ret uintptr) error {
-		if ret == 0 {
-			msg := CppLastError(p.ctxPtr)
-			if msg == "" {
-				msg = "unknown error"
-			}
-			return fmt.Errorf("parakeet-cpp: stream feed/finalize failed: %s", msg)
-		}
-		raw := goStringFromCPtr(ret)
-		CppFreeString(ret)
-		var doc streamFeedJSON
-		if err := json.Unmarshal([]byte(raw), &doc); err != nil {
-			return fmt.Errorf("parakeet-cpp: decode stream json: %w", err)
-		}
-		if doc.Text != "" {
-			full.WriteString(doc.Text)
-			results <- &pb.TranscriptStreamResponse{Delta: doc.Text}
-		}
-		seg.add(doc)
-		return nil
-	}
-
-	for off := 0; off < len(data); off += streamChunkSamples {
-		if err := ctx.Err(); err != nil {
-			return status.Error(codes.Canceled, "transcription cancelled")
-		}
-		end := min(off+streamChunkSamples, len(data))
-		chunk := data[off:end]
-		if err := consume(CppStreamFeedJSON(stream, chunk, int32(len(chunk)))); err != nil {
-			return err
-		}
-	}
-	if err := consume(CppStreamFinalizeJSON(stream)); err != nil {
+	if err := p.flushTail(stream, emit); err != nil {
 		return err
 	}
-	seg.flush() // close any trailing utterance that never saw an EOU
+	seg.flush() // close a trailing utterance that never saw an <EOU>
 
-	text := strings.TrimSpace(full.String())
+	// final.Text is the exact concatenation of the streamed deltas (full is
+	// their accumulation), so concat(deltas) == FinalResult.Text holds even
+	// when the model prepends a leading space to the first word (SentencePiece
+	// detokenization). This matches the whisper backend's streaming contract.
+	// The single-segment fallback stays trimmed.
+	fullText := full.String()
 	segments := seg.segments()
-	if len(segments) == 0 && text != "" {
-		segments = append(segments, &pb.TranscriptSegment{Id: 0, Text: text})
+	if trimmed := strings.TrimSpace(fullText); len(segments) == 0 && trimmed != "" {
+		segments = append(segments, &pb.TranscriptSegment{Id: 0, Text: trimmed})
 	}
 	results <- &pb.TranscriptStreamResponse{
 		FinalResult: &pb.TranscriptResult{
-			Text:     text,
+			Text:     fullText,
 			Segments: segments,
 			Duration: duration,
+			Eou:      boundary.ended(),
 		},
 	}
 	return nil
@@ -803,6 +826,10 @@ func (p *ParakeetCpp) Free() error {
 		close(p.batStop)
 		p.batStop = nil
 	}
+	// engineMu so an in-flight streaming call (which locks per C call and
+	// re-checks ctxPtr under the lock) can never feed into a freed ctx.
+	p.engineMu.Lock()
+	defer p.engineMu.Unlock()
 	if p.ctxPtr != 0 {
 		CppFree(p.ctxPtr)
 		p.ctxPtr = 0

backend/go/parakeet-cpp/goparakeetcpp_test.go

@@ -14,6 +14,8 @@ import (
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
 )
 
 func TestParakeetCpp(t *testing.T) {
@@ -201,6 +203,29 @@ var _ = Describe("ParakeetCpp", func() {
 	})
 
 	Context("AudioTranscriptionStream", func() {
+		It("returns the typed Unimplemented signal for non-streaming models (no offline fallback)", func() {
+			// stream_begin == 0 means the loaded model is not a cache-aware
+			// streaming model. The backend must surface that, not silently
+			// decode offline and fake a one-shot "stream".
+			savedBegin, savedBeginLang := CppStreamBegin, CppStreamBeginLang
+			defer func() { CppStreamBegin, CppStreamBeginLang = savedBegin, savedBeginLang }()
+			CppStreamBeginLang = nil
+			CppStreamBegin = func(ctx uintptr) uintptr { return 0 }
+
+			p := &ParakeetCpp{ctxPtr: 1}
+			results := make(chan *pb.TranscriptStreamResponse, 8)
+			err := p.AudioTranscriptionStream(context.Background(),
+				&pb.TranscriptRequest{Dst: "ignored.wav"}, results)
+			Expect(status.Code(err)).To(Equal(codes.Unimplemented))
+
+			// Honest signal: nothing was emitted — no faked batch result.
+			var emitted []*pb.TranscriptStreamResponse
+			for r := range results {
+				emitted = append(emitted, r)
+			}
+			Expect(emitted).To(BeEmpty())
+		})
+
 		It("streams deltas and a closing FinalResult from a cache-aware model", func() {
 			// Streaming needs a cache-aware streaming model (e.g.
 			// realtime_eou); the offline test model would fail stream_begin.

backend/go/parakeet-cpp/live.go

@@ -0,0 +1,186 @@
+package main
+
+import (
+	"strings"
+	"time"
+
+	"github.com/mudler/LocalAI/pkg/grpc/grpcerrors"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// liveSampleRate is the only PCM rate the parakeet C streaming API accepts.
+const liveSampleRate = 16000
+
+// AudioTranscriptionLive drives one cache-aware streaming session over audio
+// fed incrementally by the caller (the realtime API's semantic_vad turn
+// detection). Contract:
+//
+//   - the first request must carry a Config; a Config mid-stream resets the
+//     decode session (free + begin) and drops accumulated transcript state;
+//   - a Ready ack is sent right after a successful stream_begin so callers
+//     can degrade synchronously when the model has no streaming support
+//     (LiveTranscriptionUnsupported, codes.Unimplemented);
+//   - every feed that produced output is forwarded as {delta, eou, words};
+//     the <EOU>/<EOB> flag is the model's own utterance boundary and the
+//     decoder auto-resets after it, so one session spans many utterances;
+//   - closing the send side finalizes: the held-back tail chunk is flushed
+//     (the last ~2 encoder frames of words only appear here) and a terminal
+//     FinalResult carries the full transcript Text only. Per-utterance
+//     segments, duration, and the terminal <EOU> flag are NOT produced here —
+//     the realtime core consumes the streamed per-feed tokens and the final
+//     Text; those batch fields are the file path's concern (see
+//     AudioTranscriptionStream).
+//
+// Engine access is serialized per C call (streamBegin/streamFeed*/streamFree
+// take engineMu internally), never for the session lifetime — unary
+// transcription keeps flowing between feeds.
+func (p *ParakeetCpp) AudioTranscriptionLive(in <-chan *pb.TranscriptLiveRequest, out chan<- *pb.TranscriptLiveResponse) error {
+	defer close(out)
+
+	if p.ctxPtr == 0 {
+		return grpcerrors.ModelNotLoaded("parakeet-cpp")
+	}
+
+	first, ok := <-in
+	if !ok {
+		return nil // caller closed without sending anything
+	}
+	cfg := first.GetConfig()
+	if cfg == nil {
+		return status.Error(codes.InvalidArgument, "parakeet-cpp: first live message must carry a config")
+	}
+	if err := validateLiveConfig(cfg); err != nil {
+		return err
+	}
+
+	stream, err := p.streamBegin(cfg.GetLanguage())
+	if err != nil {
+		return err
+	}
+	if stream == 0 {
+		return grpcerrors.LiveTranscriptionUnsupported("parakeet-cpp",
+			"loaded model is not a cache-aware streaming model")
+	}
+	// stream is reassigned on a mid-stream Config reset; free whatever is
+	// current when the RPC unwinds.
+	defer func() { p.streamFree(stream) }()
+
+	out <- &pb.TranscriptLiveResponse{Ready: true}
+
+	var (
+		full    strings.Builder
+		fedSecs float64
+
+		// behindSec accumulates how far decode wall time has fallen behind
+		// the audio it was fed. A live caller feeds in real time, so a
+		// persistent positive backlog means every downstream signal —
+		// including the <EOU> the turn detector waits on — arrives that many
+		// seconds late. Warned once per session; reset by a Config reset.
+		behindSec    float64
+		behindWarned bool
+	)
+
+	// emit forwards one decode increment: it streams the per-feed tokens the
+	// realtime turn detector consumes (delta/eou/eob/words) and accumulates the
+	// running transcript for the closing FinalResult. No segmentation or
+	// boundary latch here — the live consumer reads only the streamed tokens
+	// and the final Text; per-utterance segments and the terminal <EOU> flag
+	// are an offline-path concern (see AudioTranscriptionStream / boundary.go).
+	emit := func(r streamFeedResult) error {
+		if r.Delta != "" {
+			full.WriteString(r.Delta)
+		}
+		if r.Delta != "" || r.Eou || r.Eob || len(r.Words) > 0 {
+			out <- &pb.TranscriptLiveResponse{
+				Delta: r.Delta,
+				Eou:   r.Eou,
+				Eob:   r.Eob,
+				Words: liveWordsToProto(r.Words),
+			}
+		}
+		return nil
+	}
+
+	for req := range in {
+		switch payload := req.GetPayload().(type) {
+		case *pb.TranscriptLiveRequest_Config:
+			if err := validateLiveConfig(payload.Config); err != nil {
+				return err
+			}
+			// Reset: a fresh decode session, dropping accumulated state.
+			p.streamFree(stream)
+			stream, err = p.streamBegin(payload.Config.GetLanguage())
+			if err != nil {
+				return err
+			}
+			if stream == 0 {
+				return grpcerrors.LiveTranscriptionUnsupported("parakeet-cpp",
+					"loaded model is not a cache-aware streaming model")
+			}
+			full.Reset()
+			fedSecs = 0
+		case *pb.TranscriptLiveRequest_Audio:
+			pcm := payload.Audio.GetPcm()
+			audioSec := float64(len(pcm)) / liveSampleRate
+			fedSecs += audioSec
+			start := time.Now()
+			// nil ctx: a live session is bounded by this request channel, not a
+			// context — cancellation is the caller closing the stream.
+			if err := p.feedSlices(nil, stream, pcm, emit); err != nil {
+				return err
+			}
+			wallSec := time.Since(start).Seconds()
+			behindSec += wallSec - audioSec
+			if behindSec < 0 {
+				behindSec = 0
+			}
+			xlog.Debug("parakeet-cpp: live feed",
+				"audio_ms", int(audioSec*1000), "wall_ms", int(wallSec*1000),
+				"behind_ms", int(behindSec*1000), "fed_s", fedSecs)
+			if behindSec > 1 && !behindWarned {
+				behindWarned = true
+				xlog.Warn("parakeet-cpp: live decode is falling behind real time; "+
+					"end-of-utterance signals will arrive late",
+					"behind_s", behindSec, "fed_s", fedSecs)
+			}
+		}
+	}
+
+	// Send side closed: flush the streaming tail and emit the final transcript.
+	// The live FinalResult carries only Text — the authoritative full-turn
+	// transcript the realtime core commits. Per-utterance segments, duration,
+	// and the terminal <EOU> flag are not produced on the live path.
+	if err := p.flushTail(stream, emit); err != nil {
+		return err
+	}
+	out <- &pb.TranscriptLiveResponse{
+		FinalResult: &pb.TranscriptResult{Text: strings.TrimSpace(full.String())},
+	}
+	return nil
+}
+
+func validateLiveConfig(cfg *pb.TranscriptLiveConfig) error {
+	if sr := cfg.GetSampleRate(); sr != 0 && sr != liveSampleRate {
+		return status.Errorf(codes.InvalidArgument,
+			"parakeet-cpp: unsupported live sample_rate %d (only %d)", sr, liveSampleRate)
+	}
+	return nil
+}
+
+func liveWordsToProto(words []transcriptWord) []*pb.TranscriptWord {
+	if len(words) == 0 {
+		return nil
+	}
+	out := make([]*pb.TranscriptWord, len(words))
+	for i, w := range words {
+		out[i] = &pb.TranscriptWord{
+			Start: secondsToNanos(w.Start),
+			End:   secondsToNanos(w.End),
+			Text:  w.W,
+		}
+	}
+	return out
+}

backend/go/parakeet-cpp/live_test.go

@@ -0,0 +1,417 @@
+package main
+
+import (
+	"sync"
+	"time"
+	"unsafe"
+
+	"github.com/mudler/LocalAI/pkg/grpc/grpcerrors"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
+)
+
+// The live-RPC specs drive AudioTranscriptionLive entirely against stubbed
+// Cpp* package vars (the same seam batcher_test.go uses), so they run
+// without libparakeet.so.
+
+// liveCstrPool hands out NUL-terminated C-style strings backed by Go memory
+// and keeps them alive for the duration of a spec (goStringFromCPtr reads
+// through the raw pointer; Go's GC must not collect the backing array while
+// a stub's return value is in flight).
+type liveCstrPool struct {
+	mu   sync.Mutex
+	bufs [][]byte
+}
+
+func (p *liveCstrPool) cstr(s string) uintptr {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	b := append([]byte(s), 0)
+	p.bufs = append(p.bufs, b)
+	return uintptr(unsafe.Pointer(&b[0]))
+}
+
+// liveStubs swaps every C entry point the live path touches and returns a
+// restore func for AfterEach.
+func liveStubs() (restore func()) {
+	savedBegin, savedBeginLang := CppStreamBegin, CppStreamBeginLang
+	savedFeed, savedFeedJSON := CppStreamFeed, CppStreamFeedJSON
+	savedFinalize, savedFinalizeJSON := CppStreamFinalize, CppStreamFinalizeJSON
+	savedFree, savedLastError := CppStreamFree, CppLastError
+	savedFreeString := CppFreeString
+	return func() {
+		CppStreamBegin, CppStreamBeginLang = savedBegin, savedBeginLang
+		CppStreamFeed, CppStreamFeedJSON = savedFeed, savedFeedJSON
+		CppStreamFinalize, CppStreamFinalizeJSON = savedFinalize, savedFinalizeJSON
+		CppStreamFree, CppLastError = savedFree, savedLastError
+		CppFreeString = savedFreeString
+	}
+}
+
+// runLive starts the RPC on its own goroutine and returns the request
+// channel plus a collector for everything the backend emitted.
+func runLive(p *ParakeetCpp) (chan *pb.TranscriptLiveRequest, chan *pb.TranscriptLiveResponse, chan error) {
+	in := make(chan *pb.TranscriptLiveRequest)
+	out := make(chan *pb.TranscriptLiveResponse, 32)
+	errCh := make(chan error, 1)
+	go func() { errCh <- p.AudioTranscriptionLive(in, out) }()
+	return in, out, errCh
+}
+
+func liveConfig(lang string) *pb.TranscriptLiveRequest {
+	return &pb.TranscriptLiveRequest{
+		Payload: &pb.TranscriptLiveRequest_Config{Config: &pb.TranscriptLiveConfig{Language: lang}},
+	}
+}
+
+func liveAudio(pcm []float32) *pb.TranscriptLiveRequest {
+	return &pb.TranscriptLiveRequest{
+		Payload: &pb.TranscriptLiveRequest_Audio{Audio: &pb.TranscriptLiveAudio{Pcm: pcm}},
+	}
+}
+
+func collectLive(out chan *pb.TranscriptLiveResponse) []*pb.TranscriptLiveResponse {
+	var got []*pb.TranscriptLiveResponse
+	for r := range out {
+		got = append(got, r)
+	}
+	return got
+}
+
+var _ = Describe("AudioTranscriptionLive (stubbed C API)", func() {
+	var (
+		pool    *liveCstrPool
+		restore func()
+		p       *ParakeetCpp
+	)
+
+	BeforeEach(func() {
+		pool = &liveCstrPool{}
+		restore = liveStubs()
+		p = &ParakeetCpp{ctxPtr: 1}
+
+		CppStreamBeginLang = nil
+		CppStreamBegin = func(ctx uintptr) uintptr { return 7 }
+		CppStreamFree = func(s uintptr) {}
+		CppFreeString = func(s uintptr) {}
+		CppLastError = func(ctx uintptr) string { return "stub error" }
+		CppStreamFeed = nil
+		CppStreamFeedJSON = nil
+		CppStreamFinalize = nil
+		CppStreamFinalizeJSON = nil
+	})
+
+	AfterEach(func() { restore() })
+
+	It("rejects a stream whose first message is not a config", func() {
+		in, out, errCh := runLive(p)
+		in <- liveAudio([]float32{0.1})
+		close(in)
+
+		err := <-errCh
+		Expect(status.Code(err)).To(Equal(codes.InvalidArgument))
+		Expect(collectLive(out)).To(BeEmpty())
+	})
+
+	It("rejects a non-16k sample rate", func() {
+		in, _, errCh := runLive(p)
+		in <- &pb.TranscriptLiveRequest{
+			Payload: &pb.TranscriptLiveRequest_Config{Config: &pb.TranscriptLiveConfig{SampleRate: 8000}},
+		}
+		close(in)
+		Expect(status.Code(<-errCh)).To(Equal(codes.InvalidArgument))
+	})
+
+	It("returns the typed Unimplemented signal for non-streaming models, before any ack", func() {
+		CppStreamBegin = func(ctx uintptr) uintptr { return 0 }
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		close(in)
+
+		err := <-errCh
+		Expect(grpcerrors.IsLiveTranscriptionUnsupported(err)).To(BeTrue())
+		Expect(collectLive(out)).To(BeEmpty())
+	})
+
+	It("streams deltas, eou flags and words on the JSON path and finalizes on close", func() {
+		var freed []uintptr
+		CppStreamFree = func(s uintptr) { freed = append(freed, s) }
+		feeds := 0
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr {
+			feeds++
+			switch feeds {
+			case 1:
+				return pool.cstr(`{"text":"hello ","eou":0,"frame_sec":0.08,` +
+					`"words":[{"w":"hello","start":0.1,"end":0.4,"conf":0.9}]}`)
+			default:
+				return pool.cstr(`{"text":"world","eou":1,"frame_sec":0.08,` +
+					`"words":[{"w":"world","start":0.5,"end":0.8,"conf":0.9}]}`)
+			}
+		}
+		CppStreamFinalizeJSON = func(s uintptr) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("en")
+		in <- liveAudio(make([]float32, 100))
+		in <- liveAudio(make([]float32, 200))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		Expect(got).To(HaveLen(4)) // ready, two deltas, final
+
+		Expect(got[0].Ready).To(BeTrue())
+
+		Expect(got[1].Delta).To(Equal("hello "))
+		Expect(got[1].Eou).To(BeFalse())
+		Expect(got[1].Words).To(HaveLen(1))
+		Expect(got[1].Words[0].Text).To(Equal("hello"))
+
+		Expect(got[2].Delta).To(Equal("world"))
+		Expect(got[2].Eou).To(BeTrue())
+
+		final := got[3].FinalResult
+		Expect(final).NotTo(BeNil())
+		Expect(final.Text).To(Equal("hello world"))
+		// The live FinalResult carries only Text. Per-utterance segments,
+		// duration and the terminal eou flag are an offline-path concern (see
+		// boundary.go / AudioTranscriptionStream); the realtime core reads the
+		// streamed per-feed tokens above plus this Text.
+		Expect(final.Eou).To(BeFalse())
+		Expect(final.Segments).To(BeEmpty())
+		Expect(final.Duration).To(BeZero())
+
+		Expect(freed).To(Equal([]uintptr{7}))
+	})
+
+	It("falls back to the text feed (eou out-param) when the JSON entry points are absent", func() {
+		feeds := 0
+		CppStreamFeed = func(s uintptr, pcm []float32, n int32, eouOut unsafe.Pointer) uintptr {
+			feeds++
+			if feeds == 2 {
+				*(*int32)(eouOut) = 1
+				return pool.cstr("done")
+			}
+			return pool.cstr("first ")
+		}
+		CppStreamFinalize = func(s uintptr) uintptr { return pool.cstr("") }
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+		in <- liveAudio(make([]float32, 10))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		Expect(got).To(HaveLen(4))
+		Expect(got[1].Delta).To(Equal("first "))
+		Expect(got[1].Eou).To(BeFalse())
+		Expect(got[2].Delta).To(Equal("done"))
+		Expect(got[2].Eou).To(BeTrue())
+		Expect(got[3].FinalResult.Text).To(Equal("first done"))
+	})
+
+	It("forwards <EOB> as eob — a backchannel, never an eou (ABI v5 JSON)", func() {
+		feeds := 0
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr {
+			feeds++
+			if feeds == 1 {
+				return pool.cstr(`{"text":"uh-huh","eou":0,"eob":1,"frame_sec":0.08,` +
+					`"words":[{"w":"uh-huh","start":0.1,"end":0.3,"conf":0.9}]}`)
+			}
+			return pool.cstr(`{"text":"the turn","eou":1,"eob":0,"frame_sec":0.08,` +
+				`"words":[{"w":"the","start":0.5,"end":0.6,"conf":0.9},{"w":"turn","start":0.6,"end":0.8,"conf":0.9}]}`)
+		}
+		CppStreamFinalizeJSON = func(s uintptr) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"eob":0,"frame_sec":0.08,"words":[]}`)
+		}
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+		in <- liveAudio(make([]float32, 10))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		Expect(got).To(HaveLen(4))
+		Expect(got[1].Eob).To(BeTrue())
+		Expect(got[1].Eou).To(BeFalse(), "a backchannel must not masquerade as a turn boundary")
+		Expect(got[2].Eou).To(BeTrue())
+	})
+
+	It("maps the v5 eou_out bitmask on the text path (bit0 <EOU>, bit1 <EOB>)", func() {
+		feeds := 0
+		CppStreamFeed = func(s uintptr, pcm []float32, n int32, eouOut unsafe.Pointer) uintptr {
+			feeds++
+			if feeds == 1 {
+				*(*int32)(eouOut) = 2 // <EOB> only
+				return pool.cstr("uh-huh")
+			}
+			*(*int32)(eouOut) = 1 // <EOU>
+			return pool.cstr(" done")
+		}
+		CppStreamFinalize = func(s uintptr) uintptr { return pool.cstr("") }
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+		in <- liveAudio(make([]float32, 10))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		Expect(got).To(HaveLen(4))
+		Expect(got[1].Eob).To(BeTrue())
+		Expect(got[1].Eou).To(BeFalse())
+		Expect(got[2].Eou).To(BeTrue())
+		Expect(got[2].Eob).To(BeFalse())
+	})
+
+	It("accumulates trailing text after an EOU into the final transcript", func() {
+		feeds := 0
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr {
+			feeds++
+			if feeds == 1 {
+				return pool.cstr(`{"text":"turn one","eou":1,"frame_sec":0.08,"words":[]}`)
+			}
+			return pool.cstr(`{"text":" and more","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+		CppStreamFinalizeJSON = func(s uintptr) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+		in <- liveAudio(make([]float32, 10))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		final := got[len(got)-1].FinalResult
+		Expect(final.Text).To(Equal("turn one and more"))
+	})
+
+	It("resets the decode session on a mid-stream config", func() {
+		var begun, freed int
+		CppStreamBegin = func(ctx uintptr) uintptr { begun++; return uintptr(10 + begun) }
+		CppStreamFree = func(s uintptr) { freed++ }
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr {
+			return pool.cstr(`{"text":"x","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+		CppStreamFinalizeJSON = func(s uintptr) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+		in <- liveConfig("") // reset
+		in <- liveAudio(make([]float32, 10))
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+
+		got := collectLive(out)
+		final := got[len(got)-1].FinalResult
+		Expect(final.Text).To(Equal("x"), "pre-reset transcript dropped")
+		Expect(begun).To(Equal(2))
+		Expect(freed).To(Equal(2), "old session freed on reset, new one on unwind")
+	})
+
+	It("does not hold engineMu between feeds (unary work interleaves with a live session)", func() {
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+		CppStreamFinalizeJSON = func(s uintptr) uintptr {
+			return pool.cstr(`{"text":"","eou":0,"frame_sec":0.08,"words":[]}`)
+		}
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+
+		// The session is open and idle between feeds: the engine lock must be
+		// acquirable, which is what lets batched unary transcription proceed
+		// mid-session. Under stream-lifetime locking this probe would block
+		// until the stream ended and the Eventually would time out.
+		locked := make(chan struct{})
+		go func() {
+			p.engineMu.Lock()
+			p.engineMu.Unlock() //nolint:staticcheck // probe: acquire-release proves availability
+			close(locked)
+		}()
+		Eventually(locked, time.Second).Should(BeClosed())
+
+		close(in)
+		Expect(<-errCh).NotTo(HaveOccurred())
+		collectLive(out)
+	})
+
+	It("errors out and reads last_error under the lock when a feed fails", func() {
+		CppStreamFeedJSON = func(s uintptr, pcm []float32, n int32) uintptr { return 0 }
+
+		in, out, errCh := runLive(p)
+		in <- liveConfig("")
+		in <- liveAudio(make([]float32, 10))
+
+		err := <-errCh
+		Expect(err).To(MatchError(ContainSubstring("stub error")))
+		got := collectLive(out)
+		Expect(got).To(HaveLen(1)) // just the ready ack
+		close(in)
+	})
+})
+
+var _ = Describe("stripEouMarker", func() {
+	It("strips a trailing <EOU> and reports it", func() {
+		text, eou := stripEouMarker("it is certainly very like the old portrait<EOU>")
+		Expect(text).To(Equal("it is certainly very like the old portrait"))
+		Expect(eou).To(BeTrue())
+	})
+
+	It("strips a trailing <EOB> WITHOUT reporting an utterance end", func() {
+		// A decode ending on a backchannel must not confirm the
+		// retranscribe gate — the user was acknowledging, not yielding.
+		text, eou := stripEouMarker("uh-huh<EOB>")
+		Expect(text).To(Equal("uh-huh"))
+		Expect(eou).To(BeFalse())
+	})
+
+	It("leaves marker-free text alone", func() {
+		text, eou := stripEouMarker("plain transcript")
+		Expect(text).To(Equal("plain transcript"))
+		Expect(eou).To(BeFalse())
+	})
+
+	It("does not strip a marker in the middle of the text", func() {
+		text, eou := stripEouMarker("a<EOU>b")
+		Expect(text).To(Equal("a<EOU>b"))
+		Expect(eou).To(BeFalse())
+	})
+})
+
+var _ = Describe("transcriptResultFromDoc EOU handling", func() {
+	It("strips the offline marker from text and sets the result flag", func() {
+		doc := transcriptJSON{Text: "the old portrait<EOU>"}
+		res := transcriptResultFromDoc(doc, &pb.TranscriptRequest{}, 0)
+		Expect(res.Text).To(Equal("the old portrait"))
+		Expect(res.Eou).To(BeTrue())
+		Expect(res.Segments).To(HaveLen(1))
+		Expect(res.Segments[0].Text).To(Equal("the old portrait"))
+	})
+
+	It("reports eou=false for marker-free decodes", func() {
+		doc := transcriptJSON{Text: "no marker here"}
+		res := transcriptResultFromDoc(doc, &pb.TranscriptRequest{}, 0)
+		Expect(res.Text).To(Equal("no marker here"))
+		Expect(res.Eou).To(BeFalse())
+	})
+})

backend/go/parakeet-cpp/segments_test.go

@@ -106,7 +106,7 @@ var _ = Describe("transcriptResultFromDoc (multi-segment)", func() {
 var _ = Describe("streaming segment assembly", func() {
 	It("closes a segment with start/end from its words on EOU", func() {
 		acc := &streamSegmenter{}
-		acc.add(streamFeedJSON{Text: "hello world", Eou: 1, Words: []transcriptWord{
+		acc.add(streamFeedResult{Delta: "hello world", Eou: true, Words: []transcriptWord{
 			{W: "hello", Start: 0.0, End: 0.4}, {W: "world", Start: 0.4, End: 0.9},
 		}})
 		segs := acc.segments()
@@ -118,9 +118,9 @@ var _ = Describe("streaming segment assembly", func() {
 
 	It("buffers words across feeds until EOU", func() {
 		acc := &streamSegmenter{}
-		acc.add(streamFeedJSON{Text: "hi", Eou: 0, Words: []transcriptWord{{W: "hi", Start: 0, End: 0.3}}})
+		acc.add(streamFeedResult{Delta: "hi", Words: []transcriptWord{{W: "hi", Start: 0, End: 0.3}}})
 		Expect(acc.segments()).To(BeEmpty())
-		acc.add(streamFeedJSON{Text: "there", Eou: 1, Words: []transcriptWord{{W: "there", Start: 0.3, End: 0.7}}})
+		acc.add(streamFeedResult{Delta: "there", Eou: true, Words: []transcriptWord{{W: "there", Start: 0.3, End: 0.7}}})
 		Expect(acc.segments()).To(HaveLen(1))
 		Expect(acc.segments()[0].Text).To(Equal("hi there"))
 	})
@@ -129,7 +129,7 @@ var _ = Describe("streaming segment assembly", func() {
 	// field; a backchannel must still close the segment as it did in v4.
 	It("closes a segment on EOB (backchannel) too", func() {
 		acc := &streamSegmenter{}
-		acc.add(streamFeedJSON{Text: "uh huh", Eou: 0, Eob: 1, Words: []transcriptWord{
+		acc.add(streamFeedResult{Delta: "uh huh", Eob: true, Words: []transcriptWord{
 			{W: "uh", Start: 0.0, End: 0.2}, {W: "huh", Start: 0.2, End: 0.5},
 		}})
 		segs := acc.segments()
@@ -137,4 +137,18 @@ var _ = Describe("streaming segment assembly", func() {
 		Expect(segs[0].Text).To(Equal("uh huh"))
 		Expect(segs[0].End).To(Equal(secondsToNanos(0.5)))
 	})
+
+	// Older text-only libparakeet.so: no per-word timings, so a segment is cut
+	// from the delta text on each <EOU>/<EOB> (no timestamps), one per utterance.
+	It("falls back to text segments when the feed carries no words", func() {
+		acc := &streamSegmenter{}
+		acc.add(streamFeedResult{Delta: "first turn", Eou: true})
+		acc.add(streamFeedResult{Delta: "second turn", Eou: true})
+		segs := acc.segments()
+		Expect(segs).To(HaveLen(2))
+		Expect(segs[0].Text).To(Equal("first turn"))
+		Expect(segs[1].Text).To(Equal("second turn"))
+		Expect(segs[0].Start).To(Equal(int64(0)), "no per-word timing on the text path")
+		Expect(segs[0].End).To(Equal(int64(0)))
+	})
 })

backend/go/qwen3-tts-cpp/CMakeLists.txt

@@ -30,7 +30,7 @@ target_include_directories(goqwen3ttscpp PRIVATE ${QWENTTS_DIR}/src)
 target_include_directories(goqwen3ttscpp SYSTEM PRIVATE ${QWENTTS_DIR}/ggml/include)
 
 # Link GPU backends if the upstream ggml created them.
-foreach(backend blas cuda metal vulkan sycl)
+foreach(backend blas cuda hip metal vulkan sycl)
     if(TARGET ggml-${backend})
         target_link_libraries(goqwen3ttscpp PRIVATE ggml-${backend})
         if(backend STREQUAL "cuda")

backend/go/qwen3-tts-cpp/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
-QWEN3TTS_CPP_VERSION?=9dbe7ea26a01b30fccb117ae5e86807c1dc23d42
+QWEN3TTS_CPP_VERSION?=d17c33d4ee2f56d15f9ca8a1bb82f7389305f838
 SO_TARGET?=libgoqwen3ttscpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -24,7 +24,14 @@ else ifeq ($(BUILD_TYPE),openblas)
 else ifeq ($(BUILD_TYPE),clblas)
 	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
 else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=ON
 else ifeq ($(OS),Darwin)

backend/go/qwen3-tts-cpp/README.md

@@ -0,0 +1,85 @@
+# Qwen3-TTS C++ backend
+
+This backend runs Qwen3-TTS GGUF models through
+[qwentts.cpp](https://github.com/ServeurpersoCom/qwentts.cpp). It supports
+24 kHz speech generation, streaming, named speakers, voice design, and
+reference-audio cloning depending on the model variant.
+
+## Gallery models
+
+The following Base models accept LocalAI Voice Library profiles:
+
+- `qwen3-tts-cpp`
+- `qwen3-tts-cpp-0.6b-base-q4`
+- `qwen3-tts-cpp-1.7b-base`
+- `qwen3-tts-cpp-1.7b-base-q4`
+
+Gallery models containing `customvoice` or `voicedesign` implement those Qwen
+modes instead and are not advertised as raw reference-audio models.
+
+Install a Base model with:
+
+```bash
+local-ai models install qwen3-tts-cpp
+```
+
+## Model configuration
+
+Base filenames are detected automatically. Set `tts.voice_cloning` only when a
+verified private conversion has a name that does not identify it as a Base or
+VoiceClone model:
+
+```yaml
+name: private-qwen-voice
+backend: qwen3-tts-cpp
+parameters:
+  model: qwen-private/talker.gguf
+known_usecases:
+  - tts
+tts:
+  voice_cloning: true
+  audio_path: voices/default-reference.wav  # optional model-wide fallback
+```
+
+The tokenizer GGUF is auto-discovered when its filename contains `tokenizer`
+and it is stored beside the talker. Otherwise set
+`options: ["tokenizer:qwen-private/tokenizer.gguf"]`.
+
+`tts.voice_cloning: false` removes a model from Voice Library compatibility
+results and rejects saved `localai://voice-profiles/...` references. It does
+not disable Qwen's named-speaker or VoiceDesign modes. Setting it to `true`
+cannot add cloning to a backend that lacks LocalAI's reference-audio contract.
+
+Request precedence is: a request `voice`, then `tts.voice`, then
+`tts.audio_path`. A saved profile supplies its private WAV and exact transcript
+for that request without changing the model YAML.
+
+## API example
+
+Create or select a profile in **Operate → Voice Library**, then pass its stable
+URI to either speech endpoint:
+
+```bash
+curl http://localhost:8080/v1/audio/speech \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "qwen3-tts-cpp",
+    "input": "This request uses a saved reference voice.",
+    "voice": "localai://voice-profiles/PROFILE_ID"
+  }' \
+  --output speech.wav
+```
+
+## Native end-to-end test
+
+The labeled test loads real GGUFs, synthesizes speech, streams audio, and
+exercises cloning with a generated 24 kHz reference WAV:
+
+```bash
+make -C backend/go/qwen3-tts-cpp qwen3-tts-cpp
+
+QWEN3TTS_MODEL=/path/to/qwen-talker-0.6b-base-Q8_0.gguf \
+QWEN3TTS_CODEC=/path/to/qwen-tokenizer-12hz-Q8_0.gguf \
+QWEN3TTS_LIBRARY=backend/go/qwen3-tts-cpp/libgoqwen3ttscpp-fallback.so \
+  go test ./backend/go/qwen3-tts-cpp -ginkgo.label-filter=e2e
+```

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=9956436c925a367daeab097598b1ea1f32d3503f
+STABLEDIFFUSION_GGML_VERSION?=b5d812008eb7082a238fc589444544b3278187ae
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/stablediffusion-ggml/cpp/gosd.cpp

@@ -660,9 +660,16 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     ctx_params.diffusion_conv_direct = diffusion_conv_direct;
     ctx_params.vae_conv_direct = vae_conv_direct;
     ctx_params.force_sdxl_vae_conv_scale = force_sdxl_vae_conv_scale;
-    ctx_params.chroma_use_dit_mask = chroma_use_dit_mask;
-    ctx_params.chroma_use_t5_mask = chroma_use_t5_mask;
-    ctx_params.chroma_t5_mask_pad = chroma_t5_mask_pad;
+    // Chroma knobs: upstream dropped the dedicated chroma_use_dit_mask /
+    // chroma_use_t5_mask / chroma_t5_mask_pad struct fields and now reads them
+    // from the generic model_args key=value spec (parse_key_value_args). Emit
+    // them there so the existing chroma options keep working. This string must
+    // outlive new_sd_ctx() below.
+    std::string model_args_spec =
+        "chroma_use_dit_mask=" + std::string(chroma_use_dit_mask ? "true" : "false") +
+        ",chroma_use_t5_mask=" + std::string(chroma_use_t5_mask ? "true" : "false") +
+        ",chroma_t5_mask_pad=" + std::to_string(chroma_t5_mask_pad);
+    ctx_params.model_args = model_args_spec.c_str();
     sd_ctx_t* sd_ctx = new_sd_ctx(&ctx_params);
 
     if (sd_ctx == NULL) {
@@ -798,6 +805,7 @@ void sd_img_gen_params_set_seed(sd_img_gen_params_t *params, int64_t seed) {
 int gen_image(sd_img_gen_params_t *p, int steps, char *dst, float cfg_scale, char *src_image, float strength, char *mask_image, char* ref_images[], int ref_images_count) {
 
     sd_image_t* results;
+    int num_results_out = 0;
 
     std::vector<int> skip_layers = {7, 8, 9};
 
@@ -994,10 +1002,14 @@ int gen_image(sd_img_gen_params_t *p, int steps, char *dst, float cfg_scale, cha
             sd_ctx_params_to_str(&ctx_params),
             sd_img_gen_params_to_str(p));
 
-    results = generate_image(sd_c, p);
+    bool gen_ok = generate_image(sd_c, p, &results, &num_results_out);
 
     std::free(p);
 
+    if (!gen_ok || num_results_out == 0) {
+        results = NULL;
+    }
+
     if (results == NULL) {
         fprintf (stderr, "NO results\n");
         if (input_image_buffer) free(input_image_buffer);

backend/go/vibevoice-cpp/CMakeLists.txt

@@ -50,7 +50,7 @@ target_include_directories(govibevoicecpp SYSTEM PRIVATE ${VIBEVOICE_DIR}/third_
 # Link GPU backends if available — vibevoice's own CMake already links
 # these to the libvibevoice STATIC library, but we re-link them on the
 # MODULE so resolved symbols include all backend kernels.
-foreach(backend blas cuda metal vulkan)
+foreach(backend blas cuda hip metal vulkan)
     if(TARGET ggml-${backend})
         target_link_libraries(govibevoicecpp PRIVATE ggml-${backend})
         string(TOUPPER ${backend} BACKEND_UPPER)

backend/go/vibevoice-cpp/Makefile

@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1)
 # already do for ik_llama.cpp / llama.cpp / whisper.cpp). Floating on
 # `master` led to silent ABI breaks reaching CI — pin it.
 VIBEVOICE_REPO?=https://github.com/mudler/vibevoice.cpp
-VIBEVOICE_CPP_VERSION?=ad856bda6b1311b7f3d7c4a667be43eeb8a8249a
+VIBEVOICE_CPP_VERSION?=000e37282bc5bb09edc20f7047a47924122ba3a0
 SO_TARGET?=libgovibevoicecpp.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
@@ -29,7 +29,14 @@ else ifeq ($(BUILD_TYPE),openblas)
 else ifeq ($(BUILD_TYPE),clblas)
 	CMAKE_ARGS+=-DGGML_CLBLAST=ON -DCLBlast_DIR=/some/path
 else ifeq ($(BUILD_TYPE),hipblas)
-	CMAKE_ARGS+=-DGGML_HIPBLAS=ON -DVIBEVOICE_GGML_HIPBLAS=ON
+	# This ggml only understands GGML_HIP (GGML_HIPBLAS was removed upstream),
+	# so passing GGML_HIPBLAS silently produced a CPU-only build (see #10666).
+	ROCM_HOME ?= /opt/rocm
+	ROCM_PATH ?= /opt/rocm
+	export CXX=$(ROCM_HOME)/llvm/bin/clang++
+	export CC=$(ROCM_HOME)/llvm/bin/clang
+	AMDGPU_TARGETS ?= gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201
+	CMAKE_ARGS+=-DGGML_HIP=ON -DAMDGPU_TARGETS=$(AMDGPU_TARGETS)
 else ifeq ($(BUILD_TYPE),vulkan)
 	CMAKE_ARGS+=-DGGML_VULKAN=ON -DVIBEVOICE_GGML_VULKAN=ON
 else ifeq ($(OS),Darwin)

backend/go/vibevoice-cpp/govibevoicecpp.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"context"
+	"encoding/binary"
 	"encoding/json"
 	"fmt"
 	"io"
@@ -10,7 +11,9 @@ import (
 	"path/filepath"
 	"runtime"
 	"strings"
+	"unsafe"
 
+	"github.com/ebitengine/purego"
 	laudio "github.com/mudler/LocalAI/pkg/audio"
 	"github.com/mudler/LocalAI/pkg/grpc/base"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
@@ -113,12 +116,76 @@ var (
 		refAudioPaths []*byte, nRefAudioPaths int32,
 		dstWav string,
 		nSteps int32, cfgScale float32, maxSpeechFrames int32, seed uint32) int32
+	// CppTTSStream drives vv_capi_tts_stream: it synthesizes `text` and
+	// invokes the C callback `cb` once per decoded PCM window instead of
+	// writing a file. `cb` is the address of a purego callback (see
+	// streamCB); `user` is an opaque pointer handed back to every
+	// callback invocation - we route via the package-level activeStream
+	// instead, so it is always nil here.
+	CppTTSStream func(text, voicePath string,
+		nSteps int32, cfgScale float32, maxFrames int32, seed uint32,
+		cb uintptr, user unsafe.Pointer) int32
 	CppASR func(srcWav string, outJSON []byte, capacity uint64,
 		maxNewTokens int32) int32
 	CppUnload  func()
 	CppVersion func() string
 )
 
+// streamState carries the destination channel for one in-flight
+// TTSStream call. vibevoice's engine is a single process-global, and
+// backend calls are serialized through base.SingleThread, so a single
+// package-level pointer is safe: only one TTSStream runs at a time.
+type streamState struct {
+	results chan []byte
+}
+
+// activeStream points at the streamState for the currently-running
+// TTSStream. The C callback (streamCB) and the deliverPCMForTest hook
+// read it to find the channel. Guarded by base.SingleThread
+// serialization; TTSStream sets it and clears it in a defer.
+var activeStream *streamState
+
+// pushPCM copies a transient int16 PCM window into a fresh little-endian
+// []byte and pushes it onto the active stream. The C buffer handed to
+// the callback is only valid for the duration of the call, so we must
+// copy before returning. A nil/empty input or a missing active stream
+// is a no-op.
+func pushPCM(pcm []int16) {
+	s := activeStream
+	if s == nil || len(pcm) == 0 {
+		return
+	}
+	buf := make([]byte, len(pcm)*2)
+	for i, v := range pcm {
+		binary.LittleEndian.PutUint16(buf[i*2:], uint16(v))
+	}
+	s.results <- buf
+}
+
+// streamCB is the ONE reusable purego callback bound to the C ABI's
+// vv_pcm_cb. purego cannot free callbacks and enforces a process-global
+// limit, so we create exactly one at package init and reuse it for every
+// TTSStream call - the per-call state lives in activeStream, not here.
+// purego marshals the C `const int16_t*` first argument straight into a
+// Go *int16, so we can unsafe.Slice it without a uintptr round-trip
+// (keeps go vet clean); pushPCM copies the transient buffer out and
+// returns 0 to keep synthesizing.
+var streamCB = purego.NewCallback(func(samples *int16, n int32, _ uintptr) uintptr {
+	if activeStream == nil || samples == nil || n <= 0 {
+		return 0
+	}
+	pcm := unsafe.Slice(samples, int(n))
+	pushPCM(pcm)
+	return 0
+})
+
+// deliverPCMForTest exercises the exact copy-and-push path streamCB runs
+// against activeStream, but from a Go []int16 - so unit tests can
+// validate the callback -> channel framing without the C library.
+func deliverPCMForTest(samples []int16) {
+	pushPCM(samples)
+}
+
 // VibevoiceCpp speaks gRPC against vibevoice.cpp's flat C ABI. The
 // engine is a single global, so we serialize calls through SingleThread.
 type VibevoiceCpp struct {
@@ -404,14 +471,16 @@ func (v *VibevoiceCpp) callASR(srcWav string, maxNewTokens int32) (string, error
 	return string(buf[:rc]), nil
 }
 
-// TTSStream is the streaming counterpart to TTS. vibevoice's C ABI is
-// file-only (vv_capi_tts writes a complete WAV), so we synthesize to
-// a tempfile, then emit a streaming-WAV header followed by the PCM
-// body in chunks. The main reason this exists at all is the gRPC
-// server wrapper (pkg/grpc/server.go:TTSStream) blocks on a channel
-// that only this method can close - if we leave the default Base
-// stub in place, every TTSStream call hangs until the client
-// deadline.
+// TTSStream is the streaming counterpart to TTS. It drives
+// vv_capi_tts_stream, which synthesizes `text` and invokes our C
+// callback (streamCB) once per decoded PCM window instead of writing a
+// file - so the client starts receiving audio while the model is still
+// generating. We first emit a streaming-WAV header, install the results
+// channel as the active stream, then let the callback push each PCM
+// window (copied to little-endian bytes) onto that channel. The gRPC
+// server wrapper (pkg/grpc/server.go:TTSStream) blocks on the channel
+// until this method closes it, so `defer close(results)` is mandatory
+// even on the error paths.
 func (v *VibevoiceCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error {
 	defer close(results)
 	if v.ttsModel == "" {
@@ -421,28 +490,6 @@ func (v *VibevoiceCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error
 		return fmt.Errorf("vibevoice-cpp: TTSStream requires text")
 	}
 
-	tmp, err := os.CreateTemp("", "vibevoice-cpp-stream-*.wav")
-	if err != nil {
-		return fmt.Errorf("vibevoice-cpp: tempfile: %w", err)
-	}
-	dst := tmp.Name()
-	_ = tmp.Close()
-	defer func() { _ = os.Remove(dst) }()
-
-	if err := v.TTS(&pb.TTSRequest{
-		Text:     req.Text,
-		Voice:    req.Voice,
-		Dst:      dst,
-		Language: req.Language,
-	}); err != nil {
-		return err
-	}
-
-	wav, err := os.ReadFile(dst)
-	if err != nil {
-		return fmt.Errorf("vibevoice-cpp: read tempfile: %w", err)
-	}
-
 	// Streaming WAV header: declare 0xFFFFFFFF for chunk sizes so HTTP
 	// clients can start playback before they see the full PCM.
 	const streamingSize = 0xFFFFFFFF
@@ -455,18 +502,38 @@ func (v *VibevoiceCpp) TTSStream(req *pb.TTSRequest, results chan []byte) error
 	}
 	results <- hdrBuf
 
-	// PCM body: send in ~64 KB slices so the client gets multiple
-	// reply chunks (e2e harness asserts >=2 frames).
-	pcm := laudio.StripWAVHeader(wav)
-	const chunkBytes = 64 * 1024
-	for off := 0; off < len(pcm); off += chunkBytes {
-		end := off + chunkBytes
-		if end > len(pcm) {
-			end = len(pcm)
-		}
-		chunk := make([]byte, end-off)
-		copy(chunk, pcm[off:end])
-		results <- chunk
+	// vv_capi_tts_stream takes a single voice_path (realtime-0.5B path);
+	// unlike vv_capi_tts it has no ref_audio array. Resolve the per-call
+	// override when it names a voice gguf, otherwise fall back to the
+	// load-time default that already went to vv_capi_load.
+	voice := v.voice
+	if reqVoice := strings.TrimSpace(req.Voice); reqVoice != "" && !isRefAudioOverride(reqVoice) {
+		voice = resolvePath(reqVoice, v.modelRoot)
+	}
+
+	if req.Language != nil && *req.Language != "" {
+		fmt.Fprintf(os.Stderr,
+			"[vibevoice-cpp] note: TTSRequest.language=%q ignored - vibevoice picks language from the voice prompt\n",
+			*req.Language)
+	}
+
+	const (
+		defaultSteps     = 20
+		defaultMaxFrames = 200
+	)
+	defaultCfg := float32(1.3)
+
+	// Serialized by base.SingleThread, so a single package-level
+	// activeStream is race-free: exactly one TTSStream runs at a time.
+	// The callback reads it to find `results`; clear it on the way out.
+	activeStream = &streamState{results: results}
+	defer func() { activeStream = nil }()
+
+	rc := CppTTSStream(req.Text, voice,
+		int32(defaultSteps), defaultCfg, int32(defaultMaxFrames), 0,
+		streamCB, nil)
+	if rc != 0 {
+		return fmt.Errorf("vibevoice-cpp: vv_capi_tts_stream failed (rc=%d)", rc)
 	}
 	return nil
 }

backend/go/vibevoice-cpp/main.go

@@ -37,6 +37,7 @@ func main() {
 	libFuncs := []LibFuncs{
 		{&CppLoad, "vv_capi_load"},
 		{&CppTTS, "vv_capi_tts"},
+		{&CppTTSStream, "vv_capi_tts_stream"},
 		{&CppASR, "vv_capi_asr"},
 		{&CppUnload, "vv_capi_unload"},
 		{&CppVersion, "vv_capi_version"},

backend/go/vibevoice-cpp/vibevoicecpp_integration_test.go

@@ -0,0 +1,262 @@
+package main
+
+// Real end-to-end streaming integration test for the vibevoice-cpp
+// backend. It drives the actual Go -> purego -> C path against the real
+// model, proving that TTSStream delivers audio incrementally (measurable
+// time-to-first-audio) while TTS only yields anything after full
+// synthesis. Gated behind VIBEVOICE_IT=1 so normal `go test` and CI stay
+// unaffected - it needs the built engine .so and ~1.7 GB of model files.
+//
+// Run:
+//
+//	VIBEVOICE_IT=1 \
+//	VIBEVOICECPP_LIBRARY=<abs path to libgovibevoicecpp-fallback.so> \
+//	  go test ./... -run TestVibevoiceCpp -v -timeout 600s
+//
+// Optional overrides (default to the staged bundle under
+// ~/_git/vibevoice-models):
+//
+//	VIBEVOICE_IT_MODEL      vibevoice-realtime-0.5B-q8_0.gguf (abs path)
+//	VIBEVOICE_IT_TOKENIZER  tokenizer.gguf (abs path)
+//	VIBEVOICE_IT_VOICE      voice-en-Carter_man.gguf (abs path)
+
+import (
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"sync"
+	"time"
+
+	"github.com/ebitengine/purego"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// itDefaultModelDir is where Task 6 staged the model bundle. Individual
+// files can be overridden via the VIBEVOICE_IT_* env vars.
+const itDefaultModelDir = "/home/mudler/_git/vibevoice-models"
+
+// itLibLoaded guards the one-time purego Dlopen + RegisterLibFunc into
+// the package-level Cpp* vars. purego cannot free a loaded library and
+// the Cpp* symbols are process-global, so we bind exactly once.
+var itLibLoaded sync.Once
+
+// integrationOrSkip enforces the VIBEVOICE_IT=1 gate and binds the C ABI
+// symbols into the package vars the backend calls (mirrors main.go's
+// libFuncs list). It Skip()s the spec when the gate is off or the .so /
+// model files are missing, so the suite stays green in every other run.
+func integrationOrSkip() (model, tokenizer, voice string) {
+	if os.Getenv("VIBEVOICE_IT") != "1" {
+		Skip("VIBEVOICE_IT!=1, skipping real-model streaming integration test")
+	}
+
+	lib := os.Getenv("VIBEVOICECPP_LIBRARY")
+	if lib == "" {
+		Skip("VIBEVOICECPP_LIBRARY not set, cannot dlopen the engine .so")
+	}
+	if _, err := os.Stat(lib); err != nil {
+		Skip("engine .so not found at " + lib)
+	}
+
+	model = itEnvOr("VIBEVOICE_IT_MODEL", filepath.Join(itDefaultModelDir, "vibevoice-realtime-0.5B-q8_0.gguf"))
+	tokenizer = itEnvOr("VIBEVOICE_IT_TOKENIZER", filepath.Join(itDefaultModelDir, "tokenizer.gguf"))
+	voice = itEnvOr("VIBEVOICE_IT_VOICE", filepath.Join(itDefaultModelDir, "voice-en-Carter_man.gguf"))
+	for _, p := range []string{model, tokenizer, voice} {
+		if _, err := os.Stat(p); err != nil {
+			Skip("model file missing: " + p)
+		}
+	}
+
+	itLibLoaded.Do(func() {
+		handle, err := purego.Dlopen(lib, purego.RTLD_NOW|purego.RTLD_GLOBAL)
+		Expect(err).ToNot(HaveOccurred(), "dlopen %s", lib)
+		// Mirror the libFuncs list in main.go verbatim so the test binds
+		// the exact same symbols the production backend binary does.
+		purego.RegisterLibFunc(&CppLoad, handle, "vv_capi_load")
+		purego.RegisterLibFunc(&CppTTS, handle, "vv_capi_tts")
+		purego.RegisterLibFunc(&CppTTSStream, handle, "vv_capi_tts_stream")
+		purego.RegisterLibFunc(&CppASR, handle, "vv_capi_asr")
+		purego.RegisterLibFunc(&CppUnload, handle, "vv_capi_unload")
+		purego.RegisterLibFunc(&CppVersion, handle, "vv_capi_version")
+	})
+	return model, tokenizer, voice
+}
+
+func itEnvOr(key, def string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return def
+}
+
+// itParseWavPCM returns the int16 PCM samples from a standard WAV file by
+// locating the "data" sub-chunk, so it works regardless of how many
+// bytes of header/metadata the engine wrote.
+func itParseWavPCM(b []byte) []int16 {
+	// Find the "data" sub-chunk id, then its 4-byte little-endian size.
+	idx := -1
+	for i := 12; i+8 <= len(b); i += 2 {
+		if string(b[i:i+4]) == "data" {
+			idx = i
+			break
+		}
+	}
+	Expect(idx).To(BeNumerically(">=", 0), "no data chunk in WAV")
+	pcmStart := idx + 8
+	size := int(b[idx+4]) | int(b[idx+5])<<8 | int(b[idx+6])<<16 | int(b[idx+7])<<24
+	if size <= 0 || pcmStart+size > len(b) {
+		size = len(b) - pcmStart
+	}
+	n := size / 2
+	out := make([]int16, n)
+	for i := 0; i < n; i++ {
+		out[i] = int16(uint16(b[pcmStart+i*2]) | uint16(b[pcmStart+i*2+1])<<8)
+	}
+	return out
+}
+
+// itRMS computes the root-mean-square of int16 PCM as a sanity signal:
+// > 0 means non-silent, and NaN/Inf-free means well-formed samples.
+func itRMS(pcm []int16) float64 {
+	if len(pcm) == 0 {
+		return 0
+	}
+	var sumSq float64
+	for _, s := range pcm {
+		v := float64(s)
+		sumSq += v * v
+	}
+	return sqrt(sumSq / float64(len(pcm)))
+}
+
+// sqrt via Newton's method to avoid dragging math into the file's tiny
+// need (keeps the dependency surface of the test minimal).
+func sqrt(x float64) float64 {
+	if x <= 0 {
+		return 0
+	}
+	z := x
+	for i := 0; i < 40; i++ {
+		z = 0.5 * (z + x/z)
+	}
+	return z
+}
+
+var _ = Describe("VibeVoice-cpp real-model streaming (VIBEVOICE_IT=1)", Ordered, func() {
+	// The paragraph is deliberately long (multiple sentences) so the
+	// engine decodes several streaming windows and TTFA is meaningfully
+	// earlier than full synthesis.
+	const paragraph = "The quick brown fox jumps over the lazy dog near the riverbank. " +
+		"A gentle breeze carried the sound of distant bells across the quiet valley. " +
+		"Streaming synthesis lets you hear the very first words while the rest is still being generated."
+
+	var v *VibevoiceCpp
+
+	BeforeAll(func() {
+		model, tokenizer, voice := integrationOrSkip()
+
+		v = &VibevoiceCpp{}
+		err := v.Load(&pb.ModelOptions{
+			ModelFile: model,
+			ModelPath: filepath.Dir(model),
+			Options: []string{
+				"tokenizer=" + tokenizer,
+				"voice=" + voice,
+			},
+			Threads: 4,
+		})
+		Expect(err).ToNot(HaveOccurred(), "Load must succeed with the real model")
+	})
+
+	It("streams audio incrementally and beats the batch path to first audio", func() {
+		// ---- Streaming run -------------------------------------------
+		results := make(chan []byte, 256)
+		streamErr := make(chan error, 1)
+
+		start := time.Now()
+		go func() {
+			streamErr <- v.TTSStream(&pb.TTSRequest{Text: paragraph}, results)
+		}()
+
+		var (
+			header      []byte
+			ttfa        time.Duration
+			totalStream time.Duration
+			chunkCount  int
+			pcmBytes    int
+			firstPCM    bool
+		)
+		for buf := range results {
+			if header == nil {
+				header = buf
+				continue
+			}
+			if len(buf) == 0 {
+				continue
+			}
+			if !firstPCM {
+				ttfa = time.Since(start)
+				firstPCM = true
+			}
+			chunkCount++
+			pcmBytes += len(buf)
+		}
+		totalStream = time.Since(start)
+		Expect(<-streamErr).ToNot(HaveOccurred(), "TTSStream returned an error")
+
+		// First message must be a 44-byte streaming WAV header.
+		Expect(header).To(HaveLen(44), "first stream message must be the WAV header")
+		Expect(string(header[0:4])).To(Equal("RIFF"))
+		Expect(string(header[8:12])).To(Equal("WAVE"))
+
+		// Streaming invariants: multiple windows, real audio, early
+		// delivery (first audio strictly before the stream completes).
+		Expect(chunkCount).To(BeNumerically(">=", 2), "expected multiple streamed PCM windows")
+		Expect(pcmBytes).To(BeNumerically(">", 0), "no PCM bytes streamed")
+		Expect(firstPCM).To(BeTrue(), "never received a PCM chunk")
+		Expect(ttfa).To(BeNumerically("<", totalStream), "TTFA must precede stream completion")
+
+		// ---- Batch baseline ------------------------------------------
+		tmp, err := os.MkdirTemp("", "vv-it-batch-*")
+		Expect(err).ToNot(HaveOccurred())
+		DeferCleanup(func() { _ = os.RemoveAll(tmp) })
+		dst := filepath.Join(tmp, "batch.wav")
+
+		batchStart := time.Now()
+		Expect(v.TTS(&pb.TTSRequest{Text: paragraph, Dst: dst})).To(Succeed())
+		totalBatch := time.Since(batchStart)
+
+		wav, err := os.ReadFile(dst)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(len(wav)).To(BeNumerically(">", 44), "batch wav has no PCM payload")
+		Expect(string(wav[0:4])).To(Equal("RIFF"))
+		Expect(string(wav[8:12])).To(Equal("WAVE"))
+		batchPCM := itParseWavPCM(wav)
+		Expect(len(batchPCM)).To(BeNumerically(">", 0), "batch produced no samples")
+		rms := itRMS(batchPCM)
+		Expect(rms).To(BeNumerically(">", 0), "batch audio is silent")
+		Expect(rms).To(BeNumerically("<", 40000), "batch rms out of int16 range (corrupt samples)")
+
+		// ---- Headline numbers ----------------------------------------
+		// Emit to both GinkgoWriter and stderr: GinkgoWriter needs
+		// -ginkgo.v to surface, stderr is always captured so the headline
+		// TTFA vs batch numbers are never lost in an unattended run.
+		ratio := float64(ttfa) / float64(totalBatch)
+		report := fmt.Sprintf("\n================ vibevoice-cpp streaming TTFA ================\n"+
+			"input words        : ~%d\n"+
+			"TTFA (first audio) : %v\n"+
+			"total_stream       : %v\n"+
+			"total_batch        : %v\n"+
+			"stream chunks      : %d\n"+
+			"stream PCM bytes   : %d\n"+
+			"batch samples/rms  : %d / %.1f\n"+
+			"TTFA / total_batch : %.3f (first audio in this fraction of batch's deliver time)\n"+
+			"==============================================================\n\n",
+			len(paragraph)/6, ttfa, totalStream, totalBatch, chunkCount, pcmBytes, len(batchPCM), rms, ratio)
+		for _, out := range []io.Writer{GinkgoWriter, os.Stderr} {
+			_, _ = io.WriteString(out, report)
+		}
+	})
+})

backend/go/vibevoice-cpp/vibevoicecpp_stream_test.go

@@ -0,0 +1,46 @@
+package main
+
+import (
+	"encoding/binary"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("TTSStream callback framing", func() {
+	// The real C callback copies a transient int16 PCM buffer out of the
+	// engine into a fresh []byte and pushes it onto the active stream's
+	// results channel as little-endian bytes. deliverPCMForTest runs that
+	// exact copy-and-push path against a []int16 so we can validate the
+	// framing without the C library (full audio e2e is a later task).
+	It("copies int16 PCM from the C callback into the results channel as LE bytes", func() {
+		samples := []int16{0, 1, -1, 32767, -32768, 1234, -4321}
+
+		prev := activeStream
+		DeferCleanup(func() { activeStream = prev })
+
+		s := &streamState{results: make(chan []byte, 1)}
+		activeStream = s
+
+		deliverPCMForTest(samples)
+
+		var got []byte
+		Eventually(s.results).Should(Receive(&got))
+		Expect(got).To(HaveLen(len(samples) * 2))
+
+		want := make([]byte, len(samples)*2)
+		for i, v := range samples {
+			binary.LittleEndian.PutUint16(want[i*2:], uint16(v))
+		}
+		Expect(got).To(Equal(want))
+	})
+
+	It("is a no-op when there is no active stream", func() {
+		prev := activeStream
+		DeferCleanup(func() { activeStream = prev })
+		activeStream = nil
+
+		// Must not panic when no stream is installed.
+		Expect(func() { deliverPCMForTest([]int16{1, 2, 3}) }).ToNot(Panic())
+	})
+})

backend/go/voice-detect/Makefile

@@ -1,6 +1,6 @@
 # voice-detect backend Makefile.
 #
-# Upstream pin lives below as VOICEDETECT_VERSION?=3d51077... (.github/bump_deps.sh
+# Upstream pin lives below as VOICEDETECT_VERSION?=1db1759572c90faef6f3a78c36b5941a096a9f89
 # can find and update it - matches the parakeet.cpp / whisper.cpp / ds4 convention).
 #
 # Local dev shortcut: if you already have an out-of-tree voice-detect.cpp build,
@@ -13,7 +13,7 @@
 # The default target below does the proper clone-at-pin + cmake build so CI does
 # not need a side-checkout.
 
-VOICEDETECT_VERSION?=3d510772357538c5182808ac7de2278b84824e24
+VOICEDETECT_VERSION?=1db1759572c90faef6f3a78c36b5941a096a9f89
 VOICEDETECT_REPO?=https://github.com/mudler/voice-detect.cpp
 
 GOCMD?=go

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=0ae02cdb2c7317b50991367c165736ce42ed96ac
+WHISPER_CPP_VERSION?=080bbbe85230f624f0b52127f1ae1218247989f9
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/index.yaml

@@ -178,6 +178,35 @@
     nvidia-cuda-12: "cuda12-parakeet-cpp"
     nvidia-l4t-cuda-12: "nvidia-l4t-arm64-parakeet-cpp"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-parakeet-cpp"
+- &mosstranscribecpp
+  name: "moss-transcribe-cpp"
+  alias: "moss-transcribe-cpp"
+  license: mit
+  icon: https://avatars.githubusercontent.com/u/95302084
+  description: |
+    moss-transcribe.cpp is a C++/ggml port of OpenMOSS MOSS-Transcribe-Diarize, an end-to-end audio understanding model.
+    It does joint long-form transcription, speaker diarization and timestamping in a single pass, emitting a speaker-labelled,
+    time-aligned transcript, and runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan, Apple Metal and NVIDIA Jetson (L4T) targets.
+  urls:
+    - https://github.com/mudler/moss-transcribe.cpp
+  tags:
+    - audio-transcription
+    - CPU
+    - GPU
+    - CUDA
+    - HIP
+  capabilities:
+    default: "cpu-moss-transcribe-cpp"
+    nvidia: "cuda12-moss-transcribe-cpp"
+    intel: "intel-sycl-f16-moss-transcribe-cpp"
+    metal: "metal-moss-transcribe-cpp"
+    amd: "rocm-moss-transcribe-cpp"
+    vulkan: "vulkan-moss-transcribe-cpp"
+    nvidia-l4t: "nvidia-l4t-arm64-moss-transcribe-cpp"
+    nvidia-cuda-13: "cuda13-moss-transcribe-cpp"
+    nvidia-cuda-12: "cuda12-moss-transcribe-cpp"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-transcribe-cpp"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-transcribe-cpp"
 - &ced
   name: "ced"
   alias: "ced"
@@ -795,6 +824,30 @@
     nvidia-cuda-12: "cuda12-vllm-omni"
     nvidia-cuda-13: "cuda13-vllm-omni"
     nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-vllm-omni"
+- &longcat-video
+  name: "longcat-video"
+  alias: "longcat-video"
+  license: mit
+  urls:
+    - https://github.com/meituan-longcat/LongCat-Video
+  tags:
+    - text-to-video
+    - image-to-video
+    - audio-to-video
+    - avatar-generation
+    - video-generation
+    - CUDA
+  icon: https://raw.githubusercontent.com/meituan-longcat/LongCat-Video/main/assets/longcat-video_logo.svg
+  description: |
+    LongCat-Video generation for text, image, and audio-conditioned avatars.
+    Supports LongCat-Video and LongCat-Video-Avatar-1.5, including multi-segment
+    talking-head continuation and an SDPA path for NVIDIA Blackwell ARM64 systems.
+    Requires Linux with an NVIDIA CUDA GPU; CPU, ROCm, and macOS are unsupported.
+  capabilities:
+    nvidia: "cuda12-longcat-video"
+    nvidia-cuda-12: "cuda12-longcat-video"
+    nvidia-cuda-13: "cuda13-longcat-video"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-longcat-video"
 - &mlx
   name: "mlx"
   icon: https://avatars.githubusercontent.com/u/102832242?s=200&v=4
@@ -2784,6 +2837,121 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-parakeet-cpp"
   mirrors:
     - localai/localai-backends:master-gpu-nvidia-cuda-13-parakeet-cpp
+## moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "moss-transcribe-cpp-development"
+  capabilities:
+    default: "cpu-moss-transcribe-cpp-development"
+    nvidia: "cuda12-moss-transcribe-cpp-development"
+    intel: "intel-sycl-f16-moss-transcribe-cpp-development"
+    metal: "metal-moss-transcribe-cpp-development"
+    amd: "rocm-moss-transcribe-cpp-development"
+    vulkan: "vulkan-moss-transcribe-cpp-development"
+    nvidia-l4t: "nvidia-l4t-arm64-moss-transcribe-cpp-development"
+    nvidia-cuda-13: "cuda13-moss-transcribe-cpp-development"
+    nvidia-cuda-12: "cuda12-moss-transcribe-cpp-development"
+    nvidia-l4t-cuda-12: "nvidia-l4t-arm64-moss-transcribe-cpp-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-moss-transcribe-cpp-development"
+- !!merge <<: *mosstranscribecpp
+  name: "nvidia-l4t-arm64-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "nvidia-l4t-arm64-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda13-nvidia-l4t-arm64-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda13-nvidia-l4t-arm64-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cpu-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-cpu-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cpu-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-cpu-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-cpu-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "metal-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-metal-darwin-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "metal-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-metal-darwin-arm64-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda12-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda12-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "rocm-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-rocm-hipblas-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-rocm-hipblas-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "rocm-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-rocm-hipblas-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "intel-sycl-f32-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f32-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f32-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "intel-sycl-f32-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f32-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f32-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "intel-sycl-f16-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-intel-sycl-f16-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-intel-sycl-f16-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "intel-sycl-f16-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-intel-sycl-f16-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-intel-sycl-f16-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "vulkan-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-vulkan-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-vulkan-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "vulkan-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-vulkan-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-vulkan-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda13-moss-transcribe-cpp"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-moss-transcribe-cpp
+- !!merge <<: *mosstranscribecpp
+  name: "cuda13-moss-transcribe-cpp-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-moss-transcribe-cpp"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-moss-transcribe-cpp
 ## ced
 - !!merge <<: *ced
   name: "ced-development"
@@ -3461,6 +3629,44 @@
   uri: "quay.io/go-skynet/local-ai-backends:master-gpu-rocm-hipblas-vllm-omni"
   mirrors:
     - localai/localai-backends:master-gpu-rocm-hipblas-vllm-omni
+# longcat-video
+- !!merge <<: *longcat-video
+  name: "longcat-video-development"
+  capabilities:
+    nvidia: "cuda12-longcat-video-development"
+    nvidia-cuda-12: "cuda12-longcat-video-development"
+    nvidia-cuda-13: "cuda13-longcat-video-development"
+    nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-longcat-video-development"
+- !!merge <<: *longcat-video
+  name: "cuda12-longcat-video"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-longcat-video"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-12-longcat-video
+- !!merge <<: *longcat-video
+  name: "cuda13-longcat-video"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-longcat-video"
+  mirrors:
+    - localai/localai-backends:latest-gpu-nvidia-cuda-13-longcat-video
+- !!merge <<: *longcat-video
+  name: "cuda13-nvidia-l4t-arm64-longcat-video"
+  uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-longcat-video"
+  mirrors:
+    - localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-longcat-video
+- !!merge <<: *longcat-video
+  name: "cuda12-longcat-video-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-longcat-video"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-12-longcat-video
+- !!merge <<: *longcat-video
+  name: "cuda13-longcat-video-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-longcat-video"
+  mirrors:
+    - localai/localai-backends:master-gpu-nvidia-cuda-13-longcat-video
+- !!merge <<: *longcat-video
+  name: "cuda13-nvidia-l4t-arm64-longcat-video-development"
+  uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-longcat-video"
+  mirrors:
+    - localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-longcat-video
 # rfdetr
 - !!merge <<: *rfdetr
   name: "rfdetr-development"

backend/python/README.md

@@ -27,6 +27,7 @@ The Python backends use a unified build system based on `libbackend.sh` that pro
 
 ### Computer Vision
 - **diffusers** - Stable Diffusion and image generation
+- **longcat-video** - CUDA video and speech-driven avatar generation with LongCat-Video
 - **mlx-vlm** - Vision-language models for Apple Silicon
 - **rfdetr** - Object detection models
 

backend/python/chatterbox/backend.py

@@ -199,7 +199,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
             if "language" in self.options:
                 kwargs["language_id"] = self.options["language"]
-            if self.AudioPath is not None:
+            if request.voice and os.path.isfile(request.voice):
+                kwargs["audio_prompt_path"] = request.voice
+            elif self.AudioPath is not None:
                 kwargs["audio_prompt_path"] = self.AudioPath
 
             # add options to kwargs
@@ -211,6 +213,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             # strings on the wire and are coerced to float/int/bool.
             if hasattr(request, "params") and request.params:
                 for key, value in request.params.items():
+                    if key == "ref_text":
+                        continue
                     kwargs[key] = coerce_param_value(value)
 
             # Check if text exceeds 250 characters

backend/python/common/grpc_auth.py

@@ -11,6 +11,8 @@ import os
 
 import grpc
 
+from parent_watch import start_parent_death_watcher
+
 
 class _AbortHandler(grpc.RpcMethodHandler):
     """A method handler that immediately aborts with UNAUTHENTICATED."""
@@ -70,6 +72,13 @@ def get_auth_interceptors(*, aio: bool = False):
 
     Returns an empty list when LOCALAI_GRPC_AUTH_TOKEN is not set.
     """
+    # Arm the best-effort parent-death backstop here: this is the single helper
+    # every LocalAI Python backend invokes exactly once while building its gRPC
+    # server (mirroring how the Go watcher arms in pkg/grpc's shared serve path).
+    # start_parent_death_watcher() is idempotent and a no-op when disabled or on
+    # unsupported platforms — see parent_watch.py.
+    start_parent_death_watcher()
+
     token = os.environ.get("LOCALAI_GRPC_AUTH_TOKEN", "")
     if not token:
         return []

backend/python/common/libbackend.sh

@@ -397,10 +397,29 @@ function ensureVenv() {
 
 function runProtogen() {
     ensureVenv
+
+    # Match grpcio-tools to the grpcio already installed by the backend's
+    # requirements. grpcio and grpcio-tools are released in lockstep, and the
+    # protoc that grpcio-tools bundles stamps a Protobuf "gencode" version into
+    # backend_pb2.py. Left unpinned, `uv pip install grpcio-tools` pulls the
+    # newest release, whose newer gencode (e.g. 7.35.0) trips Protobuf's
+    # runtime >= gencode guarantee at import time when a backend caps the
+    # protobuf runtime lower (vLLM pins it to 6.33.6), crashing the backend with
+    # "grpc service not ready" before it ever loads a model. Pinning
+    # grpcio-tools to the installed grpcio version keeps the gencode in step with
+    # the runtime. Falls back to unpinned when grpcio isn't installed yet.
+    # See mudler/LocalAI#10718.
+    local grpcio_tools_spec="grpcio-tools"
+    local grpcio_version
+    grpcio_version="$(python -c 'import importlib.metadata as m; print(m.version("grpcio"))' 2>/dev/null || true)"
+    if [ -n "${grpcio_version}" ]; then
+        grpcio_tools_spec="grpcio-tools==${grpcio_version}"
+    fi
+
     if [ "x${USE_PIP}" == "xtrue" ]; then
-        pip install grpcio-tools
+        pip install "${grpcio_tools_spec}"
     else
-        uv pip install grpcio-tools
+        uv pip install "${grpcio_tools_spec}"
     fi
     pushd "${EDIR}" >/dev/null
         # use the venv python (ensures correct interpreter & sys.path)

backend/python/common/mlx_utils.py

@@ -20,7 +20,15 @@ def split_reasoning(text, think_start, think_end):
     Returns ``(reasoning_content, remaining_text)``. When ``think_start`` is
     empty or not found, returns ``("", text)`` unchanged.
     """
-    if not think_start or not text or think_start not in text:
+    if not think_start or not text:
+        return "", text
+    if think_start not in text:
+        # Models like Qwen3.5 open assistant turns already INSIDE thinking, so
+        # the generated text carries only the closing tag. Everything before it
+        # is reasoning that would otherwise leak into the content.
+        if think_end and think_end in text:
+            head, _, tail = text.partition(think_end)
+            return head.strip(), tail.strip()
         return "", text
     pattern = re.compile(
         re.escape(think_start) + r"(.*?)" + re.escape(think_end or ""),

backend/python/common/mlx_utils_test.py

@@ -0,0 +1,75 @@
+"""Unit tests for the mlx/mlx-vlm shared helpers (mlx_utils.py).
+
+Run standalone (Python standard library only, no backend venv needed):
+    python3 -m unittest mlx_utils_test
+
+These mirror the server-less helper tests in backend/python/mlx/test.py
+(TestSharedHelpers), but live here so they run on any platform: the mlx
+test module imports grpc/backend_pb2 at import time and needs the MLX venv,
+whereas mlx_utils only needs the standard library.
+"""
+
+import types
+import unittest
+
+from mlx_utils import parse_tool_calls, split_reasoning
+
+
+class TestSplitReasoning(unittest.TestCase):
+    def test_both_tags(self):
+        r, c = split_reasoning(
+            "<think>step 1\nstep 2</think>The answer is 42.", "<think>", "</think>"
+        )
+        self.assertEqual(r, "step 1\nstep 2")
+        self.assertEqual(c, "The answer is 42.")
+
+    def test_implicit_opener_only_closing_tag(self):
+        # Qwen3.5 opens the assistant turn already inside thinking, so the
+        # output carries only the closing tag; everything before it is reasoning.
+        r, c = split_reasoning(
+            "The user is asking about the weather.\n</think>\n\nThe weather in Rome is sunny.",
+            "<think>",
+            "</think>",
+        )
+        self.assertEqual(r, "The user is asking about the weather.")
+        self.assertEqual(c, "The weather in Rome is sunny.")
+
+    def test_no_tags_at_all(self):
+        r, c = split_reasoning("just text", "<think>", "</think>")
+        self.assertEqual(r, "")
+        self.assertEqual(c, "just text")
+
+    def test_empty_think_end_and_no_opener_match(self):
+        # No think_end to anchor on, and the opener is absent → return unchanged.
+        r, c = split_reasoning("no opener here", "<think>", "")
+        self.assertEqual(r, "")
+        self.assertEqual(c, "no opener here")
+
+    def test_empty_text(self):
+        r, c = split_reasoning("", "<think>", "</think>")
+        self.assertEqual(r, "")
+        self.assertEqual(c, "")
+
+
+class TestParseToolCalls(unittest.TestCase):
+    def test_with_shim(self):
+        tm = types.SimpleNamespace(
+            tool_call_start="<tool_call>",
+            tool_call_end="</tool_call>",
+            parse_tool_call=lambda body, tools: {
+                "name": "get_weather",
+                "arguments": {"location": body.strip()},
+            },
+        )
+        calls, remaining = parse_tool_calls(
+            "Sure: <tool_call>Paris</tool_call>", tm, tools=None
+        )
+        self.assertEqual(len(calls), 1)
+        self.assertEqual(calls[0]["name"], "get_weather")
+        self.assertEqual(calls[0]["arguments"], '{"location": "Paris"}')
+        self.assertEqual(calls[0]["index"], 0)
+        self.assertNotIn("<tool_call>", remaining)
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/common/parent_watch.py

@@ -0,0 +1,149 @@
+"""Parent-death watcher (best-effort backstop) for LocalAI Python backends.
+
+LocalAI spawns each backend as a child process and, on a clean shutdown, tears
+it down itself (SIGTERM -> grace -> SIGKILL). That graceful path only runs when
+LocalAI receives a catchable signal and lives long enough to run its handlers.
+If LocalAI is SIGKILLed (e.g. a supervising process's grace period elapses
+first), that teardown never runs and this backend would be reparented to init
+and linger, holding GPU/VRAM and its listen port.
+
+The watcher here is a best-effort backstop for exactly that case: it does NOT
+replace the graceful teardown, it only covers the "parent vanished without
+cleaning up" path. It detects reparenting: when the process that spawned this
+backend dies, the kernel reparents us to the nearest sub-reaper or to init
+(PID 1), so os.getppid() stops matching the value captured at startup. This
+getppid() approach is portable across Linux/macOS (unlike the Linux-only
+PR_SET_PDEATHSIG), which is why it is used here, mirroring the Go backends'
+pkg/grpc/parentwatch.go and the C++ backends' parent_watch.h. It is disabled on
+Windows, which has no equivalent orphan-reparenting semantics.
+
+Env vars (shared verbatim across the Go, C++ and Python backends):
+  LOCALAI_BACKEND_PARENT_WATCH           enabled by default; a falsey value
+                                         ("false"/"0"/"no"/"off", case-insensitive)
+                                         disables it.
+  LOCALAI_BACKEND_PARENT_WATCH_INTERVAL  poll interval as a Go-style duration
+                                         string ("500ms", "2s", "1m") or a bare
+                                         number of seconds. Defaults to 2s.
+"""
+
+import os
+import sys
+import threading
+
+ENV_PARENT_WATCH = "LOCALAI_BACKEND_PARENT_WATCH"
+ENV_PARENT_WATCH_INTERVAL = "LOCALAI_BACKEND_PARENT_WATCH_INTERVAL"
+
+_DEFAULT_INTERVAL_SECONDS = 2.0
+
+# Guard so repeated calls (e.g. get_auth_interceptors invoked more than once)
+# only ever arm a single watcher thread per process.
+_started = False
+_started_lock = threading.Lock()
+
+
+def _enabled():
+    """Report whether the watcher should run in this process."""
+    # Windows does not reparent orphans to a well-known init PID, so the
+    # getppid() heuristic used here doesn't apply there.
+    if os.name == "nt" or sys.platform.startswith("win"):
+        return False
+    val = os.environ.get(ENV_PARENT_WATCH, "").strip().lower()
+    if val in ("false", "0", "no", "off"):
+        return False
+    return True
+
+
+def _interval_seconds():
+    """Return the configured poll interval in seconds, or the default.
+
+    Accepts Go-style duration strings ("500ms", "2s", "1m") for cross-language
+    parity, or a bare number interpreted as seconds.
+    """
+    raw = os.environ.get(ENV_PARENT_WATCH_INTERVAL, "").strip()
+    if not raw:
+        return _DEFAULT_INTERVAL_SECONDS
+    # Split numeric prefix from unit suffix.
+    i = 0
+    while i < len(raw) and (raw[i].isdigit() or raw[i] == "." or (i == 0 and raw[i] in "+-")):
+        i += 1
+    if i == 0:
+        return _DEFAULT_INTERVAL_SECONDS
+    try:
+        num = float(raw[:i])
+    except ValueError:
+        return _DEFAULT_INTERVAL_SECONDS
+    unit = raw[i:].lower()
+    if unit == "ms":
+        seconds = num / 1000.0
+    elif unit in ("s", ""):
+        seconds = num
+    elif unit == "m":
+        seconds = num * 60.0
+    else:
+        return _DEFAULT_INTERVAL_SECONDS
+    return seconds if seconds > 0 else _DEFAULT_INTERVAL_SECONDS
+
+
+def _parent_died(orig_ppid):
+    """Report whether this process has been reparented away from orig_ppid.
+
+    Reparenting is the standard POSIX signal that the original parent (here, the
+    LocalAI process that spawned this backend) has exited: the orphan is handed
+    to the nearest sub-reaper or to init (PID 1), so os.getppid() no longer
+    matches the value captured at startup.
+    """
+    ppid = os.getppid()
+    return ppid != orig_ppid or ppid == 1
+
+
+def _watch(orig_ppid, interval, on_death):
+    """Poll until _parent_died reports the original parent is gone, then call
+    on_death. Blocks, so run it on its own (daemon) thread."""
+    import time
+
+    while True:
+        time.sleep(interval)
+        if _parent_died(orig_ppid):
+            on_death()
+            return
+
+
+def start_parent_death_watcher():
+    """Install the best-effort safety net described in this module's docstring.
+
+    No-op when disabled, on Windows, when already orphaned at startup
+    (os.getppid() <= 1), or if already started. This is a backstop alongside —
+    never a replacement for — LocalAI's graceful teardown.
+    """
+    global _started
+    if not _enabled():
+        return
+    with _started_lock:
+        if _started:
+            return
+        orig_ppid = os.getppid()
+        # A parent of 1 (or less) at startup means we were already orphaned (or
+        # launched directly under init) — there is no original parent to watch.
+        if orig_ppid <= 1:
+            return
+        interval = _interval_seconds()
+
+        def on_death():
+            print(
+                "backend parent process (pid {}) exited without stopping this "
+                "backend; self-terminating to avoid orphaning".format(orig_ppid),
+                file=sys.stderr,
+                flush=True,
+            )
+            # Immediate, non-cleanup exit: this is a shutdown safety net and the
+            # normal graceful path is already gone.
+            os._exit(1)
+
+        thread = threading.Thread(
+            target=_watch,
+            args=(orig_ppid, interval, on_death),
+            name="parent-death-watcher",
+            daemon=True,
+        )
+        thread.start()
+        _started = True

backend/python/common/parent_watch_test.py

@@ -0,0 +1,150 @@
+"""Unit tests for the parent-death watcher (parent_watch.py).
+
+Run standalone (Python standard library only, no backend venv needed):
+    python3 -m unittest parent_watch_test
+
+The core test (test_detects_reparent) builds a genuine two-level process tree
+(test -> middle -> grandchild) with os.fork, lets the middle process die, and
+asserts the grandchild's parent_watch._watch detects the reparenting and
+self-terminates — mirroring the Go test in pkg/grpc/parentwatch_test.go and the
+C++ test in backend/cpp/llama-cpp/parent_watch_test.cpp.
+"""
+
+import os
+import sys
+import tempfile
+import threading
+import time
+import unittest
+
+import parent_watch
+
+
+class TestParentWatchEnvParsing(unittest.TestCase):
+    def setUp(self):
+        self._saved = {
+            k: os.environ.get(k)
+            for k in (parent_watch.ENV_PARENT_WATCH, parent_watch.ENV_PARENT_WATCH_INTERVAL)
+        }
+        for k in self._saved:
+            os.environ.pop(k, None)
+
+    def tearDown(self):
+        for k, v in self._saved.items():
+            if v is None:
+                os.environ.pop(k, None)
+            else:
+                os.environ[k] = v
+
+    def test_interval_default(self):
+        self.assertEqual(parent_watch._interval_seconds(), 2.0)
+
+    def test_interval_units(self):
+        cases = {"500ms": 0.5, "2s": 2.0, "1m": 60.0, "3": 3.0, "0.5s": 0.5}
+        for raw, expected in cases.items():
+            os.environ[parent_watch.ENV_PARENT_WATCH_INTERVAL] = raw
+            self.assertAlmostEqual(parent_watch._interval_seconds(), expected, msg=raw)
+
+    def test_interval_garbage_falls_back(self):
+        os.environ[parent_watch.ENV_PARENT_WATCH_INTERVAL] = "garbage"
+        self.assertEqual(parent_watch._interval_seconds(), 2.0)
+
+    @unittest.skipIf(os.name == "nt" or sys.platform.startswith("win"), "POSIX only")
+    def test_enabled_default(self):
+        self.assertTrue(parent_watch._enabled())
+
+    @unittest.skipIf(os.name == "nt" or sys.platform.startswith("win"), "POSIX only")
+    def test_disabled_by_falsey(self):
+        for val in ("false", "0", "no", "off", "OFF", " False "):
+            os.environ[parent_watch.ENV_PARENT_WATCH] = val
+            self.assertFalse(parent_watch._enabled(), msg=val)
+
+    @unittest.skipIf(os.name == "nt" or sys.platform.startswith("win"), "POSIX only")
+    def test_enabled_by_truthy(self):
+        for val in ("true", "1", "yes", "on"):
+            os.environ[parent_watch.ENV_PARENT_WATCH] = val
+            self.assertTrue(parent_watch._enabled(), msg=val)
+
+
+@unittest.skipIf(os.name == "nt" or sys.platform.startswith("win"), "fork/reparent is POSIX only")
+class TestParentWatchReparent(unittest.TestCase):
+    def _wait_for_file(self, path, timeout=10.0):
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            if os.path.exists(path):
+                return True
+            time.sleep(0.02)
+        return False
+
+    def test_detects_reparent(self):
+        tmpdir = tempfile.mkdtemp(prefix="parentwatch_test_")
+        ready_file = os.path.join(tmpdir, "ready")
+        exited_file = os.path.join(tmpdir, "exited")
+
+        middle = os.fork()
+        if middle == 0:
+            # ---- middle process ----
+            grandchild = os.fork()
+            if grandchild == 0:
+                # ---- grandchild process: arm the REAL watcher against middle ----
+                orig_ppid = os.getppid()
+
+                def on_death():
+                    with open(exited_file, "w") as f:
+                        f.write("1")
+                    os._exit(7)
+
+                threading.Thread(
+                    target=parent_watch._watch,
+                    args=(orig_ppid, 0.05, on_death),
+                    daemon=True,
+                ).start()
+
+                # Safety valve: never linger if something goes wrong.
+                def bail():
+                    time.sleep(30)
+                    os._exit(2)
+
+                threading.Thread(target=bail, daemon=True).start()
+
+                # Signal readiness only after the watcher captured orig_ppid.
+                with open(ready_file, "w") as f:
+                    f.write(str(os.getpid()))
+                while True:
+                    time.sleep(1)
+            else:
+                # middle: wait until grandchild is ready, then exit to orphan it.
+                if not self._wait_for_file(ready_file):
+                    os._exit(5)
+                os._exit(0)
+
+        # ---- test (top) process ----
+        os.waitpid(middle, 0)  # reap middle only; grandchild is orphaned
+
+        self.assertTrue(os.path.exists(ready_file), "grandchild never signaled readiness")
+        self.assertTrue(
+            self._wait_for_file(exited_file),
+            "watcher did not detect parent death within timeout",
+        )
+
+        # Best-effort cleanup: kill the grandchild if it somehow survived.
+        try:
+            with open(ready_file) as f:
+                pid = int(f.read().strip())
+            if pid > 1:
+                os.kill(pid, 9)
+        except (OSError, ValueError):
+            pass
+        for p in (ready_file, exited_file):
+            try:
+                os.remove(p)
+            except OSError:
+                pass
+        try:
+            os.rmdir(tmpdir)
+        except OSError:
+            pass
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/common/python_utils.py

@@ -58,7 +58,18 @@ def messages_to_dicts(proto_messages):
             d["reasoning_content"] = msg.reasoning_content
         if msg.tool_calls:
             try:
-                d["tool_calls"] = json.loads(msg.tool_calls)
+                tool_calls = json.loads(msg.tool_calls)
+                # Chat templates (e.g. Qwen) iterate function.arguments as a
+                # mapping, but the OpenAI wire format carries it as a JSON
+                # string — decode it back so the template's .items() works.
+                for tc in tool_calls:
+                    fn = tc.get("function") if isinstance(tc, dict) else None
+                    if isinstance(fn, dict) and isinstance(fn.get("arguments"), str):
+                        try:
+                            fn["arguments"] = json.loads(fn["arguments"])
+                        except json.JSONDecodeError:
+                            pass
+                d["tool_calls"] = tool_calls
             except json.JSONDecodeError:
                 pass
         result.append(d)

backend/python/common/python_utils_test.py

@@ -0,0 +1,122 @@
+"""Unit tests for the shared python backend helpers (python_utils.py).
+
+Run standalone (Python standard library only, no backend venv needed):
+    python3 -m unittest python_utils_test
+
+These mirror the server-less helper tests in backend/python/mlx/test.py
+(TestSharedHelpers), but live here so they run on any platform: the mlx
+test module imports grpc/backend_pb2 at import time and needs the MLX venv,
+whereas python_utils has no third-party dependency. Proto Message objects
+are faked with types.SimpleNamespace (real proto fields default to "").
+"""
+
+import json
+import types
+import unittest
+
+from python_utils import messages_to_dicts, parse_options
+
+
+def _msg(**fields):
+    """Fake a proto Message: every unset field is the empty string, as protobuf."""
+    defaults = {
+        "role": "",
+        "content": "",
+        "name": "",
+        "tool_call_id": "",
+        "reasoning_content": "",
+        "tool_calls": "",
+    }
+    defaults.update(fields)
+    return types.SimpleNamespace(**defaults)
+
+
+class TestParseOptions(unittest.TestCase):
+    def test_type_inference(self):
+        opts = parse_options(
+            ["temperature:0.7", "max_tokens:128", "trust:true", "name:hello", "no_colon_skipped"]
+        )
+        self.assertEqual(opts["temperature"], 0.7)
+        self.assertEqual(opts["max_tokens"], 128)
+        self.assertIs(opts["trust"], True)
+        self.assertEqual(opts["name"], "hello")
+        self.assertNotIn("no_colon_skipped", opts)
+
+
+class TestMessagesToDicts(unittest.TestCase):
+    def test_basic_fields(self):
+        out = messages_to_dicts(
+            [
+                _msg(role="user", content="hi"),
+                _msg(role="tool", content="42", tool_call_id="call_1", name="f"),
+            ]
+        )
+        self.assertEqual(out[0], {"role": "user", "content": "hi"})
+        self.assertEqual(out[1]["tool_call_id"], "call_1")
+        self.assertEqual(out[1]["name"], "f")
+
+    def test_tool_call_arguments_string_decoded_to_mapping(self):
+        # OpenAI wire format ships function.arguments as a JSON *string*; chat
+        # templates iterate it as a mapping, so it must come back as a dict.
+        out = messages_to_dicts(
+            [
+                _msg(
+                    role="assistant",
+                    tool_calls=json.dumps(
+                        [
+                            {
+                                "id": "call_1",
+                                "type": "function",
+                                "function": {
+                                    "name": "get_weather",
+                                    "arguments": '{"location": "Rome"}',
+                                },
+                            }
+                        ]
+                    ),
+                )
+            ]
+        )
+        args = out[0]["tool_calls"][0]["function"]["arguments"]
+        self.assertEqual(args, {"location": "Rome"})
+        self.assertEqual(dict(args.items()), {"location": "Rome"})
+
+    def test_tool_call_arguments_already_mapping_is_idempotent(self):
+        out = messages_to_dicts(
+            [
+                _msg(
+                    role="assistant",
+                    tool_calls=json.dumps(
+                        [{"function": {"name": "f", "arguments": {"a": 1}}}]
+                    ),
+                )
+            ]
+        )
+        self.assertEqual(out[0]["tool_calls"][0]["function"]["arguments"], {"a": 1})
+
+    def test_tool_call_arguments_invalid_json_left_as_string(self):
+        out = messages_to_dicts(
+            [
+                _msg(
+                    role="assistant",
+                    tool_calls=json.dumps(
+                        [{"function": {"name": "f", "arguments": "not-json"}}]
+                    ),
+                )
+            ]
+        )
+        self.assertEqual(out[0]["tool_calls"][0]["function"]["arguments"], "not-json")
+
+    def test_tool_call_without_function_key(self):
+        out = messages_to_dicts(
+            [_msg(role="assistant", tool_calls=json.dumps([{"id": "call_1"}]))]
+        )
+        self.assertEqual(out[0]["tool_calls"], [{"id": "call_1"}])
+
+    def test_tool_calls_invalid_json_dropped(self):
+        out = messages_to_dicts([_msg(role="assistant", tool_calls="{not json")])
+        self.assertNotIn("tool_calls", out[0])
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/python/coqui/backend.py

@@ -79,14 +79,19 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             if self.tts.is_multi_lingual and lang is None:
                return backend_pb2.Result(success=False, message=f"Model is multi-lingual, but no language was provided")
 
-            # if model is multi-speaker, use speaker_wav or the speaker_id from request.voice
-            if self.tts.is_multi_speaker and self.AudioPath is None and request.voice is None:
+            # A path-shaped per-request voice is a cloning reference; otherwise
+            # preserve Coqui's named-speaker behavior.
+            request_voice = request.voice if request.voice else ""
+            speaker_wav = request_voice if os.path.isfile(request_voice) else self.AudioPath
+            if self.tts.is_multi_speaker and speaker_wav is None and not request_voice:
                 return backend_pb2.Result(success=False, message=f"Model is multi-speaker, but no speaker was provided")
 
-            if self.tts.is_multi_speaker and request.voice is not None:
-               self.tts.tts_to_file(text=request.text, speaker=request.voice, language=lang, file_path=request.dst)
+            if speaker_wav is not None:
+                self.tts.tts_to_file(text=request.text, speaker_wav=speaker_wav, language=lang, file_path=request.dst)
+            elif self.tts.is_multi_speaker and request_voice:
+               self.tts.tts_to_file(text=request.text, speaker=request_voice, language=lang, file_path=request.dst)
             else:
-                self.tts.tts_to_file(text=request.text, speaker_wav=self.AudioPath, language=lang, file_path=request.dst)
+                self.tts.tts_to_file(text=request.text, language=lang, file_path=request.dst)
         except Exception as err:
             return backend_pb2.Result(success=False, message=f"Unexpected {err=}, {type(err)=}")
         return backend_pb2.Result(success=True)

backend/python/diffusers/backend.py

@@ -864,8 +864,13 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 **kwargs
             ).images[0]
 
-        # save the result
-        image.save(request.dst)
+        # save the result. Save as PNG explicitly instead of letting Pillow
+        # infer the encoder from the extension: the core passes an absolute
+        # staging path ending in .tmp (e.g. /staging/localai-output-*.tmp),
+        # which Pillow can't map to a format and would raise
+        # "unknown file extension: .tmp". LocalAI serves generated images as
+        # PNG regardless of the temp path.
+        image.save(request.dst, format="PNG")
 
         return backend_pb2.Result(message="Media generated", success=True)
 

backend/python/faster-qwen3-tts/backend.py

@@ -83,20 +83,23 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
         return backend_pb2.Result(message="Model loaded successfully", success=True)
 
     def _get_ref_audio_path(self, request):
-        if not self.audio_path:
+        # A per-request voice path is the canonical LocalAI voice-profile
+        # contract. Keep AudioPath as the backwards-compatible YAML fallback.
+        audio_path = request.voice if hasattr(request, "voice") and request.voice else self.audio_path
+        if not audio_path:
             return None
-        if os.path.isabs(self.audio_path):
-            return self.audio_path
+        if os.path.isabs(audio_path):
+            return audio_path
         if self.model_file:
             model_file_base = os.path.dirname(self.model_file)
-            ref_path = os.path.join(model_file_base, self.audio_path)
+            ref_path = os.path.join(model_file_base, audio_path)
             if os.path.exists(ref_path):
                 return ref_path
         if self.model_path:
-            ref_path = os.path.join(self.model_path, self.audio_path)
+            ref_path = os.path.join(self.model_path, audio_path)
             if os.path.exists(ref_path):
                 return ref_path
-        return self.audio_path
+        return audio_path
 
     def TTS(self, request, context):
         try:
@@ -122,13 +125,13 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                     success=False,
                     message="AudioPath is required for voice clone (set in LoadModel)"
                 )
-            ref_text = self.options.get("ref_text")
-            if not ref_text and hasattr(request, 'ref_text') and request.ref_text:
-                ref_text = request.ref_text
+            ref_text = request.params.get("ref_text") if hasattr(request, "params") else None
+            if not ref_text:
+                ref_text = self.options.get("ref_text")
             if not ref_text:
                 return backend_pb2.Result(
                     success=False,
-                    message="ref_text is required for voice clone (set via LoadModel Options, e.g. ref_text:Your reference transcript)"
+                    message="ref_text is required for voice clone (set in request.params or LoadModel options)"
                 )
 
             chunk_size = self.options.get("chunk_size")

backend/python/fish-speech/backend.py

@@ -267,6 +267,8 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
     def _get_ref_audio_path(self, voice_name=None):
         """Get reference audio path from voices dict or stored AudioPath."""
+        if voice_name and os.path.isfile(voice_name):
+            return voice_name
         if voice_name and voice_name in self.voices:
             audio_path = self.voices[voice_name]["audio"]
 
@@ -332,7 +334,19 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
             references = []
             voice_name = request.voice if request.voice else None
 
-            if voice_name and voice_name in self.voices:
+            if voice_name and os.path.isfile(voice_name):
+                ref_audio_path = self._get_ref_audio_path(voice_name)
+                with open(ref_audio_path, "rb") as f:
+                    audio_bytes = f.read()
+                ref_text = request.params.get("ref_text", "") if hasattr(request, "params") else ""
+                references.append(
+                    ServeReferenceAudio(audio=audio_bytes, text=ref_text)
+                )
+                print(
+                    f"[INFO] Using per-request reference audio: {ref_audio_path}",
+                    file=sys.stderr,
+                )
+            elif voice_name and voice_name in self.voices:
                 ref_audio_path = self._get_ref_audio_path(voice_name)
                 if ref_audio_path and os.path.exists(ref_audio_path):
                     with open(ref_audio_path, "rb") as f:
@@ -350,7 +364,9 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
                 if ref_audio_path and os.path.exists(ref_audio_path):
                     with open(ref_audio_path, "rb") as f:
                         audio_bytes = f.read()
-                    ref_text = self.options.get("ref_text", "")
+                    ref_text = request.params.get("ref_text", "") if hasattr(request, "params") else ""
+                    if not ref_text:
+                        ref_text = self.options.get("ref_text", "")
                     references.append(
                         ServeReferenceAudio(audio=audio_bytes, text=ref_text)
                     )

backend/python/fish-speech/install.sh

@@ -13,6 +13,17 @@ fi
 # fish-speech uses pyrootutils which requires a .project-root marker
 touch "${backend_dir}/.project-root"
 
+# On darwin arm64 the transitive `tokenizers` dep compiles its Rust extension
+# from source (Linux uses prebuilt manylinux wheels, so it never compiles
+# there). The pinned tokenizers crate that fish-speech's stack resolves to
+# contains a `&T` -> `&mut T` cast that trips the now-deny-by-default
+# `invalid_reference_casting` lint in the macOS runner's newer Rust toolchain,
+# breaking the build (seen in the v4.5.5 release CI fish-speech darwin/metal
+# job). Allow that lint so the unchanged third-party crate compiles as before.
+# Append rather than clobber any pre-existing RUSTFLAGS; harmless on Linux
+# where no Rust compile happens.
+export RUSTFLAGS="${RUSTFLAGS:-} -A invalid_reference_casting"
+
 installRequirements
 
 # Clone fish-speech source (the pip package doesn't include inference modules)

backend/python/kokoro/requirements.txt

@@ -3,4 +3,5 @@ protobuf
 certifi
 packaging==24.1
 pip
-chardet
+chardet
+click

backend/python/longcat-video/.gitignore

@@ -0,0 +1,6 @@
+backend_pb2.py
+backend_pb2_grpc.py
+lib/
+python/
+sources/
+venv/

backend/python/longcat-video/Makefile

@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: MIT
+
+LONGCAT_VIDEO_VERSION?=6b3f4b8582a8bc3f20f795735f5383716c4ba794
+LONGCAT_VIDEO_REPO?=https://github.com/meituan-longcat/LongCat-Video
+LONGCAT_SOURCE_STAMP=sources/LongCat-Video/.localai-$(LONGCAT_VIDEO_VERSION)
+
+.PHONY: all
+all: $(LONGCAT_SOURCE_STAMP)
+	bash install.sh
+
+$(LONGCAT_SOURCE_STAMP): patches/0001-sdpa-attention-fallback.patch
+	rm -rf sources/LongCat-Video
+	mkdir -p sources/LongCat-Video
+	cd sources/LongCat-Video && git init -q && \
+		git remote add origin $(LONGCAT_VIDEO_REPO) && \
+		git fetch --depth 1 origin $(LONGCAT_VIDEO_VERSION) && \
+		git checkout --detach FETCH_HEAD && \
+		git apply ../../patches/0001-sdpa-attention-fallback.patch && \
+		rm -rf .git && \
+		touch .localai-$(LONGCAT_VIDEO_VERSION)
+
+.PHONY: run
+run: all
+	bash run.sh
+
+.PHONY: test
+test: all
+	bash test.sh
+
+.PHONY: protogen-clean
+protogen-clean:
+	$(RM) backend_pb2.py backend_pb2_grpc.py
+
+.PHONY: clean
+clean: protogen-clean
+	rm -rf __pycache__ lib python sources venv

backend/python/longcat-video/README.md

@@ -0,0 +1,44 @@
+# LongCat Video backend
+
+This backend serves Meituan's `LongCat-Video` and
+`LongCat-Video-Avatar-1.5` checkpoints through LocalAI's `GenerateVideo`
+RPC. It supports:
+
+- text-to-video and image-to-video with `LongCat-Video`;
+- audio + text-to-avatar and portrait + audio-to-avatar with Avatar 1.5;
+- multi-segment avatar continuation for speech longer than one segment;
+- PyTorch SDPA when FlashAttention is unavailable, including CUDA 13 ARM64
+  systems such as NVIDIA DGX Spark.
+
+Install the `longcat-video` or `longcat-video-avatar-1.5` recipe from the
+LocalAI Model Gallery. See the [LongCat user guide](../../../docs/content/features/longcat-video.md)
+for Studio and API examples, hardware requirements, and manual configuration.
+
+The upstream source is pinned in `Makefile` and patched at build time. The
+patch adds only the missing SDPA attention branches; model and source licenses
+remain MIT.
+
+## Model options
+
+| Option | Default | Description |
+| --- | --- | --- |
+| `attention_backend` | `sdpa` | `sdpa`, `auto`, `flash2`, `flash3`, or `xformers`. The packaged backend guarantees only `sdpa`. |
+| `use_distill` | `true` for Avatar, `false` for base | Loads the checkpoint's fast distillation LoRA. |
+| `use_int8` | `false` | Loads Avatar 1.5's INT8 DiT. BF16 has a lower load-time peak on unified-memory systems. |
+| `base_model` | `meituan-longcat/LongCat-Video` | Base components used by Avatar 1.5. |
+| `max_segments` | `8` | Maximum avatar continuation segments accepted per request. |
+| `resolution` | `480p` | Image-conditioned generation resolution (`480p` or `720p`). |
+
+Per-request `params` may set `num_segments`, `audio_guidance_scale`,
+`offload_kv_cache`, `ref_img_index`, `mask_frame_range`, and `resolution`.
+
+Gallery and imported configs declare `known_input_modalities` and
+`known_output_modalities`. Keep those declarations in manual configs as well;
+they let model discovery distinguish base image-conditioned video from Avatar
+audio conditioning without inspecting the backend or checkpoint name.
+
+LongCat is CUDA-only and very large. Avatar 1.5 also loads tokenizer,
+text-encoder, and VAE components from the base checkpoint. Keep ample unified
+memory and storage available; no CPU or macOS backend image is published. The
+initial backend supports one GPU per process; tensor parallel sizes above one
+are rejected explicitly.

backend/python/longcat-video/backend.py

@@ -0,0 +1,916 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: MIT
+
+import argparse
+import datetime
+import gc
+import math
+import os
+import signal
+import subprocess
+import sys
+import tempfile
+import traceback
+from concurrent import futures
+
+import grpc
+
+import backend_pb2
+import backend_pb2_grpc
+
+from longcat_utils import (
+    BASE_MODEL_ID,
+    MODEL_KIND_AVATAR,
+    MODEL_KIND_BASE,
+    attention_overrides,
+    avatar_segments_for_duration,
+    avatar_segments_for_frames,
+    classify_model,
+    normalize_model_source,
+    normalize_num_frames,
+    parse_options,
+    require_bool,
+    require_float,
+    require_int,
+    select_known_options,
+    validate_dimensions,
+)
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "common"))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "common"))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "sources", "LongCat-Video"))
+
+from grpc_auth import get_auth_interceptors
+
+
+MAX_WORKERS = int(os.environ.get("PYTHON_GRPC_MAX_WORKERS", "1"))
+
+DEFAULT_NEGATIVE_PROMPT = (
+    "Close-up, bright tones, overexposed, static, blurred details, subtitles, "
+    "paintings, low quality, JPEG compression residue, ugly, incomplete, extra "
+    "fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, "
+    "misshapen limbs, fused fingers, still picture, messy background, three legs, "
+    "many people in the background, walking backwards"
+)
+
+LOAD_OPTIONS = {
+    "attention_backend",
+    "base_model",
+    "max_segments",
+    "resolution",
+    "use_distill",
+    "use_int8",
+}
+
+REQUEST_PARAMS = {
+    "audio_guidance_scale",
+    "mask_frame_range",
+    "num_segments",
+    "offload_kv_cache",
+    "ref_img_index",
+    "resolution",
+}
+
+BASE_CHECKPOINT_PATTERNS = [
+    "config.json",
+    "model_index.json",
+    "dit/**",
+    "lora/cfg_step_lora.safetensors",
+    "scheduler/**",
+    "text_encoder/**",
+    "tokenizer/**",
+    "vae/**",
+]
+
+AVATAR_BASE_PATTERNS = [
+    "config.json",
+    "model_index.json",
+    "text_encoder/**",
+    "tokenizer/**",
+    "vae/**",
+]
+
+AVATAR_COMMON_PATTERNS = [
+    "config.json",
+    "lora/dmd_lora.safetensors",
+    "model_index.json",
+    "scheduler/**",
+    "whisper-large-v3/config.json",
+    "whisper-large-v3/model.safetensors",
+    "whisper-large-v3/preprocessor_config.json",
+]
+
+
+class BackendServicer(backend_pb2_grpc.BackendServicer):
+    def __init__(self):
+        self.model_kind = None
+        self.pipeline = None
+        self.options = {}
+        self.device_index = 0
+        self.cp_split_hw = None
+        self._dist_store_dir = None
+
+    def Health(self, request, context):
+        return backend_pb2.Reply(message=b"OK")
+
+    def LoadModel(self, request, context):
+        model = request.Model
+        if request.ModelFile and os.path.isdir(request.ModelFile):
+            model = request.ModelFile
+
+        model_kind = classify_model(model)
+        if model_kind is None:
+            return self._fail(
+                context,
+                grpc.StatusCode.INVALID_ARGUMENT,
+                "longcat-video only accepts LongCat-Video or LongCat-Video-Avatar-1.5 checkpoints",
+            )
+
+        try:
+            options, ignored = select_known_options(
+                parse_options(request.Options), LOAD_OPTIONS
+            )
+            if ignored:
+                # The server injects llama.cpp serving defaults (cache_reuse,
+                # parallel) onto every config; ignore what we don't understand
+                # rather than fail to load.
+                print(
+                    f"longcat-video ignoring unknown model option(s): {', '.join(ignored)}",
+                    file=sys.stderr,
+                )
+
+            self._import_torch()
+            if not self.torch.cuda.is_available():
+                return self._fail(
+                    context,
+                    grpc.StatusCode.FAILED_PRECONDITION,
+                    "longcat-video requires an NVIDIA CUDA GPU",
+                )
+            if request.TensorParallelSize > 1:
+                return self._fail(
+                    context,
+                    grpc.StatusCode.UNIMPLEMENTED,
+                    "longcat-video currently supports one GPU per backend process",
+                )
+            self._import_runtime()
+
+            attention_name = str(options.get("attention_backend", "sdpa")).lower()
+            attention_overrides(attention_name)
+            resolution = str(options.get("resolution", "480p")).lower()
+            if resolution not in {"480p", "720p"}:
+                raise ValueError("resolution must be 480p or 720p")
+
+            use_distill_default = model_kind == MODEL_KIND_AVATAR
+            use_distill = require_bool(
+                options.get("use_distill", use_distill_default),
+                "use_distill",
+            )
+            use_int8 = require_bool(options.get("use_int8", False), "use_int8")
+            if model_kind == MODEL_KIND_BASE and use_int8:
+                raise ValueError(
+                    "use_int8 is supported only by LongCat-Video-Avatar-1.5"
+                )
+
+            self.options = {
+                **options,
+                "attention_backend": attention_name,
+                "resolution": resolution,
+                "use_distill": use_distill,
+                "use_int8": use_int8,
+                "max_segments": require_int(
+                    options.get("max_segments", 8),
+                    "max_segments",
+                    minimum=1,
+                    maximum=64,
+                ),
+            }
+
+            self._release_model()
+            self._ensure_distributed()
+            if model_kind == MODEL_KIND_BASE:
+                self._load_base_model(model)
+            else:
+                self._load_avatar_model(model)
+            self.model_kind = model_kind
+            print(
+                f"Loaded {normalize_model_source(model)} as {model_kind} "
+                f"with attention_backend={attention_name}",
+                file=sys.stderr,
+            )
+            return backend_pb2.Result(message="Model loaded successfully", success=True)
+        except ValueError as err:
+            self._release_model()
+            return self._fail(context, grpc.StatusCode.INVALID_ARGUMENT, str(err))
+        except Exception as err:
+            self._release_model()
+            print(f"Error loading LongCat model: {err}", file=sys.stderr)
+            traceback.print_exc()
+            return self._fail(
+                context,
+                grpc.StatusCode.INTERNAL,
+                f"failed to load LongCat model: {err}",
+            )
+
+    def Free(self, request, context):
+        self._release_model()
+        return backend_pb2.Result(message="Model released", success=True)
+
+    def GenerateVideo(self, request, context):
+        if self.pipeline is None or self.model_kind is None:
+            return self._fail(
+                context,
+                grpc.StatusCode.FAILED_PRECONDITION,
+                "model is not loaded",
+            )
+        if not request.prompt.strip():
+            return self._fail(
+                context,
+                grpc.StatusCode.INVALID_ARGUMENT,
+                "prompt is required",
+            )
+        if not request.dst:
+            return self._fail(
+                context,
+                grpc.StatusCode.INVALID_ARGUMENT,
+                "output destination is required",
+            )
+        if request.end_image:
+            return self._fail(
+                context,
+                grpc.StatusCode.INVALID_ARGUMENT,
+                "longcat-video does not support end_image conditioning",
+            )
+
+        request_state = {"finished": False}
+
+        def interrupt_if_cancelled():
+            if not request_state["finished"] and self.pipeline is not None:
+                self.pipeline._interrupt = True
+
+        try:
+            params, ignored_params = select_known_options(
+                dict(request.params), REQUEST_PARAMS
+            )
+            if ignored_params:
+                print(
+                    f"longcat-video ignoring unknown request param(s): {', '.join(ignored_params)}",
+                    file=sys.stderr,
+                )
+
+            os.makedirs(os.path.dirname(request.dst) or ".", mode=0o750, exist_ok=True)
+            if hasattr(context, "add_callback"):
+                context.add_callback(interrupt_if_cancelled)
+
+            if request.start_image and not os.path.isfile(request.start_image):
+                raise ValueError("start_image is not a readable staged file")
+            if request.num_frames < 0:
+                raise ValueError("num_frames must not be negative")
+
+            if self.model_kind == MODEL_KIND_BASE:
+                if request.audio:
+                    raise ValueError(
+                        "audio input requires a LongCat-Video-Avatar-1.5 model"
+                    )
+                self._generate_base(request, params)
+            else:
+                self._generate_avatar(request, params, context)
+
+            return backend_pb2.Result(
+                message="Video generated successfully", success=True
+            )
+        except ValueError as err:
+            return self._fail(context, grpc.StatusCode.INVALID_ARGUMENT, str(err))
+        except Exception as err:
+            print(f"Error generating LongCat video: {err}", file=sys.stderr)
+            traceback.print_exc()
+            return self._fail(
+                context,
+                grpc.StatusCode.INTERNAL,
+                f"LongCat video generation failed: {err}",
+            )
+        finally:
+            request_state["finished"] = True
+            if self.pipeline is not None:
+                self.pipeline._interrupt = False
+
+    def _import_torch(self):
+        if hasattr(self, "torch"):
+            return
+
+        import torch
+
+        self.torch = torch
+
+    def _import_runtime(self):
+        if hasattr(self, "LongCatVideoPipeline"):
+            return
+
+        import imageio.v2 as imageio
+        import imageio_ffmpeg
+        import librosa
+        import numpy as np
+        import torch.distributed as dist
+        from diffusers.utils import load_image
+        from huggingface_hub import snapshot_download
+        from PIL import Image
+        from transformers import AutoTokenizer, UMT5EncoderModel
+
+        from longcat_video.audio_process import (
+            get_audio_encoder,
+            get_audio_feature_extractor,
+        )
+        from longcat_video.context_parallel import context_parallel_util
+        from longcat_video.modules.autoencoder_kl_wan import AutoencoderKLWan
+        from longcat_video.modules.avatar.longcat_video_dit_avatar import (
+            LongCatVideoAvatarTransformer3DModel,
+        )
+        from longcat_video.modules.longcat_video_dit import (
+            LongCatVideoTransformer3DModel,
+        )
+        from longcat_video.modules.quantization import load_quantized_dit
+        from longcat_video.modules.scheduling_flow_match_euler_discrete import (
+            FlowMatchEulerDiscreteScheduler,
+        )
+        from longcat_video.pipeline_longcat_video import LongCatVideoPipeline
+        from longcat_video.pipeline_longcat_video_avatar import (
+            LongCatVideoAvatarPipeline,
+        )
+
+        self.imageio = imageio
+        self.imageio_ffmpeg = imageio_ffmpeg
+        self.librosa = librosa
+        self.np = np
+        self.dist = dist
+        self.load_image = load_image
+        self.snapshot_download = snapshot_download
+        self.Image = Image
+        self.AutoTokenizer = AutoTokenizer
+        self.UMT5EncoderModel = UMT5EncoderModel
+        self.get_audio_encoder = get_audio_encoder
+        self.get_audio_feature_extractor = get_audio_feature_extractor
+        self.context_parallel_util = context_parallel_util
+        self.AutoencoderKLWan = AutoencoderKLWan
+        self.LongCatVideoAvatarTransformer3DModel = LongCatVideoAvatarTransformer3DModel
+        self.LongCatVideoTransformer3DModel = LongCatVideoTransformer3DModel
+        self.load_quantized_dit = load_quantized_dit
+        self.FlowMatchEulerDiscreteScheduler = FlowMatchEulerDiscreteScheduler
+        self.LongCatVideoPipeline = LongCatVideoPipeline
+        self.LongCatVideoAvatarPipeline = LongCatVideoAvatarPipeline
+
+    def _ensure_distributed(self):
+        self.torch.cuda.set_device(self.device_index)
+        if not self.dist.is_initialized():
+            self._dist_store_dir = tempfile.mkdtemp(prefix="localai-longcat-dist-")
+            init_file = os.path.join(self._dist_store_dir, "store")
+            self.dist.init_process_group(
+                backend="nccl",
+                init_method=f"file://{init_file}",
+                rank=0,
+                world_size=1,
+                timeout=datetime.timedelta(hours=24),
+            )
+            self.context_parallel_util.init_context_parallel(
+                context_parallel_size=1,
+                global_rank=0,
+                world_size=1,
+            )
+            self.cp_split_hw = self.context_parallel_util.get_optimal_split(1)
+
+    def _resolve_checkpoint(self, model, patterns):
+        source = normalize_model_source(model)
+        if os.path.isdir(source):
+            return source
+        print(f"Downloading required files for {source}", file=sys.stderr)
+        return self.snapshot_download(repo_id=source, allow_patterns=patterns)
+
+    def _load_base_model(self, model):
+        checkpoint = self._resolve_checkpoint(model, BASE_CHECKPOINT_PATTERNS)
+        dtype = self.torch.bfloat16
+        overrides = attention_overrides(self.options["attention_backend"])
+
+        tokenizer = self.AutoTokenizer.from_pretrained(
+            checkpoint,
+            subfolder="tokenizer",
+        )
+        text_encoder = self.UMT5EncoderModel.from_pretrained(
+            checkpoint,
+            subfolder="text_encoder",
+            torch_dtype=dtype,
+            low_cpu_mem_usage=True,
+        )
+        vae = self.AutoencoderKLWan.from_pretrained(
+            checkpoint,
+            subfolder="vae",
+            torch_dtype=dtype,
+            low_cpu_mem_usage=True,
+        )
+        scheduler = self.FlowMatchEulerDiscreteScheduler.from_pretrained(
+            checkpoint,
+            subfolder="scheduler",
+        )
+        dit = self.LongCatVideoTransformer3DModel.from_pretrained(
+            checkpoint,
+            subfolder="dit",
+            cp_split_hw=self.cp_split_hw,
+            torch_dtype=dtype,
+            low_cpu_mem_usage=True,
+            **overrides,
+        )
+        if self.options["use_distill"]:
+            dit.load_lora(
+                os.path.join(checkpoint, "lora", "cfg_step_lora.safetensors"),
+                "cfg_step_lora",
+            )
+            dit.enable_loras(["cfg_step_lora"])
+
+        self.pipeline = self.LongCatVideoPipeline(
+            tokenizer=tokenizer,
+            text_encoder=text_encoder,
+            vae=vae,
+            scheduler=scheduler,
+            dit=dit,
+        )
+        self.pipeline.to(self.device_index)
+
+    def _load_avatar_model(self, model):
+        avatar_patterns = list(AVATAR_COMMON_PATTERNS)
+        model_subfolder = (
+            "base_model_int8" if self.options["use_int8"] else "base_model"
+        )
+        avatar_patterns.append(f"{model_subfolder}/**")
+        checkpoint = self._resolve_checkpoint(model, avatar_patterns)
+
+        base_model = self.options.get("base_model")
+        if not base_model and os.path.isdir(normalize_model_source(model)):
+            sibling = os.path.join(
+                os.path.dirname(normalize_model_source(model)), "LongCat-Video"
+            )
+            if os.path.isdir(sibling):
+                base_model = sibling
+        base_model = base_model or BASE_MODEL_ID
+        if classify_model(str(base_model)) != MODEL_KIND_BASE:
+            raise ValueError("base_model must point to a LongCat-Video checkpoint")
+        base_checkpoint = self._resolve_checkpoint(base_model, AVATAR_BASE_PATTERNS)
+
+        dtype = self.torch.bfloat16
+        overrides = attention_overrides(self.options["attention_backend"])
+        tokenizer = self.AutoTokenizer.from_pretrained(
+            base_checkpoint,
+            subfolder="tokenizer",
+        )
+        text_encoder = self.UMT5EncoderModel.from_pretrained(
+            base_checkpoint,
+            subfolder="text_encoder",
+            torch_dtype=dtype,
+            low_cpu_mem_usage=True,
+        )
+        vae = self.AutoencoderKLWan.from_pretrained(
+            base_checkpoint,
+            subfolder="vae",
+            torch_dtype=dtype,
+            low_cpu_mem_usage=True,
+        )
+        scheduler = self.FlowMatchEulerDiscreteScheduler.from_pretrained(
+            checkpoint,
+            subfolder="scheduler",
+        )
+
+        if self.options["use_int8"]:
+            previous_dtype = self.torch.get_default_dtype()
+            self.torch.set_default_dtype(dtype)
+            try:
+                dit = self.load_quantized_dit(
+                    checkpoint,
+                    subfolder="base_model_int8",
+                    cp_split_hw=self.cp_split_hw,
+                    **overrides,
+                )
+            finally:
+                self.torch.set_default_dtype(previous_dtype)
+        else:
+            dit = self.LongCatVideoAvatarTransformer3DModel.from_pretrained(
+                checkpoint,
+                subfolder="base_model",
+                cp_split_hw=self.cp_split_hw,
+                torch_dtype=dtype,
+                low_cpu_mem_usage=True,
+                **overrides,
+            )
+
+        if self.options["use_distill"]:
+            dit.load_lora(
+                os.path.join(checkpoint, "lora", "dmd_lora.safetensors"),
+                "dmd",
+                multiplier=1.0,
+                lora_network_dim=128,
+                lora_network_alpha=64,
+            )
+            dit.enable_loras(["dmd"])
+
+        audio_checkpoint = os.path.join(checkpoint, "whisper-large-v3")
+        audio_encoder = self.get_audio_encoder(
+            audio_checkpoint,
+            MODEL_KIND_AVATAR + "-v1.5",
+        ).to(self.device_index)
+        audio_feature_extractor = self.get_audio_feature_extractor(
+            audio_checkpoint,
+            MODEL_KIND_AVATAR + "-v1.5",
+        )
+        self.pipeline = self.LongCatVideoAvatarPipeline(
+            tokenizer=tokenizer,
+            text_encoder=text_encoder,
+            vae=vae,
+            scheduler=scheduler,
+            dit=dit,
+            audio_encoder=audio_encoder,
+            audio_feature_extractor=audio_feature_extractor,
+            model_type="avatar-v1.5",
+        )
+        self.pipeline.to(self.device_index)
+
+    def _generate_base(self, request, params):
+        use_distill = self.options["use_distill"]
+        frames = normalize_num_frames(request.num_frames)
+        steps = (
+            16
+            if use_distill
+            else require_int(
+                request.step or 50,
+                "step",
+                minimum=1,
+                maximum=200,
+            )
+        )
+        guidance_scale = (
+            1.0
+            if use_distill
+            else require_float(
+                request.cfg_scale or 4.0,
+                "cfg_scale",
+                minimum=0.0,
+                maximum=30.0,
+            )
+        )
+        fps = require_int(request.fps or 15, "fps", minimum=1, maximum=60)
+        seed = request.seed if request.seed > 0 else 42
+        negative_prompt = request.negative_prompt or DEFAULT_NEGATIVE_PROMPT
+        generator = self.torch.Generator(device=self.device_index).manual_seed(seed)
+
+        if request.start_image:
+            resolution = self._resolution(params)
+            image = self.load_image(request.start_image)
+            output = self.pipeline.generate_i2v(
+                image=image,
+                prompt=request.prompt,
+                negative_prompt=negative_prompt,
+                resolution=resolution,
+                num_frames=frames,
+                num_inference_steps=steps,
+                use_distill=use_distill,
+                guidance_scale=guidance_scale,
+                generator=generator,
+            )[0]
+        else:
+            width, height = validate_dimensions(request.width, request.height)
+            output = self.pipeline.generate_t2v(
+                prompt=request.prompt,
+                negative_prompt=negative_prompt,
+                height=height,
+                width=width,
+                num_frames=frames,
+                num_inference_steps=steps,
+                use_distill=use_distill,
+                guidance_scale=guidance_scale,
+                generator=generator,
+            )[0]
+
+        self._save_video(output, request.dst, fps)
+
+    def _generate_avatar(self, request, params, context):
+        if not request.audio:
+            raise ValueError("audio is required for LongCat-Video-Avatar-1.5")
+        if not os.path.isfile(request.audio):
+            raise ValueError("audio input is not a readable staged file")
+
+        use_distill = self.options["use_distill"]
+        steps = (
+            8
+            if use_distill
+            else require_int(
+                request.step or 50,
+                "step",
+                minimum=1,
+                maximum=200,
+            )
+        )
+        text_guidance = (
+            1.0
+            if use_distill
+            else require_float(
+                request.cfg_scale or 4.0,
+                "cfg_scale",
+                minimum=0.0,
+                maximum=30.0,
+            )
+        )
+        audio_guidance = (
+            1.0
+            if use_distill
+            else require_float(
+                params.get("audio_guidance_scale", 4.0),
+                "audio_guidance_scale",
+                minimum=0.0,
+                maximum=20.0,
+            )
+        )
+        seed = request.seed if request.seed > 0 else 42
+        generator = self.torch.Generator(device=self.device_index).manual_seed(seed)
+        negative_prompt = request.negative_prompt or DEFAULT_NEGATIVE_PROMPT
+        resolution = self._resolution(params)
+
+        speech, sample_rate = self.librosa.load(request.audio, sr=16000, mono=True)
+        if speech.size == 0:
+            raise ValueError("audio contains no samples")
+        audio_duration = len(speech) / sample_rate
+        segments = self._avatar_segments(request, params, audio_duration)
+
+        segment_frames = 93
+        conditioning_frames = 13
+        avatar_fps = 25
+        generated_duration = (
+            segment_frames + (segments - 1) * (segment_frames - conditioning_frames)
+        ) / avatar_fps
+        pad_samples = max(
+            0, math.ceil((generated_duration - audio_duration) * sample_rate)
+        )
+        if pad_samples:
+            speech = self.np.pad(speech, (0, pad_samples))
+
+        full_audio_embedding = self.pipeline.get_audio_embedding(
+            speech,
+            fps=avatar_fps,
+            device=self.device_index,
+            sample_rate=sample_rate,
+            model_type="avatar-v1.5",
+        )
+        if not self.torch.isfinite(full_audio_embedding).all():
+            raise ValueError("audio encoder returned non-finite values")
+
+        indices = self.torch.arange(5) - 2
+
+        def audio_window(start_index):
+            centers = self.torch.arange(
+                start_index,
+                start_index + segment_frames,
+            ).unsqueeze(1) + indices.unsqueeze(0)
+            centers = self.torch.clamp(
+                centers,
+                min=0,
+                max=full_audio_embedding.shape[0] - 1,
+            )
+            return full_audio_embedding[centers][None, ...].to(self.device_index)
+
+        audio_start = 0
+        common = {
+            "prompt": request.prompt,
+            "negative_prompt": negative_prompt,
+            "num_frames": segment_frames,
+            "num_inference_steps": steps,
+            "text_guidance_scale": text_guidance,
+            "audio_guidance_scale": audio_guidance,
+            "output_type": "both",
+            "generator": generator,
+            "audio_emb": audio_window(audio_start),
+            "use_distill": use_distill,
+        }
+
+        if request.start_image:
+            output, latent = self.pipeline.generate_ai2v(
+                image=self.load_image(request.start_image),
+                resolution=resolution,
+                **common,
+            )
+        else:
+            width, height = validate_dimensions(request.width, request.height)
+            output, latent = self.pipeline.generate_at2v(
+                height=height,
+                width=width,
+                **common,
+            )
+
+        video = self._frames_to_pil(output[0])
+        width, height = video[0].size
+        current_video = video
+        reference_latent = latent[:, :, :1].clone()
+        all_frames = list(video)
+
+        for segment in range(1, segments):
+            if hasattr(context, "is_active") and not context.is_active():
+                raise RuntimeError("request was cancelled")
+            print(
+                f"Generating avatar segment {segment + 1}/{segments}", file=sys.stderr
+            )
+            audio_start += segment_frames - conditioning_frames
+            output, latent = self.pipeline.generate_avc(
+                video=current_video,
+                video_latent=latent,
+                prompt=request.prompt,
+                negative_prompt=negative_prompt,
+                height=height,
+                width=width,
+                num_frames=segment_frames,
+                num_cond_frames=conditioning_frames,
+                num_inference_steps=steps,
+                text_guidance_scale=text_guidance,
+                audio_guidance_scale=audio_guidance,
+                generator=generator,
+                output_type="both",
+                use_kv_cache=True,
+                offload_kv_cache=require_bool(
+                    params.get("offload_kv_cache", False),
+                    "offload_kv_cache",
+                ),
+                enhance_hf=not use_distill,
+                audio_emb=audio_window(audio_start),
+                ref_latent=reference_latent,
+                ref_img_index=require_int(
+                    params.get("ref_img_index", 10),
+                    "ref_img_index",
+                    minimum=-30,
+                    maximum=30,
+                ),
+                mask_frame_range=require_int(
+                    params.get("mask_frame_range", 3),
+                    "mask_frame_range",
+                    minimum=0,
+                    maximum=32,
+                ),
+                use_distill=use_distill,
+            )
+            current_video = self._frames_to_pil(output[0])
+            all_frames.extend(current_video[conditioning_frames:])
+
+        self._save_avatar_video(all_frames, request.audio, request.dst, avatar_fps)
+
+    def _avatar_segments(self, request, params, audio_duration):
+        if "num_segments" in params:
+            segments = require_int(
+                params["num_segments"],
+                "num_segments",
+                minimum=1,
+            )
+        elif request.num_frames > 0:
+            segments = avatar_segments_for_frames(request.num_frames)
+        else:
+            segments = avatar_segments_for_duration(audio_duration)
+
+        max_segments = self.options["max_segments"]
+        if segments > max_segments:
+            raise ValueError(
+                f"request needs {segments} avatar segments, but max_segments is {max_segments}; "
+                "trim the audio or raise the model's max_segments option"
+            )
+        return segments
+
+    def _resolution(self, params):
+        resolution = str(params.get("resolution", self.options["resolution"])).lower()
+        if resolution not in {"480p", "720p"}:
+            raise ValueError("resolution must be 480p or 720p")
+        return resolution
+
+    def _frames_to_pil(self, frames):
+        images = []
+        for frame in frames:
+            array = self.np.asarray(frame)
+            if self.np.issubdtype(array.dtype, self.np.floating):
+                array = self.np.clip(array, 0.0, 1.0) * 255
+            images.append(self.Image.fromarray(array.astype(self.np.uint8)))
+        return images
+
+    def _save_video(self, frames, path, fps):
+        writer = self.imageio.get_writer(
+            path,
+            format="FFMPEG",
+            mode="I",
+            fps=fps,
+            codec="libx264",
+            macro_block_size=1,
+            ffmpeg_params=[
+                "-crf",
+                "18",
+                "-pix_fmt",
+                "yuv420p",
+                "-movflags",
+                "+faststart",
+                "-f",
+                "mp4",
+            ],
+        )
+        try:
+            for frame in frames:
+                array = self.np.asarray(frame)
+                if self.np.issubdtype(array.dtype, self.np.floating):
+                    array = self.np.clip(array, 0.0, 1.0) * 255
+                writer.append_data(array.astype(self.np.uint8))
+        finally:
+            writer.close()
+
+    def _save_avatar_video(self, frames, audio_path, dst, fps):
+        output_dir = os.path.dirname(dst) or "."
+        handle, silent_path = tempfile.mkstemp(
+            prefix="longcat-silent-",
+            suffix=".mp4",
+            dir=output_dir,
+        )
+        os.close(handle)
+        try:
+            self._save_video(frames, silent_path, fps)
+            command = [
+                self.imageio_ffmpeg.get_ffmpeg_exe(),
+                "-y",
+                "-i",
+                silent_path,
+                "-i",
+                audio_path,
+                "-map",
+                "0:v:0",
+                "-map",
+                "1:a:0",
+                "-c:v",
+                "copy",
+                "-c:a",
+                "aac",
+                "-b:a",
+                "192k",
+                "-shortest",
+                "-movflags",
+                "+faststart",
+                "-f",
+                "mp4",
+                dst,
+            ]
+            subprocess.run(
+                command,
+                check=True,
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.PIPE,
+                text=True,
+            )
+        except subprocess.CalledProcessError as err:
+            details = (err.stderr or "ffmpeg failed")[-2000:]
+            raise RuntimeError(f"failed to mux avatar audio: {details}") from err
+        finally:
+            try:
+                os.remove(silent_path)
+            except FileNotFoundError:
+                pass
+
+    def _release_model(self):
+        self.pipeline = None
+        self.model_kind = None
+        gc.collect()
+        if hasattr(self, "torch") and self.torch.cuda.is_available():
+            self.torch.cuda.empty_cache()
+            self.torch.cuda.ipc_collect()
+
+    @staticmethod
+    def _fail(context, code, message):
+        if context is not None:
+            context.set_code(code)
+            context.set_details(message)
+        return backend_pb2.Result(message=message, success=False)
+
+
+def serve(address):
+    server = grpc.server(
+        futures.ThreadPoolExecutor(max_workers=MAX_WORKERS),
+        options=[
+            ("grpc.max_message_length", 64 * 1024 * 1024),
+            ("grpc.max_send_message_length", 64 * 1024 * 1024),
+            ("grpc.max_receive_message_length", 64 * 1024 * 1024),
+        ],
+        interceptors=get_auth_interceptors(),
+    )
+    backend_pb2_grpc.add_BackendServicer_to_server(BackendServicer(), server)
+    server.add_insecure_port(address)
+    server.start()
+    print(f"LongCat Video backend listening on {address}", file=sys.stderr)
+
+    def stop_server(signum, frame):
+        del signum, frame
+        server.stop(0)
+
+    signal.signal(signal.SIGINT, stop_server)
+    signal.signal(signal.SIGTERM, stop_server)
+    server.wait_for_termination()
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Run the LongCat Video gRPC backend")
+    parser.add_argument(
+        "--addr",
+        default="localhost:50051",
+        help="address on which to serve the backend",
+    )
+    arguments = parser.parse_args()
+    serve(arguments.addr)

backend/python/longcat-video/install.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: MIT
+set -euo pipefail
+
+PYTHON_VERSION="3.12"
+PYTHON_PATCH="12"
+PY_STANDALONE_TAG="20251120"
+
+backend_dir=$(dirname "$0")
+if [ -d "${backend_dir}/common" ]; then
+    source "${backend_dir}/common/libbackend.sh"
+else
+    source "${backend_dir}/../common/libbackend.sh"
+fi
+
+installRequirements

backend/python/longcat-video/longcat_utils.py

@@ -0,0 +1,199 @@
+# SPDX-License-Identifier: MIT
+
+import json
+import math
+import os
+from urllib.parse import urlparse
+
+
+BASE_MODEL_ID = "meituan-longcat/LongCat-Video"
+AVATAR_MODEL_ID = "meituan-longcat/LongCat-Video-Avatar-1.5"
+MODEL_KIND_BASE = "base"
+MODEL_KIND_AVATAR = "avatar"
+
+ATTENTION_OVERRIDES = {
+    "auto": {},
+    "sdpa": {
+        "enable_flashattn2": False,
+        "enable_flashattn3": False,
+        "enable_xformers": False,
+    },
+    "flash2": {
+        "enable_flashattn2": True,
+        "enable_flashattn3": False,
+        "enable_xformers": False,
+    },
+    "flash3": {
+        "enable_flashattn2": False,
+        "enable_flashattn3": True,
+        "enable_xformers": False,
+    },
+    "xformers": {
+        "enable_flashattn2": False,
+        "enable_flashattn3": False,
+        "enable_xformers": True,
+    },
+}
+
+
+def parse_options(values):
+    options = {}
+    for raw in values:
+        if ":" not in raw:
+            options[raw.strip()] = True
+            continue
+        key, value = raw.split(":", 1)
+        key = key.strip()
+        value = value.strip()
+        if not key:
+            continue
+        lower = value.lower()
+        if lower in {"true", "false"}:
+            options[key] = lower == "true"
+            continue
+        try:
+            options[key] = int(value)
+            continue
+        except ValueError:
+            pass
+        try:
+            options[key] = float(value)
+            continue
+        except ValueError:
+            pass
+        options[key] = value
+    return options
+
+
+def select_known_options(options, known):
+    """Split parsed options into the subset this backend understands and the
+    unknown keys to ignore.
+
+    LocalAI injects serving defaults (e.g. the llama.cpp cache_reuse / parallel
+    options) onto every model config regardless of backend. A backend should
+    tolerate options it does not understand rather than refuse to load, matching
+    the other LocalAI Python backends; the caller logs the ignored keys.
+
+    Returns (kept, ignored) where kept preserves the known entries and ignored is
+    the sorted list of dropped keys.
+    """
+    ignored = sorted(key for key in options if key not in known)
+    kept = {key: value for key, value in options.items() if key in known}
+    return kept, ignored
+
+
+def require_bool(value, name):
+    if isinstance(value, bool):
+        return value
+    if isinstance(value, str) and value.lower() in {"true", "false"}:
+        return value.lower() == "true"
+    raise ValueError(f"{name} must be true or false")
+
+
+def require_int(value, name, minimum=None, maximum=None):
+    try:
+        parsed = int(value)
+    except (TypeError, ValueError) as err:
+        raise ValueError(f"{name} must be an integer") from err
+    if minimum is not None and parsed < minimum:
+        raise ValueError(f"{name} must be at least {minimum}")
+    if maximum is not None and parsed > maximum:
+        raise ValueError(f"{name} must be at most {maximum}")
+    return parsed
+
+
+def require_float(value, name, minimum=None, maximum=None):
+    try:
+        parsed = float(value)
+    except (TypeError, ValueError) as err:
+        raise ValueError(f"{name} must be a number") from err
+    if minimum is not None and parsed < minimum:
+        raise ValueError(f"{name} must be at least {minimum}")
+    if maximum is not None and parsed > maximum:
+        raise ValueError(f"{name} must be at most {maximum}")
+    return parsed
+
+
+def attention_overrides(name):
+    try:
+        return dict(ATTENTION_OVERRIDES[name])
+    except KeyError as err:
+        choices = ", ".join(ATTENTION_OVERRIDES)
+        raise ValueError(f"attention_backend must be one of: {choices}") from err
+
+
+def _model_name_from_directory(path):
+    for filename in ("model_index.json", "config.json"):
+        config_path = os.path.join(path, filename)
+        try:
+            with open(config_path, "r", encoding="utf-8") as config_file:
+                model_name = json.load(config_file).get("model_name", "")
+        except (FileNotFoundError, OSError, ValueError, TypeError):
+            continue
+        if model_name:
+            return model_name
+    return ""
+
+
+def normalize_model_source(model):
+    value = model.rstrip("/")
+    for prefix in ("huggingface://", "hf://"):
+        if value.startswith(prefix):
+            return value[len(prefix) :]
+    parsed = urlparse(value)
+    if parsed.scheme in {"http", "https"} and parsed.netloc.lower() == "huggingface.co":
+        parts = [part for part in parsed.path.split("/") if part]
+        if len(parts) >= 2:
+            return "/".join(parts[:2])
+    return value
+
+
+def classify_model(model):
+    if not model:
+        return None
+    normalized = normalize_model_source(model)
+    if os.path.isdir(normalized):
+        name = _model_name_from_directory(normalized).lower()
+        if name == "longcat-video":
+            return MODEL_KIND_BASE
+        if name == "longcat-video-avatar-1.5":
+            return MODEL_KIND_AVATAR
+        return None
+
+    normalized = normalized.lower()
+    if normalized == BASE_MODEL_ID.lower():
+        return MODEL_KIND_BASE
+    if normalized == AVATAR_MODEL_ID.lower():
+        return MODEL_KIND_AVATAR
+    return None
+
+
+def normalize_num_frames(value, default=93):
+    frames = default if not value or value < 1 else value
+    return max(1, ((frames - 1) // 4) * 4 + 1)
+
+
+def avatar_segments_for_frames(frames):
+    if not frames or frames <= 93:
+        return 1
+    return 1 + math.ceil((frames - 93) / 80)
+
+
+def avatar_segments_for_duration(duration_seconds, fps=25):
+    if duration_seconds <= 0:
+        return 1
+    return avatar_segments_for_frames(math.ceil(duration_seconds * fps))
+
+
+def validate_dimensions(width, height):
+    width = width or 832
+    height = height or 480
+    if width < 256 or height < 256:
+        raise ValueError("width and height must each be at least 256")
+    if width > 1280 or height > 768:
+        raise ValueError("width and height must not exceed 1280x768")
+    if width % 16 != 0 or height % 16 != 0:
+        raise ValueError("width and height must be divisible by 16")
+    if width * height > 1280 * 768:
+        raise ValueError("requested video dimensions exceed the 1280x768 pixel limit")
+    return width, height

backend/python/longcat-video/patches/0001-sdpa-attention-fallback.patch

@@ -0,0 +1,75 @@
+diff --git a/longcat_video/modules/attention.py b/longcat_video/modules/attention.py
+index bb5630f..9b9f3cc 100644
+--- a/longcat_video/modules/attention.py
++++ b/longcat_video/modules/attention.py
+@@ -2,6 +2,7 @@ from typing import List, Optional
+
+ import torch
+ import torch.nn as nn
++import torch.nn.functional as F
+
+ from einops import rearrange
+
+@@ -100,7 +101,8 @@ class Attention(nn.Module):
+             x = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=None,)
+             x = rearrange(x, "B M H K -> B H M K")
+         else:
+-            raise RuntimeError("Unsupported attention operations.")
++            # Keep a dependency-free path for systems without optional kernels.
++            x = F.scaled_dot_product_attention(q, k, v, scale=self.scale)
+
+         return x
+
+@@ -245,8 +247,22 @@ class MultiHeadCrossAttention(nn.Module):
+             attn_bias = xformers.ops.fmha.attn_bias.BlockDiagonalMask.from_seqlens([N] * B, kv_seqlen)
+             x = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=attn_bias)
+         else:
+-            raise RuntimeError("Unsupported attention operations.")
+-
++            # Preserve the variable-length block boundaries without materializing
++            # a dense attention mask.
++            blocks = []
++            offset = 0
++            for batch_index, key_count in enumerate(kv_seqlen):
++                query = q[0][batch_index * N:(batch_index + 1) * N]
++                key = k[0][offset:offset + key_count]
++                value = v[0][offset:offset + key_count]
++                output = F.scaled_dot_product_attention(
++                    query.transpose(0, 1),
++                    key.transpose(0, 1),
++                    value.transpose(0, 1),
++                )
++                blocks.append(output.transpose(0, 1))
++                offset += key_count
++            x = torch.cat(blocks, dim=0)
+
+         x = x.view(B, -1, C)
+         x = self.proj(x)
+diff --git a/longcat_video/modules/avatar/attention.py b/longcat_video/modules/avatar/attention.py
+index a169a7a..df9a469 100644
+--- a/longcat_video/modules/avatar/attention.py
++++ b/longcat_video/modules/avatar/attention.py
+@@ -2,6 +2,7 @@ from typing import List, Optional
+
+ import torch
+ import torch.nn as nn
++import torch.nn.functional as F
+
+ from einops import rearrange
+
+@@ -111,7 +112,8 @@ class Attention(nn.Module):
+             x = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=None,)
+             x = rearrange(x, "B M H K -> B H M K")
+         else:
+-            raise RuntimeError("Unsupported attention operations.")
++            # Keep a dependency-free path for systems without optional kernels.
++            x = F.scaled_dot_product_attention(q, k, v, scale=self.scale)
+
+         return x
+
+@@ -429,2 +431,5 @@ class SingleStreamAttention(nn.Module):
++        else:
++            # This branch uses the native PyTorch kernel when optional kernels are off.
++            x = F.scaled_dot_product_attention(q, encoder_k, encoder_v, scale=self.scale)
+
+         # linear transform