feat(gallery): default to index.localai.io with GitHub as a mirror (#11409)

The primary is a caching mirror of the same two files, so an install
resolves its gallery from infrastructure the project controls. The GitHub
URI stays as a mirror, so behaviour is unchanged whenever the primary is
unreachable.

Docs and agent guides that either state the shipped defaults or hand out a
copy-pasteable gallery list are updated to match, so following them no
longer silently demotes an install off the new primary.

Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint]

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

.agents/backend-signing.md

@@ -16,8 +16,7 @@ side (`pkg/oci/cosignverify` plus the gallery YAML).
   per-arch manifest before checking signatures.
 - **Storage:** Signatures are written as OCI 1.1 referrers
   (`--registry-referrers-mode=oci-1-1`) in the new Sigstore bundle format
-  (current cosign releases do this by default; no `--new-bundle-format`
-  flag). No `:sha256-<hex>.sig` tag clutter.
+  (`--new-bundle-format`). No `:sha256-<hex>.sig` tag clutter.
 - **Consumer:** `pkg/oci/cosignverify` discovers the bundle via the
   referrers API, hands it to `sigstore-go`, and verifies it against the
   policy declared in the gallery YAML (`Gallery.Verification`).
@@ -34,14 +33,15 @@ to sign. The job needs:
 
 - `permissions: { id-token: write, contents: read }` at the job level so
   the runner can exchange its GitHub OIDC token for a Fulcio cert.
-- `sigstore/cosign-installer@v3` step (current cosign releases already
-  default to the new bundle format).
+- `sigstore/cosign-installer@v3` step (the pinned cosign v2 release needs
+  `--new-bundle-format` explicitly).
 - After each `docker buildx imagetools create`, resolve the resulting
   list digest with `docker buildx imagetools inspect <tag> --format
   '{{.Manifest.Digest}}'` and sign:
 
 ```sh
 cosign sign --yes --recursive \
+  --new-bundle-format \
   --registry-referrers-mode=oci-1-1 \
   "${REGISTRY_REPO}@${DIGEST}"
 ```
@@ -67,10 +67,12 @@ entry (`backend/index.yaml`):
 
 ```yaml
 - name: localai
-  url: github:mudler/LocalAI/backend/index.yaml@master
+  url: https://index.localai.io/backends
+  mirrors:
+    - github:mudler/LocalAI/backend/index.yaml@master
   verification:
     issuer: "https://token.actions.githubusercontent.com"
-    identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/heads/master$"
+    identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$"
     # Optional revocation cutoff; advance during incident response.
     # not_before: "2026-06-01T00:00:00Z"
 ```

.agents/ci-caching.md

@@ -198,7 +198,7 @@ Two properties this relies on:
 
 The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
 
-`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `github:mudler/LocalAI/gallery/index.yaml@master`. A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users through GitHub immediately whether or not an image is rebuilt.
+`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `https://index.localai.io/models` (a caching mirror of `gallery/index.yaml` on master, with `github:mudler/LocalAI/gallery/index.yaml@master` as the fallback mirror). A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users over the network immediately whether or not an image is rebuilt.
 
 Two properties to preserve if you touch it:
 

.github/workflows/backend_merge.yml

@@ -71,8 +71,8 @@ jobs:
 
       # cosign signs each pushed manifest list with --recursive so the
       # index and every per-arch entry get an attached Sigstore bundle.
-      # Recent cosign releases always emit the new bundle format, so
-      # there's no extra CLI flag to opt into it.
+      # The pinned cosign v2 release needs --new-bundle-format explicitly;
+      # the verifier only consumes OCI 1.1 Sigstore bundle referrers.
       - name: Install cosign
         if: github.event_name != 'pull_request'
         uses: sigstore/cosign-installer@v3
@@ -159,6 +159,7 @@ jobs:
           # manifest before checking signatures need the per-arch
           # signatures, not just the list-level one.
           cosign sign --yes --recursive \
+            --new-bundle-format \
             --registry-referrers-mode=oci-1-1 \
             "quay.io/go-skynet/local-ai-backends@${digest}"
 
@@ -185,6 +186,7 @@ jobs:
           ' <<< "$DOCKER_METADATA_OUTPUT_JSON")
           digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}')
           cosign sign --yes --recursive \
+            --new-bundle-format \
             --registry-referrers-mode=oci-1-1 \
             "localai/localai-backends@${digest}"
 

.github/workflows/refresh-site-counters.yml

@@ -23,7 +23,7 @@ jobs:
   refresh:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7
 
       - name: Read the counts off the GitHub API
         env:

.github/workflows/stalebot.yml

@@ -11,7 +11,7 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@1e223db275d687790206a7acac4d1a11bd6fe629 # v9
+      - uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v9
         with:
           stale-issue-message: 'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
           stale-pr-message: 'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=238ab6a9e321c17de8e120559f57efeedaeb1345
+AUDIO_CPP_VERSION?=7efbb58def443722ea540d931dd3debee3e4d5e8
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ds4/Makefile

@@ -1,10 +1,10 @@
 # ds4 backend Makefile.
 #
-# Upstream pin lives below as DS4_VERSION?=6747e7718dd08f00b680d0c16231f2d59ec3747e
+# Upstream pin lives below as DS4_VERSION?=b0309611041655f4e45671cfd9c9886aff161406
 # (.github/bump_deps.sh) can find and update it - matches the
 # llama-cpp / ik-llama-cpp / turboquant convention.
 
-DS4_VERSION?=6747e7718dd08f00b680d0c16231f2d59ec3747e
+DS4_VERSION?=b0309611041655f4e45671cfd9c9886aff161406
 DS4_REPO?=https://github.com/antirez/ds4
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=6b55d2c7504f482e7c8ec6cbf22a19f3778c522b
+IK_LLAMA_VERSION?=cf1aa57e1a0fabfd015831718fc99d1aec01ada5
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=ec730908a418b6032f9e69ded6186d3f042a7747
+CRISPASR_VERSION?=21901d3f7c23554f072964828363e49ddbc2dc68
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/crispasr/gocrispasr.go

@@ -67,7 +67,16 @@ const defaultTTSSampleRate = 24000
 // resampling, so the WAV header must match it. Returns ok=false for non-piper
 // models (key absent) or an unreadable file, letting the caller fall back to
 // defaultTTSSampleRate.
-func piperSampleRate(modelPath string) (int, bool) {
+func piperSampleRate(modelPath string) (rate int, ok bool) {
+	// A malformed metadata length can make gguf-parser-go panic before it can
+	// return an error. Keep a bad voice file from crash-looping the backend.
+	defer func() {
+		if recover() != nil {
+			rate = 0
+			ok = false
+		}
+	}()
+
 	// Only scalar architecture keys are read, so skip the large array metadata
 	// (phoneme map) and mmap the header - same rationale as pkg/vram's reader.
 	f, err := gguf.ParseGGUFFile(modelPath, gguf.UseMMap(), gguf.SkipLargeMetadata())
@@ -78,7 +87,7 @@ func piperSampleRate(modelPath string) (int, bool) {
 	if !ok || kv.ValueType != gguf.GGUFMetadataValueTypeUint32 {
 		return 0, false
 	}
-	rate := int(kv.ValueUint32())
+	rate = int(kv.ValueUint32())
 	if rate <= 0 {
 		return 0, false
 	}

backend/go/crispasr/gocrispasr_samplerate_test.go

@@ -3,6 +3,7 @@ package main
 import (
 	"bytes"
 	"encoding/binary"
+	"math"
 	"os"
 	"path/filepath"
 
@@ -102,6 +103,24 @@ var _ = Describe("piper sample rate", func() {
 			_, ok := piperSampleRate(p)
 			Expect(ok).To(BeFalse())
 		})
+
+		It("returns ok=false instead of panicking on a malformed string length", func() {
+			p := filepath.Join(GinkgoT().TempDir(), "malformed.gguf")
+			var b bytes.Buffer
+			b.WriteString("GGUF")
+			Expect(binary.Write(&b, binary.LittleEndian, uint32(3))).To(Succeed())
+			Expect(binary.Write(&b, binary.LittleEndian, uint64(0))).To(Succeed())
+			Expect(binary.Write(&b, binary.LittleEndian, uint64(1))).To(Succeed())
+			key := "general.name"
+			Expect(binary.Write(&b, binary.LittleEndian, uint64(len(key)))).To(Succeed())
+			b.WriteString(key)
+			Expect(binary.Write(&b, binary.LittleEndian, ggufTypeString)).To(Succeed())
+			Expect(binary.Write(&b, binary.LittleEndian, uint64(math.MaxInt64))).To(Succeed())
+			Expect(os.WriteFile(p, b.Bytes(), 0o644)).To(Succeed())
+
+			_, ok := piperSampleRate(p)
+			Expect(ok).To(BeFalse())
+		})
 	})
 
 	// End-to-end through the built .so. Gated on CRISPASR_PIPER_MODEL_PATH (a

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=ea7f0c87cfe4c673263b4c201c596c7f1cbe2528
+STABLEDIFFUSION_GGML_VERSION?=c6beeef35526c6dc94b74a7fb69f9d2e6a2a7a12
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/vllm-cpp/Makefile

@@ -96,6 +96,12 @@ endif
 UNAME_S := $(shell uname -s)
 ifeq ($(UNAME_S),Darwin)
 	LIB=libvllm.dylib
+	# Apple Clang diagnoses a pair of constant-folded array bounds in the Metal
+	# build as a GNU extension. Disable that diagnostic for both Objective-C and
+	# C++ because vllm.cpp appends target-local -Werror after these global flags.
+	CMAKE_ARGS+=-DCMAKE_CXX_FLAGS=-Wno-gnu-folding-constant
+	CMAKE_ARGS+=-DCMAKE_OBJC_FLAGS=-Wno-gnu-folding-constant
+	CMAKE_ARGS+=-DCMAKE_OBJCXX_FLAGS=-Wno-gnu-folding-constant
 else
 	LIB=libvllm.so
 endif
@@ -133,7 +139,26 @@ MLX_STAMP=
 MLX_CMAKE_ARGS=
 endif
 
+# govllmcpp.go mirrors vllm.h by hand, and the only guard against the two
+# drifting apart is the vllm_abi_version check inside registerLib - which fires
+# at runtime, on the user's machine, taking down every model load (issue
+# #11379). Compare the two here instead, so moving VLLM_CPP_VERSION past the
+# mirrors turns the build red while the header is still around to diff.
+abi-check: sources/vllm.cpp
+	@engine=$$(sed -n 's/^#define VLLM_ABI_VERSION \([0-9][0-9]*\).*/\1/p' sources/vllm.cpp/include/vllm.h); \
+	backend=$$(sed -n 's/^const abiVersion = \([0-9][0-9]*\).*/\1/p' govllmcpp.go); \
+	if [ -z "$$engine" ] || [ -z "$$backend" ]; then \
+		echo "vllm-cpp: cannot read the ABI version (engine='$$engine' backend='$$backend')" >&2; exit 1; \
+	fi; \
+	if [ "$$engine" != "$$backend" ]; then \
+		echo "vllm-cpp: ABI mismatch: vllm.cpp $(VLLM_CPP_VERSION) is v$$engine, govllmcpp.go mirrors v$$backend." >&2; \
+		echo "  Update the struct mirrors and abiVersion in govllmcpp.go (and the offsets in vllmcpp_test.go) to v$$engine." >&2; \
+		exit 1; \
+	fi; \
+	echo "vllm-cpp: ABI v$$engine matches the pinned engine"
+
 $(LIB): sources/vllm.cpp $(MLX_STAMP)
+	$(MAKE) abi-check
 	mkdir -p build && \
 	cd build && \
 	cmake ../sources/vllm.cpp $(CMAKE_ARGS) $(MLX_CMAKE_ARGS) && \
@@ -154,6 +179,8 @@ clean: purge
 purge:
 	rm -rf build
 
+.PHONY: abi-check
+
 .NOTPARALLEL:
 
 # The unit specs are pure Go (struct mirrors, option mapping, load

backend/go/vllm-cpp/README.md

@@ -6,7 +6,7 @@ safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
 inference time.
 
 The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
-ABI v2) through purego:
+ABI v10) through purego:
 
 - `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
   directory (`config.json` + safetensors). `context_size` maps to
@@ -29,6 +29,12 @@ ABI v2) through purego:
   LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
   choice constraints are also exposed by the ABI.
 
+The struct mirrors in `govllmcpp.go` are hand-written against one ABI version,
+and the engine refuses to load against any other. Moving `VLLM_CPP_VERSION` in
+the Makefile therefore means updating `abiVersion` plus the mirrors (and their
+offsets in `vllmcpp_test.go`) in the same change; `make abi-check` compares the
+pinned header against the bindings and the library build runs it first.
+
 Model config example:
 
 ```yaml

backend/go/vllm-cpp/backend.go

@@ -109,6 +109,16 @@ func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
 
 	v.opts = parseOptions(opts)
 
+	// A DFlash draft is a second checkpoint the engine opens by path, and the
+	// engine never downloads one. Resolve it against LocalAI's models directory
+	// now so a repo-id spelling works, and so a missing draft fails here with an
+	// actionable message rather than as an HF-cache miss inside the load.
+	resolvedSpec, err := resolveDraftModelPath(v.opts.speculativeConfig, opts.ModelPath)
+	if err != nil {
+		return err
+	}
+	v.opts.speculativeConfig = resolvedSpec
+
 	mp := defaultModelParams()
 	if v.opts.blockSize > 0 {
 		mp.BlockSize = v.opts.blockSize
@@ -116,34 +126,62 @@ func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
 	if v.opts.numBlocks > 0 {
 		mp.NumBlocks = v.opts.numBlocks
 	}
+	// Sequence-length precedence, narrowest source last: context_size is the
+	// generic LocalAI knob every backend honours, max_model_len is the
+	// vLLM-specific one, and engine_args.max_model_len is the explicit
+	// vllm-cpp override.
 	if opts.ContextSize > 0 {
 		mp.MaxModelLen = opts.ContextSize
 	}
+	if opts.MaxModelLen > 0 {
+		mp.MaxModelLen = opts.MaxModelLen
+	}
+	if v.opts.maxModelLen > 0 {
+		mp.MaxModelLen = v.opts.maxModelLen
+	}
 	if v.opts.maxNumSeqs > 0 {
 		mp.MaxNumSeqs = v.opts.maxNumSeqs
 	}
+	if v.opts.maxNumBatchedTokens > 0 {
+		mp.MaxNumBatchedTokens = v.opts.maxNumBatchedTokens
+	}
+	mp.EnablePrefixCaching = v.opts.enablePrefixCaching
+	mp.EnableJumpForward = v.opts.enableJumpForward
 
+	// Every string below is borrowed by C for the duration of the load call
+	// only (the library copies what it keeps), so the backing slices just have
+	// to outlive vllmEngineLoad - hence the single KeepAlive after it.
 	modelC := cString(model)
 	mp.ModelPath = uintptr(unsafe.Pointer(&modelC[0])) // #nosec G103 -- borrowed by C for the load call only
-	var toolParserC, reasoningParserC []byte
-	if v.opts.toolParser != "" {
-		toolParserC = cString(v.opts.toolParser)
-		mp.ToolParser = uintptr(unsafe.Pointer(&toolParserC[0])) // #nosec G103 -- borrowed by C for the load call only
-	}
-	if v.opts.reasoningParser != "" {
-		reasoningParserC = cString(v.opts.reasoningParser)
-		mp.ReasoningParser = uintptr(unsafe.Pointer(&reasoningParserC[0])) // #nosec G103 -- borrowed by C for the load call only
+	keep := [][]byte{modelC}
+	setStr := func(dst *uintptr, s string) {
+		if s == "" {
+			return
+		}
+		b := cString(s)
+		keep = append(keep, b)
+		*dst = uintptr(unsafe.Pointer(&b[0])) // #nosec G103 -- borrowed by C for the load call only
 	}
+	setStr(&mp.ToolParser, v.opts.toolParser)
+	setStr(&mp.ReasoningParser, v.opts.reasoningParser)
+	setStr(&mp.SpeculativeConfig, v.opts.speculativeConfig)
+	setStr(&mp.KVTransferConfig, v.opts.kvTransferConfig)
+	setStr(&mp.SchedulingPolicy, v.opts.schedulingPolicy)
+	setStr(&mp.TokenizerConfigPath, v.opts.tokenizerConfigPath)
 
 	xlog.Info("[vllm-cpp] Load", "model", model, "engine", vllmVersion(),
 		"blockSize", mp.BlockSize, "numBlocks", mp.NumBlocks,
-		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs)
+		"maxModelLen", mp.MaxModelLen, "maxNumSeqs", mp.MaxNumSeqs,
+		"maxNumBatchedTokens", mp.MaxNumBatchedTokens,
+		"prefixCaching", triStateName(mp.EnablePrefixCaching),
+		"jumpForward", triStateName(mp.EnableJumpForward),
+		"schedulingPolicy", v.opts.schedulingPolicy,
+		"speculativeConfig", v.opts.speculativeConfig,
+		"kvTransferConfig", v.opts.kvTransferConfig)
 
 	var engine uintptr
 	rc := vllmEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
-	runtime.KeepAlive(modelC)
-	runtime.KeepAlive(toolParserC)
-	runtime.KeepAlive(reasoningParserC)
+	runtime.KeepAlive(keep)
 	if rc != vllmOK {
 		return fmt.Errorf("vllm-cpp: engine load failed: %s", vllmLastError())
 	}

backend/go/vllm-cpp/govllmcpp.go

@@ -1,6 +1,6 @@
 package main
 
-// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v2).
+// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v10).
 //
 // The structs below are hand-mirrored PODs of the C declarations, with
 // explicit padding so the Go layout matches the C layout on linux/darwin
@@ -17,29 +17,65 @@ import (
 	"github.com/ebitengine/purego"
 )
 
-// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h).
-const abiVersion = 5
+// abiVersion is the VLLM_ABI_VERSION this file mirrors (vllm.h). It must track
+// the header of the VLLM_CPP_VERSION pinned in the Makefile: the build checks
+// the two against each other, because a mismatch is only caught at runtime by
+// registerLib, where it takes the backend down on every load (issue #11379).
+const abiVersion = 10
+
+// The ABI's tri-state toggles (enable_prefix_caching ABI v7,
+// enable_jump_forward ABI v10) share one encoding: 0 is NOT "off", it is
+// "defer" - to the model capability for prefix caching, to the environment for
+// jump forward. Only 2 is an explicit off.
+const (
+	triStateDefer int32 = 0
+	triStateOn    int32 = 1
+	triStateOff   int32 = 2
+)
+
+// triStateName renders a tri-state for the load log line, where "0" would
+// otherwise read as "off" rather than "whatever the default resolves to".
+func triStateName(state int32) string {
+	switch state {
+	case triStateOn:
+		return "on"
+	case triStateOff:
+		return "off"
+	default:
+		return "model-default"
+	}
+}
 
 // vllm_status (vllm.h).
 const (
 	vllmOK = 0
 )
 
-// cModelParams mirrors vllm_model_params.
+// cModelParams mirrors vllm_model_params. The int32 fields sit in pairs so the
+// interior needs no padding on LP64, but the struct is 8-aligned (it holds
+// pointers) and ends on a lone int32, so the trailing pad is explicit. Offsets
+// and total size are asserted in vllmcpp_test.go.
 type cModelParams struct {
 	ModelPath           uintptr // const char*
-	TokenizerConfigPath uintptr // const char*
+	TokenizerConfigPath uintptr // const char*; NULL = <model_dir>/... (ABI v9)
 	BlockSize           int32
 	NumBlocks           int32
 	MaxModelLen         int32
 	MaxNumSeqs          int32
 	ToolParser          uintptr // const char*; NULL = auto-detect (ABI v4)
 	ReasoningParser     uintptr // const char*; NULL = auto-detect (ABI v5)
+	SpeculativeConfig   uintptr // const char* JSON; NULL = no speculation (ABI v6)
+	EnablePrefixCaching int32   // tri-state 0/1/2 (ABI v7)
+	MaxNumBatchedTokens int32   // <= 0 = per-arch default (ABI v9)
+	SchedulingPolicy    uintptr // const char*; NULL = "fcfs" (ABI v9)
+	KVTransferConfig    uintptr // const char* JSON; NULL = no connector (ABI v9)
+	EnableJumpForward   int32   // tri-state 0/1/2 (ABI v10)
+	_                   [4]byte // trailing pad to the struct's 8-byte alignment
 }
 
-// cSamplingParams mirrors vllm_sampling_params (ABI v2, structured fields
-// included). Padding matches the C compiler's: the uint64 seed is 8-aligned,
-// and each pointer following an int32 is 8-aligned.
+// cSamplingParams mirrors vllm_sampling_params (structured fields included).
+// Padding matches the C compiler's: the uint64 seed is 8-aligned, and each
+// pointer following an int32 is 8-aligned.
 type cSamplingParams struct {
 	Temperature          float32
 	TopP                 float32
@@ -65,6 +101,12 @@ type cSamplingParams struct {
 	StructuredGrammar    uintptr // const char*
 	StructuredJSONObject int32
 	_                    [4]byte
+	// ABI v8 tail. LocalAI installs no custom logits processor, but the fields
+	// MUST be mirrored: the C side reads them off the pointer we hand it, so a
+	// Go struct that stopped at StructuredJSONObject would have the engine read
+	// 16 bytes past our allocation and call whatever garbage sat there.
+	LogitsProcessor         uintptr // vllm_logits_processor; NULL = none
+	LogitsProcessorUserData uintptr // void*
 }
 
 // cCompletion mirrors vllm_completion.

backend/go/vllm-cpp/options.go

@@ -1,30 +1,80 @@
 package main
 
-// Engine-sizing knobs carried through the model config's free-form
-// `options:` list ("key:value" entries), mirroring how the other in-house
-// backends pass engine-specific settings that have no proto field.
+// Load-time engine configuration, from two config surfaces:
+//
+//   - `engine_args:` (ModelOptions.EngineArgs, a JSON object) is the canonical
+//     one. Keys are spelled exactly as vLLM's own CLI flags, so a config written
+//     against vLLM works verbatim here - `speculative_config` and
+//     `kv_transfer_config` in particular take the same JSON documents vLLM's
+//     --speculative-config / --kv-transfer-config accept, and are handed to the
+//     engine unparsed.
+//   - `options:` (the free-form "key:value" list) is the older surface this
+//     backend shipped with. It is still honoured so existing configs keep
+//     working; engine_args wins on any key set in both.
+//
+// Anything unrecognised is ignored rather than fatal: the engine validates the
+// documents it is given and reports a precise error at load, and a config that
+// also carries knobs for a different backend must not fail the load here.
 
 import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path"
+	"path/filepath"
 	"strconv"
 	"strings"
 
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	"github.com/mudler/xlog"
 )
 
 type loadOptions struct {
 	blockSize  int32 // KV block size (tokens/block); engine default 32.
 	numBlocks  int32 // KV blocks to allocate; engine default 256.
 	maxNumSeqs int32 // max concurrent sequences; engine default 8.
+	// Max sequence length. Also settable through the model config's
+	// context_size / max_model_len; see Load for the precedence.
+	maxModelLen int32
+	// Per-step chunked-prefill token budget (ABI v9). 0 = the engine's
+	// bounded per-arch default.
+	maxNumBatchedTokens int32
+	// Automatic prefix caching tri-state (ABI v7): 0 = the model-capability
+	// default, 1 = force on, 2 = force off.
+	enablePrefixCaching int32
+	// Jump-forward decoding tri-state (ABI v10), SGLang's grammar-speed subset:
+	// 0 = defer to the environment (VT_ENABLE_JUMP_FORWARD, default off),
+	// 1 = force on, 2 = force off.
+	enableJumpForward int32
+	// Scheduler admission policy (ABI v9): "" = fcfs, else fcfs|priority|lpm.
+	schedulingPolicy string
 	// Engine-side parser selection (ABI v4/v5). Empty = the engine
 	// auto-detects from the chat template; "none" disables the reasoning
 	// split; unknown names fail the first chat call.
 	toolParser      string
 	reasoningParser string
+	// Speculative decoding (ABI v6), as vLLM's --speculative-config JSON:
+	// {"method":"mtp"|"dflash"|"ngram", ...}. Empty = no speculation.
+	speculativeConfig string
+	// External KV connector / LMCache (ABI v9), as vLLM's --kv-transfer-config
+	// JSON. Empty = no connector.
+	kvTransferConfig string
+	// Override for the tokenizer_config.json the chat template is read from
+	// (ABI v9). Empty = <model_dir>/tokenizer_config.json.
+	tokenizerConfigPath string
 }
 
 func parseOptions(opts *pb.ModelOptions) loadOptions {
 	lo := loadOptions{}
-	for _, o := range opts.GetOptions() {
+	applyOptionsList(&lo, opts.GetOptions())
+	applyEngineArgs(&lo, opts.GetEngineArgs())
+	return lo
+}
+
+// applyOptionsList reads the legacy free-form "key:value" list. strings.Cut
+// splits on the FIRST colon only, so a JSON object value survives intact.
+func applyOptionsList(lo *loadOptions, options []string) {
+	for _, o := range options {
 		k, v, found := strings.Cut(o, ":")
 		if !found {
 			continue
@@ -36,13 +86,211 @@ func parseOptions(opts *pb.ModelOptions) loadOptions {
 			lo.numBlocks = parseInt32(v, lo.numBlocks)
 		case "max_num_seqs":
 			lo.maxNumSeqs = parseInt32(v, lo.maxNumSeqs)
-		case "tool_parser":
+		case "max_num_batched_tokens":
+			lo.maxNumBatchedTokens = parseInt32(v, lo.maxNumBatchedTokens)
+		case "max_model_len":
+			lo.maxModelLen = parseInt32(v, lo.maxModelLen)
+		case "scheduling_policy", "schedule_policy":
+			lo.schedulingPolicy = strings.TrimSpace(v)
+		case "tool_parser", "tool_call_parser":
 			lo.toolParser = strings.TrimSpace(v)
 		case "reasoning_parser":
 			lo.reasoningParser = strings.TrimSpace(v)
+		case "speculative_config":
+			lo.speculativeConfig = strings.TrimSpace(v)
+		case "kv_transfer_config":
+			lo.kvTransferConfig = strings.TrimSpace(v)
+		case "tokenizer_config", "tokenizer_config_path":
+			lo.tokenizerConfigPath = strings.TrimSpace(v)
+		case "enable_prefix_caching", "enable_radix_attention":
+			if b, err := strconv.ParseBool(strings.TrimSpace(v)); err == nil {
+				lo.enablePrefixCaching = boolTriState(b)
+			}
+		case "enable_jump_forward":
+			if b, err := strconv.ParseBool(strings.TrimSpace(v)); err == nil {
+				lo.enableJumpForward = boolTriState(b)
+			}
 		}
 	}
-	return lo
+}
+
+// applyEngineArgs overlays the `engine_args:` JSON object. A document that does
+// not parse is logged and skipped: engine_args is shared with the other engines
+// (the vLLM and SGLang backends read the same field), so a stray key must not
+// take the model down.
+func applyEngineArgs(lo *loadOptions, engineArgs string) {
+	if strings.TrimSpace(engineArgs) == "" {
+		return
+	}
+	var args map[string]any
+	if err := json.Unmarshal([]byte(engineArgs), &args); err != nil {
+		xlog.Warn("[vllm-cpp] ignoring unparseable engine_args", "error", err)
+		return
+	}
+	for k, v := range args {
+		switch k {
+		case "block_size":
+			lo.blockSize = jsonInt32(v, lo.blockSize)
+		case "num_blocks":
+			lo.numBlocks = jsonInt32(v, lo.numBlocks)
+		case "max_num_seqs":
+			lo.maxNumSeqs = jsonInt32(v, lo.maxNumSeqs)
+		case "max_num_batched_tokens":
+			lo.maxNumBatchedTokens = jsonInt32(v, lo.maxNumBatchedTokens)
+		case "max_model_len":
+			lo.maxModelLen = jsonInt32(v, lo.maxModelLen)
+		case "scheduling_policy", "schedule_policy":
+			lo.schedulingPolicy = jsonString(v, lo.schedulingPolicy)
+		case "tool_parser", "tool_call_parser":
+			lo.toolParser = jsonString(v, lo.toolParser)
+		case "reasoning_parser":
+			lo.reasoningParser = jsonString(v, lo.reasoningParser)
+		case "tokenizer_config", "tokenizer_config_path":
+			lo.tokenizerConfigPath = jsonString(v, lo.tokenizerConfigPath)
+		case "speculative_config":
+			lo.speculativeConfig = jsonDocument(v, lo.speculativeConfig, k)
+		case "kv_transfer_config":
+			lo.kvTransferConfig = jsonDocument(v, lo.kvTransferConfig, k)
+		case "enable_prefix_caching", "enable_radix_attention":
+			if b, ok := v.(bool); ok {
+				lo.enablePrefixCaching = boolTriState(b)
+			}
+		case "enable_jump_forward":
+			if b, ok := v.(bool); ok {
+				lo.enableJumpForward = boolTriState(b)
+			}
+		default:
+			xlog.Debug("[vllm-cpp] ignoring unknown engine_args key", "key", k)
+		}
+	}
+}
+
+// boolTriState maps a YAML/JSON boolean onto the ABI's tri-state encoding. An
+// explicit `false` must reach the engine as force-OFF (2), NOT as the 0 that
+// means "defer". The difference is real in both directions: prefix caching
+// defaults ON for dense archs and OFF for hybrid ones, and jump forward defers
+// to VT_ENABLE_JUMP_FORWARD.
+func boolTriState(on bool) int32 {
+	if on {
+		return triStateOn
+	}
+	return triStateOff
+}
+
+// jsonDocument normalises an object-valued engine_args entry to a JSON string
+// for the C ABI. YAML nesting arrives as a map (the natural spelling); a
+// pre-encoded JSON string is accepted too, since a config round-tripped through
+// a flat store may carry it that way.
+func jsonDocument(v any, fallback string, key string) string {
+	switch t := v.(type) {
+	case string:
+		if strings.TrimSpace(t) == "" {
+			return fallback
+		}
+		return t
+	default:
+		buf, err := json.Marshal(t)
+		if err != nil {
+			xlog.Warn("[vllm-cpp] ignoring unencodable engine_args value", "key", key, "error", err)
+			return fallback
+		}
+		return string(buf)
+	}
+}
+
+func jsonString(v any, fallback string) string {
+	s, ok := v.(string)
+	if !ok {
+		return fallback
+	}
+	return strings.TrimSpace(s)
+}
+
+// jsonInt32 accepts the float64 a JSON number decodes to, plus the string
+// spelling a YAML config may produce. Non-positive values keep the fallback:
+// every knob this covers uses "<= 0 means the engine default".
+func jsonInt32(v any, fallback int32) int32 {
+	switch t := v.(type) {
+	case float64:
+		if t <= 0 || t > 1<<31-1 {
+			return fallback
+		}
+		return int32(t)
+	case string:
+		return parseInt32(t, fallback)
+	default:
+		return fallback
+	}
+}
+
+// resolveDraftModelPath rewrites a DFlash draft reference into an absolute path
+// the engine can actually open.
+//
+// The engine resolves `speculative_config.model` against a directory containing
+// config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/
+// snapshots/* - and it NEVER downloads. LocalAI keeps models in its own
+// directory, so a bare HF repo id (the spelling the vLLM docs teach) misses the
+// HF cache and dies deep in the load with "draft checkpoint not found", which
+// reads like a broken checkpoint rather than a missing download.
+//
+// So: try the reference as given, then the last path segment under the models
+// dir (`z-lab/Qwen3.6-27B-DFlash` -> `<models>/Qwen3.6-27B-DFlash`, which is
+// what LocalAI's own downloader produces), then the whole reference under the
+// models dir. If none exist, fail HERE with a message naming both what was
+// asked for and where we looked.
+//
+// mtp and ngram carry no separate draft checkpoint, so they pass through. A
+// document that does not parse also passes through: the engine owns config
+// validation and produces the better error.
+func resolveDraftModelPath(speculativeConfig, modelsDir string) (string, error) {
+	if strings.TrimSpace(speculativeConfig) == "" {
+		return speculativeConfig, nil
+	}
+	var spec map[string]any
+	if err := json.Unmarshal([]byte(speculativeConfig), &spec); err != nil {
+		return speculativeConfig, nil
+	}
+	if method, _ := spec["method"].(string); !strings.EqualFold(method, "dflash") {
+		return speculativeConfig, nil
+	}
+
+	ref, _ := spec["model"].(string)
+	ref = strings.TrimSpace(ref)
+	if ref == "" {
+		return "", fmt.Errorf(
+			"vllm-cpp: speculative_config method %q requires a \"model\" key naming the draft checkpoint", "dflash")
+	}
+
+	candidates := []string{ref}
+	if modelsDir != "" {
+		if base := path.Base(filepath.ToSlash(ref)); base != "" && base != "." && base != "/" {
+			candidates = append(candidates, filepath.Join(modelsDir, base))
+		}
+		candidates = append(candidates, filepath.Join(modelsDir, filepath.FromSlash(ref)))
+	}
+
+	for _, c := range candidates {
+		if _, err := os.Stat(filepath.Join(c, "config.json")); err != nil {
+			continue
+		}
+		abs, err := filepath.Abs(c)
+		if err != nil {
+			abs = c
+		}
+		spec["model"] = abs
+		out, err := json.Marshal(spec)
+		if err != nil {
+			return "", fmt.Errorf("vllm-cpp: re-encoding speculative_config: %w", err)
+		}
+		xlog.Info("[vllm-cpp] resolved DFlash draft checkpoint", "reference", ref, "path", abs)
+		return string(out), nil
+	}
+
+	return "", fmt.Errorf(
+		"vllm-cpp: DFlash draft checkpoint %q not found (looked in: %s). "+
+			"The engine does not download drafts - install the draft model into LocalAI first, "+
+			"or set speculative_config.model to an absolute path to a directory containing config.json",
+		ref, strings.Join(candidates, ", "))
 }
 
 func parseInt32(s string, fallback int32) int32 {

backend/go/vllm-cpp/vllmcpp_test.go

@@ -16,10 +16,17 @@ func TestVllmCpp(t *testing.T) {
 	RunSpecs(t, "vllm-cpp suite")
 }
 
-// The Go POD mirrors must match the C struct layout of vllm.h (ABI v2)
+// The Go POD mirrors must match the C struct layout of vllm.h (ABI v10)
 // byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
 // amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
 var _ = Describe("C ABI struct mirrors", func() {
+	It("declares the ABI version the pinned engine reports", func() {
+		// VLLM_ABI_VERSION in the vllm.h of VLLM_CPP_VERSION (Makefile).
+		// Moving the pin past this without growing the mirrors below ships a
+		// backend that refuses every load at startup (issue #11379).
+		Expect(abiVersion).To(Equal(10))
+	})
+
 	It("cModelParams matches vllm_model_params", func() {
 		var p cModelParams
 		Expect(unsafe.Offsetof(p.ModelPath)).To(Equal(uintptr(0)))
@@ -30,10 +37,18 @@ var _ = Describe("C ABI struct mirrors", func() {
 		Expect(unsafe.Offsetof(p.MaxNumSeqs)).To(Equal(uintptr(28)))
 		Expect(unsafe.Offsetof(p.ToolParser)).To(Equal(uintptr(32)))
 		Expect(unsafe.Offsetof(p.ReasoningParser)).To(Equal(uintptr(40)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.SpeculativeConfig)).To(Equal(uintptr(48)))
+		Expect(unsafe.Offsetof(p.EnablePrefixCaching)).To(Equal(uintptr(56)))
+		Expect(unsafe.Offsetof(p.MaxNumBatchedTokens)).To(Equal(uintptr(60)))
+		Expect(unsafe.Offsetof(p.SchedulingPolicy)).To(Equal(uintptr(64)))
+		Expect(unsafe.Offsetof(p.KVTransferConfig)).To(Equal(uintptr(72)))
+		Expect(unsafe.Offsetof(p.EnableJumpForward)).To(Equal(uintptr(80)))
+		// 88, not 84: the struct is 8-aligned (it holds pointers), so the
+		// trailing int32 is padded out. Go pads identically.
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(88)))
 	})
 
-	It("cSamplingParams matches vllm_sampling_params (ABI v2)", func() {
+	It("cSamplingParams matches vllm_sampling_params (ABI v8)", func() {
 		var p cSamplingParams
 		Expect(unsafe.Offsetof(p.Temperature)).To(Equal(uintptr(0)))
 		Expect(unsafe.Offsetof(p.TopP)).To(Equal(uintptr(4)))
@@ -55,7 +70,9 @@ var _ = Describe("C ABI struct mirrors", func() {
 		Expect(unsafe.Offsetof(p.NStructuredChoice)).To(Equal(uintptr(96)))
 		Expect(unsafe.Offsetof(p.StructuredGrammar)).To(Equal(uintptr(104)))
 		Expect(unsafe.Offsetof(p.StructuredJSONObject)).To(Equal(uintptr(112)))
-		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(120)))
+		Expect(unsafe.Offsetof(p.LogitsProcessor)).To(Equal(uintptr(120)))
+		Expect(unsafe.Offsetof(p.LogitsProcessorUserData)).To(Equal(uintptr(128)))
+		Expect(unsafe.Sizeof(p)).To(Equal(uintptr(136)))
 	})
 
 	It("cCompletion matches vllm_completion", func() {
@@ -68,6 +85,23 @@ var _ = Describe("C ABI struct mirrors", func() {
 	})
 })
 
+// Pin/mirror skew is the failure mode this backend is most exposed to: the Go
+// PODs above are hand-written against one VLLM_ABI_VERSION, and the Makefile
+// pins the vllm.cpp commit that produces it. This spec catches drift without
+// needing model weights - set VLLM_CPP_LIBRARY to a built libvllm and it binds
+// every symbol and compares the library's reported ABI against the mirrors'.
+var _ = Describe("real library ABI handshake", func() {
+	It("binds every symbol and reports the ABI the mirrors were written against", func() {
+		lib := os.Getenv("VLLM_CPP_LIBRARY")
+		if lib == "" {
+			Skip("VLLM_CPP_LIBRARY not set; skipping the real-library handshake")
+		}
+		Expect(registerLib(lib)).To(Succeed())
+		Expect(vllmABIVersion()).To(Equal(int32(abiVersion)))
+		Expect(vllmVersion()).NotTo(BeEmpty())
+	})
+})
+
 var _ = Describe("parseOptions", func() {
 	It("extracts the engine sizing knobs", func() {
 		lo := parseOptions(&pb.ModelOptions{Options: []string{
@@ -83,6 +117,129 @@ var _ = Describe("parseOptions", func() {
 		}})
 		Expect(lo).To(Equal(loadOptions{}))
 	})
+
+	It("carries a speculative_config JSON value through the legacy options list", func() {
+		// strings.Cut splits on the FIRST colon only, so a JSON object value
+		// survives the "key:value" spelling intact.
+		lo := parseOptions(&pb.ModelOptions{Options: []string{
+			`speculative_config:{"method":"mtp","num_speculative_tokens":1}`,
+		}})
+		Expect(lo.speculativeConfig).To(Equal(`{"method":"mtp","num_speculative_tokens":1}`))
+	})
+})
+
+var _ = Describe("engine_args", func() {
+	It("maps every load knob onto the C model params", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{
+			"block_size": 64,
+			"num_blocks": 1024,
+			"max_model_len": 16384,
+			"max_num_seqs": 32,
+			"max_num_batched_tokens": 8192,
+			"enable_prefix_caching": true,
+			"scheduling_policy": "lpm",
+			"tool_parser": "qwen3",
+			"reasoning_parser": "deepseek_r1",
+			"tokenizer_config": "/models/tok/tokenizer_config.json"
+		}`})
+		Expect(lo.blockSize).To(Equal(int32(64)))
+		Expect(lo.numBlocks).To(Equal(int32(1024)))
+		Expect(lo.maxModelLen).To(Equal(int32(16384)))
+		Expect(lo.maxNumSeqs).To(Equal(int32(32)))
+		Expect(lo.maxNumBatchedTokens).To(Equal(int32(8192)))
+		Expect(lo.enablePrefixCaching).To(Equal(int32(1)))
+		Expect(lo.schedulingPolicy).To(Equal("lpm"))
+		Expect(lo.toolParser).To(Equal("qwen3"))
+		Expect(lo.reasoningParser).To(Equal("deepseek_r1"))
+		Expect(lo.tokenizerConfigPath).To(Equal("/models/tok/tokenizer_config.json"))
+	})
+
+	It("re-marshals a nested speculative_config object to JSON for the engine", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{
+			"speculative_config": {"method": "mtp", "num_speculative_tokens": 1}
+		}`})
+		Expect(lo.speculativeConfig).To(MatchJSON(`{"method":"mtp","num_speculative_tokens":1}`))
+	})
+
+	It("re-marshals a nested kv_transfer_config object (LMCache) to JSON", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{
+			"kv_transfer_config": {
+				"kv_connector": "LMCacheConnector",
+				"kv_role": "kv_both",
+				"kv_connector_extra_config": {"host": "127.0.0.1", "port": 65432}
+			}
+		}`})
+		Expect(lo.kvTransferConfig).To(MatchJSON(`{
+			"kv_connector":"LMCacheConnector",
+			"kv_role":"kv_both",
+			"kv_connector_extra_config":{"host":"127.0.0.1","port":65432}
+		}`))
+	})
+
+	It("accepts a pre-encoded JSON string for the object-valued knobs", func() {
+		// A config written by hand (or round-tripped through a flat store) may
+		// carry the object as a string; both spellings reach the engine the same.
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{
+			"speculative_config": "{\"method\":\"ngram\",\"num_speculative_tokens\":4}"
+		}`})
+		Expect(lo.speculativeConfig).To(MatchJSON(`{"method":"ngram","num_speculative_tokens":4}`))
+	})
+
+	It("maps enable_prefix_caching false onto the force-OFF tri-state", func() {
+		// The C ABI tri-state is 0=model default, 1=on, 2=off, so an explicit
+		// `false` must NOT collapse to the 0 that means "let the model decide".
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{"enable_prefix_caching": false}`})
+		Expect(lo.enablePrefixCaching).To(Equal(int32(2)))
+	})
+
+	It("leaves the prefix-caching tri-state at the model default when unset", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{"max_num_seqs": 4}`})
+		Expect(lo.enablePrefixCaching).To(Equal(int32(0)))
+	})
+
+	It("accepts the radix-attention alias upstream documents for prefix caching", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{"enable_radix_attention": true}`})
+		Expect(lo.enablePrefixCaching).To(Equal(int32(1)))
+	})
+
+	It("maps enable_jump_forward onto its own tri-state", func() {
+		// ABI v10. Same tri-state shape as prefix caching, and the same trap:
+		// an explicit false must be force-OFF (2), not the 0 that defers to the
+		// environment.
+		on := parseOptions(&pb.ModelOptions{EngineArgs: `{"enable_jump_forward": true}`})
+		Expect(on.enableJumpForward).To(Equal(int32(1)))
+		off := parseOptions(&pb.ModelOptions{EngineArgs: `{"enable_jump_forward": false}`})
+		Expect(off.enableJumpForward).To(Equal(int32(2)))
+		unset := parseOptions(&pb.ModelOptions{EngineArgs: `{"max_num_seqs": 4}`})
+		Expect(unset.enableJumpForward).To(Equal(int32(0)))
+	})
+
+	It("reads enable_jump_forward from the legacy options list too", func() {
+		lo := parseOptions(&pb.ModelOptions{Options: []string{"enable_jump_forward:true"}})
+		Expect(lo.enableJumpForward).To(Equal(int32(1)))
+	})
+
+	It("lets engine_args override the legacy options list", func() {
+		lo := parseOptions(&pb.ModelOptions{
+			Options:    []string{"max_num_seqs:8", "block_size:16"},
+			EngineArgs: `{"max_num_seqs": 64}`,
+		})
+		Expect(lo.maxNumSeqs).To(Equal(int32(64))) // engine_args wins
+		Expect(lo.blockSize).To(Equal(int32(16)))  // untouched keys survive
+	})
+
+	It("ignores malformed engine_args rather than failing the load", func() {
+		lo := parseOptions(&pb.ModelOptions{
+			Options:    []string{"max_num_seqs:8"},
+			EngineArgs: `{not json`,
+		})
+		Expect(lo.maxNumSeqs).To(Equal(int32(8)))
+	})
+
+	It("ignores unknown keys", func() {
+		lo := parseOptions(&pb.ModelOptions{EngineArgs: `{"gpu_memory_utilization": 0.9}`})
+		Expect(lo).To(Equal(loadOptions{}))
+	})
 })
 
 var _ = Describe("samplingFromPredict", func() {
@@ -135,6 +292,91 @@ var _ = Describe("samplingFromPredict", func() {
 	})
 })
 
+// The engine resolves speculative_config.model against a local directory or
+// ~/.cache/huggingface/hub ONLY - it never downloads. LocalAI keeps models in
+// its own directory, so a bare repo id would miss the HF cache and fail deep in
+// the load with a confusing "draft checkpoint not found". Resolve it here.
+var _ = Describe("resolveDraftModelPath", func() {
+	var modelsDir string
+
+	BeforeEach(func() {
+		modelsDir = GinkgoT().TempDir()
+	})
+
+	// draftDir creates a plausible draft checkpoint under models/.
+	draftDir := func(name string) string {
+		d := filepath.Join(modelsDir, name)
+		Expect(os.MkdirAll(d, 0o750)).To(Succeed())
+		Expect(os.WriteFile(filepath.Join(d, "config.json"), []byte("{}"), 0o600)).To(Succeed())
+		return d
+	}
+
+	It("rewrites a repo id to the matching directory in the models dir", func() {
+		want := draftDir("Qwen3.6-27B-DFlash")
+		spec := `{"method":"dflash","model":"z-lab/Qwen3.6-27B-DFlash"}`
+		out, err := resolveDraftModelPath(spec, modelsDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(MatchJSON(`{"method":"dflash","model":"` + want + `"}`))
+	})
+
+	It("rewrites a models-dir-relative path", func() {
+		want := draftDir("drafts__dflash")
+		spec := `{"method":"dflash","model":"drafts__dflash"}`
+		out, err := resolveDraftModelPath(spec, modelsDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(ContainSubstring(want))
+	})
+
+	It("leaves an absolute path that already resolves alone", func() {
+		abs := draftDir("elsewhere")
+		spec := `{"method":"dflash","model":"` + abs + `"}`
+		out, err := resolveDraftModelPath(spec, modelsDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(MatchJSON(spec))
+	})
+
+	It("fails with an actionable error when the draft is nowhere on disk", func() {
+		// Silently passing the repo id through would surface as an HF-cache
+		// miss inside the engine, which reads as "your model is broken".
+		spec := `{"method":"dflash","model":"z-lab/Not-Downloaded"}`
+		_, err := resolveDraftModelPath(spec, modelsDir)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("z-lab/Not-Downloaded"))
+		Expect(err.Error()).To(ContainSubstring(modelsDir))
+	})
+
+	It("requires a model key for dflash", func() {
+		_, err := resolveDraftModelPath(`{"method":"dflash"}`, modelsDir)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("model"))
+	})
+
+	It("leaves mtp and ngram configs untouched", func() {
+		// Neither has a separate draft checkpoint to resolve.
+		for _, spec := range []string{
+			`{"method":"mtp"}`,
+			`{"method":"ngram","num_speculative_tokens":4}`,
+		} {
+			out, err := resolveDraftModelPath(spec, modelsDir)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(out).To(MatchJSON(spec))
+		}
+	})
+
+	It("passes a malformed document through for the engine to reject", func() {
+		// The engine owns config validation and produces the better message.
+		out, err := resolveDraftModelPath(`{not json`, modelsDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(Equal(`{not json`))
+	})
+
+	It("is a no-op on an empty config", func() {
+		out, err := resolveDraftModelPath("", modelsDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(out).To(BeEmpty())
+	})
+})
+
 var _ = Describe("validModelPath", func() {
 	It("accepts a .gguf file", func() {
 		dir := GinkgoT().TempDir()

core/cli/run_socket_activation_linux.go

@@ -24,6 +24,14 @@ func systemdActivatedListeners() ([]net.Listener, error) {
 		}
 	}()
 
+	// A half-populated environment is not an activation attempt. Container runtimes
+	// started from a socket-activated system unit leak a bare LISTEN_PID into every
+	// container they spawn, and systemd's own sd_listen_fds() treats either variable
+	// being absent as "not activated" rather than as an error.
+	if listenPID == "" || listenFDs == "" {
+		return nil, nil
+	}
+
 	pid, err := strconv.Atoi(listenPID)
 	if err != nil {
 		return nil, fmt.Errorf("invalid LISTEN_PID %q: %w", listenPID, err)

core/cli/run_socket_activation_test.go

@@ -85,6 +85,34 @@ var _ = Describe("systemdActivatedListeners", func() {
 		Expect(os.Getenv("LISTEN_FDNAMES")).To(BeEmpty())
 	})
 
+	It("binds normally when the environment leaks LISTEN_PID without LISTEN_FDS", func() {
+		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()))).To(Succeed())
+		Expect(os.Unsetenv("LISTEN_FDS")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_PID")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_PID")).To(BeEmpty())
+	})
+
+	It("binds normally when the environment leaks LISTEN_FDS without LISTEN_PID", func() {
+		Expect(os.Unsetenv("LISTEN_PID")).To(Succeed())
+		Expect(os.Setenv("LISTEN_FDS", "1")).To(Succeed())
+		DeferCleanup(func() {
+			_ = os.Unsetenv("LISTEN_FDS")
+		})
+
+		listeners, err := systemdActivatedListeners()
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(listeners).To(BeEmpty())
+		Expect(os.Getenv("LISTEN_FDS")).To(BeEmpty())
+	})
+
 	It("reports malformed activation metadata instead of silently binding another socket", func() {
 		Expect(os.Setenv("LISTEN_PID", strconv.Itoa(os.Getpid()))).To(Succeed())
 		Expect(os.Setenv("LISTEN_FDS", "not-a-number")).To(Succeed())

core/config/gallery.go

@@ -1,5 +1,7 @@
 package config
 
+import "slices"
+
 // GalleryVerification declares the keyless-cosign signature policy that
 // every OCI backend image fetched from this gallery must satisfy.
 //
@@ -31,7 +33,39 @@ type GalleryVerification struct {
 }
 
 type Gallery struct {
-	URL          string               `json:"url" yaml:"url"`
+	URL string `json:"url" yaml:"url"`
+	// Mirrors are tried in order when URL cannot be fetched. They are a
+	// fallback for availability, not a load-balancing pool: the primary is
+	// always preferred, and a mirror is only consulted after the one before
+	// it fails. Any URI the gallery loader understands works here
+	// (https://, github:, file://).
+	Mirrors      []string             `json:"mirrors,omitempty" yaml:"mirrors,omitempty"`
 	Name         string               `json:"name" yaml:"name"`
 	Verification *GalleryVerification `json:"verification,omitempty" yaml:"verification,omitempty"`
 }
+
+// Equal reports whether two gallery entries describe the same gallery.
+//
+// Mirrors made Gallery non-comparable with ==, so callers that used to rely
+// on that (the runtime settings registry diffs the live gallery list against
+// the option-less baseline to decide whether env/CLI claimed the setting)
+// need an explicit value comparison. Verification is compared by value:
+// under == it was compared by pointer identity, which would have called two
+// structurally identical policies different.
+func (g Gallery) Equal(other Gallery) bool {
+	if g.URL != other.URL || g.Name != other.Name {
+		return false
+	}
+	if !slices.Equal(g.Mirrors, other.Mirrors) {
+		return false
+	}
+	if g.Verification == nil || other.Verification == nil {
+		return g.Verification == other.Verification
+	}
+	return *g.Verification == *other.Verification
+}
+
+// GalleriesEqual compares two gallery lists element-wise, in order.
+func GalleriesEqual(a, b []Gallery) bool {
+	return slices.EqualFunc(a, b, Gallery.Equal)
+}

core/config/gallery_test.go

@@ -0,0 +1,171 @@
+package config_test
+
+import (
+	"encoding/json"
+
+	"github.com/mudler/LocalAI/core/config"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"gopkg.in/yaml.v3"
+)
+
+var _ = Describe("Gallery mirrors", func() {
+	// Galleries are configured as a JSON list in LOCALAI_GALLERIES and edited
+	// as raw JSON in the settings UI, so both directions must round-trip or a
+	// user silently loses their mirrors the next time they save.
+	It("round-trips through JSON", func() {
+		const in = `[{"url":"https://primary.example/index.yaml","name":"localai",` +
+			`"mirrors":["github:mudler/LocalAI/gallery/index.yaml@master","file:///srv/index.yaml"]}]`
+
+		var galleries []config.Gallery
+		Expect(json.Unmarshal([]byte(in), &galleries)).To(Succeed())
+		Expect(galleries).To(HaveLen(1))
+
+		// Order is load-bearing: mirrors are an ordered fallback chain, not a set.
+		want := []string{"github:mudler/LocalAI/gallery/index.yaml@master", "file:///srv/index.yaml"}
+		Expect(galleries[0].Mirrors).To(Equal(want))
+
+		out, err := json.Marshal(galleries)
+		Expect(err).ToNot(HaveOccurred())
+		var again []config.Gallery
+		Expect(json.Unmarshal(out, &again)).To(Succeed())
+		Expect(again[0].Mirrors).To(Equal(want), "mirrors lost or reordered on round-trip: %s", out)
+		Expect(again[0].URL).To(Equal("https://primary.example/index.yaml"))
+		Expect(again[0].Name).To(Equal("localai"))
+	})
+
+	It("round-trips through YAML", func() {
+		const in = "- url: https://primary.example/index.yaml\n" +
+			"  name: localai\n" +
+			"  mirrors:\n" +
+			"    - github:mudler/LocalAI/gallery/index.yaml@master\n" +
+			"    - https://fallback.example/index.yaml\n"
+
+		var galleries []config.Gallery
+		Expect(yaml.Unmarshal([]byte(in), &galleries)).To(Succeed())
+		want := []string{"github:mudler/LocalAI/gallery/index.yaml@master", "https://fallback.example/index.yaml"}
+		Expect(galleries[0].Mirrors).To(Equal(want))
+
+		out, err := yaml.Marshal(galleries)
+		Expect(err).ToNot(HaveOccurred())
+		var again []config.Gallery
+		Expect(yaml.Unmarshal(out, &again)).To(Succeed())
+		Expect(again[0].Mirrors).To(Equal(want), "mirrors lost or reordered on YAML round-trip: %s", out)
+	})
+
+	// omitempty keeps existing configs byte-identical when they declare no
+	// mirrors, so this change cannot churn anyone's stored settings.
+	Context("a gallery without mirrors", func() {
+		It("marshals to unchanged JSON", func() {
+			out, err := json.Marshal(config.Gallery{URL: "https://x/index.yaml", Name: "n"})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(out)).To(Equal(`{"url":"https://x/index.yaml","name":"n"}`), "want no mirrors key")
+		})
+
+		It("marshals to unchanged YAML", func() {
+			y, err := yaml.Marshal(config.Gallery{URL: "https://x/index.yaml", Name: "localai"})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(y)).To(Equal("url: https://x/index.yaml\nname: localai\n"), "want no mirrors key")
+		})
+	})
+})
+
+// The runtime settings registry diffs gallery lists to decide whether the
+// persisted settings differ from the startup baseline. A Gallery carrying a
+// slice is no longer comparable with ==, so that diff must still notice a
+// change confined to the mirror list — otherwise editing mirrors in the
+// settings UI would be dropped as a no-op.
+var _ = Describe("GalleriesEqual", func() {
+	base := []config.Gallery{{URL: "https://x/index.yaml", Name: "n"}}
+	withMirror := []config.Gallery{{URL: "https://x/index.yaml", Name: "n", Mirrors: []string{"github:mudler/LocalAI/gallery/index.yaml@master"}}}
+
+	It("reports lists that differ only by mirrors as unequal", func() {
+		Expect(config.GalleriesEqual(base, withMirror)).To(BeFalse())
+	})
+
+	It("reports identical mirror-less lists as equal", func() {
+		Expect(config.GalleriesEqual(base, []config.Gallery{{URL: "https://x/index.yaml", Name: "n"}})).To(BeTrue())
+	})
+
+	It("reports identical mirrored lists as equal", func() {
+		Expect(config.GalleriesEqual(withMirror, []config.Gallery{
+			{URL: "https://x/index.yaml", Name: "n", Mirrors: []string{"github:mudler/LocalAI/gallery/index.yaml@master"}},
+		})).To(BeTrue())
+	})
+
+	// Reordering the fallback chain is a real change, not a no-op.
+	It("does not ignore mirror ordering", func() {
+		a := []config.Gallery{{URL: "u", Mirrors: []string{"m1", "m2"}}}
+		b := []config.Gallery{{URL: "u", Mirrors: []string{"m2", "m1"}}}
+		Expect(config.GalleriesEqual(a, b)).To(BeFalse())
+	})
+
+	// A nil mirror list and an empty one both mean "no mirrors".
+	It("does not distinguish nil from empty mirrors", func() {
+		Expect(config.GalleriesEqual(
+			[]config.Gallery{{URL: "u"}},
+			[]config.Gallery{{URL: "u", Mirrors: []string{}}})).To(BeTrue())
+	})
+
+	It("reports lists of different length as unequal", func() {
+		Expect(config.GalleriesEqual(base, nil)).To(BeFalse())
+	})
+
+	// Equal replaced ==, so it has to keep covering every field == covered:
+	// missing one would make an env/CLI-set gallery list look like the default.
+	Context("comparing every field", func() {
+		It("does not ignore URL", func() {
+			Expect(config.GalleriesEqual(
+				[]config.Gallery{{URL: "https://a/index.yaml", Name: "n"}},
+				[]config.Gallery{{URL: "https://b/index.yaml", Name: "n"}})).To(BeFalse())
+		})
+
+		It("does not ignore Name", func() {
+			Expect(config.GalleriesEqual(
+				[]config.Gallery{{URL: "https://a/index.yaml", Name: "one"}},
+				[]config.Gallery{{URL: "https://a/index.yaml", Name: "two"}})).To(BeFalse())
+		})
+
+		// The verification pointer must be compared by value, not identity.
+		It("compares the verification block by value, not pointer identity", func() {
+			v1 := &config.GalleryVerification{Issuer: "i"}
+			v2 := &config.GalleryVerification{Issuer: "i"}
+			Expect(config.GalleriesEqual(
+				[]config.Gallery{{URL: "u", Verification: v1}},
+				[]config.Gallery{{URL: "u", Verification: v2}})).To(BeTrue())
+		})
+
+		It("does not ignore a differing verification block", func() {
+			Expect(config.GalleriesEqual(
+				[]config.Gallery{{URL: "u", Verification: &config.GalleryVerification{Issuer: "i"}}},
+				[]config.Gallery{{URL: "u", Verification: &config.GalleryVerification{Issuer: "other"}}})).To(BeFalse())
+		})
+
+		It("does not ignore a verification block appearing", func() {
+			Expect(config.GalleriesEqual(
+				[]config.Gallery{{URL: "u"}},
+				[]config.Gallery{{URL: "u", Verification: &config.GalleryVerification{Issuer: "i"}}})).To(BeFalse())
+		})
+
+		// GalleryVerification has five string fields; a value comparison must
+		// notice a change in any of them, not just the first.
+		DescribeTable("notices a change in any verification field",
+			func(mutate func(*config.GalleryVerification)) {
+				full := config.GalleryVerification{
+					Issuer: "i", IssuerRegex: "ir", Identity: "id", IdentityRegex: "idr", NotBefore: "2026-05-01T00:00:00Z",
+				}
+				other := full
+				mutate(&other)
+				Expect(config.GalleriesEqual(
+					[]config.Gallery{{URL: "u", Verification: &full}},
+					[]config.Gallery{{URL: "u", Verification: &other}})).To(BeFalse(),
+					"GalleriesEqual ignored a verification change: %+v vs %+v", full, other)
+			},
+			Entry("issuer", func(v *config.GalleryVerification) { v.Issuer = "x" }),
+			Entry("issuer regex", func(v *config.GalleryVerification) { v.IssuerRegex = "x" }),
+			Entry("identity", func(v *config.GalleryVerification) { v.Identity = "x" }),
+			Entry("identity regex", func(v *config.GalleryVerification) { v.IdentityRegex = "x" }),
+			Entry("not before", func(v *config.GalleryVerification) { v.NotBefore = "2030-01-01T00:00:00Z" }),
+		)
+	})
+})

core/config/runtime_settings_registry.go

@@ -296,19 +296,21 @@ var runtimeSettingsFields = []fieldSpec{
 		func(o *ApplicationConfig) bool { return o.Federated },
 		func(o *ApplicationConfig, v bool) { o.Federated = v }),
 
-	// Galleries. Gallery is comparable (string fields + a pointer), so
-	// slices.Equal gives element-wise comparison against the baseline's
-	// default gallery list.
+	// Galleries. Gallery holds a Mirrors slice, so it is not comparable with
+	// == and slices.Equal does not apply; GalleriesEqual walks the list
+	// element-wise against the baseline's default gallery list, mirrors
+	// included, so a list that differs only by its mirrors still counts as
+	// env/CLI-set.
 	fieldEq("galleries",
 		func(s *RuntimeSettings) **[]Gallery { return &s.Galleries },
 		func(o *ApplicationConfig) []Gallery { return o.Galleries },
 		func(o *ApplicationConfig, v []Gallery) { o.Galleries = v },
-		slices.Equal),
+		GalleriesEqual),
 	fieldEq("backend_galleries",
 		func(s *RuntimeSettings) **[]Gallery { return &s.BackendGalleries },
 		func(o *ApplicationConfig) []Gallery { return o.BackendGalleries },
 		func(o *ApplicationConfig, v []Gallery) { o.BackendGalleries = v },
-		slices.Equal),
+		GalleriesEqual),
 	field("autoload_galleries",
 		func(s *RuntimeSettings) **bool { return &s.AutoloadGalleries },
 		func(o *ApplicationConfig) bool { return o.AutoloadGalleries },

core/config/runtime_settings_startup.go

@@ -12,8 +12,13 @@ import (
 // the "${galleries}" / "${backends}" vars, and DefaultRuntimeBaseline uses
 // them to tell "kong default" apart from "user-configured" at settings-load
 // time - they must stay the single source for both.
-const DefaultGalleriesJSON = `[{"name":"localai", "url":"github:mudler/LocalAI/gallery/index.yaml@master"}]`
-const DefaultBackendGalleriesJSON = `[{"name":"localai", "url":"github:mudler/LocalAI/backend/index.yaml@master"}]`
+//
+// The primary is served by index-server (github.com/localai-org/index-server),
+// a caching mirror of the files below. The GitHub URI stays as a mirror so an
+// install still resolves its gallery unchanged whenever the primary is
+// unreachable - see the fallback chain in core/gallery/gallery_mirrors.go.
+const DefaultGalleriesJSON = `[{"name":"localai", "url":"https://index.localai.io/models", "mirrors":["github:mudler/LocalAI/gallery/index.yaml@master"]}]`
+const DefaultBackendGalleriesJSON = `[{"name":"localai", "url":"https://index.localai.io/backends", "mirrors":["github:mudler/LocalAI/backend/index.yaml@master"]}]`
 
 func mustGalleries(jsonList string) []Gallery {
 	var g []Gallery

core/config/runtime_settings_startup_test.go

@@ -0,0 +1,83 @@
+package config_test
+
+import (
+	"encoding/json"
+
+	"github.com/mudler/LocalAI/core/config"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("default galleries", func() {
+	It("serves the model gallery from index.localai.io with GitHub as a mirror", func() {
+		var galleries []config.Gallery
+		Expect(json.Unmarshal([]byte(config.DefaultGalleriesJSON), &galleries)).To(Succeed())
+		Expect(galleries).To(HaveLen(1))
+		Expect(galleries[0].Name).To(Equal("localai"))
+		Expect(galleries[0].URL).To(Equal("https://index.localai.io/models"))
+		Expect(galleries[0].Mirrors).To(Equal([]string{"github:mudler/LocalAI/gallery/index.yaml@master"}))
+	})
+
+	It("serves the backend gallery from index.localai.io with GitHub as a mirror", func() {
+		var galleries []config.Gallery
+		Expect(json.Unmarshal([]byte(config.DefaultBackendGalleriesJSON), &galleries)).To(Succeed())
+		Expect(galleries).To(HaveLen(1))
+		Expect(galleries[0].Name).To(Equal("localai"))
+		Expect(galleries[0].URL).To(Equal("https://index.localai.io/backends"))
+		Expect(galleries[0].Mirrors).To(Equal([]string{"github:mudler/LocalAI/backend/index.yaml@master"}))
+	})
+
+	// The mirror is the whole reason this default is safe to ship: if
+	// index.localai.io is unreachable, an install must still resolve its
+	// gallery from GitHub exactly as it did before.
+	It("always leaves a reachable fallback behind the primary", func() {
+		for _, raw := range []string{config.DefaultGalleriesJSON, config.DefaultBackendGalleriesJSON} {
+			var galleries []config.Gallery
+			Expect(json.Unmarshal([]byte(raw), &galleries)).To(Succeed())
+			for _, g := range galleries {
+				Expect(g.Mirrors).ToNot(BeEmpty(), "default %q has no mirror", g.Name)
+			}
+		}
+	})
+
+	// mustGalleries panics on malformed JSON, and these constants are also
+	// fed to kong as defaults, so a typo would be a startup crash rather
+	// than a config error.
+	It("parses through the same path startup uses", func() {
+		baseline := config.DefaultRuntimeBaseline()
+		Expect(baseline.Galleries).To(HaveLen(1))
+		Expect(baseline.BackendGalleries).To(HaveLen(1))
+		Expect(baseline.Galleries[0].URL).To(Equal("https://index.localai.io/models"))
+		Expect(baseline.BackendGalleries[0].URL).To(Equal("https://index.localai.io/backends"))
+	})
+
+	// The baseline decides "kong default" vs "user-configured", and since
+	// the defaults grew a Mirrors slice that comparison runs through
+	// GalleriesEqual rather than ==. A default install must still let the
+	// persisted file apply, and any override - including one that differs
+	// from the default only by its mirrors - must still be seen as env-set.
+	Context("as the option-less startup baseline", func() {
+		It("treats an untouched default list as not env-set, so the file applies", func() {
+			o := config.DefaultRuntimeBaseline()
+			saved := []config.Gallery{{Name: "mine", URL: "https://example.com/index.yaml"}}
+			savedBackends := []config.Gallery{{Name: "mine-backends", URL: "https://example.com/backends.yaml"}}
+			o.ApplyRuntimeSettingsAtStartup(&config.RuntimeSettings{
+				Galleries:        &saved,
+				BackendGalleries: &savedBackends,
+			})
+			Expect(o.Galleries).To(Equal(saved))
+			Expect(o.BackendGalleries).To(Equal(savedBackends))
+		})
+
+		It("treats LOCALAI_GALLERIES as env-set even when it only drops the mirror", func() {
+			o := config.DefaultRuntimeBaseline()
+			// Same primary as the default, mirror removed: env-set.
+			o.Galleries = []config.Gallery{{Name: "localai", URL: "https://index.localai.io/models"}}
+			saved := []config.Gallery{{Name: "file", URL: "https://file.example/index.yaml"}}
+			o.ApplyRuntimeSettingsAtStartup(&config.RuntimeSettings{Galleries: &saved})
+			Expect(o.Galleries[0].Name).To(Equal("localai"),
+				"an env list differing only by mirrors must still win over the file")
+		})
+	})
+})

core/config/vllm_spec.go

@@ -0,0 +1,117 @@
+package config
+
+// Speculative-decoding auto-defaults for the vllm-cpp backend, the safetensors
+// counterpart of the GGUF/llama.cpp hook in mtp.go.
+//
+// The two engines detect and spell the same feature differently. llama.cpp
+// reads `<arch>.nextn_predict_layers` out of the GGUF header and takes
+// `spec_type:draft-mtp` in `options:`; vllm.cpp reads `mtp_num_hidden_layers`
+// out of the checkpoint's config.json and takes vLLM's own
+// `--speculative-config` JSON, which LocalAI carries in `engine_args`. The
+// engine resolves the draft depth and the default k itself, so the config only
+// has to name the method.
+
+import (
+	"encoding/json"
+
+	"github.com/mudler/xlog"
+)
+
+// hfSpecConfig is the subset of a HuggingFace config.json that decides whether
+// speculative decoding can be auto-enabled.
+type hfSpecConfig struct {
+	ModelType string `json:"model_type"`
+	// MtpNumHiddenLayers is the MTP head depth (upstream speculative.py reads
+	// it as n_predict for the qwen3_5 / qwen3_5_moe families).
+	MtpNumHiddenLayers uint32 `json:"mtp_num_hidden_layers"`
+	// DFlashConfig marks a z-lab DFlash DRAFT checkpoint (mask_token_id +
+	// target_layer_ids). Its presence means this repo is a draft, not a
+	// servable target.
+	DFlashConfig json.RawMessage `json:"dflash_config"`
+	// TextConfig is where multimodal checkpoints nest the language-model
+	// config, and therefore the MTP depth.
+	TextConfig *hfSpecConfig `json:"text_config"`
+}
+
+// parseHFSpecConfig decodes the speculative-relevant subset of a config.json.
+// A document that does not parse yields nothing rather than an error: detection
+// is best-effort and must never break an import.
+func parseHFSpecConfig(configJSON []byte) (hfSpecConfig, bool) {
+	if len(configJSON) == 0 {
+		return hfSpecConfig{}, false
+	}
+	var c hfSpecConfig
+	if err := json.Unmarshal(configJSON, &c); err != nil {
+		xlog.Debug("[vllm-spec] config.json did not parse; skipping detection", "error", err)
+		return hfSpecConfig{}, false
+	}
+	return c, true
+}
+
+// IsDFlashDraftConfig reports whether a HuggingFace config.json describes a
+// DFlash DRAFT checkpoint. Unlike MTP - whose head ships inside the target
+// checkpoint's `mtp.*` tensors - a DFlash draft is its own repo that can only
+// run paired with a target it verifies against, so it must never be configured
+// as a standalone model.
+func IsDFlashDraftConfig(configJSON []byte) bool {
+	c, ok := parseHFSpecConfig(configJSON)
+	if !ok {
+		return false
+	}
+	return len(c.DFlashConfig) > 0 ||
+		(c.TextConfig != nil && len(c.TextConfig.DFlashConfig) > 0)
+}
+
+// HasSafetensorsMTPHead reports whether a HuggingFace config.json declares a
+// self-speculating Multi-Token Prediction head, returning its depth. The depth
+// is informational: vllm.cpp resolves n_predict and the default
+// num_speculative_tokens from the checkpoint itself.
+//
+// DFlash drafts are excluded for the same reason `gemma4-assistant` GGUFs are
+// excluded from the llama.cpp hook: they carry head metadata but cannot
+// self-speculate.
+//
+// NOTE this is a safetensors-only signal. vllm.cpp rejects an MTP config over a
+// GGUF source, because the `mtp.*` draft tensors only exist in the safetensors
+// checkpoint - so the GGUF import path must not use this.
+func HasSafetensorsMTPHead(configJSON []byte) (uint32, bool) {
+	c, ok := parseHFSpecConfig(configJSON)
+	if !ok {
+		return 0, false
+	}
+	if IsDFlashDraftConfig(configJSON) {
+		return 0, false
+	}
+	n := c.MtpNumHiddenLayers
+	if n == 0 && c.TextConfig != nil {
+		n = c.TextConfig.MtpNumHiddenLayers
+	}
+	return n, n > 0
+}
+
+// ApplyVLLMSpeculativeDefaults enables MTP speculative decoding in cfg's
+// engine_args when nothing is configured there yet. It is a no-op when the user
+// already set a speculative_config, so an explicit choice (a different method,
+// an explicit k, a DFlash draft) is never clobbered.
+//
+// `layers` is the detected head depth and is only used for the diagnostic log
+// line - the engine derives the real k from the checkpoint.
+func ApplyVLLMSpeculativeDefaults(cfg *ModelConfig, layers uint32) {
+	if cfg == nil {
+		return
+	}
+	if _, set := cfg.EngineArgs["speculative_config"]; set {
+		xlog.Debug("[vllm-spec] MTP head detected but speculative_config already configured; leaving user choice intact",
+			"name", cfg.Name, "mtp_num_hidden_layers", layers)
+		return
+	}
+	if cfg.EngineArgs == nil {
+		cfg.EngineArgs = map[string]any{}
+	}
+	// Only the method: vllm.cpp defaults num_speculative_tokens to the
+	// checkpoint's own n_predict (speculative.py:865-875), which is the right
+	// value far more reliably than anything guessable here.
+	cfg.EngineArgs["speculative_config"] = map[string]any{"method": "mtp"}
+	xlog.Info("[vllm-spec] MTP head detected; enabling mtp speculative decoding",
+		"name", cfg.Name, "mtp_num_hidden_layers", layers)
+}

core/config/vllm_spec_test.go

@@ -0,0 +1,117 @@
+package config_test
+
+import (
+	. "github.com/mudler/LocalAI/core/config"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp speculative-decoding auto-defaults", func() {
+	Context("HasSafetensorsMTPHead", func() {
+		It("detects a top-level mtp_num_hidden_layers", func() {
+			n, ok := HasSafetensorsMTPHead([]byte(`{
+				"model_type": "qwen3_5_moe",
+				"mtp_num_hidden_layers": 1
+			}`))
+			Expect(ok).To(BeTrue())
+			Expect(n).To(Equal(uint32(1)))
+		})
+
+		It("detects the head nested under text_config", func() {
+			// Multimodal checkpoints nest the language-model config, which is
+			// where the MTP depth lives (mirrors the engine's own resolution
+			// off config.raw text_config).
+			n, ok := HasSafetensorsMTPHead([]byte(`{
+				"model_type": "qwen3_5_moe",
+				"text_config": {"mtp_num_hidden_layers": 2}
+			}`))
+			Expect(ok).To(BeTrue())
+			Expect(n).To(Equal(uint32(2)))
+		})
+
+		It("reports no head when the key is absent", func() {
+			n, ok := HasSafetensorsMTPHead([]byte(`{"model_type": "llama"}`))
+			Expect(ok).To(BeFalse())
+			Expect(n).To(BeZero())
+		})
+
+		It("reports no head for a zero depth", func() {
+			_, ok := HasSafetensorsMTPHead([]byte(`{"mtp_num_hidden_layers": 0}`))
+			Expect(ok).To(BeFalse())
+		})
+
+		It("ignores a DFlash draft checkpoint", func() {
+			// A DFlash draft is a SEPARATE checkpoint that cannot serve alone:
+			// it needs a target to verify against. Same exclusion the GGUF path
+			// makes for gemma4-assistant drafts.
+			_, ok := HasSafetensorsMTPHead([]byte(`{
+				"model_type": "qwen3_dflash",
+				"mtp_num_hidden_layers": 1,
+				"dflash_config": {"mask_token_id": 151666, "target_layer_ids": [0, 1]}
+			}`))
+			Expect(ok).To(BeFalse())
+		})
+
+		It("reports no head on unparseable JSON", func() {
+			_, ok := HasSafetensorsMTPHead([]byte(`{not json`))
+			Expect(ok).To(BeFalse())
+		})
+
+		It("reports no head on empty input", func() {
+			_, ok := HasSafetensorsMTPHead(nil)
+			Expect(ok).To(BeFalse())
+		})
+	})
+
+	Context("IsDFlashDraftConfig", func() {
+		It("recognises a draft by its dflash_config block", func() {
+			Expect(IsDFlashDraftConfig([]byte(`{
+				"dflash_config": {"mask_token_id": 151666, "target_layer_ids": [0]}
+			}`))).To(BeTrue())
+		})
+
+		It("does not flag an ordinary checkpoint", func() {
+			Expect(IsDFlashDraftConfig([]byte(`{"model_type": "qwen3_5_moe"}`))).To(BeFalse())
+		})
+	})
+
+	Context("ApplyVLLMSpeculativeDefaults", func() {
+		It("writes the mtp method into engine_args", func() {
+			cfg := &ModelConfig{Name: "qwen"}
+			ApplyVLLMSpeculativeDefaults(cfg, 1)
+			Expect(cfg.EngineArgs).To(HaveKey("speculative_config"))
+			spec, ok := cfg.EngineArgs["speculative_config"].(map[string]any)
+			Expect(ok).To(BeTrue())
+			Expect(spec["method"]).To(Equal("mtp"))
+		})
+
+		It("leaves an existing speculative_config alone", func() {
+			cfg := &ModelConfig{
+				Name: "qwen",
+				LLMConfig: LLMConfig{
+					EngineArgs: map[string]any{
+						"speculative_config": map[string]any{"method": "ngram", "num_speculative_tokens": 4},
+					},
+				},
+			}
+			ApplyVLLMSpeculativeDefaults(cfg, 1)
+			spec := cfg.EngineArgs["speculative_config"].(map[string]any)
+			Expect(spec["method"]).To(Equal("ngram"))
+		})
+
+		It("preserves unrelated engine_args keys", func() {
+			cfg := &ModelConfig{
+				Name:      "qwen",
+				LLMConfig: LLMConfig{EngineArgs: map[string]any{"max_num_seqs": 32}},
+			}
+			ApplyVLLMSpeculativeDefaults(cfg, 1)
+			Expect(cfg.EngineArgs).To(HaveKeyWithValue("max_num_seqs", 32))
+			Expect(cfg.EngineArgs).To(HaveKey("speculative_config"))
+		})
+
+		It("tolerates a nil config", func() {
+			Expect(func() { ApplyVLLMSpeculativeDefaults(nil, 1) }).ToNot(Panic())
+		})
+	})
+})

core/gallery/estimate_warm.go

@@ -9,6 +9,7 @@ import (
 	"time"
 
 	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/concurrency"
 	"github.com/mudler/LocalAI/pkg/system"
 	"github.com/mudler/LocalAI/pkg/vram"
 	"github.com/mudler/xlog"
@@ -101,7 +102,7 @@ func WarmEstimateCache(ctx context.Context, galleries []config.Gallery, systemSt
 		return
 	}
 
-	go func() {
+	concurrency.SafeGo(func() {
 		started := time.Now()
 
 		models, err := AvailableGalleryModelsCached(galleries, systemState)
@@ -131,7 +132,7 @@ func WarmEstimateCache(ctx context.Context, galleries []config.Gallery, systemSt
 
 		for i := 0; i < cfg.Concurrency; i++ {
 			wg.Add(1)
-			go func() {
+			concurrency.SafeGo(func() {
 				defer wg.Done()
 				for m := range cursor {
 					// Per entry, not for the run: one unreachable weight file
@@ -164,7 +165,7 @@ func WarmEstimateCache(ctx context.Context, galleries []config.Gallery, systemSt
 
 					cancel()
 				}
-			}()
+			})
 		}
 
 	feed:
@@ -183,7 +184,7 @@ func WarmEstimateCache(ctx context.Context, galleries []config.Gallery, systemSt
 			return
 		}
 		xlog.Info("gallery caches warmed", "estimates", warmed, "variants", warmedVariants, "of", len(models), "took", time.Since(started).Round(time.Second))
-	}()
+	})
 }
 
 // EstimateWarmConfigFromEnv reads the warm-up bounds from the environment,

core/gallery/estimate_warm_test.go

@@ -1,11 +1,20 @@
 package gallery_test
 
 import (
+	"bytes"
 	"context"
+	"encoding/binary"
+	"math"
+	"net/http"
+	"net/http/httptest"
 	"os"
+	"path/filepath"
+	"time"
 
+	gguf "github.com/gpustack/gguf-parser-go"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	"gopkg.in/yaml.v3"
 
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/core/gallery"
@@ -57,6 +66,46 @@ var _ = Describe("VRAM estimate warm-up", func() {
 		Consistently(func() bool { return true }, "100ms").Should(BeTrue())
 	})
 
+	It("does not crash the server when remote GGUF metadata is malformed", func() {
+		payload := warmMalformedGGUF()
+		requested := make(chan struct{})
+		server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			select {
+			case <-requested:
+			default:
+				close(requested)
+			}
+			http.ServeContent(w, r, "model.gguf", time.Time{}, bytes.NewReader(payload))
+		}))
+		DeferCleanup(server.Close)
+
+		galleryPath := filepath.Join(state.Model.ModelsPath, "malformed-gallery.yaml")
+		index, err := yaml.Marshal([]gallery.GalleryModel{{Metadata: gallery.Metadata{
+			Name: "malformed-gguf",
+			AdditionalFiles: []gallery.File{{
+				Filename: "model.gguf",
+				URI:      server.URL + "/model.gguf",
+			}},
+		}}})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(os.WriteFile(galleryPath, index, 0600)).To(Succeed())
+
+		cfg := gallery.DefaultEstimateWarmConfig
+		cfg.Limit = 1
+		cfg.Concurrency = 1
+		cfg.Contexts = []uint32{8192}
+		gallery.WarmEstimateCache(context.Background(), []config.Gallery{{
+			Name: "malformed",
+			URL:  "file://" + galleryPath,
+		}}, state, cfg)
+
+		Eventually(requested, "2s").Should(BeClosed())
+		// The warm-up is detached. Give its parser time to consume the response;
+		// before the recovery boundary, that goroutine panicked and killed the
+		// entire test process (and the LocalAI server in production).
+		Consistently(func() bool { return true }, "300ms").Should(BeTrue())
+	})
+
 	Describe("configuration from the environment", func() {
 		AfterEach(func() {
 			os.Unsetenv("LOCALAI_VRAM_WARM_LIMIT")
@@ -113,3 +162,19 @@ var _ = Describe("VRAM estimate warm-up", func() {
 	})
 
 })
+
+func warmMalformedGGUF() []byte {
+	payload := make([]byte, 0, 128)
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMagicGGUFLe))
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFVersionV3))
+	payload = binary.LittleEndian.AppendUint64(payload, 0)
+	payload = binary.LittleEndian.AppendUint64(payload, 1)
+	key := "tokenizer.ggml.tokens"
+	payload = binary.LittleEndian.AppendUint64(payload, uint64(len(key)))
+	payload = append(payload, key...)
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeArray))
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeString))
+	payload = binary.LittleEndian.AppendUint64(payload, 1)
+	payload = binary.LittleEndian.AppendUint64(payload, math.MaxUint64)
+	return payload
+}

core/gallery/gallery.go

@@ -612,17 +612,24 @@ func getGalleryElements[T GalleryElement](gallery config.Gallery, basePath strin
 		}
 	}
 
-	uri := downloader.URI(gallery.URL)
-
 	if len(models) == 0 {
-		err := uri.ReadWithCallback(basePath, func(url string, d []byte) error {
-			galleryCache.Set(cacheKey, galleryCacheEntry{
-				yamlEntry:   d,
-				lastUpdated: time.Now(),
-			})
-			return yaml.Unmarshal(d, &models)
-		})
+		// The cache key stays the gallery's identity rather than the URL that
+		// answered: a mirror serves the same index, so a mirror-served fetch
+		// must populate the entry the primary would have filled.
+		body, servedBy, err := fetchGalleryIndex(context.Background(), gallery, basePath)
 		if err != nil {
+			return models, fmt.Errorf("failed to read gallery elements: %w", err)
+		}
+		if servedBy != gallery.URL {
+			// A mirror's URL, or the path of the last known good copy on disk
+			// when nothing was reachable at all — either way, not the primary.
+			xlog.Info("gallery served by a fallback source", "gallery", gallery.Name, "source", servedBy)
+		}
+		galleryCache.Set(cacheKey, galleryCacheEntry{
+			yamlEntry:   body,
+			lastUpdated: time.Now(),
+		})
+		if err := yaml.Unmarshal(body, &models); err != nil {
 			if yamlErr, ok := err.(*yaml.TypeError); ok {
 				xlog.Debug("YAML errors", "errors", strings.Join(yamlErr.Errors, "\n"), "models", models)
 			}

core/gallery/gallery_mirrors.go

@@ -0,0 +1,274 @@
+package gallery
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"os"
+	"path/filepath"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/downloader"
+	"github.com/mudler/LocalAI/pkg/xsync"
+	"github.com/mudler/xlog"
+	"gopkg.in/yaml.v3"
+)
+
+// galleryFetchTimeout bounds a single candidate attempt. GitHub's raw endpoint
+// degrades by getting slow far more often than by returning an error, so the
+// timeout — not the mirror list — is what actually gets a user to a working
+// gallery on a bad day.
+//
+// It is deliberately far longer than a healthy fetch needs. The downloader only
+// ever bounded the response headers, never the body, so this is the first
+// whole-transfer deadline this path has had: too tight a value would fail slow
+// links that work today and then park a perfectly healthy source in cooldown
+// for ten minutes. The default index is ~2.2 MB, so 120s tolerates a sustained
+// ~19 KB/s — below any link that could go on to install a model.
+//
+// A var rather than a const so tests can shorten it.
+var galleryFetchTimeout = 120 * time.Second
+
+// galleryFailureCooldown keeps a candidate that just failed out of the rotation
+// for a while. Without it, every gallery listing pays the full timeout against
+// a dead host before reaching a mirror that works.
+const galleryFailureCooldown = 10 * time.Minute
+
+// galleryFailures records when each candidate URL last failed. It is
+// package-level and shared by every gallery: the point is that a host which is
+// down stays skipped across listings, and the URL is what identifies it.
+var galleryFailures = xsync.NewSyncedMap[string, time.Time]()
+
+// galleryCandidates returns the URLs to try, primary first. Empty and repeated
+// entries are dropped so a copy-pasted config cannot make us dial the same
+// dead host three times.
+//
+// Deliberately no SSRF validation here. validateGalleryConfigURL guards
+// GetGalleryConfigFromURL because that URL arrives in a request body; these
+// come from the operator's own gallery configuration (LOCALAI_GALLERIES or the
+// admin-gated POST /api/settings), the same place the primary URL has always
+// come from, and the index fetch has never validated the primary. A mirror is
+// no more privileged than the URL it backs up, so validating mirrors while the
+// primary goes unchecked would buy nothing and would break the deployment
+// mirrors exist for: an index served from a host on the LAN. file:// mirrors
+// remain confined to the models directory by the downloader's basePath check.
+func galleryCandidates(g config.Gallery) []string {
+	seen := make(map[string]struct{}, len(g.Mirrors)+1)
+	out := make([]string, 0, len(g.Mirrors)+1)
+
+	for _, candidate := range append([]string{g.URL}, g.Mirrors...) {
+		if candidate == "" {
+			continue
+		}
+		if _, dup := seen[candidate]; dup {
+			continue
+		}
+		seen[candidate] = struct{}{}
+		out = append(out, candidate)
+	}
+	return out
+}
+
+// inCooldown reports whether a candidate failed recently enough to skip.
+//
+// Exists and Get take the lock separately, so a concurrent Delete between them
+// yields the zero time and reads as "not in cooldown". That is the harmless
+// direction: the cost is one extra dial, never a skipped source.
+func inCooldown(url string) bool {
+	if !galleryFailures.Exists(url) {
+		return false
+	}
+	failedAt := galleryFailures.Get(url)
+	if failedAt.IsZero() || time.Since(failedAt) >= galleryFailureCooldown {
+		galleryFailures.Delete(url)
+		return false
+	}
+	return true
+}
+
+// galleryCachePath is where the last known good copy of an index lives.
+//
+// Deliberately not inside basePath: getGalleryElements' caller treats every
+// <name>.yaml in the models directory as an installed model config, so a cached
+// index there would be misread as a model. The sibling cache directory follows
+// the precedent in core/services/worker/file_staging.go. The name is a digest
+// of the gallery URL so the model and the backend gallery — often fetched with
+// sibling base paths — cannot overwrite each other.
+//
+// A non-absolute basePath yields no path at all: "", "." and "models" all
+// resolve the sibling against the process' working directory, which is not
+// somewhere LocalAI should be dropping files. Only an absolute models
+// directory names a location we can reason about.
+func galleryCachePath(basePath, url string) string {
+	if !filepath.IsAbs(basePath) {
+		return ""
+	}
+	sum := sha256.Sum256([]byte(url))
+	return filepath.Join(basePath, "..", "cache", "gallery", hex.EncodeToString(sum[:])+".yaml")
+}
+
+// isUsableGalleryIndex reports whether body is worth keeping as the last known
+// good copy.
+//
+// HTTP 200 does not mean "index": a captive portal, a corporate proxy or a CDN
+// error page all answer 200 with HTML, and the fetch path has no other reason
+// to look at the bytes — the parse only happens later, in getGalleryElements.
+// Persisting on status alone therefore lets an interception page overwrite a
+// good copy, and the next offline start — the one case this cache exists for —
+// would serve that page instead of the gallery it already had.
+//
+// An empty document is rejected for the same reason. It parses fine, so a
+// probe that only checked the parse would let a source that answers with a
+// blank body replace a populated index with one that lists nothing; from the
+// user's side an empty gallery and an unparseable one are the same outage. A
+// genuinely empty index is worth nothing offline anyway, so there is no case
+// where keeping it beats keeping what came before.
+//
+// The shape check is deliberately shallow — a top-level YAML sequence — because
+// this is a guard against "not an index at all", not a schema validator.
+// getGalleryElements still does the real typed unmarshal.
+func isUsableGalleryIndex(body []byte) bool {
+	var probe []any
+	if err := yaml.Unmarshal(body, &probe); err != nil {
+		return false
+	}
+	return len(probe) > 0
+}
+
+// persistGalleryIndex stores a freshly fetched index for the next time nothing
+// is reachable.
+//
+// Every failure here is logged at debug and otherwise ignored: the copy is an
+// optimisation, and a read-only or full disk must not turn a gallery that was
+// fetched perfectly well into a failed listing.
+func persistGalleryIndex(basePath, url string, body []byte) {
+	path := galleryCachePath(basePath, url)
+	if path == "" {
+		return
+	}
+	if !isUsableGalleryIndex(body) {
+		xlog.Debug("refusing to cache a response that is not a gallery index",
+			"url", url, "bytes", len(body))
+		return
+	}
+	// 0o750: the cache is LocalAI's own bookkeeping, so nothing outside the
+	// server's user and group has any reason to traverse it.
+	if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
+		xlog.Debug("could not create gallery cache directory", "path", path, "error", err)
+		return
+	}
+	// Write via a temporary file so an interrupted write cannot leave a
+	// truncated index that the next offline start would try to parse.
+	tmp, err := os.CreateTemp(filepath.Dir(path), ".gallery-*.tmp")
+	if err != nil {
+		xlog.Debug("could not stage gallery cache", "path", path, "error", err)
+		return
+	}
+	tmpName := tmp.Name()
+	if _, err := tmp.Write(body); err != nil {
+		// The write already failed; a close or unlink error on the way out
+		// changes nothing about the outcome and has nowhere useful to go.
+		_ = tmp.Close()
+		_ = os.Remove(tmpName)
+		xlog.Debug("could not write gallery cache", "path", path, "error", err)
+		return
+	}
+	if err := tmp.Close(); err != nil {
+		_ = os.Remove(tmpName)
+		xlog.Debug("could not flush gallery cache", "path", path, "error", err)
+		return
+	}
+	if err := os.Rename(tmpName, path); err != nil {
+		_ = os.Remove(tmpName)
+		xlog.Debug("could not install gallery cache", "path", path, "error", err)
+	}
+}
+
+// fetchGalleryIndex returns the raw index bytes and the URL that served them,
+// trying each candidate in order.
+//
+// A candidate in cooldown is skipped, unless every candidate is in cooldown —
+// in which case the cooldown is ignored rather than failing outright, because
+// refusing to serve a gallery we might be able to reach is worse than one slow
+// request.
+//
+// If no candidate answers, the last known good copy on disk is served and its
+// path is returned as the source. Nothing else in the chain helps a machine
+// that has no network at all.
+func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string) ([]byte, string, error) {
+	candidates := galleryCandidates(g)
+	if len(candidates) == 0 {
+		return nil, "", fmt.Errorf("gallery %q has no URL", g.Name)
+	}
+
+	attempt := make([]string, 0, len(candidates))
+	for _, c := range candidates {
+		if !inCooldown(c) {
+			attempt = append(attempt, c)
+		}
+	}
+	if len(attempt) == 0 {
+		attempt = candidates
+	}
+
+	var lastErr error
+	for _, candidate := range attempt {
+		attemptCtx, cancel := context.WithTimeout(ctx, galleryFetchTimeout)
+
+		var body []byte
+		err := downloader.URI(candidate).ReadWithAuthorizationAndCallback(
+			attemptCtx, basePath, "",
+			func(_ string, d []byte) error {
+				body = d
+				return nil
+			})
+		cancel()
+
+		if err == nil {
+			// A source that answers is usable again immediately; leaving the
+			// record behind would keep a recovered host skipped.
+			galleryFailures.Delete(candidate)
+			// Keyed on the gallery's own URL rather than the candidate that
+			// answered: a mirror serves the same index, so a mirror-served
+			// fetch must refresh the copy an offline run will look for.
+			persistGalleryIndex(basePath, g.URL, body)
+			return body, candidate, nil
+		}
+
+		lastErr = err
+		// Only blame the source for its own failures. If the caller gave up —
+		// a browser disconnecting mid-listing, once a request context is wired
+		// through here — recording that would blackhole every candidate for ten
+		// minutes over something the sources had no part in.
+		if ctx.Err() == nil {
+			galleryFailures.Set(candidate, time.Now())
+		}
+		xlog.Warn("gallery source unreachable, trying the next one",
+			"gallery", g.Name, "url", candidate, "error", err)
+	}
+
+	// Every source failed. A copy from a previous run is much better than no
+	// gallery at all — this is what lets an offline or airgapped machine still
+	// list what it already knows about.
+	cachePath := galleryCachePath(basePath, g.URL)
+	if cachePath != "" {
+		// #nosec G304 -- cachePath is galleryCachePath's own construction: a
+		// hex sha256 of the URL under the fixed <basePath>/../cache/gallery
+		// directory, with a non-absolute basePath already rejected. No part of
+		// it is caller-supplied text, so there is nothing to traverse with.
+		if body, readErr := os.ReadFile(cachePath); readErr == nil {
+			xlog.Warn("all gallery sources failed, serving the last known good copy",
+				"gallery", g.Name, "path", cachePath, "error", lastErr)
+			return body, cachePath, nil
+		}
+	}
+
+	// Report what was configured and what was skipped, not just what we dialled:
+	// "all 1 source(s) failed" on a gallery with three mirrors reads as a
+	// misconfiguration and sends the operator looking for the missing mirrors,
+	// when the truth is that two of them are in cooldown.
+	return nil, "", fmt.Errorf("all %d source(s) for gallery %q failed (%d configured, %d skipped as recently failed) and no cached copy exists, last error: %w",
+		len(attempt), g.Name, len(candidates), len(candidates)-len(attempt), lastErr)
+}

core/gallery/gallery_mirrors_test.go

@@ -0,0 +1,677 @@
+package gallery
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"gopkg.in/yaml.v3"
+)
+
+// TestMain gives this package its own temporary root so the gallery index cache
+// cannot escape it.
+//
+// The cache is a sibling of the models directory (<models>/../cache/gallery),
+// which is right in production but leaks under test: a models directory made
+// with os.MkdirTemp("", …) gets one directly under the system temp directory,
+// so the sibling resolves to /tmp/cache — a path no test framework cleans up,
+// left behind after every run. Pointing TMPDIR at a directory we remove
+// ourselves contains the sibling without having to rewrite every call site,
+// and covers any added later.
+func TestMain(m *testing.M) {
+	root, err := os.MkdirTemp("", "localai-gallery-tests-*")
+	if err != nil {
+		panic(err)
+	}
+	// os.TempDir consults TMPDIR on every call, so this applies to temp
+	// directories created from here on.
+	if err := os.Setenv("TMPDIR", root); err != nil {
+		panic(err)
+	}
+
+	code := m.Run()
+
+	// Not deferred: os.Exit does not run deferred functions. Nothing useful
+	// can be done about a failure to clean up a temporary directory at this
+	// point, and the exit code must stay the suite's.
+	_ = os.RemoveAll(root)
+	os.Exit(code)
+}
+
+// resetGalleryFailures and expireGalleryFailure exist so specs can drive the
+// cooldown without sleeping. They live here because nothing in the production
+// path ever needs to reach into the failure map.
+func resetGalleryFailures() {
+	for _, k := range galleryFailures.Keys() {
+		galleryFailures.Delete(k)
+	}
+}
+
+func expireGalleryFailure(url string, at time.Time) {
+	galleryFailures.Set(url, at)
+}
+
+// tempModelsDir returns an absolute models directory whose parent is private to
+// the calling spec, so the sibling cache (<models>/../cache/gallery) is
+// isolated too. A bare temp directory would put every spec's cache in one
+// shared place, where the specs that count files in it see each other's.
+func tempModelsDir() string {
+	GinkgoHelper()
+	root, err := os.MkdirTemp("", "gallery-mirrors-spec-*")
+	Expect(err).ToNot(HaveOccurred())
+	DeferCleanup(func() { _ = os.RemoveAll(root) })
+
+	models := filepath.Join(root, "models")
+	Expect(os.MkdirAll(models, 0o750)).To(Succeed())
+	return models
+}
+
+// countingServer serves body with status, counting the requests it actually
+// received. The counter is atomic because the handler runs on the server's
+// goroutine while the assertions run on the spec's.
+func countingServer(status int, body string) (*httptest.Server, *atomic.Int64) {
+	GinkgoHelper()
+	var hits atomic.Int64
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		hits.Add(1)
+		if status >= 400 {
+			http.Error(w, body, status)
+			return
+		}
+		_, _ = w.Write([]byte(body))
+	}))
+	DeferCleanup(srv.Close)
+	return srv, &hits
+}
+
+var _ = Describe("galleryCandidates", func() {
+	It("orders the primary first", func() {
+		Expect(galleryCandidates(config.Gallery{
+			URL:     "https://primary/index.yaml",
+			Mirrors: []string{"https://a/index.yaml", "https://b/index.yaml"},
+		})).To(Equal([]string{"https://primary/index.yaml", "https://a/index.yaml", "https://b/index.yaml"}))
+	})
+
+	It("drops empty and duplicate entries", func() {
+		Expect(galleryCandidates(config.Gallery{
+			URL:     "https://primary/index.yaml",
+			Mirrors: []string{"", "https://primary/index.yaml", "https://a/index.yaml", "https://a/index.yaml"},
+		})).To(Equal([]string{"https://primary/index.yaml", "https://a/index.yaml"}),
+			"want the primary then the single distinct mirror")
+	})
+
+	// A gallery whose primary URL is empty still has usable mirrors; dropping
+	// the empty must not drop the rest with it.
+	It("keeps the mirrors when the primary is empty", func() {
+		Expect(galleryCandidates(config.Gallery{Mirrors: []string{"https://a/index.yaml"}})).
+			To(Equal([]string{"https://a/index.yaml"}))
+	})
+})
+
+var _ = Describe("fetchGalleryIndex", func() {
+	BeforeEach(resetGalleryFailures)
+
+	It("falls back to a mirror when the primary fails", func() {
+		primary, _ := countingServer(http.StatusInternalServerError, "down")
+		mirror, _ := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		body, served, err := fetchGalleryIndex(context.Background(), config.Gallery{
+			URL:     primary.URL,
+			Mirrors: []string{mirror.URL},
+		}, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(served).To(Equal(mirror.URL))
+		Expect(string(body)).To(Equal("- name: from-mirror\n"))
+	})
+
+	It("prefers the primary when it works", func() {
+		mirror, mirrorHits := countingServer(http.StatusOK, "- name: from-mirror\n")
+		primary, _ := countingServer(http.StatusOK, "- name: from-primary\n")
+
+		body, served, err := fetchGalleryIndex(context.Background(), config.Gallery{
+			URL:     primary.URL,
+			Mirrors: []string{mirror.URL},
+		}, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(served).To(Equal(primary.URL))
+		Expect(string(body)).To(Equal("- name: from-primary\n"))
+		Expect(mirrorHits.Load()).To(BeZero(), "mirror was contacted even though the primary answered")
+	})
+
+	It("errors when every candidate fails", func() {
+		down, hits := countingServer(http.StatusInternalServerError, "down")
+
+		_, _, err := fetchGalleryIndex(context.Background(), config.Gallery{
+			URL:     down.URL,
+			Mirrors: []string{down.URL + "/other"},
+		}, tempModelsDir())
+		Expect(err).To(HaveOccurred(), "want an error when nothing can serve the index")
+		Expect(hits.Load()).To(BeEquivalentTo(2), "want both candidates tried")
+	})
+
+	It("errors for a gallery with neither a URL nor mirrors", func() {
+		_, _, err := fetchGalleryIndex(context.Background(), config.Gallery{Name: "empty"}, tempModelsDir())
+		Expect(err).To(HaveOccurred())
+	})
+
+	// An HTTP error page is not an index. Without this the downloader hands
+	// back a 404 body as if it were content, the fallback never triggers, and
+	// the junk gets cached for an hour.
+	It("treats an HTTP error status as a failure", func() {
+		primary, _ := countingServer(http.StatusNotFound, "no such index")
+		mirror, _ := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		body, served, err := fetchGalleryIndex(context.Background(), config.Gallery{
+			URL:     primary.URL,
+			Mirrors: []string{mirror.URL},
+		}, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(served).To(Equal(mirror.URL), "a 404 body was taken for an index")
+		Expect(string(body)).To(Equal("- name: from-mirror\n"))
+	})
+
+	// A caller that has already given up must not be dragged through the whole
+	// candidate list.
+	It("honours the caller's context", func() {
+		srv, hits := countingServer(http.StatusOK, "- name: from-primary\n")
+
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+
+		_, _, err := fetchGalleryIndex(ctx, config.Gallery{URL: srv.URL}, tempModelsDir())
+		Expect(err).To(HaveOccurred(), "want an error when the caller's context is already cancelled")
+		Expect(hits.Load()).To(BeZero(), "server dialled despite a cancelled context")
+		// The source did nothing wrong. Blaming it would blackhole a healthy
+		// candidate for ten minutes because a browser tab closed.
+		Expect(inCooldown(srv.URL)).To(BeFalse(),
+			"caller cancellation was recorded as a failure of the source")
+	})
+
+	// A candidate that accepts the connection and then never answers is the
+	// failure mode the per-attempt timeout exists for: without it the whole
+	// listing hangs on one bad host and the mirrors are never reached.
+	It("gives up on a hanging candidate", func() {
+		release := make(chan struct{})
+		var hangHits atomic.Int64
+		hang := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			hangHits.Add(1)
+			select {
+			case <-release:
+			case <-r.Context().Done():
+			}
+		}))
+		DeferCleanup(func() {
+			close(release)
+			hang.Close()
+		})
+
+		mirror, mirrorHits := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		restore := galleryFetchTimeout
+		galleryFetchTimeout = 100 * time.Millisecond
+		DeferCleanup(func() { galleryFetchTimeout = restore })
+
+		g := config.Gallery{URL: hang.URL, Mirrors: []string{mirror.URL}}
+		basePath := tempModelsDir()
+
+		type outcome struct {
+			served string
+			err    error
+		}
+		done := make(chan outcome, 1)
+		go func() {
+			defer GinkgoRecover()
+			_, served, err := fetchGalleryIndex(context.Background(), g, basePath)
+			done <- outcome{served, err}
+		}()
+
+		// The assertion has to be bounded: an unbounded attempt does not fail,
+		// it hangs, and a hung spec is a useless signal.
+		var got outcome
+		Eventually(done, 30*time.Second).Should(Receive(&got),
+			"fetch never returned — a hanging candidate is not bounded by a per-attempt timeout")
+		Expect(got.err).ToNot(HaveOccurred())
+		Expect(got.served).To(Equal(mirror.URL))
+
+		Expect(hangHits.Load()).To(BeEquivalentTo(1), "hanging candidate should be dialled once")
+		Expect(mirrorHits.Load()).To(BeEquivalentTo(1), "the timed-out attempt did not fall through")
+		// A timeout is the source's own failure, unlike caller cancellation.
+		Expect(inCooldown(hang.URL)).To(BeTrue(), "a candidate that timed out was not put in cooldown")
+	})
+
+	// "all 1 source(s) failed" on a three-mirror gallery reads as a
+	// misconfiguration; the operator needs to see that the rest were skipped.
+	It("reports how many sources were configured and skipped when all fail", func() {
+		down, _ := countingServer(http.StatusInternalServerError, "down")
+		g := config.Gallery{
+			URL:     down.URL,
+			Name:    "localai",
+			Mirrors: []string{down.URL + "/a", down.URL + "/b"},
+		}
+
+		// Two of the three are already in cooldown, so only one is dialled.
+		expireGalleryFailure(down.URL+"/a", time.Now())
+		expireGalleryFailure(down.URL+"/b", time.Now())
+
+		_, _, err := fetchGalleryIndex(context.Background(), g, tempModelsDir())
+		Expect(err).To(HaveOccurred(), "want an error when nothing can serve the index")
+		Expect(err.Error()).To(And(
+			ContainSubstring("3 configured"),
+			ContainSubstring("2 skipped"),
+		), "the error does not say how many sources were configured and skipped")
+	})
+})
+
+var _ = Describe("the gallery source cooldown", func() {
+	BeforeEach(resetGalleryFailures)
+
+	// A dead primary must not be re-dialled on every call. Without this, a
+	// gallery listing in the UI pays the full timeout against a dead host every
+	// refresh.
+	It("skips a failed candidate while it is cooling down", func() {
+		primary, hits := countingServer(http.StatusInternalServerError, "down")
+		mirror, _ := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		g := config.Gallery{URL: primary.URL, Mirrors: []string{mirror.URL}}
+		for i := 0; i < 3; i++ {
+			_, _, err := fetchGalleryIndex(context.Background(), g, tempModelsDir())
+			Expect(err).ToNot(HaveOccurred(), "fetch %d", i)
+		}
+		Expect(hits.Load()).To(BeEquivalentTo(1), "primary re-dialled — cooldown is not holding")
+	})
+
+	It("expires", func() {
+		primary, hits := countingServer(http.StatusInternalServerError, "down")
+		mirror, _ := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		g := config.Gallery{URL: primary.URL, Mirrors: []string{mirror.URL}}
+		_, _, err := fetchGalleryIndex(context.Background(), g, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+
+		// Age the recorded failure past the cooldown rather than sleeping.
+		expireGalleryFailure(primary.URL, time.Now().Add(-2*galleryFailureCooldown))
+		_, _, err = fetchGalleryIndex(context.Background(), g, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(hits.Load()).To(BeEquivalentTo(2), "primary was not re-dialled — cooldown never expired")
+	})
+
+	// Refusing to serve a gallery because every source is in cooldown is worse
+	// than paying for one slow request, so the cooldown is ignored when it
+	// would leave nothing to try.
+	It("is ignored when every candidate is cooling down", func() {
+		primary, primaryHits := countingServer(http.StatusInternalServerError, "down")
+		mirror, mirrorHits := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		// Put both candidates in cooldown without dialling them.
+		expireGalleryFailure(primary.URL, time.Now())
+		expireGalleryFailure(mirror.URL, time.Now())
+
+		_, served, err := fetchGalleryIndex(context.Background(), config.Gallery{
+			URL:     primary.URL,
+			Mirrors: []string{mirror.URL},
+		}, tempModelsDir())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(served).To(Equal(mirror.URL))
+		Expect(primaryHits.Load()).To(BeEquivalentTo(1), "cooldown should have been ignored, not obeyed")
+		Expect(mirrorHits.Load()).To(BeEquivalentTo(1), "cooldown should have been ignored, not obeyed")
+	})
+
+	// A source that answers is out of cooldown immediately, otherwise a host
+	// that blipped once stays skipped for ten minutes after it has recovered.
+	It("is cleared by a successful fetch", func() {
+		srv, hits := countingServer(http.StatusOK, "- name: ok\n")
+
+		expireGalleryFailure(srv.URL, time.Now())
+		g := config.Gallery{URL: srv.URL}
+		for i := 0; i < 2; i++ {
+			_, _, err := fetchGalleryIndex(context.Background(), g, tempModelsDir())
+			Expect(err).ToNot(HaveOccurred(), "fetch %d", i)
+		}
+		Expect(hits.Load()).To(BeEquivalentTo(2), "a successful fetch must clear the cooldown")
+		Expect(inCooldown(srv.URL)).To(BeFalse(), "candidate still in cooldown after answering")
+	})
+})
+
+// getGalleryElements is the choke point every gallery listing goes through, so
+// the fallback has to be reachable from there and not just from the helper.
+var _ = Describe("getGalleryElements", func() {
+	BeforeEach(resetGalleryFailures)
+
+	It("falls back to a mirror", func() {
+		primary, _ := countingServer(http.StatusInternalServerError, "down")
+		mirror, _ := countingServer(http.StatusOK, "- name: mirror-model\n  description: served by a mirror\n")
+
+		g := config.Gallery{Name: "mirror-fallback-spec", URL: primary.URL, Mirrors: []string{mirror.URL}}
+		DeferCleanup(func() { galleryCache.Delete(g.Name + "-" + g.URL) })
+
+		models, err := getGalleryElements(g, tempModelsDir(), func(*GalleryModel) bool { return false })
+		Expect(err).ToNot(HaveOccurred())
+		Expect(models).To(HaveLen(1))
+		Expect(models[0].Name).To(Equal("mirror-model"))
+
+		// The cache identifies the gallery, not whichever source answered, so a
+		// mirror-served fetch must populate the entry the primary URL would hit.
+		Expect(galleryCache.Exists(g.Name + "-" + g.URL)).To(BeTrue(),
+			"mirror-served index was not cached under the gallery's own key")
+	})
+})
+
+var _ = Describe("galleryCachePath", func() {
+	// The cache must never land in the models directory, where a *.yaml file is
+	// interpreted as an installed model config.
+	It("is outside the models directory", func() {
+		base := tempModelsDir()
+		got := galleryCachePath(base, "https://example/index.yaml")
+		Expect(filepath.Dir(got)).ToNot(Equal(base), "cache path is inside the models directory")
+
+		// Nor anywhere below it: the models directory is walked and listed, and
+		// a cache subdirectory in there is LocalAI's own litter in the user's
+		// models.
+		rel, err := filepath.Rel(base, got)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(rel).To(HavePrefix(".."), "cache path %q is under the models directory", got)
+	})
+
+	// The model gallery and the backend gallery are both fetched, often under
+	// the same parent directory. Keying the file on the URL is what stops one
+	// from being served as the other.
+	It("distinguishes galleries", func() {
+		base := tempModelsDir()
+		models := galleryCachePath(base, "https://example/index.yaml")
+		backends := galleryCachePath(base, "https://example/backends.yaml")
+		Expect(models).ToNot(Equal(backends), "one gallery would overwrite the other")
+		Expect(galleryCachePath(base, "https://example/index.yaml")).To(Equal(models),
+			"the same gallery URL produced two different cache paths")
+	})
+
+	// Without a models directory there is no sensible place for the cache, and
+	// a relative path would write next to the process' working directory.
+	It("yields nothing without a models directory", func() {
+		Expect(galleryCachePath("", "https://example/index.yaml")).To(BeEmpty())
+		// Must not panic or write anywhere either.
+		persistGalleryIndex("", "https://example/index.yaml", []byte("- name: x\n"))
+	})
+
+	// A relative models directory is the same failure as an empty one: "." and
+	// "models" both resolve against whatever directory the process happens to
+	// be running in, which is exactly what the guard exists to prevent.
+	DescribeTable("rejects a relative models directory",
+		func(base string) {
+			Expect(galleryCachePath(base, "https://example/index.yaml")).To(BeEmpty(),
+				"it resolves against the working directory")
+			// And nothing may be written next to the working directory either.
+			persistGalleryIndex(base, "https://example/index.yaml", []byte("- name: x\n"))
+		},
+		Entry("the working directory itself", "."),
+		Entry("a bare relative name", "models"),
+		Entry("an explicitly relative path", "./models"),
+		Entry("a parent-relative path", "../models"),
+	)
+
+	// Sanity: the guard must still let a real absolute models directory through.
+	It("accepts an absolute models directory", func() {
+		Expect(galleryCachePath(tempModelsDir(), "https://example/index.yaml")).ToNot(BeEmpty())
+	})
+})
+
+var _ = Describe("the last known good gallery index", func() {
+	BeforeEach(resetGalleryFailures)
+
+	It("is written after a successful fetch", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte("- name: cached\n"))
+		}))
+		DeferCleanup(srv.Close)
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		body, err := os.ReadFile(galleryCachePath(base, srv.URL))
+		Expect(err).ToNot(HaveOccurred(), "no cached copy written")
+		Expect(string(body)).To(Equal("- name: cached\n"))
+	})
+
+	// The offline case: nothing is reachable, but a previous run left a copy.
+	It("is served when every source fails", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte("- name: cached\n"))
+		}))
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		srv.Close() // now nothing is reachable
+		resetGalleryFailures()
+
+		body, served, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred(), "want the cached copy")
+		Expect(string(body)).To(Equal("- name: cached\n"))
+		Expect(served).To(Equal(galleryCachePath(base, srv.URL)))
+	})
+
+	It("cannot rescue a fetch when there is no copy and no network", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
+		url := srv.URL
+		srv.Close()
+
+		_, _, err := fetchGalleryIndex(context.Background(),
+			config.Gallery{URL: url, Name: "localai"}, tempModelsDir())
+		Expect(err).To(HaveOccurred(), "want an error when there is neither a source nor a cached copy")
+	})
+
+	// The cache is an optimisation. A read-only or full disk must not turn a
+	// gallery that was fetched perfectly well into a failure.
+	It("does not fail the fetch when it cannot be written", func() {
+		srv, _ := countingServer(http.StatusOK, "- name: live\n")
+
+		base := tempModelsDir()
+		// A regular file where the cache directory needs to be: every write
+		// below it fails, and nothing can repair it at runtime.
+		Expect(os.WriteFile(filepath.Join(base, "..", "cache"), []byte("not a directory"), 0o600)).To(Succeed())
+
+		body, served, err := fetchGalleryIndex(context.Background(),
+			config.Gallery{URL: srv.URL, Name: "localai"}, base)
+		Expect(err).ToNot(HaveOccurred(), "a cache write failure failed the whole fetch")
+		Expect(served).To(Equal(srv.URL))
+		Expect(string(body)).To(Equal("- name: live\n"))
+	})
+
+	// The cache is keyed on the gallery, not on whichever source answered, so a
+	// mirror-served fetch refreshes the copy an offline run will look for.
+	It("is keyed on the gallery URL even when a mirror served it", func() {
+		primary, _ := countingServer(http.StatusInternalServerError, "down")
+		mirror, _ := countingServer(http.StatusOK, "- name: from-mirror\n")
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: primary.URL, Mirrors: []string{mirror.URL}, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		body, err := os.ReadFile(galleryCachePath(base, g.URL))
+		Expect(err).ToNot(HaveOccurred(), "no copy cached under the gallery's own URL")
+		Expect(string(body)).To(Equal("- name: from-mirror\n"))
+
+		_, err = os.ReadFile(galleryCachePath(base, mirror.URL))
+		Expect(err).To(HaveOccurred(),
+			"the copy was cached under the mirror's URL, where an offline run will not look for it")
+	})
+
+	// A reachable source always wins over the copy on disk, and the copy is
+	// refreshed with what it served — otherwise the first fetch a machine ever
+	// makes would be the only one it remembers.
+	It("loses to a live fetch, and is refreshed by it", func() {
+		served := "- name: old\n"
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte(served))
+		}))
+		DeferCleanup(srv.Close)
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		served = "- name: new\n"
+		body, from, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(body)).To(Equal("- name: new\n"), "want the live index from the source")
+		Expect(from).To(Equal(srv.URL))
+
+		onDisk, err := os.ReadFile(galleryCachePath(base, srv.URL))
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(onDisk)).To(Equal("- name: new\n"),
+			"the cached copy was not refreshed with what the source served")
+	})
+
+	// A failed fetch must leave the copy alone: writing a failure's empty body
+	// over it would destroy the only gallery an offline machine has. The staged
+	// write must not litter the cache directory either.
+	It("survives a failed fetch, and leaves no staging file behind", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte("- name: cached\n"))
+		}))
+		DeferCleanup(srv.Close)
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		cacheDir := filepath.Dir(galleryCachePath(base, g.URL))
+		entries, err := os.ReadDir(cacheDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(entries).To(HaveLen(1), "want just the index — a staging file was left behind")
+
+		srv.Close()
+		resetGalleryFailures()
+		_, _, err = fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred(), "fallback")
+
+		body, err := os.ReadFile(galleryCachePath(base, g.URL))
+		Expect(err).ToNot(HaveOccurred(), "the cached copy is gone after a failed fetch")
+		Expect(string(body)).To(Equal("- name: cached\n"), "want it untouched by a failed fetch")
+
+		entries, err = os.ReadDir(cacheDir)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(entries).To(HaveLen(1), "want just the index after a failed fetch")
+	})
+
+	// A captive portal, a corporate proxy or a CDN error page all answer HTTP
+	// 200 with HTML. Persisting on status alone lets one of those overwrite the
+	// copy an offline start depends on, which is the worst possible time to
+	// discover it.
+	It("is not overwritten by an HTML page served with status 200", func() {
+		served := "- name: cached\n"
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte(served))
+		}))
+		DeferCleanup(srv.Close)
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		// Now the same URL answers 200 with an interception page.
+		served = "<html><head><title>Sign in to the network</title></head>\n<body>Please authenticate</body></html>\n"
+		body, from, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+		// The live body is still handed back — rejecting it here would hide the
+		// failure from the caller that actually parses it.
+		Expect(from).To(Equal(srv.URL))
+		Expect(string(body)).To(Equal(served), "want the live response")
+
+		onDisk, err := os.ReadFile(galleryCachePath(base, g.URL))
+		Expect(err).ToNot(HaveOccurred(), "the cached copy is gone")
+		Expect(string(onDisk)).To(Equal("- name: cached\n"), "a 200 HTML page overwrote the good index")
+	})
+
+	// The point of the probe is what happens next: once the network is gone,
+	// the offline path must still find a copy it can parse.
+	It("still parses as a gallery index after an unparseable body was served", func() {
+		served := "- name: cached\n  description: the good index\n"
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte(served))
+		}))
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		// A proxy starts answering 200 with something that is not YAML at all.
+		served = "\t<html>\n\t  <body>502 Bad Gateway</body>\n</html>\n"
+		_, _, err = fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		srv.Close() // and now the machine is offline
+		resetGalleryFailures()
+
+		body, from, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred(), "offline fallback")
+		Expect(from).To(Equal(galleryCachePath(base, g.URL)), "want the cached copy")
+
+		// Readable by the offline path means parseable, not merely present.
+		var models []GalleryModel
+		Expect(yaml.Unmarshal(body, &models)).To(Succeed(),
+			"the offline copy no longer parses as a gallery index")
+		Expect(models).To(HaveLen(1))
+		Expect(models[0].Name).To(Equal("cached"))
+	})
+
+	// An empty document parses fine but is not an index. Replacing a populated
+	// copy with one that lists nothing is the same outage as replacing it with
+	// garbage, and an empty index is worth nothing offline, so the older copy
+	// wins.
+	DescribeTable("is not overwritten by an empty index",
+		func(empty string) {
+			served := "- name: cached\n"
+			srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				_, _ = w.Write([]byte(served))
+			}))
+			DeferCleanup(srv.Close)
+
+			base := tempModelsDir()
+			g := config.Gallery{URL: srv.URL, Name: "localai"}
+			_, _, err := fetchGalleryIndex(context.Background(), g, base)
+			Expect(err).ToNot(HaveOccurred())
+
+			served = empty
+			_, _, err = fetchGalleryIndex(context.Background(), g, base)
+			Expect(err).ToNot(HaveOccurred())
+
+			onDisk, err := os.ReadFile(galleryCachePath(base, g.URL))
+			Expect(err).ToNot(HaveOccurred(), "the cached copy is gone after an empty body %q", empty)
+			Expect(string(onDisk)).To(Equal("- name: cached\n"), "want the populated index kept")
+		},
+		Entry("no body at all", ""),
+		Entry("an empty sequence", "[]\n"),
+		Entry("a bare document marker", "---\n"),
+	)
+
+	// A machine with nothing cached and an interception page in front of it has
+	// no gallery: the junk must not be written, so the next offline start still
+	// has nothing rather than something unparseable.
+	It("is not created at all when the first fetch is unparseable", func() {
+		srv, _ := countingServer(http.StatusOK, "<html><body>hello</body></html>")
+
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "localai"}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base)
+		Expect(err).ToNot(HaveOccurred())
+
+		_, err = os.Stat(galleryCachePath(base, g.URL))
+		Expect(err).To(HaveOccurred(), "an HTML page was written as the last known good gallery index")
+	})
+})

core/gallery/importers/llama-cpp.go

@@ -298,7 +298,15 @@ func (i *LlamaCPPImporter) Import(details Details) (gallery.ModelConfig, error)
 	// imported configs already carry spec_type:draft-mtp before the model is
 	// ever loaded - users see it in the YAML preview rather than discovering
 	// it after the first start.
-	maybeApplyMTPDefaults(&modelConfig, details, &cfg)
+	//
+	// vllm-cpp is excluded on both counts: `spec_type:*` are llama.cpp option
+	// keys it does not read, and vllm.cpp rejects an MTP config over a GGUF
+	// source outright (the `mtp.*` draft tensors exist only in the safetensors
+	// checkpoint). Its MTP auto-config runs in the vllm importer instead, over
+	// the safetensors config.json.
+	if backend != "vllm-cpp" {
+		maybeApplyMTPDefaults(&modelConfig, details, &cfg)
+	}
 
 	data, err := yaml.Marshal(modelConfig)
 	if err != nil {
@@ -401,7 +409,10 @@ func maybeApplyMTPDefaults(modelConfig *config.ModelConfig, details Details, cfg
 		}
 	}()
 
-	f, err := gguf.ParseGGUFFileRemote(ctx, probeURL)
+	// MTP markers are architecture scalars. Avoid allocating tokenizer and
+	// other large arrays from an untrusted remote header; panic recovery cannot
+	// contain a fatal out-of-memory condition.
+	f, err := gguf.ParseGGUFFileRemote(ctx, probeURL, gguf.SkipLargeMetadata())
 	if err != nil {
 		xlog.Debug("[mtp-importer] failed to read remote GGUF header for MTP detection", "uri", probeURL, "error", err)
 		return

core/gallery/importers/vllm.go

@@ -1,13 +1,21 @@
 package importers
 
 import (
+	"context"
 	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
 	"path/filepath"
 	"strings"
+	"time"
 
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/core/gallery"
 	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/pkg/downloader"
+	"github.com/mudler/LocalAI/pkg/httpclient"
+	"github.com/mudler/xlog"
 	"go.yaml.in/yaml/v2"
 )
 
@@ -107,6 +115,12 @@ func (i *VLLMImporter) Import(details Details) (gallery.ModelConfig, error) {
 		// vllm python backend, so use_tokenizer_template carries over), but
 		// tool/reasoning parsing is the engine's own autoparser pipeline -
 		// the vllm-python tool_parser/reasoning_parser options don't apply.
+		//
+		// Auto-detect a Multi-Token Prediction head, the safetensors analogue
+		// of the llama-cpp importer's GGUF hook, so a freshly imported
+		// Qwen3.5 / Qwen3.6 config already carries speculative decoding in its
+		// engine_args instead of leaving the throughput on the table.
+		maybeApplyVLLMSpeculativeDefaults(&modelConfig, details)
 	} else {
 		// Auto-detect tool_parser and reasoning_parser for known model families.
 		// Surfacing them in the generated YAML lets users see and edit the choices.
@@ -132,3 +146,89 @@ func (i *VLLMImporter) Import(details Details) (gallery.ModelConfig, error) {
 		ConfigFile:  string(data),
 	}, nil
 }
+
+// maxSpecConfigProbeBytes caps the config.json body we read. Real ones are a
+// few KB; the cap keeps a hostile or mislabelled URL from streaming into the
+// importer.
+const maxSpecConfigProbeBytes = 1 << 20 // 1 MiB
+
+// specConfigProbeTimeout bounds the config.json fetch. Detection is an
+// optimisation, so it must never hold an import open for long.
+const specConfigProbeTimeout = 30 * time.Second
+
+// specConfigFetcher is the seam the config.json probe goes through, so tests can
+// drive the whole import path without a network round trip.
+var specConfigFetcher = fetchProbeBody
+
+// maybeApplyVLLMSpeculativeDefaults fetches the repository's config.json and,
+// when it declares a Multi-Token Prediction head, enables MTP speculative
+// decoding in the emitted engine_args. This is the safetensors counterpart of
+// the llama-cpp importer's GGUF header probe.
+//
+// Every failure is non-fatal and logged at debug: a network blip, a private
+// repo, or a config.json this doesn't understand must leave the import working
+// exactly as it did before, just without the speculative default.
+func maybeApplyVLLMSpeculativeDefaults(modelConfig *config.ModelConfig, details Details) {
+	probeURL := vllmSpecProbeURL(details)
+	if probeURL == "" {
+		return
+	}
+
+	body, err := specConfigFetcher(probeURL)
+	if err != nil {
+		xlog.Debug("[vllm-spec-importer] could not read config.json for MTP detection", "uri", probeURL, "error", err)
+		return
+	}
+
+	applySpecFromConfigJSON(modelConfig, body, details.URI)
+}
+
+// applySpecFromConfigJSON is the decision half of the probe, split out so it can
+// be exercised without a network round trip.
+func applySpecFromConfigJSON(modelConfig *config.ModelConfig, body []byte, uri string) {
+	if config.IsDFlashDraftConfig(body) {
+		// A DFlash draft cannot serve on its own - it only proposes tokens for
+		// a target model to verify. Say so rather than emitting a config that
+		// would fail at load.
+		xlog.Warn("[vllm-spec-importer] this repository is a DFlash DRAFT checkpoint, not a servable model; "+
+			"import the TARGET model and point engine_args.speculative_config at this repo "+
+			`({"method":"dflash","model":"<this repo>"})`, "uri", uri)
+		return
+	}
+
+	n, ok := config.HasSafetensorsMTPHead(body)
+	if !ok {
+		return
+	}
+	config.ApplyVLLMSpeculativeDefaults(modelConfig, n)
+}
+
+// vllmSpecProbeURL returns the HTTP(S) URL of the repository's config.json, or
+// "" when the import isn't backed by a HuggingFace repo we can fetch from (a
+// local directory import, an OCI artifact, ...).
+func vllmSpecProbeURL(details Details) string {
+	if details.HuggingFace == nil || details.HuggingFace.ModelID == "" {
+		return ""
+	}
+	return resolveHTTPProbe(downloader.HuggingFacePrefix + details.HuggingFace.ModelID + "/config.json")
+}
+
+// fetchProbeBody GETs a small remote JSON document under a short timeout.
+func fetchProbeBody(url string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), specConfigProbeTimeout)
+	defer cancel()
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	resp, err := httpclient.NewWithTimeout(specConfigProbeTimeout).Do(req)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = resp.Body.Close() }()
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
+	}
+	return io.ReadAll(io.LimitReader(resp.Body, maxSpecConfigProbeBytes))
+}

core/gallery/importers/vllm_spec_internal_test.go

@@ -0,0 +1,118 @@
+package importers
+
+import (
+	"encoding/json"
+	"errors"
+
+	"github.com/mudler/LocalAI/core/config"
+	hfapi "github.com/mudler/LocalAI/pkg/huggingface-api"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("vllm-cpp speculative auto-config (importer)", func() {
+	Context("applySpecFromConfigJSON", func() {
+		It("enables mtp when the checkpoint declares an MTP head", func() {
+			cfg := &config.ModelConfig{Name: "qwen3.5"}
+			applySpecFromConfigJSON(cfg, []byte(`{
+				"model_type": "qwen3_5_moe",
+				"mtp_num_hidden_layers": 1
+			}`), "huggingface://Qwen/Qwen3.5-A3B")
+			Expect(cfg.EngineArgs).To(HaveKeyWithValue("speculative_config",
+				map[string]any{"method": "mtp"}))
+		})
+
+		It("leaves a plain checkpoint untouched", func() {
+			cfg := &config.ModelConfig{Name: "llama"}
+			applySpecFromConfigJSON(cfg, []byte(`{"model_type": "llama"}`), "huggingface://meta/llama")
+			Expect(cfg.EngineArgs).To(BeEmpty())
+		})
+
+		It("refuses to configure a DFlash draft as a servable model", func() {
+			// The draft only proposes tokens; configuring it standalone would
+			// produce a model that cannot load.
+			cfg := &config.ModelConfig{Name: "dflash-draft"}
+			applySpecFromConfigJSON(cfg, []byte(`{
+				"model_type": "qwen3_dflash",
+				"dflash_config": {"mask_token_id": 151666, "target_layer_ids": [0, 1]}
+			}`), "huggingface://z-lab/Qwen3.6-27B-DFlash")
+			Expect(cfg.EngineArgs).To(BeEmpty())
+		})
+
+		It("survives a config.json it cannot parse", func() {
+			cfg := &config.ModelConfig{Name: "weird"}
+			Expect(func() {
+				applySpecFromConfigJSON(cfg, []byte(`<html>404</html>`), "huggingface://a/b")
+			}).ToNot(Panic())
+			Expect(cfg.EngineArgs).To(BeEmpty())
+		})
+	})
+
+	Context("Import over a repository with an MTP head", func() {
+		var restore func()
+
+		BeforeEach(func() {
+			original := specConfigFetcher
+			restore = func() { specConfigFetcher = original }
+		})
+		AfterEach(func() { restore() })
+
+		importWith := func(backend, configJSON string) string {
+			specConfigFetcher = func(string) ([]byte, error) {
+				return []byte(configJSON), nil
+			}
+			importer := &VLLMImporter{}
+			out, err := importer.Import(Details{
+				URI:         "huggingface://Qwen/Qwen3.5-A3B",
+				Preferences: json.RawMessage(`{"backend": "` + backend + `"}`),
+				HuggingFace: &hfapi.ModelDetails{ModelID: "Qwen/Qwen3.5-A3B"},
+			})
+			Expect(err).ToNot(HaveOccurred())
+			return out.ConfigFile
+		}
+
+		It("emits engine_args.speculative_config for vllm-cpp", func() {
+			yaml := importWith("vllm-cpp", `{"model_type":"qwen3_5_moe","mtp_num_hidden_layers":1}`)
+			Expect(yaml).To(ContainSubstring("engine_args:"))
+			Expect(yaml).To(ContainSubstring("speculative_config:"))
+			Expect(yaml).To(ContainSubstring("method: mtp"))
+		})
+
+		It("emits nothing speculative for the python vllm backend", func() {
+			// The python backend has its own speculative surface and its own
+			// version-dependent MTP support; this hook is vllm-cpp only.
+			yaml := importWith("vllm", `{"model_type":"qwen3_5_moe","mtp_num_hidden_layers":1}`)
+			Expect(yaml).NotTo(ContainSubstring("speculative_config"))
+		})
+
+		It("emits nothing speculative when the probe fails", func() {
+			specConfigFetcher = func(string) ([]byte, error) {
+				return nil, errors.New("network down")
+			}
+			importer := &VLLMImporter{}
+			out, err := importer.Import(Details{
+				URI:         "huggingface://Qwen/Qwen3.5-A3B",
+				Preferences: json.RawMessage(`{"backend": "vllm-cpp"}`),
+				HuggingFace: &hfapi.ModelDetails{ModelID: "Qwen/Qwen3.5-A3B"},
+			})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(out.ConfigFile).NotTo(ContainSubstring("speculative_config"))
+		})
+	})
+
+	Context("vllmSpecProbeURL", func() {
+		It("resolves the repository's config.json to an HTTPS URL", func() {
+			url := vllmSpecProbeURL(Details{
+				URI:         "huggingface://Qwen/Qwen3.5-A3B",
+				HuggingFace: &hfapi.ModelDetails{ModelID: "Qwen/Qwen3.5-A3B"},
+			})
+			Expect(url).To(ContainSubstring("Qwen/Qwen3.5-A3B"))
+			Expect(url).To(HaveSuffix("config.json"))
+			Expect(url).To(HavePrefix("https://"))
+		})
+
+		It("skips the probe when there is no HuggingFace repo behind the import", func() {
+			Expect(vllmSpecProbeURL(Details{URI: "/models/local-dir"})).To(BeEmpty())
+		})
+	})
+})

core/http/middleware/trace.go

@@ -60,6 +60,7 @@ type APIExchange struct {
 }
 
 var traceBuffer *circularbuffer.Queue[APIExchange]
+var inFlightTraces = make(map[string]APIExchange)
 var mu sync.Mutex
 var logChan = make(chan traceCommand, 100)
 var traceIDSeq atomic.Uint64
@@ -126,16 +127,17 @@ func initializeTracing(dataPath string, maxItems int) {
 					continue
 				}
 				exchange := *command.exchange
+				mu.Lock()
+				delete(inFlightTraces, exchange.ID)
+				if traceBuffer != nil {
+					traceBuffer.Enqueue(exchange)
+				}
+				mu.Unlock()
 				if command.store != nil {
 					if err := command.store.Append(exchange.ID, exchange); err != nil {
 						xlog.Warn("Failed to persist API trace", "error", err)
 					}
 				}
-				mu.Lock()
-				if traceBuffer != nil {
-					traceBuffer.Enqueue(exchange)
-				}
-				mu.Unlock()
 			}
 		}()
 	})
@@ -261,6 +263,38 @@ func TraceMiddleware(app *application.Application) echo.MiddlewareFunc {
 			// tens of MB, which then locks the admin Traces UI fetching the
 			// JSON dump faster than the 5s auto-refresh.
 			maxBodyBytes := app.ApplicationConfig().TracingMaxBodyBytes
+			requestHeaders := redactSensitiveHeaders(c.Request().Header)
+			requestBody, requestTruncated := truncateForTrace(body, maxBodyBytes)
+			exchange := APIExchange{
+				ID:        nextTraceID(),
+				Timestamp: startTime,
+				ClientIP:  c.RealIP(),
+				UserAgent: c.Request().UserAgent(),
+				Request: APIExchangeRequest{
+					Method:        c.Request().Method,
+					Path:          c.Path(),
+					Headers:       &requestHeaders,
+					Body:          &requestBody,
+					BodyTruncated: requestTruncated,
+					BodyBytes:     len(body),
+				},
+			}
+			if user := auth.GetUser(c); user != nil {
+				exchange.UserID = user.ID
+				exchange.UserName = user.Name
+			}
+			mu.Lock()
+			inFlightTraces[exchange.ID] = exchange
+			mu.Unlock()
+			queued := false
+			defer func() {
+				if queued {
+					return
+				}
+				mu.Lock()
+				delete(inFlightTraces, exchange.ID)
+				mu.Unlock()
+			}()
 
 			// Wrap response writer to capture body
 			resBody := new(bytes.Buffer)
@@ -287,47 +321,27 @@ func TraceMiddleware(app *application.Application) echo.MiddlewareFunc {
 			// the trace endpoint is admin-only but the buffer is also reachable
 			// via any heap-dump-style introspection, and tokens shouldn't
 			// outlive the request that carried them.
-			requestHeaders := redactSensitiveHeaders(c.Request().Header)
-			requestBody, requestTruncated := truncateForTrace(body, maxBodyBytes)
 			responseHeaders := redactSensitiveHeaders(c.Response().Header())
 			responseBody := make([]byte, resBody.Len())
 			copy(responseBody, resBody.Bytes())
-			exchange := APIExchange{
-				ID:        nextTraceID(),
-				Timestamp: startTime,
-				Duration:  time.Since(startTime),
-				ClientIP:  c.RealIP(),
-				UserAgent: c.Request().UserAgent(),
-				Request: APIExchangeRequest{
-					Method:        c.Request().Method,
-					Path:          c.Path(),
-					Headers:       &requestHeaders,
-					Body:          &requestBody,
-					BodyTruncated: requestTruncated,
-					BodyBytes:     len(body),
-				},
-				Response: APIExchangeResponse{
-					Status:        status,
-					Headers:       &responseHeaders,
-					Body:          &responseBody,
-					BodyTruncated: mw.truncated,
-					BodyBytes:     mw.totalBytes,
-				},
+			exchange.Duration = time.Since(startTime)
+			exchange.Response = APIExchangeResponse{
+				Status:        status,
+				Headers:       &responseHeaders,
+				Body:          &responseBody,
+				BodyTruncated: mw.truncated,
+				BodyBytes:     mw.totalBytes,
 			}
 			if handlerErr != nil {
 				exchange.Error = handlerErr.Error()
 			}
 
-			if user := auth.GetUser(c); user != nil {
-				exchange.UserID = user.ID
-				exchange.UserName = user.Name
-			}
-
 			mu.Lock()
 			store := traceStore
 			mu.Unlock()
 			select {
 			case logChan <- traceCommand{exchange: &exchange, store: store}:
+				queued = true
 			default:
 				xlog.Warn("Trace channel full, dropping trace")
 			}
@@ -345,6 +359,10 @@ func GetTraces() []APIExchange {
 		return []APIExchange{}
 	}
 	traces := traceBuffer.Values()
+	for _, exchange := range inFlightTraces {
+		exchange.Duration = time.Since(exchange.Timestamp)
+		traces = append(traces, exchange)
+	}
 	mu.Unlock()
 
 	slices.SortFunc(traces, func(a, b APIExchange) int {

core/http/middleware/trace_live_test.go

@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: MIT
+
+package middleware
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"time"
+
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/application"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/system"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("live API traces", func() {
+	newApp := func(root string) *application.Application {
+		app, err := application.New(
+			config.EnableTracing,
+			config.WithDataPath(root),
+			config.WithDisableLocalAIAssistant(true),
+			config.WithDisableStats(true),
+			config.WithSystemState(&system.SystemState{
+				Model:   system.Model{ModelsPath: root},
+				Backend: system.Backend{BackendsPath: root},
+			}),
+		)
+		Expect(err).NotTo(HaveOccurred())
+		DeferCleanup(func() { Expect(app.Shutdown()).To(Succeed()) })
+		ClearTraces()
+		return app
+	}
+
+	It("lists a request while its handler is still running", func() {
+		root := GinkgoT().TempDir()
+		app := newApp(root)
+
+		started := make(chan struct{})
+		release := make(chan struct{})
+		DeferCleanup(func() {
+			select {
+			case <-release:
+			default:
+				close(release)
+			}
+		})
+		handler := TraceMiddleware(app)(func(c echo.Context) error {
+			close(started)
+			<-release
+			return c.NoContent(http.StatusNoContent)
+		})
+
+		e := echo.New()
+		req := httptest.NewRequest(http.MethodPost, "/slow", http.NoBody)
+		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
+		rec := httptest.NewRecorder()
+		ctx := e.NewContext(req, rec)
+		ctx.SetPath("/slow")
+		done := make(chan error, 1)
+		go func() {
+			done <- handler(ctx)
+		}()
+		<-started
+
+		var running APIExchange
+		Eventually(func() bool {
+			traces := GetTraces()
+			if len(traces) != 1 {
+				return false
+			}
+			running = traces[0]
+			return running.Request.Path == "/slow"
+		}).Should(BeTrue())
+		Expect(running.Response.Status).To(Equal(0))
+		Expect(running.Duration).To(BeNumerically(">", 0))
+
+		close(release)
+		Expect(<-done).To(Succeed())
+		Eventually(func() []APIExchange { return GetTraces() }).Should(ConsistOf(
+			And(
+				HaveField("ID", running.ID),
+				HaveField("Response.Status", http.StatusNoContent),
+				HaveField("Duration", BeNumerically(">", time.Duration(0))),
+			),
+		))
+	})
+
+	It("removes an in-flight trace when the handler panics", func() {
+		app := newApp(GinkgoT().TempDir())
+		handler := TraceMiddleware(app)(func(echo.Context) error {
+			panic("handler panic")
+		})
+		e := echo.New()
+		req := httptest.NewRequest(http.MethodPost, "/panic", http.NoBody)
+		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
+		ctx := e.NewContext(req, httptest.NewRecorder())
+		ctx.SetPath("/panic")
+
+		func() {
+			defer func() { _ = recover() }()
+			_ = handler(ctx)
+		}()
+
+		Expect(GetTraces()).To(BeEmpty())
+	})
+})

core/http/react-ui/e2e/threed-gen.spec.js

@@ -125,6 +125,22 @@ async function generateOnce(page) {
   await page.locator('button[type="submit"]').click()
 }
 
+async function pasteImage(page) {
+  await page.locator('.biometrics-mediainput').focus()
+  await page.evaluate((base64) => {
+    const bytes = Uint8Array.from(atob(base64), char => char.charCodeAt(0))
+    const transfer = new DataTransfer()
+    transfer.items.add(new File([bytes], 'clipboard.png', { type: 'image/png' }))
+    const target = document.querySelector('.biometrics-mediainput')
+    target.dispatchEvent(new ClipboardEvent('paste', {
+      bubbles: true,
+      cancelable: true,
+      clipboardData: transfer,
+    }))
+  }, TINY_PNG.toString('base64'))
+  await expect(page.locator('.biometrics-mediainput__source-pill')).toContainText('Pasted image')
+}
+
 test.describe('3D generation', () => {
   test.beforeEach(async ({ page }) => {
     await mockCapabilities(page)
@@ -154,6 +170,60 @@ test.describe('3D generation', () => {
     expect(requestBody.response_format).toBe('url')
   })
 
+  test('caps auto-rotate at 30 FPS and renders still models on demand', async ({ page }) => {
+    await page.addInitScript(() => {
+      window.__glDrawTimes = []
+      const proto = window.WebGL2RenderingContext?.prototype
+      if (!proto) return
+      const drawElements = proto.drawElements
+      proto.drawElements = function (...args) {
+        window.__glDrawTimes.push(performance.now())
+        return drawElements.apply(this, args)
+      }
+    })
+    await mockGeneration(page)
+    await generateOnce(page)
+    await expect(page.getByTestId('glb-stats')).toBeVisible({ timeout: 15_000 })
+    await page.waitForTimeout(100)
+    await page.evaluate(() => { window.__glDrawTimes = [] })
+    await page.waitForTimeout(600)
+
+    const drawTimes = await page.evaluate(() => window.__glDrawTimes)
+    test.skip(drawTimes.length < 3, 'WebGL2 drawing is unavailable in this browser')
+    expect(drawTimes.length).toBeLessThanOrEqual(22)
+    const gaps = drawTimes.slice(1).map((time, index) => time - drawTimes[index]).sort((a, b) => a - b)
+    expect(gaps[Math.floor(gaps.length / 2)]).toBeGreaterThan(25)
+
+    await page.getByRole('button', { name: 'Auto-rotate' }).click()
+    await page.waitForTimeout(100)
+    const stoppedAt = await page.evaluate(() => window.__glDrawTimes.length)
+    await page.waitForTimeout(250)
+    const idleAt = await page.evaluate(() => window.__glDrawTimes.length)
+    expect(idleAt - stoppedAt).toBeLessThanOrEqual(1)
+
+    await page.getByTestId('glb-canvas').dispatchEvent('wheel', { deltaY: 20 })
+    await expect.poll(() => page.evaluate(() => window.__glDrawTimes.length)).toBeGreaterThan(idleAt)
+  })
+
+  test('pastes a conditioning image without mounting its base64 in the request panel', async ({ page }) => {
+    let requestBody = null
+    await mockGeneration(page, (body) => { requestBody = body })
+
+    await page.goto('/app/studio/threed')
+    await expect(page.getByRole('button', { name: 'trellis-test-model' })).toBeVisible({ timeout: 10_000 })
+    await pasteImage(page)
+    await page.locator('button[type="submit"]').click()
+
+    await expect(page.getByTestId('glb-stats')).toBeVisible({ timeout: 15_000 })
+    await expect(page.getByTestId('media-history-item')).toHaveCount(1)
+    const panel = page.locator('.request-panel')
+    await expect(panel).toContainText('<base64 image/png omitted>')
+    const panelText = await panel.textContent()
+    expect(panelText.length).toBeLessThan(2000)
+    expect(panelText).not.toContain(requestBody.image)
+    expect(requestBody.image).toBeTruthy()
+  })
+
   test('advanced settings map to step/texture_steps/cfg_scale/seed', async ({ page }) => {
     let requestBody = null
     await mockGeneration(page, (body) => { requestBody = body })
@@ -226,6 +296,18 @@ test.describe('3D generation', () => {
     await expect(page.getByTestId('glb-download')).toHaveAttribute('href', /^blob:/)
   })
 
+  test('new history is visible on the Studio overview without a reload', async ({ page }) => {
+    await mockGeneration(page)
+    await page.goto('/app/studio/threed')
+    await expect(page.getByRole('button', { name: 'trellis-test-model' })).toBeVisible({ timeout: 10_000 })
+    await page.locator('#threed-image-file').setInputFiles({ name: 'input.png', mimeType: 'image/png', buffer: TINY_PNG })
+    await page.locator('button[type="submit"]').click()
+    await expect(page.getByTestId('media-history-item')).toHaveCount(1, { timeout: 15_000 })
+
+    await page.locator('.studio-tab[data-tab="overview"]').click()
+    await expect(page.getByTestId('studio-recent')).toContainText('trellis-test-model')
+  })
+
   test('deleting a history entry removes it', async ({ page }) => {
     await mockGeneration(page)
     await generateOnce(page)

core/http/react-ui/e2e/traces-live.spec.js

@@ -0,0 +1,65 @@
+import { test, expect } from './coverage-fixtures.js'
+
+test('marks an API trace with no response status as in progress', async ({ page }) => {
+  await page.route('**/api/traces?*', route => route.fulfill({
+    json: [{
+      id: 'running-1',
+      timestamp: '2026-08-05T02:00:00Z',
+      duration: 2_000_000_000,
+      request: { method: 'POST', path: '/v1/chat/completions' },
+      response: { status: 0 },
+    }],
+    headers: { 'X-Total-Count': '1' },
+  }))
+  await page.route('**/api/backend-traces?*', route => route.fulfill({ json: [] }))
+
+  await page.goto('/app/traces')
+
+  const row = page.locator('tbody tr').filter({ hasText: '/v1/chat/completions' })
+  await expect(row.getByText('Running', { exact: true })).toBeVisible()
+  await expect(row.locator('[title="In progress"]')).toBeVisible()
+  await expect(row.locator('.fa-check-circle')).toHaveCount(0)
+})
+
+// Regression for #11376: switching from Backend Traces back to API Traces
+// used to crash the page. `traces` holds whichever list was fetched last, so
+// right after `setActiveTab('api')` — before the refetch effect lands — the
+// API table renders the previous tab's backend rows, which carry no
+// `response` envelope. The status column must tolerate that instead of
+// dereferencing `trace.response.status` and tearing down the React tree.
+test('switching from backend to API traces with a response-less row does not crash', async ({ page }) => {
+  const pageErrors = []
+  page.on('pageerror', (e) => pageErrors.push(e.message))
+
+  await page.route('**/api/traces?*', route => route.fulfill({
+    json: [{
+      id: 'api-1',
+      timestamp: '2026-08-05T02:00:00Z',
+      request: { method: 'POST', path: '/v1/chat/completions' },
+      response: { status: 200 },
+    }],
+    headers: { 'X-Total-Count': '1' },
+  }))
+  await page.route('**/api/backend-traces?*', route => route.fulfill({
+    json: [{
+      id: 'backend-1',
+      type: 'llm',
+      timestamp: '2026-08-05T02:00:00Z',
+      model_name: 'mock-model',
+      summary: 'generated a reply',
+    }],
+    headers: { 'X-Total-Count': '1' },
+  }))
+
+  await page.goto('/app/traces')
+  await expect(page.locator('tbody tr').filter({ hasText: '/v1/chat/completions' })).toBeVisible()
+
+  await page.getByRole('button', { name: /Backend Traces/ }).click()
+  await expect(page.locator('tbody tr').filter({ hasText: 'generated a reply' })).toBeVisible()
+
+  await page.getByRole('button', { name: /API Traces/ }).click()
+  // The stale backend row renders in the API table for one frame; the status
+  // column falls back to a neutral placeholder rather than throwing.
+  await expect(page.locator('tbody tr').filter({ hasText: '/v1/chat/completions' })).toBeVisible()
+  expect(pageErrors).toEqual([])
+})

core/http/react-ui/src/components/GlbViewer.jsx

@@ -132,6 +132,7 @@ const Q = {
 // GLBs are already Y-up (the baker swaps axes on export), so unlike the demo
 // there is no Z-up correction here — just a gentle 3/4 default view.
 const QBASE = Q.norm(Q.mul(Q.axisAngle(1, 0, 0, -0.30), Q.axisAngle(0, 1, 0, 0.55)))
+const FRAME_INTERVAL_MS = 1000 / 30
 
 /* minimal mat4 helpers (column-major) */
 const M = {
@@ -333,10 +334,12 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
     nIndices = 0
     nWire = 0
     dropTextures()
+    requestRender()
   }
 
   function resetView() {
     rot = QBASE.slice(); dist = 1.8; panX = panY = 0
+    requestRender()
   }
 
   /* input */
@@ -353,6 +356,7 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
   }
   const stopSpin = () => {
     spin = false
+    requestRender()
     if (onSpinChange) onSpinChange(false)
   }
   const onPointerDown = (e) => {
@@ -400,6 +404,7 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
       pinchDistance = nextDistance
       pinchX = nextX
       pinchY = nextY
+      requestRender()
       return
     }
 
@@ -415,12 +420,14 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
       rot = Q.norm(Q.mul(Q.axisAngle(1, 0, 0, dy * k), Q.mul(Q.axisAngle(0, 1, 0, dx * k), rot)))
       stopSpin()
     }
+    requestRender()
   }
   const onContextMenu = (e) => e.preventDefault()
   const onWheel = (e) => {
     e.preventDefault()
     dist *= Math.exp(e.deltaY * 0.001)
     dist = Math.max(0.3, Math.min(8, dist))
+    requestRender()
   }
   const onDblClick = () => resetView()
   let onSpinChange = null
@@ -439,10 +446,26 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
   gl.clearColor(0.063, 0.078, 0.094, 1)
 
   let rafId = 0
-  let last = performance.now()
+  let lastDraw = 0
+  let dirty = true
+  function requestRender() {
+    dirty = true
+    if (!disposed && !rafId) rafId = requestAnimationFrame(frame)
+  }
   function frame(now) {
+    rafId = 0
     if (disposed) return
-    const dt = (now - last) / 1000; last = now
+    // requestAnimationFrame follows the display refresh rate, which can be
+    // 120-240 Hz. Skip expensive mesh draws until the 30 FPS budget is due.
+    if (spin && lastDraw && now - lastDraw < FRAME_INTERVAL_MS) {
+      rafId = requestAnimationFrame(frame)
+      return
+    }
+    if (!spin && !dirty) return
+
+    const dt = lastDraw ? Math.min((now - lastDraw) / 1000, 0.1) : 0
+    lastDraw = now
+    dirty = false
     // auto-rotate: a slow turn about the screen-vertical axis (turntable feel)
     if (spin) rot = Q.norm(Q.mul(Q.axisAngle(0, 1, 0, dt * 0.4), rot))
 
@@ -503,13 +526,20 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
       }
       gl.bindVertexArray(null)
     }
-    rafId = requestAnimationFrame(frame)
+    // A still model is complete until input, resize, or a control invalidates
+    // it. Spinning models keep scheduling frames, subject to the cap above.
+    if (spin) rafId = requestAnimationFrame(frame)
   }
-  rafId = requestAnimationFrame(frame)
+  const resizeObserver = typeof ResizeObserver === 'undefined'
+    ? null
+    : new ResizeObserver(requestRender)
+  resizeObserver?.observe(canvas)
+  requestRender()
 
   function dispose() {
     disposed = true
     cancelAnimationFrame(rafId)
+    resizeObserver?.disconnect()
     canvas.removeEventListener('pointerdown', onPointerDown)
     canvas.removeEventListener('pointerup', onPointerUp)
     canvas.removeEventListener('pointercancel', onPointerUp)
@@ -532,8 +562,8 @@ export function createGlbViewer(canvas, { onContextLost } = {}) {
     clear,
     dispose,
     resetView,
-    setWire(v) { wire = v },
-    setSpin(v) { spin = v },
+    setWire(v) { wire = v; requestRender() },
+    setSpin(v) { spin = v; requestRender() },
     onSpinChanged(fn) { onSpinChange = fn },
   }
 }

core/http/react-ui/src/components/biometrics/MediaInput.jsx

@@ -51,9 +51,7 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
     if (tab !== 'live' && cap.active) cap.stop()
   }, [tab]) // eslint-disable-line react-hooks/exhaustive-deps
 
-  const handleFile = async (e) => {
-    const f = e.target.files?.[0]
-    if (!f) { onChange(null); return }
+  const acceptFile = async (f, source = 'file') => {
     if (maxBytes && f.size > maxBytes) {
       const error = new Error(`Selected file exceeds the ${Math.round(maxBytes / (1024 * 1024))} MiB limit`)
       if (fileRef.current) fileRef.current.value = ''
@@ -62,8 +60,11 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
       return
     }
     try {
+      const name = source === 'paste'
+        ? `pasted-image.${(f.type.split('/')[1] || 'png').replace('+xml', '')}`
+        : f.name
       if (preferBlob) {
-        onChange({ blob: f, mime: f.type, source: 'file', name: f.name })
+        onChange({ blob: f, mime: f.type, source, name })
         return
       }
       const base64 = await fileToBase64(f)
@@ -73,13 +74,30 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
         reader.onload = () => resolve(reader.result)
         reader.readAsDataURL(f)
       })
-      onChange({ base64, blob: f, dataUrl, mime: f.type, source: 'file', name: f.name })
+      onChange({ base64, blob: f, dataUrl, mime: f.type, source, name })
     } catch (error) {
       onChange(null)
       onError?.(error)
     }
   }
 
+  const handleFile = async (e) => {
+    const f = e.target.files?.[0]
+    if (!f) { onChange(null); return }
+    await acceptFile(f)
+  }
+
+  const handlePaste = async (e) => {
+    if (mode !== 'image') return
+    const item = Array.from(e.clipboardData?.items || []).find(entry => entry.type.startsWith('image/'))
+    const f = item?.getAsFile()
+      || Array.from(e.clipboardData?.files || []).find(file => file.type.startsWith('image/'))
+    if (!f) return
+    e.preventDefault()
+    setTab('file')
+    await acceptFile(f, 'paste')
+  }
+
   const handleSnap = () => {
     const shot = cap.snap()
     if (shot) onChange({ ...shot, source: 'live' })
@@ -106,7 +124,13 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
   const inputId = `${idPrefix}-${mode}-file`
 
   return (
-    <div className="biometrics-mediainput">
+    <div
+      className="biometrics-mediainput"
+      onPaste={handlePaste}
+      tabIndex={mode === 'image' ? 0 : undefined}
+      role={mode === 'image' ? 'group' : undefined}
+      aria-label={mode === 'image' ? `${label || 'Image'} upload or clipboard paste` : undefined}
+    >
       {label && <label className="form-label" htmlFor={inputId}>{label}</label>}
 
       <div className="biometrics-mediainput__tabs" role="tablist" aria-label={`${label || 'Media'} source`}>
@@ -133,6 +157,9 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
             accept={mode === 'image' ? 'image/*' : 'audio/*'}
             onChange={handleFile}
           />
+          {mode === 'image' && (
+            <p className="form-hint"><i className="fas fa-clipboard" aria-hidden="true" /> Paste an image from the clipboard</p>
+          )}
         </div>
       )}
 
@@ -184,8 +211,8 @@ export default function MediaInput({ mode, label, value, onChange, onError, maxB
             : <audio controls src={value.dataUrl} />}
           <div className="biometrics-mediainput__preview-meta">
             <span className="biometrics-mediainput__source-pill">
-              <i className={`fas ${value.source === 'live' ? (mode === 'image' ? 'fa-camera' : 'fa-microphone') : 'fa-file'}`} aria-hidden="true" />
-              {value.source === 'live' ? ' Captured' : ` ${value.name || 'Uploaded'}`}
+              <i className={`fas ${value.source === 'live' ? (mode === 'image' ? 'fa-camera' : 'fa-microphone') : value.source === 'paste' ? 'fa-clipboard' : 'fa-file'}`} aria-hidden="true" />
+              {value.source === 'live' ? ' Captured' : value.source === 'paste' ? ' Pasted image' : ` ${value.name || 'Uploaded'}`}
             </span>
             <button type="button" className="biometrics-mediainput__clear" onClick={clear} aria-label="Remove sample">
               <i className="fas fa-xmark" aria-hidden="true" />

core/http/react-ui/src/hooks/use3DHistory.js

@@ -17,6 +17,14 @@ const DB_NAME = 'localai-3d-history'
 const DB_VERSION = 1
 const STORE = 'generations'
 const MAX_ENTRIES = 20
+const historyListeners = new Set()
+let sessionEntries = []
+
+async function refreshOtherHooks(source) {
+  await Promise.all([...historyListeners]
+    .filter(listener => listener !== source)
+    .map(listener => listener()))
+}
 
 function openDb() {
   return new Promise((resolve, reject) => {
@@ -78,14 +86,19 @@ export function use3DHistory() {
 
   const refresh = useCallback(async () => {
     try {
-      setEntries(await idbGetAll())
+      sessionEntries = await idbGetAll()
+      setEntries(sessionEntries)
     } catch {
       // IndexedDB unavailable (private mode etc.) — degrade to session-only.
-      setEntries((prev) => prev)
+      setEntries(sessionEntries)
     }
   }, [])
 
-  useEffect(() => { refresh() }, [refresh])
+  useEffect(() => {
+    historyListeners.add(refresh)
+    refresh()
+    return () => { historyListeners.delete(refresh) }
+  }, [refresh])
 
   const addEntry = useCallback(async ({ model, params, inputThumb, glb, name }) => {
     const entry = { id: generateId(), createdAt: Date.now(), model, params, inputThumb, glb, name }
@@ -93,8 +106,10 @@ export function use3DHistory() {
       await idbPutAndEvict(entry)
       await refresh()
     } catch {
-      setEntries((prev) => [entry, ...prev].slice(0, MAX_ENTRIES))
+      sessionEntries = [entry, ...sessionEntries.filter(e => e.id !== entry.id)].slice(0, MAX_ENTRIES)
+      setEntries(sessionEntries)
     }
+    await refreshOtherHooks(refresh)
     return entry
   }, [refresh])
 
@@ -104,19 +119,21 @@ export function use3DHistory() {
       await idbDelete(id)
       await refresh()
     } catch {
-      setEntries((prev) => prev.filter((e) => e.id !== id))
+      sessionEntries = sessionEntries.filter((e) => e.id !== id)
+      setEntries(sessionEntries)
     }
+    await refreshOtherHooks(refresh)
   }, [refresh])
 
   const clearAll = useCallback(async () => {
     setSelectedId(null)
     try {
       await idbClear()
-    } catch {
-      // fall through to the local reset below
-    }
+    } catch { /* session-only history is cleared below */ }
+    sessionEntries = []
     setEntries([])
-  }, [])
+    await refreshOtherHooks(refresh)
+  }, [refresh])
 
   // Toggles: clicking the selected entry deselects it (back to latest result).
   const selectEntry = useCallback((id) => {

core/http/react-ui/src/pages/Settings.jsx

@@ -492,7 +492,7 @@ export default function Settings() {
                   value={settings.galleries_json || (settings.galleries ? JSON.stringify(settings.galleries, null, 2) : '')}
                   onChange={(e) => update('galleries_json', e.target.value)}
                   rows={4}
-                  placeholder={'[\n  { "url": "https://...", "name": "my-gallery" }\n]'}
+                  placeholder={'[\n  { "url": "https://...", "name": "my-gallery", "mirrors": ["https://fallback/..."] }\n]'}
                 />
               </div>
               <div className="mt-sm">
@@ -502,7 +502,7 @@ export default function Settings() {
                   value={settings.backend_galleries_json || (settings.backend_galleries ? JSON.stringify(settings.backend_galleries, null, 2) : '')}
                   onChange={(e) => update('backend_galleries_json', e.target.value)}
                   rows={4}
-                  placeholder={'[\n  { "url": "https://...", "name": "my-backends" }\n]'}
+                  placeholder={'[\n  { "url": "https://...", "name": "my-backends", "mirrors": ["https://fallback/..."] }\n]'}
                 />
               </div>
             </div>

core/http/react-ui/src/pages/ThreeDGen.jsx

@@ -100,7 +100,9 @@ export default function ThreeDGen() {
     if (guidance) body.cfg_scale = parseFloat(guidance)
     if (seed) body.seed = parseInt(seed)
 
-    setLastRequest(body)
+    // RequestPanel renders and copies its body. Keeping a multi-megabyte image
+    // there duplicates the upload in React and can starve the result render.
+    setLastRequest({ ...body, image: `<base64 ${image.mime || 'image'} omitted>` })
 
     try {
       const data = await threeDApi.generate(body)

core/http/react-ui/src/pages/Traces.jsx

@@ -664,10 +664,18 @@ export default function Traces() {
                     <td><span className="badge badge-info">{trace.request?.method || '-'}</span></td>
                     <td className="text-mono text-sm">{trace.request?.path || '-'}</td>
                     <td className="text-sub cell-clip" title={trace.user_name || trace.user_id || ''}>{trace.user_name || trace.user_id || '-'}</td>
-                    <td><span className={`badge ${(trace.response?.status || 0) < 400 ? 'badge-success' : 'badge-error'}`}>{trace.response?.status || '-'}</span></td>
+                    <td>
+                      {trace.response?.status === 0
+                        ? <span className="badge badge-info">Running</span>
+                        : trace.response?.status == null
+                        ? <span className="badge badge--soft">-</span>
+                        : <span className={`badge ${trace.response.status < 400 ? 'badge-success' : 'badge-error'}`}>{trace.response.status}</span>}
+                    </td>
                     <td><LatencyCell ns={trace.duration} max={slowestTrace} /></td>
                     <td className="text-center">
-                      {trace.error
+                      {trace.response?.status === 0
+                        ? <i className="fas fa-spinner fa-spin text-primary" title="In progress" />
+                        : trace.error
                         ? <i className="fas fa-times-circle text-error" title={trace.error} />
                         : <i className="fas fa-check-circle text-success" />}
                     </td>

docker-compose.distributed.yaml

@@ -74,6 +74,9 @@ services:
       GODEBUG: "netdns=go"
       # Paths
       MODELS_PATH: /models
+      # Avoid probing remote gallery GGUF metadata during container startup.
+      # Remove this line or set a positive limit to opt back into cache warming.
+      LOCALAI_VRAM_WARM_LIMIT: "0"
     volumes:
       - frontend_models:/models
       - frontend_data:/data

docker-compose.yaml

@@ -18,6 +18,9 @@ services:
       - .env
     environment:
       - MODELS_PATH=/models
+      # Avoid probing remote gallery GGUF metadata during container startup.
+      # Remove this line or set a positive limit to opt back into cache warming.
+      - LOCALAI_VRAM_WARM_LIMIT=0
     #  - DEBUG=true
     ## Agents (LocalAGI) - https://localai.io/features/agents/
     #  - LOCALAI_DISABLE_AGENTS=false

docs/content/advanced/vram-management.md

@@ -477,6 +477,11 @@ then on.
 | `LOCALAI_VRAM_WARM_LIMIT` | `300` | How many gallery entries to warm at startup, estimates and variants alike. Set to `0` to disable the warm-up entirely. |
 | `LOCALAI_VRAM_WARM_CONCURRENCY` | `4` | How many estimates to run at once. |
 
+The provided Docker Compose configurations set `LOCALAI_VRAM_WARM_LIMIT=0`
+as a defensive default, so container startup does not probe remote GGUF files.
+Remove that override or set it to a positive number to opt into background
+warming.
+
 ```bash
 # Air-gapped, or you would rather not make the requests at all
 LOCALAI_VRAM_WARM_LIMIT=0 local-ai run

docs/content/features/3d-generation.md

@@ -113,7 +113,7 @@ curl http://localhost:8080/3d/generations \
 
 ## WebUI
 
-The React UI includes a 3D tab in the Studio (and a `/3d` page) with an interactive PBR viewer: upload an image, pick the quality, and preview the generated mesh with orbit/pan/zoom and a wireframe toggle. Past generations are kept in the browser (IndexedDB). After generation, a single Detail slider and **Apply remeshing** button replace the preview with the exact watertight model that the GLB download exports; **Show original** switches back without regenerating.
+The React UI includes a 3D tab in the Studio (and a `/3d` page) with an interactive PBR viewer: upload or paste an image from the clipboard, pick the quality, and preview the generated mesh with orbit/pan/zoom and a wireframe toggle. Past generations are kept in the browser (IndexedDB). After generation, a single Detail slider and **Apply remeshing** button replace the preview with the exact watertight model that the GLB download exports; **Show original** switches back without regenerating.
 
 ## Notes
 

docs/content/features/backends.md

@@ -72,6 +72,44 @@ tags:
   - "text-generation"
 ```
 
+### Verifying OCI Backends
+
+Backend galleries can require keyless Sigstore signatures for every OCI image
+they provide. Add a `verification` policy to the gallery configuration, then
+enable strict integrity mode:
+
+```bash
+export LOCALAI_BACKEND_GALLERIES='[{"name":"localai","url":"https://index.localai.io/backends","mirrors":["github:mudler/LocalAI/backend/index.yaml@master"],"verification":{"issuer":"https://token.actions.githubusercontent.com","identity_regex":"^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$"}}]'
+export LOCALAI_REQUIRE_BACKEND_INTEGRITY=1
+local-ai run
+```
+
+The policy pins the Fulcio issuer and the GitHub Actions workflow identity that
+signed the image. The identity expression covers development images produced
+from `master` and release images produced from tags. Use a narrower expression
+if your deployment only accepts one release channel.
+
+Without strict mode, an OCI gallery without a verification policy installs
+with a warning. With strict mode, LocalAI refuses galleries without a policy,
+images without a compatible Sigstore bundle, and signatures that do not match
+the configured identity. Existing images published before bundle signing was
+enabled must be rebuilt or re-signed before strict deployments can install
+them.
+
+An optional `not_before` RFC3339 value revokes signatures logged before that
+time. Advance it after a signing-workflow compromise, then rebuild or re-sign
+the trusted images:
+
+```json
+{
+  "verification": {
+    "issuer": "https://token.actions.githubusercontent.com",
+    "identity_regex": "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$",
+    "not_before": "2026-08-05T00:00:00Z"
+  }
+}
+```
+
 ## Pre-installing Backends
 
 You can pre-install backends when starting LocalAI using the `LOCALAI_EXTERNAL_BACKENDS` environment variable:

docs/content/features/model-gallery.md

@@ -56,6 +56,34 @@ GALLERIES=[{"name":"<GALLERY_NAME>", "url":"<GALLERY_URL"}]
 
 The models in the gallery will be automatically indexed and available for installation.
 
+## Gallery mirrors
+
+A gallery entry can declare a `mirrors` list of alternative locations for the same index file. Mirrors exist for availability, not for load balancing: LocalAI always prefers the `url`, and only falls back to the mirrors, in the order you listed them, when the one before it cannot be fetched. If the primary works, the mirrors are never contacted.
+
+Mirrors accept any URI the gallery loader understands — `https://`, `github:`, `huggingface://` (also `hf://` and `hf.co/`), and `file://` — and the same rules apply to them as to a primary URL, so a `file://` mirror must still live inside your models directory.
+
+```json
+GALLERIES=[{"name":"localai", "url":"https://example.org/gallery/index.yaml", "mirrors":["github:mudler/LocalAI/gallery/index.yaml@master"]}]
+```
+
+Each attempt is bounded by a 120 second timeout, and a source that fails — a connection error, a timeout, or an HTTP error status such as 404 or 502 — is skipped for the next 10 minutes so a dead host is not re-dialled on every gallery listing. A source that answers is usable again immediately, and a request you cancel yourself is not counted against it. If every source happens to be inside that 10 minute window, LocalAI tries them all anyway rather than refuse to serve the gallery.
+
+{{% notice warning %}}
+**Neither mirrors nor the offline cache cover a `.ref` URL.** If a gallery's `url` ends in `.ref`, that reference file is fetched and resolved to the real index location *before* mirrors or the cached copy are consulted, and a failure to fetch it fails the gallery outright. That includes the offline case: a `.ref` gallery fails when the network is gone even if it has been fetched successfully before. Mirrors are alternates for the index, not for the reference that points at it. If you want mirror coverage or offline listings, point `url` directly at the index file.
+{{% /notice %}}
+
+The key is optional: a gallery without `mirrors` behaves exactly as before.
+
+## Offline gallery listings
+
+Every successful gallery fetch is written to a cache directory alongside your models directory (`<MODELS_PATH>/../cache/gallery/`), one file per gallery URL. If nothing can serve the index — the primary and every mirror failed, there is no network at all, the host is airgapped — LocalAI serves that last successfully fetched copy instead of failing the listing, and logs a warning saying it did so. This applies to every gallery whose `url` points directly at an index file, with or without `mirrors` — but not to a `.ref` URL, which is resolved before the cache is consulted (see the warning above).
+
+Only a response that actually parses as a gallery index is stored. A captive portal, a proxy or a CDN can answer an index request with HTTP 200 and an HTML error page; caching that would replace a working offline copy with something no listing can read. An empty index is rejected for the same reason, so the previous copy survives.
+
+Entries served this way may be stale: the copy is only as fresh as the last time the gallery could be reached, so models added or changed upstream since then will not show up, and an entry may point at a file that has since moved. A listing served from disk is a degraded mode, not a substitute for a reachable gallery.
+
+The copy is deliberately kept out of the models directory itself, where LocalAI reads a `.yaml` file as an installed model's configuration. Deleting the cache directory is safe — the next successful fetch recreates it — and a machine that has never reached a gallery has nothing cached, so its first listing still fails.
+
 ## API Reference
 
 ### Model repositories
@@ -72,13 +100,13 @@ To use additional repositories you need to start `local-ai` with the `GALLERIES`
 GALLERIES=[{"name":"<GALLERY_NAME>", "url":"<GALLERY_URL"}]
 ```
 
-For example, to enable the default `localai` repository, you can start `local-ai` with:
+For example, to spell out the default `localai` repository, you can start `local-ai` with:
 
 ```
-GALLERIES=[{"name":"localai", "url":"github:mudler/localai/gallery/index.yaml"}]
+GALLERIES=[{"name":"localai", "url":"https://index.localai.io/models", "mirrors":["github:mudler/LocalAI/gallery/index.yaml@master"]}]
 ```
 
-where `github:mudler/localai/gallery/index.yaml` will be expanded automatically to `https://raw.githubusercontent.com/mudler/LocalAI/main/index.yaml`.
+`https://index.localai.io/models` is a caching mirror of the same index file, and the `github:` entry is the fallback used whenever it cannot be reached. `github:mudler/LocalAI/gallery/index.yaml@master` is expanded automatically to `https://raw.githubusercontent.com/mudler/LocalAI/master/gallery/index.yaml`.
 
 Note: the url are expanded automatically for `github` and `huggingface`, however `https://` and `http://` prefix works as well.
 

docs/content/features/runtime-settings.md

@@ -79,8 +79,8 @@ Changes to P2P settings automatically restart the P2P stack with the new configu
 
 Manage model and backend galleries:
 
-- **Model Galleries**: JSON array of gallery objects with `url` and `name` fields
-- **Backend Galleries**: JSON array of backend gallery objects
+- **Model Galleries**: JSON array of gallery objects with `url` and `name` fields, plus an optional `mirrors` list of fallback URLs (see [Gallery mirrors]({{%relref "features/model-gallery#gallery-mirrors" %}}))
+- **Backend Galleries**: JSON array of backend gallery objects, which accept the same `mirrors` key
 - **Autoload Galleries**: Automatically load model galleries on startup
 - **Autoload Backend Galleries**: Automatically load backend galleries on startup
 
@@ -148,13 +148,15 @@ The `runtime_settings.json` file follows this structure:
   "federated": false,
   "galleries": [
     {
-      "url": "github:mudler/LocalAI/gallery/index.yaml@master",
+      "url": "https://index.localai.io/models",
+      "mirrors": ["github:mudler/LocalAI/gallery/index.yaml@master"],
       "name": "localai"
     }
   ],
   "backend_galleries": [
     {
-      "url": "github:mudler/LocalAI/backend/index.yaml@master",
+      "url": "https://index.localai.io/backends",
+      "mirrors": ["github:mudler/LocalAI/backend/index.yaml@master"],
       "name": "localai"
     }
   ],

docs/content/features/text-generation.md

@@ -918,6 +918,200 @@ options:
 The full list of registered parsers lives in `sglang.srt.function_call`
 and `sglang.srt.parser.reasoning_parser`.
 
+### vllm.cpp
+
+[vllm.cpp](https://github.com/mudler/vllm.cpp) is the LocalAI team's C++ port of
+vLLM: the same continuous-batching scheduler, paged KV cache and prefix caching,
+with no Python at inference time. It consumes either a HuggingFace safetensors
+model directory or a `.gguf` file, and applies the model's chat template,
+tool-call parsing and reasoning split engine-side.
+
+#### Setup
+
+```yaml
+name: vllm-cpp
+backend: vllm-cpp
+parameters:
+  model: "Qwen/Qwen3-4B"
+context_size: 8192
+template:
+  use_tokenizer_template: true
+```
+
+#### Configuring the engine with `engine_args`
+
+The same `engine_args:` map the vLLM and SGLang backends accept is honoured
+here, with keys spelled exactly as vLLM's own CLI flags - so a `speculative_config`
+or `kv_transfer_config` block written for vLLM works verbatim. Unknown keys are
+ignored rather than fatal; the engine validates the documents it is handed and
+reports a precise error at load.
+
+```yaml
+name: qwen35-a3b
+backend: vllm-cpp
+parameters:
+  model: "Qwen/Qwen3.5-A3B"
+context_size: 16384
+template:
+  use_tokenizer_template: true
+engine_args:
+  # KV cache sizing: num_blocks * block_size tokens of cache.
+  block_size: 32
+  num_blocks: 1024
+  # Concurrency and the per-step chunked-prefill token budget.
+  max_num_seqs: 32
+  max_num_batched_tokens: 8192
+  # Automatic prefix caching. Omit to keep the model's own default
+  # (on for dense models, off for hybrid / attention-free ones).
+  enable_prefix_caching: true
+  # Scheduler admission order: fcfs (default), priority, or lpm
+  # (cache-aware longest-prefix-match; needs prefix caching to have any effect).
+  scheduling_policy: lpm
+```
+
+| Key | Meaning | Default |
+|-----|---------|---------|
+| `block_size` | KV-cache block size, in tokens per block | 32 |
+| `num_blocks` | KV-cache blocks to allocate | 256 |
+| `max_model_len` | Max sequence length; also settable as `context_size` / `max_model_len` | model config |
+| `max_num_seqs` | Max concurrent sequences the scheduler admits | 8 |
+| `max_num_batched_tokens` | Per-step chunked-prefill token budget | per-arch (2048 dense, 4096/8192 MoE) |
+| `enable_prefix_caching` | Automatic prefix caching; `enable_radix_attention` is an accepted alias | model default |
+| `enable_jump_forward` | Jump-forward decoding, which emits grammar-forced tokens without a model step. Only affects constrained requests (`grammar`, JSON schema) | off |
+| `scheduling_policy` | `fcfs`, `priority`, or `lpm` | `fcfs` |
+| `tool_parser` / `reasoning_parser` | Force a parser instead of chat-template auto-detection | auto |
+| `tokenizer_config` | Override the `tokenizer_config.json` the chat template is read from | `<model_dir>/tokenizer_config.json` |
+| `speculative_config` | Speculative decoding (see below) | disabled |
+| `kv_transfer_config` | External KV connector / LMCache (see below) | none |
+
+Raising `max_num_batched_tokens` lets more prefill land in a single step, at the
+cost of decode latency for requests queued behind it. The default deliberately
+does not scale with `max_num_seqs`, which is what keeps a large concurrent
+prefill from blowing up the per-step activation on the hybrid architectures.
+
+`enable_prefix_caching` and `enable_jump_forward` are tri-state at the engine
+boundary: omitting the key defers to a default (the model's own capability for
+prefix caching, an environment variable for jump forward), while an explicit
+`false` forces the feature off. Those are genuinely different - prefix caching
+defaults *on* for dense models - so write the key only when you mean to override.
+
+#### Speculative decoding
+
+`speculative_config:` takes the same JSON object as vLLM's
+`--speculative-config`. Three methods are supported.
+
+> **Architecture limit.** At the current engine pin, `mtp` and `dflash` are
+> **Qwen3.5 / Qwen3.6 only**. The engine builds a widened speculative KV cache
+> directly for those families rather than through the model registry, so a
+> speculative config on any other architecture (Llama, GLM, Gemma, Mistral, ...)
+> will not work regardless of checkpoint format. `ngram` needs no draft weights
+> and is not subject to this limit.
+
+> **Format support.** `mtp` and `dflash` now work from a `.gguf` target as well
+> as safetensors. An MTP head is read from the GGUF's `nextn.*` tensors when the
+> file declares `<arch>.nextn_predict_layers`; a GGUF exported WITHOUT the head
+> (converted with `--no-mtp`, or predating llama.cpp's Qwen3.5 MTP support) is
+> refused at load naming that as the reason. A DFlash draft may itself be a
+> `dflash`-arch GGUF, and the target may be a GGUF too. `ngram` needs no draft
+> weights and works on any format.
+
+**MTP** (Multi-Token Prediction) uses a draft head shipped inside the target
+checkpoint's own `mtp.*` tensors, so there is no second model to download. It
+requires a **safetensors** checkpoint - the `mtp.*` tensors do not survive GGUF
+conversion, and an MTP config over a `.gguf` model is rejected at load.
+
+```yaml
+engine_args:
+  speculative_config:
+    method: mtp
+    # Optional; defaults to the checkpoint's own head depth, which is
+    # usually the right value. Must be a multiple of that depth.
+    num_speculative_tokens: 1
+```
+
+**DFlash** uses a separate block-diffusion drafter that proposes a whole block
+of tokens in one non-autoregressive forward pass. Unlike MTP, the draft is its
+own checkpoint, so `model:` is **required**:
+
+```yaml
+engine_args:
+  speculative_config:
+    method: dflash
+    model: z-lab/Qwen3.6-27B-DFlash
+    num_speculative_tokens: 4
+```
+
+The draft shares the *target's* `embed_tokens` and `lm_head`, so both must come
+from the same model family and the target must be safetensors.
+
+**The engine does not download the draft.** `model:` is resolved, in order,
+as a path as given, then as the last path segment under LocalAI's models
+directory (`z-lab/Qwen3.6-27B-DFlash` → `<models>/Qwen3.6-27B-DFlash`, which is
+what LocalAI's own downloader produces), then as the whole reference under the
+models directory. Install the draft into LocalAI first, or give an absolute path
+to a directory containing `config.json`. If none of those resolve, the load
+fails immediately naming every location that was tried, rather than reporting a
+missing checkpoint from inside the engine.
+
+**N-gram** needs no draft model at all - it proposes from the prompt's own
+suffix history. `num_speculative_tokens` is required:
+
+```yaml
+engine_args:
+  speculative_config:
+    method: ngram
+    num_speculative_tokens: 4
+    prompt_lookup_min: 5
+    prompt_lookup_max: 5
+```
+
+> **Auto-configuration on import.** When you import a safetensors repository
+> with `backend: vllm-cpp`, LocalAI reads the checkpoint's `config.json` and, if
+> it declares an MTP head (`mtp_num_hidden_layers`), writes
+> `speculative_config: {method: mtp}` into the generated `engine_args` for you.
+> An explicit `speculative_config` in your own config is never overwritten.
+> Importing a DFlash *draft* repository is refused with a warning: a drafter
+> cannot serve on its own, so import the target model and point
+> `speculative_config.model` at the draft.
+
+#### External KV cache with LMCache
+
+`kv_transfer_config:` takes vLLM's `--kv-transfer-config` JSON and selects an
+external KV-cache connector. The `lm://` LMCache client lets prefill KV be
+stored to and reloaded from a shared `lmcache.v1.server`, so a prefix computed
+by one replica does not have to be recomputed by the next:
+
+```yaml
+engine_args:
+  kv_transfer_config:
+    kv_connector: LMCacheConnector
+    kv_role: kv_both          # required whenever kv_connector is set
+    kv_connector_extra_config:
+      host: 127.0.0.1
+      port: 65432
+```
+
+`kv_role` is one of `kv_producer` (store only), `kv_consumer` (load only), or
+`kv_both`. An unregistered connector name, a missing role, or a malformed
+document fails the load with an explicit error rather than silently running
+without the cache.
+
+#### Legacy `options:` list
+
+Earlier versions configured this backend through the flat `options:` list, and
+those configs keep working. Every key in the table above is still read from
+there in `key:value` form, and `engine_args` wins on any key set in both:
+
+```yaml
+options:
+  - max_num_seqs:32
+  - enable_prefix_caching:true
+```
+
+New configs should prefer `engine_args:`, which is the only place the nested
+`speculative_config` / `kv_transfer_config` documents can be written naturally
+rather than as a single-line JSON string.
+
 ### Transformers
 
 [Transformers](https://huggingface.co/docs/transformers/index) is a State-of-the-art Machine Learning library for PyTorch, TensorFlow, and JAX.

docs/content/features/tracing.md

@@ -9,6 +9,11 @@ LocalAI can retain recent API exchanges and backend operations for inspection
 on the **Traces** page in the management interface. Enable tracing in runtime
 settings or with the existing tracing configuration.
 
+API requests appear while they are still running. Their elapsed duration
+updates when the page refreshes, and the result column marks them as in
+progress until the response completes. In-flight requests live only in memory;
+the completed exchange is what LocalAI adds to the bounded, persistent history.
+
 API and backend trace histories are persisted in separate directories below
 the configured data path. They are restored after a clean service restart,
 whether or not authentication is enabled.

docs/content/getting-started/linux.md

@@ -111,6 +111,11 @@ For a Podman-managed container, configure Podman to preserve and pass the
 systemd socket file descriptor into the container. The LocalAI process inside
 the container consumes the same activation protocol.
 
+Activation needs both `LISTEN_PID` and `LISTEN_FDS`. If only one of them is set,
+LocalAI ignores them and binds `--address` as usual. A container engine started
+from a socket-activated system unit can leak a bare `LISTEN_PID` into every
+container it spawns, and that is not an activation attempt.
+
 ## Next Steps
 
 - [Try it out with examples](/basics/try/)

docs/content/getting-started/models.md

@@ -140,7 +140,7 @@ local-ai run oci://localai/phi-2:latest
 ```
 
 {{% notice note %}}
-When pulling models from Ollama or OCI registries, LocalAI identifies itself with a `LocalAI/<version>` `User-Agent` header so registry operators can attribute usage to LocalAI.
+On every model download — Ollama and OCI registries, the model gallery, and plain HTTP(S) file URLs alike — LocalAI identifies itself with a `LocalAI/<version> (<os>; <arch>)` `User-Agent` header (for example `LocalAI/v3.2.1 (linux; amd64)`) so registry and gallery operators can attribute usage to LocalAI. Builds from source that carry no stamped version send `LocalAI (<os>; <arch>)` instead.
 {{% /notice %}}
 
 ### Run Models via URI

docs/data/version.json

@@ -1,3 +1,3 @@
 {
-  "version": "v4.7.1"
+  "version": "v4.8.1"
 }

gallery/index.yaml

@@ -1,4 +1,101 @@
 ---
+- &qwen3-5-9b-defiant-fable
+  name: "qwen3.5-9b-defiant-fable-mtp"
+  variants:
+    - model: qwen3.5-9b-defiant-fable
+  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+  urls:
+    - https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF
+  description: |
+    Qwen3.5 9B Defiant Fable is an Apache-2.0 multimodal fine-tune for
+    reasoning, coding, creative writing, and roleplay. It retains the 256K
+    context window and vision support of Qwen3.5 while reducing refusals.
+    This default entry uses the NEO-imatrix Q4_K_M build with multi-token
+    prediction enabled for faster generation.
+  license: apache-2.0
+  icon: https://huggingface.co/DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF/resolve/main/defiant-fable-9b.png
+  tags:
+    - llm
+    - gguf
+    - cpu
+    - gpu
+    - qwen3.5
+    - reasoning
+    - coding
+    - creative-writing
+    - uncensored
+    - vision
+    - multimodal
+    - mtp
+  last_checked: "2026-08-04"
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+      - vision
+    mmproj: llama-cpp/mmproj/qwen3.5-9b-defiant-fable/mmproj-BF16.gguf
+    options:
+      - use_jinja:true
+      - spec_type:draft-mtp
+      - spec_n_max:6
+      - spec_p_min:0.75
+    parameters:
+      model: llama-cpp/models/qwen3.5-9b-defiant-fable/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4_K_M.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/qwen3.5-9b-defiant-fable/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4_K_M.gguf
+      uri: huggingface://DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-MTP-Q4_K_M.gguf
+      sha256: d7eb4fac9389d53fa576f64a6ff53e914a00bc7705dc354d1065887565147320
+    - filename: llama-cpp/mmproj/qwen3.5-9b-defiant-fable/mmproj-BF16.gguf
+      uri: huggingface://DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF/mmproj-BF16.gguf
+      sha256: 853698ce7aa6c7ba732478bad280240969ddf7b0fcbf93900046f63903a83383
+- !!merge <<: *qwen3-5-9b-defiant-fable
+  name: "qwen3.5-9b-defiant-fable"
+  variants: []
+  description: |
+    Qwen3.5 9B Defiant Fable in the plain NEO-imatrix Q4_K_M GGUF format.
+    This fallback offers the same multimodal reasoning, coding, and creative
+    capabilities without enabling multi-token prediction.
+  tags:
+    - llm
+    - gguf
+    - cpu
+    - gpu
+    - qwen3.5
+    - reasoning
+    - coding
+    - creative-writing
+    - uncensored
+    - vision
+    - multimodal
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+      - vision
+    mmproj: llama-cpp/mmproj/qwen3.5-9b-defiant-fable/mmproj-BF16.gguf
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/qwen3.5-9b-defiant-fable/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-Q4_K_M.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/qwen3.5-9b-defiant-fable/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-Q4_K_M.gguf
+      uri: huggingface://DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF/Qwen3.5-9B-The-Defiant-Fable-Uncnr-Heretic-NEO-MAX-Q4_K_M.gguf
+      sha256: d33db5e583b9c9251402e876443791bc979f12af934bfb0630eadfb456279f84
+    - filename: llama-cpp/mmproj/qwen3.5-9b-defiant-fable/mmproj-BF16.gguf
+      uri: huggingface://DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-GGUF/mmproj-BF16.gguf
+      sha256: 853698ce7aa6c7ba732478bad280240969ddf7b0fcbf93900046f63903a83383
 - &nemotron-3-embed-1b
   name: "nemotron-3-embed-1b-q4"
   url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
@@ -1992,7 +2089,7 @@
   files:
     - filename: ds4flash.gguf
       uri: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
-      sha256: ba1d64ad8d77038124839956b614db2e889daa1a4ddc83060bb06ccb5a1d7461
+      sha256: a9aadd5a1921708c97aecaf29e6b3d5c0aa252aadc3b706d1281f68361bd52b9
 - name: "qwopus3.6-35b-a3b-coder-mtp"
   url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
   urls:
@@ -2091,6 +2188,83 @@
     - filename: llama-cpp/models/Qwen-AgentWorld-35B-A3B-GGUF/Qwen-AgentWorld-35B-A3B-UD-Q4_K_M.gguf
       sha256: e7a8eafdd8013443b6bcc4b6fb47b2d2025f772d359650b9ceb7d75971e22cad
       uri: https://huggingface.co/unsloth/Qwen-AgentWorld-35B-A3B-GGUF/resolve/main/Qwen-AgentWorld-35B-A3B-UD-Q4_K_M.gguf
+- &agents-a1-4b
+  name: "agents-a1-4b"
+  variants:
+    - model: agents-a1-4b-q8
+  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+  urls:
+    - https://huggingface.co/InternScience/Agents-A1-4B
+    - https://huggingface.co/InternScience/Agents-A1-4B-Q4_K_M-GGUF
+  description: |
+    Agents-A1-4B is InternScience's Apache-2.0 dense 4B agentic model, based on
+    Qwen3.5. It is trained for long-horizon search, engineering and scientific
+    research, instruction following, tool use, and multimodal tasks. This entry
+    uses the official Q4_K_M GGUF quantization and vision projector.
+  license: "apache-2.0"
+  tags:
+    - llm
+    - gguf
+    - vision
+    - multimodal
+    - gpu
+    - cpu
+  icon: https://huggingface.co/InternScience/Agents-A1-4B/resolve/main/figures/logo_nobg.png
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+    mmproj: llama-cpp/mmproj/Agents-A1-4B-Q4_K_M/Agents-A1-4B-mmproj.gguf
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/Agents-A1-4B-Q4_K_M/Agents-A1-4B-Q4_K_M.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/Agents-A1-4B-Q4_K_M/Agents-A1-4B-Q4_K_M.gguf
+      sha256: d93c393a9bd5139a4b5cfe24d31ef553c5a497bfb8afec178a354ecbf508f062
+      uri: huggingface://InternScience/Agents-A1-4B-Q4_K_M-GGUF/Agents-A1-4B-Q4_K_M.gguf
+    - filename: llama-cpp/mmproj/Agents-A1-4B-Q4_K_M/Agents-A1-4B-mmproj.gguf
+      sha256: 254145e7e03e9e8d3120813fac8033ffa04e411eb6d70a198833504935681084
+      uri: huggingface://InternScience/Agents-A1-4B-Q4_K_M-GGUF/Agents-A1-4B-mmproj.gguf
+- !!merge <<: *agents-a1-4b
+  name: "agents-a1-4b-q8"
+  variants: []
+  urls:
+    - https://huggingface.co/InternScience/Agents-A1-4B
+    - https://huggingface.co/InternScience/Agents-A1-4B-Q8_0-GGUF
+  description: |
+    Agents-A1-4B is InternScience's Apache-2.0 dense 4B agentic model, based on
+    Qwen3.5. It is trained for long-horizon search, engineering and scientific
+    research, instruction following, tool use, and multimodal tasks. This entry
+    uses the official Q8_0 GGUF quantization and vision projector.
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+    mmproj: llama-cpp/mmproj/Agents-A1-4B-Q8_0/Agents-A1-4B-mmproj.gguf
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/Agents-A1-4B-Q8_0/Agents-A1-4B-Q8_0.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/Agents-A1-4B-Q8_0/Agents-A1-4B-Q8_0.gguf
+      sha256: c327f66e820dae550bd230394595071c79f48c88d411b452d013ee4b5999fcea
+      uri: huggingface://InternScience/Agents-A1-4B-Q8_0-GGUF/Agents-A1-4B-Q8_0.gguf
+    - filename: llama-cpp/mmproj/Agents-A1-4B-Q8_0/Agents-A1-4B-mmproj.gguf
+      sha256: 254145e7e03e9e8d3120813fac8033ffa04e411eb6d70a198833504935681084
+      uri: huggingface://InternScience/Agents-A1-4B-Q8_0-GGUF/Agents-A1-4B-mmproj.gguf
 - name: "ornith-1.0-9b"
   variants:
     - model: ornith-1.0-9b-mtp
@@ -2730,6 +2904,86 @@
     - filename: llama-cpp/mmproj/Qwopus3.6-27B-Coder-Compat-MTP-GGUF/mmproj-F32.gguf
       sha256: 32f7ea0600c07272547da401d460f8abbd980f3a57b69d6df87be0e2505e0b9c
       uri: https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-Compat-MTP-GGUF/resolve/main/mmproj-F32.gguf
+- &qwen3-5-9b-hauhaucs-aggressive
+  name: "qwen3.5-9b-hauhaucs-aggressive"
+  variants:
+    - model: qwen3.5-9b-hauhaucs-aggressive-q8
+  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+  urls:
+    - https://huggingface.co/Qwen/Qwen3.5-9B
+    - https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
+  description: |
+    Qwen3.5 9B Aggressive is HauhauCS's refusal-removed fine-tune of the
+    multimodal Qwen3.5 9B model. It retains the base model's reasoning, tool
+    use, image and video understanding, and 262K-token native context window.
+
+    This entry uses the balanced Q4_K_M GGUF quantization and includes the
+    matching BF16 multimodal projector. The Q8_0 variant offers higher fidelity.
+  license: "apache-2.0"
+  tags:
+    - llm
+    - gguf
+    - cpu
+    - gpu
+    - qwen
+    - multimodal
+    - uncensored
+  icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg
+  last_checked: "2026-08-04"
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+    mmproj: llama-cpp/mmproj/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
+      sha256: 2ca636d9e81d3d23ca9b60c234fe185d30ec082eeba69ce770fdb0c76559a4f5
+      uri: huggingface://HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
+    - filename: llama-cpp/mmproj/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q4_K_M/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+      sha256: 05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934
+      uri: huggingface://HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+- !!merge <<: *qwen3-5-9b-hauhaucs-aggressive
+  name: "qwen3.5-9b-hauhaucs-aggressive-q8"
+  variants: []
+  description: |
+    Qwen3.5 9B Aggressive is HauhauCS's refusal-removed fine-tune of the
+    multimodal Qwen3.5 9B model. It retains the base model's reasoning, tool
+    use, image and video understanding, and 262K-token native context window.
+
+    This entry uses the higher-fidelity Q8_0 GGUF quantization and includes the
+    matching BF16 multimodal projector.
+  overrides:
+    backend: llama-cpp
+    function:
+      automatic_tool_parsing_fallback: true
+      grammar:
+        disable: true
+    known_usecases:
+      - chat
+    mmproj: llama-cpp/mmproj/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+    options:
+      - use_jinja:true
+    parameters:
+      model: llama-cpp/models/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0.gguf
+    template:
+      use_tokenizer_template: true
+  files:
+    - filename: llama-cpp/models/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0.gguf
+      sha256: 99e7f2201c0046b05d2825e4d8be6a2efad2b87b071cd55d37bdd9fbe201a58b
+      uri: huggingface://HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0.gguf
+    - filename: llama-cpp/mmproj/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q8_0/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
+      sha256: 05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934
+      uri: huggingface://HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
 # DFlash speculative-decoding pairs (upstream llama.cpp `draft-dflash`).
 # Each entry ships a full target model plus a small block-diffusion drafter
 # (z-lab DFlash, converted with upstream convert_hf_to_gguf.py, GGUF arch
@@ -40364,7 +40618,7 @@
   files:
     - filename: cohere-transcribe-q4_k.gguf
       uri: huggingface://cstr/cohere-transcribe-03-2026-GGUF/cohere-transcribe-q4_k.gguf
-      sha256: 2931fc0ac6d6708eef5389aadf1ebd5eec7b8e764bac385be585e910c0e7b410
+      sha256: 237261c543dc9124a3f08f95b48c9c672896ef0d79dc8cadce3fb4ddc09a2ef8
 - name: wav2vec2-crispasr
   url: github:mudler/LocalAI/gallery/virtual.yaml@master
   urls:

go.mod

@@ -24,7 +24,7 @@ require (
 	github.com/gofrs/flock v0.13.0
 	github.com/google/go-containerregistry v0.21.6
 	github.com/google/uuid v1.6.0
-	github.com/gpustack/gguf-parser-go v0.24.0
+	github.com/gpustack/gguf-parser-go v0.25.0
 	github.com/hpcloud/tail v1.0.0
 	github.com/ipfs/go-log v1.0.5
 	github.com/jaypipes/ghw v0.24.0

go.sum

@@ -666,8 +666,8 @@ github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A
 github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 h1:JeSE6pjso5THxAzdVpqr6/geYxZytqFMBCOtn/ujyeo=
 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674/go.mod h1:r4w70xmWCQKmi1ONH4KIaBptdivuRPyosB9RmPlGEwA=
-github.com/gpustack/gguf-parser-go v0.24.0 h1:tdJceXYp9e5RhE9RwVYIuUpir72Jz2D68NEtDXkKCKc=
-github.com/gpustack/gguf-parser-go v0.24.0/go.mod h1:y4TwTtDqFWTK+xvprOjRUh+dowgU2TKCX37vRKvGiZ0=
+github.com/gpustack/gguf-parser-go v0.25.0 h1:1AMBhMKtI24nTtn588Bq53FqNiOvEw1x9Nb4HbRrThs=
+github.com/gpustack/gguf-parser-go v0.25.0/go.mod h1:y4TwTtDqFWTK+xvprOjRUh+dowgU2TKCX37vRKvGiZ0=
 github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 h1:UH//fgunKIs4JdUbpDl1VZCDaL56wXCB/5+wF6uHfaI=
 github.com/grpc-ecosystem/go-grpc-middleware v1.4.0/go.mod h1:g5qyo/la0ALbONm6Vbp88Yd8NsDy6rZz+RcrMPxvld8=
 github.com/grpc-ecosystem/grpc-gateway v1.16.0/go.mod h1:BDjrQk3hbvj6Nolgz8mAMFbcEtjT1g+wF4CSlocrBnw=

internal/internal_suite_test.go

@@ -0,0 +1,13 @@
+package internal
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestInternal(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Internal test suite")
+}

internal/version.go

@@ -1,6 +1,9 @@
 package internal
 
-import "fmt"
+import (
+	"fmt"
+	"runtime"
+)
 
 var Version = ""
 var Commit = ""
@@ -9,10 +12,20 @@ func PrintableVersion() string {
 	return fmt.Sprintf("%s (%s)", Version, Commit)
 }
 
-// UserAgent returns the version-aware client identity used for outbound requests.
+// UserAgent returns the version-aware client identity used for outbound
+// requests to registries and galleries.
+//
+// The OS/arch suffix follows ordinary HTTP client convention (apt, pip and
+// docker all send the equivalent) and rides on requests LocalAI already makes.
+// It discloses nothing a registry cannot already infer: pulling a linux/amd64
+// manifest reveals the same thing.
+//
+// An empty Version means a source build, which is worth being able to tell
+// apart from a released one when reading server logs.
 func UserAgent() string {
+	platform := fmt.Sprintf("(%s; %s)", runtime.GOOS, runtime.GOARCH)
 	if Version == "" {
-		return "LocalAI"
+		return "LocalAI " + platform
 	}
-	return fmt.Sprintf("LocalAI/%s", Version)
+	return fmt.Sprintf("LocalAI/%s %s", Version, platform)
 }

internal/version_test.go

@@ -0,0 +1,43 @@
+package internal
+
+import (
+	"fmt"
+	"runtime"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("UserAgent", func() {
+	platform := fmt.Sprintf("(%s; %s)", runtime.GOOS, runtime.GOARCH)
+
+	BeforeEach(func() {
+		saved := Version
+		DeferCleanup(func() { Version = saved })
+	})
+
+	DescribeTable("identifies the build",
+		func(version, want string) {
+			Version = version
+			Expect(UserAgent()).To(Equal(want))
+		},
+		Entry("source build without a stamped version", "", "LocalAI "+platform),
+		Entry("released build", "v3.2.1", "LocalAI/v3.2.1 "+platform),
+	)
+
+	// The platform suffix is what distinguishes a real build from the bare
+	// fallback, so assert it is genuinely present rather than trusting only the
+	// composed strings above — those would still pass if the format string and
+	// the expectation drifted together.
+	DescribeTable("always carries the platform",
+		func(version string) {
+			Version = version
+			Expect(UserAgent()).To(And(
+				ContainSubstring(runtime.GOOS),
+				ContainSubstring(runtime.GOARCH),
+			))
+		},
+		Entry("source build", ""),
+		Entry("released build", "v9.9.9"),
+	)
+})

pkg/downloader/huggingface.go

@@ -1,10 +1,12 @@
 package downloader
 
 import (
+	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
 	"io"
+	"net/http"
 	"strings"
 
 	"github.com/mudler/LocalAI/pkg/httpclient"
@@ -30,7 +32,16 @@ func HuggingFaceScan(uri URI) (*HuggingFaceScanResult, error) {
 	if len(cleanParts) <= 4 || (cleanParts[2] != "huggingface.co" && cleanParts[2] != hfHost) {
 		return nil, ErrNonHuggingFaceFile
 	}
-	results, err := httpclient.New(httpclient.WithFollowRedirects()).Get(fmt.Sprintf("%s/api/models/%s/%s/scan", HF_ENDPOINT, cleanParts[3], cleanParts[4]))
+	// Built as an explicit request rather than the client's Get shorthand purely
+	// so it carries the same User-Agent as every other request this package
+	// makes; HuggingFace is exactly the kind of host that wants to know who is
+	// calling its API.
+	scanURL := fmt.Sprintf("%s/api/models/%s/%s/scan", HF_ENDPOINT, cleanParts[3], cleanParts[4])
+	req, err := newDownloadRequest(context.Background(), http.MethodGet, scanURL, "")
+	if err != nil {
+		return nil, err
+	}
+	results, err := httpclient.New(httpclient.WithFollowRedirects()).Do(req)
 	if err != nil {
 		return nil, err
 	}

pkg/downloader/read_status_test.go

@@ -0,0 +1,54 @@
+package downloader_test
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+
+	"github.com/mudler/LocalAI/pkg/downloader"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("ReadWithCallback", func() {
+	// ReadWithCallback used to hand the body of an error response to the
+	// callback with a nil error, so a 404 page was indistinguishable from an
+	// empty gallery index and callers had no way to notice the source was down.
+	DescribeTable("fails on an HTTP error status",
+		func(status int) {
+			srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				http.Error(w, "nope", status)
+			}))
+			DeferCleanup(srv.Close)
+
+			called := false
+			err := downloader.URI(srv.URL).ReadWithCallback(specTempDir(), func(string, []byte) error {
+				called = true
+				return nil
+			})
+
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("status code"),
+				"the error does not mention the status code")
+			Expect(called).To(BeFalse(), "the error body was passed to the callback as content")
+		},
+		Entry("404", http.StatusNotFound),
+		Entry("500", http.StatusInternalServerError),
+		Entry("502", http.StatusBadGateway),
+	)
+
+	It("succeeds on 200", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			_, _ = w.Write([]byte("- name: a\n"))
+		}))
+		DeferCleanup(srv.Close)
+
+		var got string
+		Expect(downloader.URI(srv.URL).ReadWithAuthorizationAndCallback(context.Background(), specTempDir(), "",
+			func(_ string, d []byte) error {
+				got = string(d)
+				return nil
+			})).To(Succeed())
+		Expect(got).To(Equal("- name: a\n"))
+	})
+})

pkg/downloader/uri.go

@@ -21,6 +21,7 @@ import (
 
 	"github.com/mudler/xlog"
 
+	"github.com/mudler/LocalAI/internal"
 	"github.com/mudler/LocalAI/pkg/httpclient"
 	"github.com/mudler/LocalAI/pkg/oci"
 	"github.com/mudler/LocalAI/pkg/utils"
@@ -195,6 +196,11 @@ func (uri URI) ReadWithAuthorizationAndCallback(ctx context.Context, basePath st
 	if err != nil {
 		return err
 	}
+	// pkg/oci has always identified itself; gallery and file fetches went out
+	// anonymously, indistinguishable from any other Go program. One identity
+	// across every transport is politer to the hosts serving us and makes our
+	// traffic attributable when a gallery operator asks who is hammering them.
+	req.Header.Set("User-Agent", internal.UserAgent())
 	if authorization != "" {
 		req.Header.Add("Authorization", authorization)
 	}
@@ -205,6 +211,15 @@ func (uri URI) ReadWithAuthorizationAndCallback(ctx context.Context, basePath st
 	}
 	defer response.Body.Close()
 
+	// An error page is not content. Without this check a 404 or a 502 from a
+	// CDN is handed to the callback as if it were a gallery index or a model
+	// config: it parses to nothing, gets cached, and no caller can tell the
+	// source was down. DownloadFile has always checked the status; this path
+	// never did.
+	if response.StatusCode >= 400 {
+		return fmt.Errorf("failed to read url %q, invalid status code %d", url, response.StatusCode)
+	}
+
 	// Read the response body
 	body, err := io.ReadAll(response.Body)
 	if err != nil {
@@ -427,6 +442,7 @@ func newDownloadRequest(
 	if err != nil {
 		return nil, err
 	}
+	req.Header.Set("User-Agent", internal.UserAgent())
 	if bearerToken != "" {
 		req.Header.Set("Authorization", "Bearer "+bearerToken)
 	}
@@ -465,6 +481,7 @@ func (u URI) ContentLength(ctx context.Context) (int64, error) {
 	if err != nil {
 		return 0, err
 	}
+	req.Header.Set("User-Agent", internal.UserAgent())
 	resp, err := downloadHTTPClient().Do(req)
 	if err != nil {
 		return 0, err
@@ -483,6 +500,7 @@ func (u URI) ContentLength(ctx context.Context) (int64, error) {
 	if err != nil {
 		return 0, err
 	}
+	req2.Header.Set("User-Agent", internal.UserAgent())
 	req2.Header.Set("Range", "bytes=0-0")
 	resp2, err := downloadHTTPClient().Do(req2)
 	if err != nil {

pkg/downloader/useragent_test.go

@@ -0,0 +1,155 @@
+package downloader_test
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+
+	"github.com/mudler/LocalAI/internal"
+	"github.com/mudler/LocalAI/pkg/downloader"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// stampVersion pins a recognisable build version for the duration of a spec so
+// the expected User-Agent is not the empty-version ("source build") form, which
+// would still match if the version were dropped from the header.
+func stampVersion() {
+	GinkgoHelper()
+	saved := internal.Version
+	internal.Version = "v9.9.9"
+	DeferCleanup(func() { internal.Version = saved })
+}
+
+// expectUserAgent fails when the header is not exactly what internal.UserAgent
+// produces, and separately when it does not name the build version — the second
+// check is what catches a header that is set but carries the wrong identity.
+func expectUserAgent(site, got string) {
+	GinkgoHelper()
+	Expect(got).To(Equal(internal.UserAgent()), "%s: wrong User-Agent", site)
+	Expect(got).To(ContainSubstring("LocalAI/v9.9.9"), "%s: User-Agent does not name the build version", site)
+}
+
+func specTempDir() string {
+	GinkgoHelper()
+	dir, err := os.MkdirTemp("", "downloader-useragent-spec-*")
+	Expect(err).ToNot(HaveOccurred())
+	DeferCleanup(func() { _ = os.RemoveAll(dir) })
+	return dir
+}
+
+var _ = Describe("the outbound User-Agent", func() {
+	BeforeEach(stampVersion)
+
+	// The gallery index is fetched through this package. Without a User-Agent
+	// the request is indistinguishable from any other Go program, which is both
+	// unhelpful to the hosts serving us and inconsistent with pkg/oci, which has
+	// always identified itself.
+	It("is sent by ReadWithCallback", func() {
+		var got string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			got = r.Header.Get("User-Agent")
+			_, _ = w.Write([]byte("- name: a\n"))
+		}))
+		DeferCleanup(srv.Close)
+
+		uri := downloader.URI(srv.URL)
+		Expect(uri.ReadWithCallback(specTempDir(), func(string, []byte) error { return nil })).To(Succeed())
+
+		expectUserAgent("gallery read", got)
+	})
+
+	// Model files are the bulk of what LocalAI pulls; they go through
+	// newDownloadRequest, which every download and every resume probe shares.
+	It("is sent by DownloadFile", func() {
+		seen := make(chan string, 8)
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			seen <- r.Header.Get("User-Agent")
+			w.Header().Set("Accept-Ranges", "bytes")
+			_, _ = w.Write([]byte("payload"))
+		}))
+		DeferCleanup(srv.Close)
+
+		uri := downloader.URI(srv.URL + "/file.bin")
+		target := filepath.Join(specTempDir(), "file.bin")
+		Expect(uri.DownloadFile(target, "", 1, 1, func(string, string, string, float64) {})).To(Succeed())
+
+		close(seen)
+		n := 0
+		for ua := range seen {
+			n++
+			expectUserAgent("download", ua)
+		}
+		Expect(n).ToNot(BeZero(), "server saw no requests")
+	})
+
+	// ContentLength builds its own HEAD request rather than going through
+	// newDownloadRequest, so it needs its own coverage.
+	It("is sent by ContentLength's HEAD", func() {
+		var got string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			got = r.Header.Get("User-Agent")
+			w.Header().Set("Content-Length", "7")
+			w.WriteHeader(http.StatusOK)
+		}))
+		DeferCleanup(srv.Close)
+
+		size, err := downloader.URI(srv.URL + "/file.bin").ContentLength(context.Background())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(size).To(BeEquivalentTo(7))
+
+		expectUserAgent("content-length HEAD", got)
+	})
+
+	// When the HEAD carries no Content-Length, ContentLength falls back to a
+	// one-byte Range GET built at a third, separate site.
+	It("is sent by ContentLength's Range GET fallback", func() {
+		var rangeUA string
+		var sawRange bool
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.Method == http.MethodHead {
+				// No Content-Length: this is what pushes ContentLength onto the
+				// Range fallback path.
+				w.Header().Set("Accept-Ranges", "bytes")
+				w.WriteHeader(http.StatusOK)
+				return
+			}
+			sawRange = true
+			rangeUA = r.Header.Get("User-Agent")
+			w.Header().Set("Content-Range", "bytes 0-0/4242")
+			w.WriteHeader(http.StatusPartialContent)
+			_, _ = w.Write([]byte("x"))
+		}))
+		DeferCleanup(srv.Close)
+
+		size, err := downloader.URI(srv.URL + "/file.bin").ContentLength(context.Background())
+		Expect(err).ToNot(HaveOccurred())
+		Expect(size).To(BeEquivalentTo(4242))
+		Expect(sawRange).To(BeTrue(), "server never saw the Range GET; the fallback path was not exercised")
+
+		expectUserAgent("content-length Range GET", rangeUA)
+	})
+
+	// The HuggingFace safety scan is the one outbound request in this package
+	// that does not live in uri.go, and it was the easiest one to overlook.
+	It("is sent by the HuggingFace safety scan", func() {
+		var got string
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			got = r.Header.Get("User-Agent")
+			_, _ = w.Write([]byte(`{"repositoryId":"owner/repo","scansDone":true}`))
+		}))
+		DeferCleanup(srv.Close)
+
+		savedEndpoint := downloader.HF_ENDPOINT
+		downloader.HF_ENDPOINT = srv.URL
+		DeferCleanup(func() { downloader.HF_ENDPOINT = savedEndpoint })
+
+		uri := downloader.URI(srv.URL + "/owner/repo/resolve/main/model.gguf")
+		_, err := downloader.HuggingFaceScan(uri)
+		Expect(err).ToNot(HaveOccurred())
+
+		expectUserAgent("huggingface scan", got)
+	})
+})

pkg/oci/useragent.go

@@ -4,9 +4,10 @@ import "github.com/mudler/LocalAI/internal"
 
 // UserAgent returns the User-Agent string LocalAI sends on outbound registry
 // requests (OCI registries and Ollama). It identifies the client as LocalAI
-// and, when the binary was built with a version stamp, appends it so registries
-// can attribute client-side usage to LocalAI rather than to the generic
-// User-Agent of the underlying transport library.
+// and, when the binary was built with a version stamp, appends it, followed by
+// the OS and architecture it is running on, so registries can attribute
+// client-side usage to LocalAI rather than to the generic User-Agent of the
+// underlying transport library.
 func UserAgent() string {
 	return internal.UserAgent()
 }

pkg/oci/useragent_test.go

@@ -1,6 +1,9 @@
 package oci_test
 
 import (
+	"fmt"
+	"runtime"
+
 	"github.com/mudler/LocalAI/internal"
 	. "github.com/mudler/LocalAI/pkg/oci"
 	. "github.com/onsi/ginkgo/v2"
@@ -21,12 +24,12 @@ var _ = Describe("OCI", func() {
 
 		It("identifies as LocalAI when no version is stamped", func() {
 			internal.Version = ""
-			Expect(UserAgent()).To(Equal("LocalAI"))
+			Expect(UserAgent()).To(Equal(fmt.Sprintf("LocalAI (%s; %s)", runtime.GOOS, runtime.GOARCH)))
 		})
 
 		It("appends the build version when one is stamped", func() {
 			internal.Version = "v3.2.1"
-			Expect(UserAgent()).To(Equal("LocalAI/v3.2.1"))
+			Expect(UserAgent()).To(Equal(fmt.Sprintf("LocalAI/v3.2.1 (%s; %s)", runtime.GOOS, runtime.GOARCH)))
 		})
 	})
 })

pkg/vram/gguf_reader.go

@@ -2,6 +2,7 @@ package vram
 
 import (
 	"context"
+	"fmt"
 	"strings"
 
 	gguf "github.com/gpustack/gguf-parser-go"
@@ -10,7 +11,18 @@ import (
 
 type defaultGGUFReader struct{}
 
-func (defaultGGUFReader) ReadMetadata(ctx context.Context, uri string) (*GGUFMeta, error) {
+func (defaultGGUFReader) ReadMetadata(ctx context.Context, uri string) (meta *GGUFMeta, err error) {
+	// gguf-parser-go parses lengths supplied by the file and has historically
+	// panicked on values that cannot fit in a Go slice. Metadata can come from
+	// an untrusted remote host, and this reader is also used by a background
+	// gallery worker, where an escaped panic would terminate the whole server.
+	defer func() {
+		if recovered := recover(); recovered != nil {
+			meta = nil
+			err = fmt.Errorf("read GGUF metadata: parser panic: %v", recovered)
+		}
+	}()
+
 	u := downloader.URI(uri)
 	urlStr := u.ResolveURL()
 
@@ -28,7 +40,10 @@ func (defaultGGUFReader) ReadMetadata(ctx context.Context, uri string) (*GGUFMet
 	if !u.LooksLikeHTTPURL() {
 		return nil, nil
 	}
-	f, err := gguf.ParseGGUFFileRemote(ctx, urlStr)
+	// The estimator only consumes architecture scalars. Tokenizer arrays can
+	// be very large and are unnecessary here, so avoid downloading or
+	// allocating them for remote files just as the local path does above.
+	f, err := gguf.ParseGGUFFileRemote(ctx, urlStr, gguf.SkipLargeMetadata())
 	if err != nil {
 		return nil, err
 	}

pkg/vram/gguf_reader_test.go

@@ -0,0 +1,115 @@
+package vram_test
+
+import (
+	"bytes"
+	"context"
+	"encoding/binary"
+	"math"
+	"net/http"
+	"net/http/httptest"
+	"time"
+
+	gguf "github.com/gpustack/gguf-parser-go"
+	"github.com/mudler/LocalAI/pkg/vram"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("DefaultGGUFReader", func() {
+	It("reads architecture scalars from a valid remote GGUF", func() {
+		server := serveGGUF(validRemoteGGUF())
+
+		meta, err := vram.DefaultGGUFReader().ReadMetadata(context.Background(), server.URL+"/model.gguf")
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(meta).To(Equal(&vram.GGUFMeta{
+			BlockCount:           32,
+			EmbeddingLength:      4096,
+			HeadCount:            32,
+			HeadCountKV:          8,
+			MaximumContextLength: 8192,
+		}))
+	})
+
+	It("rejects an overflowing tokenizer array without allocating it", func() {
+		server := serveGGUF(malformedGGUFArray(math.MaxUint64))
+
+		_, err := vram.DefaultGGUFReader().ReadMetadata(context.Background(), server.URL+"/model.gguf")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).NotTo(ContainSubstring("parser panic"),
+			"large tokenizer metadata should be skipped with a bounds error")
+	})
+
+	It("converts a parser panic from malformed string metadata to an error", func() {
+		server := serveGGUF(malformedGGUFString(uint64(math.MaxInt64)))
+
+		_, err := vram.DefaultGGUFReader().ReadMetadata(context.Background(), server.URL+"/model.gguf")
+
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("parser panic"))
+	})
+})
+
+func serveGGUF(payload []byte) *httptest.Server {
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		http.ServeContent(w, r, "model.gguf", time.Time{}, bytes.NewReader(payload))
+	}))
+	DeferCleanup(server.Close)
+	return server
+}
+
+func malformedGGUFString(length uint64) []byte {
+	payload := ggufHeader(1)
+	payload = appendGGUFString(payload, "general.name")
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeString))
+	payload = binary.LittleEndian.AppendUint64(payload, length)
+	return payload
+}
+
+func validRemoteGGUF() []byte {
+	payload := ggufHeader(6)
+	payload = appendGGUFStringValue(payload, "general.architecture", "llama")
+	payload = appendGGUFUint32(payload, "llama.block_count", 32)
+	payload = appendGGUFUint32(payload, "llama.embedding_length", 4096)
+	payload = appendGGUFUint32(payload, "llama.attention.head_count", 32)
+	payload = appendGGUFUint32(payload, "llama.attention.head_count_kv", 8)
+	payload = appendGGUFUint32(payload, "llama.context_length", 8192)
+	return payload
+}
+
+func malformedGGUFArray(itemLength uint64) []byte {
+	payload := ggufHeader(1)
+	payload = appendGGUFString(payload, "tokenizer.ggml.tokens")
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeArray))
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeString))
+	payload = binary.LittleEndian.AppendUint64(payload, 1)
+	payload = binary.LittleEndian.AppendUint64(payload, itemLength)
+	return payload
+}
+
+func ggufHeader(metadataCount uint64) []byte {
+	payload := make([]byte, 0, 128)
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMagicGGUFLe))
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFVersionV3))
+	payload = binary.LittleEndian.AppendUint64(payload, 0)
+	payload = binary.LittleEndian.AppendUint64(payload, metadataCount)
+	return payload
+}
+
+func appendGGUFString(payload []byte, value string) []byte {
+	payload = binary.LittleEndian.AppendUint64(payload, uint64(len(value)))
+	return append(payload, value...)
+}
+
+func appendGGUFStringValue(payload []byte, key, value string) []byte {
+	payload = appendGGUFString(payload, key)
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeString))
+	return appendGGUFString(payload, value)
+}
+
+func appendGGUFUint32(payload []byte, key string, value uint32) []byte {
+	payload = appendGGUFString(payload, key)
+	payload = binary.LittleEndian.AppendUint32(payload, uint32(gguf.GGUFMetadataValueTypeUint32))
+	return binary.LittleEndian.AppendUint32(payload, value)
+}

scripts/build/backend-signing_test.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+WORKFLOW="$(dirname "$(realpath "$0")")/../../.github/workflows/backend_merge.yml"
+
+sign_commands=$(grep -Ec -- '^[[:space:]]+cosign sign([[:space:]]|$)' "$WORKFLOW" || true)
+bundle_flags=$(grep -Ec -- '^[[:space:]]+--new-bundle-format([[:space:]]|$)' "$WORKFLOW" || true)
+if [ "$sign_commands" -ne 2 ] || [ "$bundle_flags" -ne "$sign_commands" ]; then
+  echo "FAIL: every backend signing command must request the new bundle format (commands=$sign_commands flags=$bundle_flags)"
+  exit 1
+fi
+
+echo "PASS: backend signing emits Sigstore bundles for both registries"

swagger/docs.go

@@ -3822,6 +3822,13 @@ const docTemplate = `{
         "config.Gallery": {
             "type": "object",
             "properties": {
+                "mirrors": {
+                    "description": "Mirrors are tried in order when URL cannot be fetched. They are a\nfallback for availability, not a load-balancing pool: the primary is\nalways preferred, and a mirror is only consulted after the one before\nit fails. Any URI the gallery loader understands works here\n(https://, github:, file://).",
+                    "type": "array",
+                    "items": {
+                        "type": "string"
+                    }
+                },
                 "name": {
                     "type": "string"
                 },

swagger/swagger.json

@@ -3819,6 +3819,13 @@
         "config.Gallery": {
             "type": "object",
             "properties": {
+                "mirrors": {
+                    "description": "Mirrors are tried in order when URL cannot be fetched. They are a\nfallback for availability, not a load-balancing pool: the primary is\nalways preferred, and a mirror is only consulted after the one before\nit fails. Any URI the gallery loader understands works here\n(https://, github:, file://).",
+                    "type": "array",
+                    "items": {
+                        "type": "string"
+                    }
+                },
                 "name": {
                     "type": "string"
                 },

swagger/swagger.yaml

@@ -2,6 +2,16 @@ basePath: /
 definitions:
   config.Gallery:
     properties:
+      mirrors:
+        description: |-
+          Mirrors are tried in order when URL cannot be fetched. They are a
+          fallback for availability, not a load-balancing pool: the primary is
+          always preferred, and a mirror is only consulted after the one before
+          it fails. Any URI the gallery loader understands works here
+          (https://, github:, file://).
+        items:
+          type: string
+        type: array
       name:
         type: string
       url:

website/content/blog/what-landed-in-localai-4-8.md

@@ -3,12 +3,12 @@ title: "What landed in LocalAI 4.8"
 date: 2026-08-04
 author: "Ettore Di Giacinto"
 category: "Release"
-tags: ["release", "vllm.cpp", "audio.cpp", "3d", "gallery", "distributed", "performance"]
-summary: "A new inference engine, 3D generation, one backend that serves six audio endpoints, and a web interface 3.48x lighter. 374 pull requests in twenty-one days."
+tags: ["release", "vllm.cpp", "audio.cpp", "3d", "agent", "gallery", "distributed", "performance"]
+summary: "A new inference engine, a terminal agent in the CLI, 3D generation, and a web interface 3.48x lighter. 386 pull requests in twenty-two days."
 extracss: ["blog.css"]
 ---
 
-LocalAI 4.8.0 is out, after twenty-one days and 374 merged pull requests. There are three new things LocalAI can do, and a lot of repair work on things it already did.
+LocalAI 4.8.0 is out, after twenty-two days and 386 merged pull requests. There are four new things LocalAI can do, and a lot of repair work on things it already did.
 
 The full notes list everything. This post covers the parts that change what you do day to day, with the pull request numbers so you can read the diffs.
 
@@ -36,6 +36,11 @@ The third one was `/api/traces` returning a 21 MB unpaginated blob that the UI p
 
 ## One gallery entry, several builds
 
+<figure>
+<img src="/media/v4-8-0-ui-model-variants.png" alt="The model detail pane listing every variant">
+<figcaption>One entry, four builds. LocalAI picks the largest that fits and marks it auto-selected.</figcaption>
+</figure>
+
 Installing a model no longer means reading a list of quantizations and guessing which one your card will hold. A gallery entry can now declare `variants:`, a list of references to other entries that are alternative builds of the same weights:
 
 ```yaml
@@ -63,6 +68,11 @@ It has grown features vLLM does not have, which is most of the reason the port e
 
 Tool calling is at llama.cpp parity by construction, because chat deliberately reuses the same autoparser path: full minja chat templates, `tool_choice: auto` lowered to a lazy structural-tag decode constraint, 30 tool dialects, 7 reasoning parsers, and streamed `ChatDelta` and `ToolCallDelta`.
 
+<figure>
+<img src="/media/v4-8-0-vllm-cpp-scoreboard.png" alt="Throughput of vllm.cpp relative to each reference engine, drawn as deviation from parity">
+<figcaption>llama.cpp is left out because its 1.18x is a prefill ratio, and putting that on the same axis as throughput would compare two different measurements.</figcaption>
+</figure>
+
 Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/blob/master/docs/BENCHMARKS.md), which calls ties ties and losses losses. Above 1.0 means vllm.cpp is ahead:
 
 <div class="tw">
@@ -73,16 +83,19 @@ Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/b
 <tr><td>vLLM</td><td>Qwen3.6-35B-A3B NVFP4, GB10</td><td>1.010x at c16 and 1.013x at c32, behind from c1 to c8 (0.817x at c1)</td></tr>
 <tr><td>llama.cpp</td><td>Qwen3.5-2B GGUF, CPU aarch64</td><td>prefill 1.18x, decode a tie, memory parity</td></tr>
 <tr><td>MLX-LM</td><td>Qwen3-0.6B, Apple M4</td><td>97.6% of warm total, prefill ahead</td></tr>
-<tr><td>DwarfStar (ds4)</td><td>DeepSeek-V4-Flash IQ2_XXS, one DGX Spark</td><td>16.28 vs 16.33 tok/s decode, 0.997x, a parity result</td></tr>
+<tr><td>DwarfStar (ds4)</td><td>DeepSeek-V4-Flash IQ2_XXS, one DGX Spark</td><td>18.69 vs 16.33 tok/s decode, <b>1.144x</b>, same output</td></tr>
+<tr><td>vLLM</td><td>Laguna-XS-2.1 NVFP4, GB10</td><td>44.46 vs 43.10 tok/s, <b>1.03x</b>, same output</td></tr>
 </tbody>
 </table>
 </div>
 
 The upstream page is careful about its own noise: on the 27B grid the run-to-run spread is 0.5% and c2 through c32 land between 0.7% and 1.7%, so it calls those five ties rather than wins. The concurrency-1 result is the one it stands behind.
 
-The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 16.28 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible.
+The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 18.69 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible.
 
-Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 is token-identical to vLLM's MTP and about 4% faster at concurrency 1.
+That number moved twice in a week, and the second move came from one lever. The dense Q8_0 projection tower was being read from the GGUF mmap over unified memory, which the GB10 reads about 20% slower per-GEMV than device memory. Staging that 6 GiB tower device-resident once at load, same bytes and same kernels, took decode from 16.23 to 18.69, generating the same tokens and using no more peak memory. The same change took Laguna-XS-2.1 from 87% of vLLM to 1.03x ahead of it.
+
+Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 generates the same tokens as vLLM's MTP and runs about 4% faster at concurrency 1.
 
 Configuration is a normal backend install:
 
@@ -131,6 +144,20 @@ The first engine behind it is `trellis2cpp`, an image-to-3D backend over TRELLIS
 <figcaption>trellis2-4b, 2,502,928 vertices and 5,012,118 triangles, turning in the browser. The remesh slider below it is the print path.</figcaption>
 </figure>
 
+## `local-ai chat` stopped being a REPL
+
+`local-ai chat` used to be a chat prompt in a terminal. It is now an agent, and it is the [nib](https://github.com/mudler/nib) harness compiled straight into the binary: tool use behind an approval gate, sub-agents, MCP servers, plugins and skills, auto-configured against your own instance. Nothing extra to install.
+
+```bash
+local-ai chat                      # the agent, pointed at your models
+echo "what is 2+2" | local-ai chat --cli
+local-ai chat --init zsh           # Ctrl+Space from any shell prompt
+```
+
+That last one prints a shell integration script (zsh, bash or fish), so you can pull the agent up from wherever you already are instead of opening something else.
+
+It runs shell commands now, so every tool call goes through an approval prompt you control, and read-only ones like `ls` and `cat` run without asking. If you had habits around the old REPL, a few things moved: `/clear` is gone and `/compact` is the closest thing, `/models` and `/model <name>` mean what they always meant, and switching model keeps the conversation instead of starting over ([#11291](https://github.com/mudler/LocalAI/pull/11291)).
+
 ## One backend, six audio endpoints
 
 The usual shape for audio is one backend per model family, which means a process per capability and a config file for each. `audio-cpp` wraps [audio.cpp](https://github.com/0xShug0/audio.cpp), a multi-family ggml audio engine. One backend process serves several unrelated families through a single runtime vocabulary, and works out which family a checkpoint belongs to from the GGUF's own `audiocpp.model_spec.family` metadata key. There is nothing backend-specific to write in the model config.
@@ -168,6 +195,11 @@ The `bonsai` backend serves the 1-bit (Q1_0) and ternary (Q2_0) Bonsai quantizat
 
 ## The operations bar became a page
 
+<figure>
+<img src="/media/v4-8-0-ui-activity.png" alt="The Activity page with four installs running">
+<figcaption>Four backend installs in flight, and the record of what already finished.</figcaption>
+</figure>
+
 The old operations bar rendered one row per in-flight operation above every page. Queue four model installs and a backend and it took most of the viewport, on every route, until the last one finished. It was doing two jobs at once. A global "something is happening" signal only needs one line, and the detail of what is happening needs a page of its own.
 
 The strip is now one line, permanently, showing a failure first and otherwise the least-advanced running operation, with a `+N more` pill. Its `✕` hides the strip and no longer cancels anything. That is a deliberate behavior change worth knowing about before you click it out of habit: the same glyph used to cancel a 17 GB download in one row and dismiss a message in the next. Cancelling moved to the new page, behind a button that says so.

website/layouts/index.html

@@ -19,7 +19,7 @@
           <div><b class="tnum" data-count="{{ .Site.Data.stats.stars }}">0</b><span>GitHub stars</span></div>
           <div><b class="tnum" data-count="73">0</b><span>Backends</span></div>
           <div><b class="tnum" data-count="{{ len .Site.Data.engines.engines }}">0</b><span>Engines we wrote</span></div>
-          <div><b class="tnum" data-count="1585">0</b><span>Models, one click</span></div>
+          <div><b class="tnum" data-count="1255">0</b><span>Models, one click</span></div>
         </div>
       </div>
       <div class="fd">
@@ -39,7 +39,8 @@
     <p class="kicker rv">The runtime</p>
     <h2 class="rv mt1" style="max-width:21ch">Everything else plugs into LocalAI.</h2>
     <p class="lede rv mt2">One binary with an OpenAI-compatible API in front of it. Point an existing client at it and the calls keep working, except now the model is on your machine. It also speaks the Anthropic, Ollama and ElevenLabs APIs, so most tools need a URL change and nothing else.</p>
-    <p class="lede rv mt2">Underneath, a small core pulls each engine in as a separate backend, only when a model asks for it. That is why one install covers this much ground without becoming a 9 GB download.</p>
+    <p class="lede rv mt2">The engine behind that API is swappable. One model can run on llama.cpp while the next loads on vLLM, SGLang or MLX, and the client never notices: same endpoint, same request, different engine underneath. Switching is one line in the model's config.</p>
+    <p class="lede rv mt2">A small core pulls each engine in as a separate backend, only when a model asks for it. That is why one install covers this much ground without becoming a 9 GB download.</p>
     <div class="apis rv">
       <span>OpenAI API</span><span>Anthropic API</span><span>Ollama API</span><span>ElevenLabs API</span><span>Realtime over WebRTC</span>
     </div>
@@ -57,7 +58,7 @@
       </div>
       <div class="duo__m rv">
         <figure class="screen" style="margin:0">
-          <figcaption class="screen__bar"><i></i> localai · model gallery <b>1,585 models</b></figcaption>
+          <figcaption class="screen__bar"><i></i> localai · model gallery <b>1,255 models</b></figcaption>
           <video src="/media/gallery.mp4" muted loop playsinline preload="none" data-lazy aria-label="Installing a model from the LocalAI gallery"></video>
         </figure>
       </div>
@@ -327,7 +328,7 @@
   <div class="shell">
     <div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
     <p class="kicker rv">The gallery</p>
-    <h2 class="rv mt1" style="max-width:20ch">1,585 models. No notebook, no conversion script.</h2>
+    <h2 class="rv mt1" style="max-width:20ch">1,255 models. No notebook, no conversion script.</h2>
     <div class="cards">
       <a class="cd rv" href="/docs/getting-started/models/"><p class="cd__k">Quantizations</p><h3>201 APEX builds</h3>
         <p>Every tier of every model we quantize, ranked against the hardware you actually have and installed with one click.</p><span class="cd__go">Browse the gallery →</span></a>

website/static/media/v4-8-0-vllm-cpp-scoreboard.html

@@ -0,0 +1,100 @@
+<!doctype html>
+<html>
+<head>
+<meta charset="utf-8">
+<style>
+  /* palette lifted from the two logos:
+     LocalAI  #0E2632 navy, #385360 slate, #469AAF teal, #90A8AE haze
+     vllm.cpp #3AB4CA teal, #95C4D1 light                              */
+  :root{
+    --bg:#0b1c25; --ink:#e8f1f4; --dim:#90a8ae; --faint:#5d757f;
+    --teal:#3ab4ca; --teal-hi:#7fd4e2; --amber:#e0a944; --rule:#1d3440;
+  }
+  *{margin:0;padding:0;box-sizing:border-box}
+  html,body{width:1600px;height:900px}
+  body{
+    background:radial-gradient(1250px 720px at 80% -12%, #143140 0%, var(--bg) 62%);
+    color:var(--ink);
+    font-family:-apple-system,"SF Pro Display","Segoe UI",Helvetica,Arial,sans-serif;
+    -webkit-font-smoothing:antialiased; padding:58px 84px; position:relative;
+  }
+  .eyebrow{display:flex;align-items:center;gap:14px;color:var(--teal);
+    font-weight:600;font-size:23px;letter-spacing:.14em;text-transform:uppercase}
+  .eyebrow .dot{width:11px;height:11px;border-radius:50%;background:var(--teal);
+    box-shadow:0 0 16px 2px var(--teal)}
+  h1{font-size:56px;line-height:1.06;font-weight:760;margin:16px 0 6px;letter-spacing:-.02em}
+  h1 .grad{background:linear-gradient(92deg,var(--teal),var(--teal-hi));
+    -webkit-background-clip:text;background-clip:text;color:transparent}
+  .sub{color:var(--dim);font-size:23px;margin-bottom:14px}
+  svg{width:100%;height:auto;display:block}
+  .foot{position:absolute;left:84px;right:84px;bottom:40px;display:flex;
+    justify-content:space-between;align-items:center;color:var(--faint);
+    font-size:21px;border-top:1px solid var(--rule);padding-top:16px}
+  .foot .link{color:var(--ink);font-weight:600}
+</style>
+</head>
+<body>
+  <div class="eyebrow"><span class="dot"></span>vllm.cpp &middot; throughput vs the reference engine</div>
+  <h1>Measured against <span class="grad">what each workload actually runs on</span></h1>
+  <div class="sub">Throughput relative to the reference. 1.00 is parity, bars run from it. Higher is faster.</div>
+
+  <svg id="c" viewBox="0 0 1432 585"></svg>
+
+  <div class="foot">
+    <span class="link">github.com/mudler/vllm.cpp</span>
+    <span>GB10 unless noted &middot; greedy, reference in its own production config &middot; docs/BENCHMARKS.md</span>
+  </div>
+
+<script>
+const rows = [
+  {ref:'DwarfStar (ds4)', work:'DeepSeek-V4-Flash IQ2_XXS', v:1.144, note:'18.69 vs 16.33 tok/s'},
+  {ref:'vLLM',            work:'Qwen3.6-27B NVFP4, c1',     v:1.045, note:'86.05 vs 82.32 tok/s'},
+  {ref:'vLLM',            work:'Laguna-XS-2.1 NVFP4',       v:1.030, note:'44.46 vs 43.10 tok/s'},
+  {ref:'vLLM',            work:'Qwen3.6-35B-A3B, c32',      v:1.013, note:'3030.5 vs 2993.0 tok/s'},
+  {ref:'MLX-LM',          work:'Qwen3-0.6B, Apple M4',      v:0.976, note:'97.6% of warm total'},
+];
+
+const W=1432, H=585;
+const AX=64;                   // axis strip reserved at the bottom
+const LBL=470;                 // left label gutter
+const R=150;                   // right gutter for the value
+const lo=-0.055, hi=0.165;     // deviation domain around parity
+const pw=W-LBL-R;
+const x = d => LBL + pw*((d-lo)/(hi-lo));
+const zero = x(0);
+const rowH = (H-AX)/rows.length;
+const barH = 46;
+
+let g='';
+// faint engineering grid at 2% steps
+for(let d=-0.04; d<=0.16001; d+=0.02){
+  const gx=x(d), on0=Math.abs(d)<1e-9;
+  g+=`<line x1="${gx}" y1="4" x2="${gx}" y2="${H-AX+10}" stroke="${on0?'#4a6b78':'#16303c'}" stroke-width="${on0?2:1}"/>`;
+  g+=`<text x="${gx}" y="${H-22}" fill="${on0?'#90a8ae':'#4d6570'}" font-size="17" text-anchor="middle"
+       font-weight="${on0?'700':'400'}">${(1+d).toFixed(2)}</text>`;
+}
+
+rows.forEach((r,i)=>{
+  const cy = i*rowH + rowH/2;
+  const d  = r.v-1;
+  const ahead = d>=0;
+  const col = ahead ? '#3ab4ca' : '#e0a944';
+  const x0 = ahead ? zero : x(d);
+  const w  = Math.abs(x(d)-zero);
+
+  // reference + workload, two weights on one line
+  g+=`<text x="${LBL-26}" y="${cy-4}" fill="#e8f1f4" font-size="25" font-weight="670" text-anchor="end">${r.ref}</text>`;
+  g+=`<text x="${LBL-26}" y="${cy+22}" fill="#5d757f" font-size="19" text-anchor="end">${r.work}</text>`;
+
+  g+=`<rect x="${x0}" y="${cy-barH/2}" width="${Math.max(w,2)}" height="${barH}" rx="4" fill="${col}" opacity="0.92"/>`;
+
+  // value, then the raw measurement under it
+  const vx = ahead ? x(d)+18 : zero+18;
+  g+=`<text x="${vx}" y="${cy+1}" fill="${col}" font-size="27" font-weight="700"
+       font-variant-numeric="tabular-nums">${r.v.toFixed(3)}&times;</text>`;
+  g+=`<text x="${vx}" y="${cy+23}" fill="#5d757f" font-size="17">${r.note}</text>`;
+});
+document.getElementById('c').innerHTML=g;
+</script>
+</body>
+</html>