mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-04 04:12:22 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs(blog): add release write-ups for 3.10 through 4.3 (#11330)

Browse Source 
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.

Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.


Assisted-by: Claude Code:claude-opus-5[1m]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
2026-08-04 00:13:45 +02:00
committed by GitHub
parent 88fdda6211
commit 2e14511fe2
5 changed files with 426 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
website/content/blog/what-landed-in-localai-3-10.md Normal file
View File

@@ -0,0 +1,59 @@
---
title: "LocalAI 3.10: the Anthropic and Responses APIs, and one image for every GPU"
date: 2026-01-18
author: "Ettore Di Giacinto"
category: "Release"
tags: ["release", "anthropic", "open-responses", "gpu", "moonshine"]
summary: "A /v1/messages endpoint that Claude clients can talk to unchanged, Open Responses compatibility that passes the official acceptance tests, and GPU libraries moved inside the backend containers so one image works on any hardware."
extracss: ["blog.css"]
---
Half the tooling worth using speaks a shape of API that is not OpenAI's. You find a client you like, it talks to Anthropic, and swapping it onto a local model means either rewriting the client or gluing a translation layer in front of it. Same story with the agent frameworks that went all in on the Responses API.
3.10.0 adds both surfaces natively, so the client does not have to know.
## Two more front doors
The Anthropic Messages API is served at `/v1/messages`, and at `/messages` for clients that do not prefix. Tool calling, streaming and non-streaming all work, so `anthropic-sdk-go`, LangChain and anything else built on that shape can be pointed at your instance without a code change.
The Open Responses API is at `/v1/responses`, with `/v1/responses/:id` to fetch one and `/v1/responses/:id/cancel` to stop it. It is stateful: pass a `response_id` and the conversation resumes, set `background: true` and the agent runs asynchronously while you go and do something else, then come back for the result. Streaming covers tools, images and audio.
That one passes the [official acceptance tests](https://www.openresponses.org/compliance), which was the bar I wanted to hit before shipping it.
## One image for every GPU
This is the change most likely to affect you even if you do not care about agents.
GPU libraries (CUDA, ROCm, Vulkan) now live inside the backend containers rather than in the image you pull. There is no longer a CUDA image, a ROCm image and a CPU image to choose between. You pull the image, and acceleration works if the hardware is there! Vulkan arm64 builds are in too.
It is experimental, and I want to be clear about that rather than bury it. It is a real architectural change to how every backend gets its libraries, and there will be hardware combinations we did not hit. If it does not work on yours, please file an issue, that is genuinely the most useful thing you can do for this one.
## Everything else
The backend gallery is system aware now, so it only lists backends your machine can actually run. No more scrolling past MLX entries on a Linux box.
Tool calls stream properly, including partial arguments as `input_json_delta`, and models that emit tools as XML (`<function>...</function>`) get parsed instead of dumping the markup into the message text. Both work across llama.cpp, vLLM and diffusers.
Thinking tags are extracted into a separate `reasoning` field rather than being left in the answer, in both SSE and non-SSE mode. The chat UI shows them under a Thinking tab.
There is a video generation page in the web UI with LTX-2 behind it, doing text-to-video and image-to-video with the usual `fps`, `num_frames` and `guidance_scale` controls.
There is request tracing now. `GET /api/traces` returns in-memory request and response logs, `/api/traces/clear` empties them. It is memory backed and drops old entries past a size cap, so it is for debugging an agent that is misbehaving right now, not for an audit trail.
Two new speech backends. Moonshine is an ONNX transcription engine aimed at low-end hardware, and it is the one to reach for on a Pi or an old laptop. It is quick! Pocket-TTS does lightweight TTS with voice cloning, though the cloning path needs a HuggingFace login and a registered voice model, so it is not quite copy-paste.
## Old hardware, and AMD memory
Two fixes worth calling out because they were silent failures rather than errors.
LocalAI was crashing on Intel CPUs without BMI2 (Sandy Bridge, Ivy Bridge), showing up as an `EOF` during model warmup rather than anything that pointed at the cause. It now falls back to `llama-cpp-fallback` on those chips.
On AMD, used and total VRAM were swapped when parsing `rocm-smi` output, so a dual-Radeon box reported nonsense. `HIP_VISIBLE_DEVICES` is also handled properly now, which matters if you are pinning to the discrete GPU.
## Thanks
Thanks to @richiejp, @majiayu000, @nanoandrew4, @DEVMANISHOFFL, @coffeerunhobby, @rampa3, @Nold360, @jroeber and @Divyanshupandey007 for the work in this cycle.
If the unified GPU backends misbehave on your setup, open an issue with what hardware you are on. And if you are wiring up the Anthropic or Responses endpoints and something does not match the spec, tell me, I would rather hear it from you than find out later.
[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0).

69
website/content/blog/what-landed-in-localai-4-0.md Normal file
View File

@@ -0,0 +1,69 @@
---
title: "LocalAI 4.0: agents in the core, and a React interface"
date: 2026-03-14
author: "Ettore Di Giacinto"
category: "Release"
tags: ["release", "agents", "agenthub", "mcp", "react", "webrtc"]
summary: "Native agent orchestration with the Agenthub, a rewritten interface with Canvas mode, MCP Apps with tool streaming, and two things removed."
extracss: ["blog.css"]
---
Running an agent locally has meant running two things: an inference server, and a separate orchestrator that talks to it. That is a lot of moving parts for something you wanted to try on a Tuesday evening.
4.0.0 puts the agent side in the core. You create agents, give them memory and skills, connect them to MCP servers, and start and stop them from the same interface you already use for models.
This is a major version bump, so there are two removals near the bottom of this post. Read those before you upgrade.
## Agents, and the Agenthub
Agents are managed through the React interface: create one, wire up MCP servers and skills, connect it to Slack, watch what it is doing through a new Events column in the agents list.
Memory has two options. Hybrid search backed by PostgreSQL if you already run one, or in-memory storage via Chromem if you do not want another service. Skills live in a central database rather than being pasted per agent.
The bit I am most curious to see used is [Agenthub](https://agenthub.localai.io), a community space for sharing agent configurations. You publish one, somebody else imports it into their instance and runs it against their own models on their own hardware!
## The interface is React now
The web interface has been rewritten. The old one had reached the point where adding anything meant fighting it.
Canvas mode is the new thing worth turning on: enable it in chat and code blocks and artifacts the model produces render in a preview pane on the right instead of scrolling past you as text. The System view splits Models and Backends into tabs. Traces render as accordions, which makes a long one readable. And if you try to install a model whose weights exceed your system RAM, you get a warning first rather than a locked-up machine.
## MCP Apps
Client-side MCP support is complete in this release ([#8947](https://github.com/mudler/LocalAI/pull/8947)). You pick which MCP servers to enable for a chat directly in the interface, and their tools get injected into the normal chat with streaming, so there is no separate agent mode to switch into.
If you would rather not have any of it, `LOCALAI_DISABLE_MCP` turns the whole thing off.
## Audio, video, and MLX across machines
WebRTC is wired into the Realtime API and the Talk page ([#8790](https://github.com/mudler/LocalAI/pull/8790)), which is a real improvement for latency over what was there before.
Three new audio backends: fish-speech, ace-step.cpp, and faster-qwen3-tts (CUDA only). TTS gained `sample_rate` support through post-processing, and Qwen TTS handles multiple voices.
There is also an experimental MLX distributed backend for spreading a workload across Apple machines ([#8801](https://github.com/mudler/LocalAI/pull/8801)). It is early, so expect rough edges if you try it.
## Infrastructure
Persistent data now has its own location, separate from configuration. `LOCALAI_DATA_PATH` (or `--data-path`) points at where agents, skills, tasks, jobs and the collection database live, defaulting to `data/` under the base path. If you are mounting volumes, this is the one to look at.
Shell completion scripts generate for bash, zsh and fish. There is dedicated Podman documentation now, including rootless setup.
## Two things are gone
The HuggingFace backend has been removed.
AIO images are dropped. They existed to bundle a preset of models with the runtime, and maintaining them across every hardware variant stopped being worth what they gave people. Use the main images and install models from the gallery.
## One known issue
The `diffusers` backend is not in this release. It failed to build because we exhausted our CI limits, so the previous version is still what you get if you install it.
This is an infrastructure problem, not a code one, and it is the kind of thing that will keep happening to us. If you know anybody at GitHub who could help us get better ARM runners, please reach out, I am not too proud to ask.
## Thanks
Thanks to @richiejp, @nanoandrew4, @Weathercold, @sozercan, @lukasdotcom, @loryanstrant, @bittoby and @attilagyorffy.
If you build an agent worth sharing, put it on the Agenthub. The more the merrier!
[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.0.0).

76
website/content/blog/what-landed-in-localai-4-1.md Normal file
View File

@@ -0,0 +1,76 @@
---
title: "LocalAI 4.1: more than one box, and more than one user"
date: 2026-04-02
author: "Ettore Di Giacinto"
category: "Release"
tags: ["release", "distributed", "auth", "oidc", "quotas", "fine-tuning"]
summary: "Distributed cluster mode that places requests by real free VRAM, OIDC with per-user API keys and quotas, and LoRA fine-tuning that exports straight to GGUF."
extracss: ["blog.css"]
---
Two problems show up the moment LocalAI stops being a thing you run for yourself.
The first is that you have more than one machine, and only one of them is doing any work. The second is that other people are using your instance, and you have no way to tell who is burning the GPU, or to stop them.
4.1.0 is mostly about those two.
## Running as a cluster
Distributed mode lets you point several nodes at one control plane and stop thinking about which one to call.
Routing orders nodes by available VRAM, so the request lands on the card with room for it. Node groups let you pin models to a subset of the cluster, which is how you keep a heavy diffusion model off the boxes doing embeddings. There is a min/max autoscaler with a reconciler managing node lifecycle, and you can drain a node for maintenance and resume it later through the API instead of pulling it out from under in-flight requests.
Model transfer between nodes goes over S3 or peer to peer, so a model you have already pulled once does not have to come down from the internet again on every node!
The cluster status shows up on the home page.
## Users, keys and quotas
LocalAI ships a multi-user platform now, which is the piece that makes it deployable for a team or a classroom rather than just for you.
- User management from the React interface.
- OIDC/OAuth against your own identity provider (Google, Keycloak, Authentik, whatever you already run).
- Invite mode, so registration is closed unless an admin lets somebody in.
- Per-user API keys.
- Admin impersonation, for when somebody reports a bug you cannot reproduce.
On top of that there is a quota system: set per-user limits and have them enforced, with a usage dashboard broken down per user and a predictive view of where consumption is heading.
## Fine-tuning without leaving the interface
Both of these are experimental. I would use them on something you can afford to throw away.
Fine-tuning uses HuggingFace TRL to train LoRA adapters, exports the result to GGUF automatically, and imports it back into LocalAI so you can serve what you just trained without moving files around by hand. There is a small evals framework included to check whether the thing you trained is actually better.
The quantization backend produces optimized variants of a model on the fly.
## Agents from the terminal
You can run an agent without the server now:
```sh
local-ai agent run <name>
local-ai agent list
```
`run` takes an agent from the pool registry in `pool.json`, or a single-turn `--prompt` if you just want one answer. Tool calls stream in real time, and the interleaved-thinking bug that mangled output when a model reasoned mid-tool-call is fixed.
## The rest of the interface work
The model pipeline editor is visual, so wiring models together no longer means editing YAML. Backend logs can be scoped to a single model rather than reading the whole stream. Studio pages remember past generations, so images and audio you made last week are still there. The model and backend selectors are searchable. Error toasts link straight to the trace that produced them.
## Under the hood
Inference defaults are pulled from Unsloth and applied across all endpoints and gallery models, so models arrive with sane sampling parameters instead of whatever the default happened to be. `min_p` is supported. When native tool-call parsing fails, an iterative fallback parser takes over rather than returning nothing.
Repeated log lines get collapsed. NVIDIA Jetson and Tegra are detected as first-class platforms. SYCL backends auto-disable `mmap`, which was crashing them on Intel GPUs. llama.cpp bundles `libdl`, `librt` and `libpthread` for portability. And the downloader rewrites HuggingFace URIs through `HF_ENDPOINT`, which is the one you need if you are behind a corporate mirror.
## Thanks
Thanks to @richiejp for a large chunk of this cycle, and to @tv42, @walcz-de, @majiayu000 and @ER-EPR.
There is a full setup walkthrough on video if you would rather watch than read: [youtube.com/watch?v=cMVNnlqwfw4](https://www.youtube.com/watch?v=cMVNnlqwfw4).
If you are setting up distributed mode or OIDC and hit a wall, reach out, I am happy to help you get it standing up.
[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.1.0).

117
website/content/blog/what-landed-in-localai-4-2.md Normal file
View File

@@ -0,0 +1,117 @@
---
title: "LocalAI 4.2: who spoke when, and whose face is that"
date: 2026-05-11
author: "Ettore Di Giacinto"
category: "Release"
tags: ["release", "diarization", "voice-recognition", "face-recognition", "ollama", "backends"]
summary: "A /v1/audio/diarization endpoint, voice and face recognition with liveness, a drop-in Ollama API, and eleven new backends."
extracss: ["blog.css"]
---
You record an hour of standup, run it through Whisper, and get back one long wall of text. Every word is correct. You still have no idea who said any of them, so you end up scrubbing through the audio with the transcript open in another window, guessing at voices.
4.2.0 is mostly about that class of problem. Audio and images carry more than "here are the words" or "here is a picture", and until now LocalAI had nowhere to put the rest of it.
## Who spoke when
There is a new `/v1/audio/diarization` endpoint, shaped like `/v1/audio/transcriptions` so your existing multipart code mostly carries over:
```bash
curl http://localhost:8080/v1/audio/diarization \
-H "Content-Type: multipart/form-data" \
-F file="@meeting.wav" \
-F model="vibevoice-cpp-asr" \
-F num_speakers=3
```
```json
{
"task": "diarize",
"duration": 12.34,
"num_speakers": 2,
"segments": [
{"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.00, "end": 2.34},
{"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 2.34, "end": 4.10}
]
}
```
Two backends serve it. [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) does pure diarization (pyannote-3.0 segmentation, a speaker-embedding extractor, then clustering) and never transcribes, so you do not pay for ASR you did not ask for. `vibevoice-cpp` emits speaker-labelled segments as a by-product of its long-form ASR pass, so with `include_text=true` you get a transcript per segment for free! `response_format` gives you `json`, `verbose_json`, or `rttm` if you want to feed the output to `dscore`.
One thing to know before you build on it: `SPEAKER_00` is local to a single request. Run the same meeting twice and the numbering can come out differently, and nothing promises that `SPEAKER_00` in Monday's recording is the same human as `SPEAKER_00` in Tuesday's. If you need identity across files, pair it with `/v1/voice/embed` and keep your own embedding store. Which brings me to..
## Voices and faces
`/v1/voice/*` is new ([#9500](https://github.com/mudler/LocalAI/pull/9500)): verify (are these two clips the same person?), identify (which of my enrolled speakers is this?), embed (give me the vector, I will do the rest myself), and analyze (age, gender, emotion).
```bash
local-ai models install speechbrain-ecapa-tdnn
curl -sX POST http://localhost:8080/v1/voice/verify \
-H "Content-Type: application/json" \
-d '{
"model": "speechbrain-ecapa-tdnn",
"audio1": "https://example.com/alice_1.wav",
"audio2": "https://example.com/alice_2.wav"
}'
```
```json
{"verified": true, "distance": 0.18, "threshold": 0.25}
```
The default threshold is around 0.25 for ECAPA-TDNN, and it moves per engine, so pass `threshold` explicitly if you swap the model out.
`/v1/face/*` does the same thing for faces ([#9480](https://github.com/mudler/LocalAI/pull/9480)), plus detection and demographics, and 4.2.0 adds antispoofing. Holding a printed photo or a phone screen up to the camera is the oldest attack on face auth there is, and the liveness check rejects it.
Some honest limits. Liveness is an arms race and this is not bank-grade. The demographic heads emit confident-looking numbers for age and emotion that you should read as a rough signal and not as a fact about a person. And the default `insightface` buffalo packs are released for non-commercial research use only, so if you are shipping this in a product, pick the OpenCV Zoo entry instead. That is in the docs, but people skip docs, so it is here too.
The samples never leave your machine, which is the part I actually care about. They go from your process to the backend running next to it and nowhere else. Doing biometrics against somebody else's cloud API always felt like the worst possible trade.
## Point your ollama client at LocalAI
```sh
OLLAMA_HOST=http://localhost:8080 ollama run qwen3
```
LocalAI answers the Ollama API now ([#9284](https://github.com/mudler/LocalAI/pull/9284)), so a tool that only ever learned to talk to Ollama keeps working with no code change on your side. `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps` and `/api/version` all land on the engine you were already running, and your existing `/v1/*` clients are untouched.
There is no `/api/pull` in there. Models come from the LocalAI gallery or from a URL you hand it, so `ollama run` against something you have not installed yet will not go and fetch it for you.
## Video, and an interface repaint
`stable-diffusion.ggml` generates video now ([#9420](https://github.com/mudler/LocalAI/pull/9420))! There are gallery entries for Wan 2.1 FLF2V 14B 720P and Wan i2v 720p, including first-last-frame interpolation.
The React interface got a long cycle of work. The chat is redesigned, the palette moved to Nord, and there is i18n across English, Italiano, Español, Deutsch and 简体中文. You can brand your instance too - name, tagline, logo, favicon - and the login page, sidebar, footer and browser tab all pick it up. Handy if you run LocalAI for a team and would rather it did not look like somebody's side project.
The model config editor is interactive now, with autocomplete over known fields and live validation, and it renames the file on save so you stop accumulating three copies of the same config.
## Eleven new backends
sglang, ik-llama.cpp, TurboQuant, sam.cpp, Kokoros, qwen3tts.cpp, tinygrad-multimodal (experimental, do not build anything load-bearing on it yet), vibevoice.cpp, LocalVQE, insightface, and voice-rec.
vLLM reached feature parity with llama.cpp in this cycle. The full `AsyncEngineArgs` surface is exposed as a generic YAML map, and tensor-parallel distributed workers let a single model span nodes. There are CUDA 13 builds for vLLM, vLLM-omni and sglang, plus L4T arm64 for Jetson-class boards.
## The unglamorous half
Most of the 279 pull requests here are not features. A sample of what actually went in:
- llama.cpp renamed its `common` target to `llama-common`, which broke the TurboQuant build until the detection was fixed.
- ik-llama.cpp needed a patch to `clip.cpp` for the new `ggml_quantize_chunk` signature, plus adapting to the `common_grammar` struct in `sampling.h`.
- `mlx-vlm` is pinned to v0.4.4 to unblock CUDA builds.
- vLLM dropped the flash-attn wheel to avoid a torch 2.10 ABI mismatch.
- Whisper transcriptions can be cancelled by the client, through the ggml `abort_callback`, so aborting a request frees the GPU instead of letting it run to completion in the background.
- faster-whisper emits word-level timestamps.
- gfx1151 (Strix Halo / Ryzen AI MAX) works, with `AMDGPU_TARGETS` exposed as a build-arg.
On the security side: an unsafe `sprintf()` came out of the C++ grpc-server, env-supplied API keys are stripped from Settings API requests before they get persisted so they cannot leak back out through the config, and deleting a user on PostgreSQL cascades across everything they owned instead of leaving orphaned rows behind.
Distributed mode got a hardening pass. Round-robin across replicas of the same model, "Upgrade All" scoped to the nodes that actually have the backend installed, NATS `backend.upgrade` split off from install, and correct VRAM/RAM reporting on NVIDIA unified-memory hosts.
## Thanks
This one had a lot of hands on it. Thanks to @richiejp for the model config editor, Kokoros and a pile of build fixes, @Anai-Guo, @russell, @leinasi2014, @keithmattix for gfx1151, @orbisai0security and @SAY-5 for the security work, @walcz-de, @thelittlefireman, @sec171, @pjbrzozowski, @mvanhorn, @arteven, @Dennisadira, @eglia, @arbrick, @neurocis and @ER-EPR.
If you are wiring up diarization or the voice endpoints and get stuck, open an issue or reach out, I am genuinely happy to help you get it working.
[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.2.0).

105
website/content/blog/what-landed-in-localai-4-3.md Normal file
View File

@@ -0,0 +1,105 @@
---
title: "LocalAI 4.3: signed backends, and the prompt cache that was off"
date: 2026-05-24
author: "Ettore Di Giacinto"
category: "Release"
tags: ["release", "security", "cosign", "prompt-cache", "distributed", "usage"]
summary: "Keyless cosign verification for backend OCI images, the llama.cpp prompt cache enabled by default, per-API-key usage attribution, and the replica-pinning bug that kept a second node idle."
extracss: ["blog.css"]
---
Here is a gap that had been sitting in LocalAI for a while. The gallery YAML tells LocalAI which OCI image to pull for a backend, and then LocalAI pulls it. Nothing checked that the bytes coming back were the bytes we built. A compromised registry, or somebody in the middle, and you would never know.
4.3.0 closes that, and fixes a default that had been quietly costing everybody a lot of prefill time.
## Signed backends
Every backend image merged by CI is now signed with [sigstore](https://www.sigstore.dev/)/cosign, keyless via Fulcio and Rekor, including each per-arch entry under the manifest list ([#9823](https://github.com/mudler/LocalAI/pull/9823)). It uses OCI 1.1 referrers rather than the legacy `:tag.sig` convention.
On your side, verification runs against a policy that the gallery declares:
```yaml
verification:
issuer_regex: "^https://token\\.actions\\.githubusercontent\\.com$"
identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@.*$"
not_before: "2026-05-22T00:00:00Z"
```
A few details that took some thinking.
`not_before` is the revocation lever. Keyless Fulcio certificates are ephemeral, so there is nothing to revoke on the signing side. Revocation has to be policy side: move the date forward in the gallery YAML and every signature older than it stops validating.
The TUF trusted root is cached process-wide, so installing ten backends from one gallery does one fetch instead of ten.
Digest pinning closes the window between verifying and pulling, which is otherwise a TOCTOU you could drive a truck through.
Strict mode is `--require-backend-integrity`, or `LOCALAI_REQUIRE_BACKEND_INTEGRITY=true`. It turns a missing policy or an empty SHA256 from a warning into a hard failure.
Now the honest part: strict mode is opt-in and off by default, and until a gallery ships a `verification:` block, installs go through with a warning. The default `backend/index.yaml` does not have the blocks populated yet, that is the next step. So today this is machinery that works and is not yet enforcing much. Turn on strict mode in production once your gallery is populated, not before, or you will just break your own installs.
## The prompt cache was off
`llama-cpp` has a server-side prompt cache. LocalAI was not enabling it. So every agent turn, every coding-assistant call, every OpenAI-compatible CLI with a long system prompt, re-prefilled that whole prompt from scratch.
On the reported workload, a repeated system prompt took 5 to 8 minutes per call before this change and seconds after it. Your numbers will depend on how long your prompt is and what hardware you are on.
Two defaults flipped ([#9925](https://github.com/mudler/LocalAI/pull/9925), [#9951](https://github.com/mudler/LocalAI/pull/9951)):
1. `kv_unified` is now `true` in `grpc-server.cpp`. The old `false` was silently force-disabling `cache_idle_slots` at server init, so the host prompt cache got allocated and then never written across requests. That is the one that actually explains the behaviour.
2. `prompt_cache_all` defaults to `true` at the YAML layer, matching upstream llama.cpp's own default in `common.h`. The per-request `cache_prompt` knob is on out of the box.
You can opt out with `options: ["kv_unified:false"]` or `prompt_cache_all: false`, and there are new keys (`cache_idle_slots`, `checkpoint_every_nt`) if you want to tune it. The model configuration docs got a worked example for the repeated-system-prompt case and an explanation of how `kv_unified`, `cache_ram` and `cache_idle_slots` interact, because they interact in ways that are not obvious.
## Who is burning the GPU
The usage page could tell you how many tokens were spent. It could not tell you who spent them ([#9920](https://github.com/mudler/LocalAI/pull/9920)).
`usage_records` gained a `Source` column (`apikey`, `web`, `legacy`) plus the API key id and name, with an idempotent backfill of older rows on `InitDB`. The auth middleware passes the resolved key and the request source through, and usage middleware snapshots the key id and name at write time, so a key you revoke later still reads correctly in history (it renders as `(revoked)` rather than vanishing).
Two new endpoints:
```
GET /api/auth/usage/sources # your own
GET /api/auth/admin/usage/sources # everyone, with user_id / api_key_id filters
```
The admin view truncates at 200 keys. The React usage page gained a Sources tab with a source-mix ribbon, a top-7-plus-Other time chart, and a sortable table. Web interface session traffic is split per user instead of being lumped into one global row.
## Distributed v3, and one good bug
This one is worth writing down because the symptom and the cause were far apart.
An operator reported this:
```
dgx-spark1 loaded in_flight=6
nvidia-thor1 loaded in_flight=0
```
Two replicas of the same model, one taking everything, one idle forever. The round-robin was there and looked correct.
The cause: `ModelLoader.Load` cached a `*Model` whose embedded `InFlightTrackingClient` was bound to a single `(nodeID, replicaIndex)`. The first request picked a node and got wrapped. Every request after that reused the wrapper, so it kept going to whichever node won the first pick, even after the reconciler scaled the model out. The routing code was fine. It just was not being consulted again!
`SmartRouter.Route` now runs per request ([#9968](https://github.com/mudler/LocalAI/pull/9968)), the `in_flight ASC, last_used ASC, available_vram DESC` ordering actually fires, and replica selection lives in one place (`PickBestReplica`) with a spec asserting the SQL `ORDER BY` and the Go picker agree on a seeded dataset. `probeHealth` is memoized per `(nodeID, addr)` with a 30 second TTL and `singleflight` coalescing, because llama.cpp serializes `HealthCheck` against in-flight `Predict` and a burst of new requests would otherwise stall on it.
Two other distributed changes.
`POST /api/nodes/:id/backends/install` used to block for up to 3 minutes while the worker pulled the image, which froze the Backends picker in the interface. It returns HTTP 202 and a `jobID` immediately now ([#9928](https://github.com/mudler/LocalAI/pull/9928)). Install and upgrade timeouts are configurable via `LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT` and `LOCALAI_NATS_BACKEND_UPGRADE_TIMEOUT`, defaulting to 15 minutes instead of the hardcoded 3. A NATS round-trip timeout while the worker is still pulling reports as `running_on_worker` rather than a hard failure.
Workers also publish debounced install progress (~250ms) that the master forwards into the operations status ([#9958](https://github.com/mudler/LocalAI/pull/9958)), so distributed installs show per-byte progress the same way local ones do. Old workers stay silent and new masters tolerate the silence, so mixed-version clusters keep working.
## Smaller things
`LOCALAI_TRACING_MAX_BODY_BYTES` caps trace payload size, which stops the admin Traces page from trying to render a 40 MB embedding response.
There is a `flake.nix` with a dev shell for NixOS users who do not want to go through Docker.
The `vllm`, `sglang` and `vllm-omni` L4T13 backends are back for Jetson and DGX boxes, switched to PyPI aarch64+cu130 wheels to fix the torch 2.10 ABI mismatch.
A distributed test harness landed in `tests/distributed/`, aimed at catching the class of regression the replica-pinning bug belonged to.
## Thanks
If you run LocalAI in production, the two things to look at here are strict mode (once your gallery has a `verification:` block) and whether the prompt cache change speeds up your workload. I would like to hear numbers from real setups, mine are one data point.
[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.3.0).