diff --git a/docs/content/blog/2024-04-24-reranker-api.md b/docs/content/blog/2024-04-24-reranker-api.md new file mode 100644 index 000000000..b35ac828a --- /dev/null +++ b/docs/content/blog/2024-04-24-reranker-api.md @@ -0,0 +1,10 @@ ++++ +title = "Reranker API" +date = 2024-04-24 +description = "A new reranker backend implementing the Jina rerankers API." +url = "/blog/reranker-api/" ++++ + +A new reranker backend lands, implementing the Jina rerankers API, in [PR #2121](https://github.com/mudler/LocalAI/pull/2121). + +See [Reranker]({{% relref "features/reranker" %}}). diff --git a/docs/content/blog/2024-05-14-distributed-and-p2p-inferencing.md b/docs/content/blog/2024-05-14-distributed-and-p2p-inferencing.md new file mode 100644 index 000000000..a96b2ae3e --- /dev/null +++ b/docs/content/blog/2024-05-14-distributed-and-p2p-inferencing.md @@ -0,0 +1,13 @@ ++++ +title = "Distributed and decentralized P2P inferencing" +date = 2024-05-14 +description = "Distributed llama.cpp inferencing, followed by fully decentralized peer-to-peer inference." +url = "/blog/distributed-and-p2p-inferencing/" ++++ + +Two changes that set up everything LocalAI later built on top of: + +- [Distributed llama.cpp inferencing](https://github.com/mudler/LocalAI/pull/2324), splitting a model across machines. +- [Totally decentralized, private, distributed peer-to-peer inference](https://github.com/mudler/LocalAI/pull/2343). + +See [Distributed inferencing]({{% relref "features/distributed_inferencing" %}}). diff --git a/docs/content/blog/2024-08-02-p2p-federation-and-swarms.md b/docs/content/blog/2024-08-02-p2p-federation-and-swarms.md new file mode 100644 index 000000000..2eb2cdf57 --- /dev/null +++ b/docs/content/blog/2024-08-02-p2p-federation-and-swarms.md @@ -0,0 +1,15 @@ ++++ +title = "P2P dashboard, federated mode and AI swarms" +date = 2024-08-02 +description = "A P2P dashboard, federation, AI swarms, global community pools, FLUX-1 support and the P2P Explorer." +url = "/blog/p2p-federation-and-swarms/" ++++ + +The peer-to-peer work matured over July and August: + +- [A P2P dashboard, federated mode and AI swarms](https://github.com/mudler/LocalAI/pull/2723). +- [Global community pools](https://github.com/mudler/LocalAI/issues/3113), for sharing federated instances and workers. +- FLUX-1 support. +- The [P2P Explorer](https://explorer.localai.io). + +See [Distributed inferencing]({{% relref "features/distributed_inferencing" %}}). diff --git a/docs/content/blog/2024-10-01-examples-moved-out.md b/docs/content/blog/2024-10-01-examples-moved-out.md new file mode 100644 index 000000000..4274feb8d --- /dev/null +++ b/docs/content/blog/2024-10-01-examples-moved-out.md @@ -0,0 +1,8 @@ ++++ +title = "Examples move to LocalAI-examples" +date = 2024-10-01 +description = "The examples directory leaves the main repository and gets its own home." +url = "/blog/examples-moved-out/" ++++ + +The examples have moved out of the main repository into [LocalAI-examples](https://github.com/mudler/LocalAI-examples), where they can be versioned and maintained independently of the runtime. diff --git a/docs/content/blog/2024-11-20-vad-and-bark-cpp.md b/docs/content/blog/2024-11-20-vad-and-bark-cpp.md new file mode 100644 index 000000000..5f1b69c7f --- /dev/null +++ b/docs/content/blog/2024-11-20-vad-and-bark-cpp.md @@ -0,0 +1,9 @@ ++++ +title = "Voice Activity Detection and bark.cpp" +date = 2024-11-20 +description = "Silero-based Voice Activity Detection, plus a bark.cpp backend for audio generation." +url = "/blog/vad-and-bark-cpp/" ++++ + +- [Voice Activity Detection](https://github.com/mudler/LocalAI/pull/4204), via a Silero VAD backend. See [Voice activity detection]({{% relref "features/voice-activity-detection" %}}). +- [A bark.cpp backend](https://github.com/mudler/LocalAI/pull/4287) for audio generation. diff --git a/docs/content/blog/2024-12-03-stablediffusion-cpp-backend.md b/docs/content/blog/2024-12-03-stablediffusion-cpp-backend.md new file mode 100644 index 000000000..1d7db6633 --- /dev/null +++ b/docs/content/blog/2024-12-03-stablediffusion-cpp-backend.md @@ -0,0 +1,10 @@ ++++ +title = "stablediffusion.cpp backend (ggml)" +date = 2024-12-03 +description = "A ggml-based stablediffusion.cpp backend for image generation." +url = "/blog/stablediffusion-cpp-backend/" ++++ + +A ggml-based `stablediffusion.cpp` backend lands for image generation, in [PR #4289](https://github.com/mudler/LocalAI/pull/4289). + +See [Image generation]({{% relref "features/image-generation" %}}). diff --git a/docs/content/blog/2025-07-24-modular-backend-architecture.md b/docs/content/blog/2025-07-24-modular-backend-architecture.md new file mode 100644 index 000000000..39f6b4c98 --- /dev/null +++ b/docs/content/blog/2025-07-24-modular-backend-architecture.md @@ -0,0 +1,12 @@ ++++ +title = "Backends move outside the main binary" +date = 2025-07-24 +description = "All backends migrate out of the main binary, leaving a lightweight modular core that pulls engines on demand." +url = "/blog/modular-backend-architecture/" ++++ + +All backends have been migrated outside the main binary. The core stays small, and each backend is an isolated service installed on demand. + +This is the architecture LocalAI still runs on: install, update or remove engines independently, and mix CPU, NVIDIA, AMD, Intel, Apple Silicon, Vulkan and Jetson in one deployment. + +See [Backends]({{% relref "features/backends" %}}) and the [v3.2.0 release notes](https://github.com/mudler/LocalAI/releases/tag/v3.2.0). diff --git a/docs/content/blog/2025-08-12-apple-silicon-backends.md b/docs/content/blog/2025-08-12-apple-silicon-backends.md new file mode 100644 index 000000000..ca5ea390f --- /dev/null +++ b/docs/content/blog/2025-08-12-apple-silicon-backends.md @@ -0,0 +1,10 @@ ++++ +title = "MLX, MLX-VLM, Diffusers and llama.cpp on Apple Silicon" +date = 2025-08-12 +description = "Apple Silicon gains first-class backend coverage." +url = "/blog/apple-silicon-backends/" ++++ + +MLX, MLX-VLM, Diffusers and llama.cpp are now supported on Apple Silicon, giving Mac users the same backend choice available elsewhere. + +Released as part of [v3.4.0](https://github.com/mudler/LocalAI/releases/tag/v3.4.0). diff --git a/docs/content/blog/2025-09-03-launcher-and-extended-backends.md b/docs/content/blog/2025-09-03-launcher-and-extended-backends.md new file mode 100644 index 000000000..8d9a95468 --- /dev/null +++ b/docs/content/blog/2025-09-03-launcher-and-extended-backends.md @@ -0,0 +1,13 @@ ++++ +title = "New launcher, extended backend support, MLX-Audio and WAN 2.2" +date = 2025-09-03 +description = "A desktop launcher for macOS and Linux, wider backend coverage for Mac and Nvidia L4T, MLX-Audio and WAN 2.2." +url = "/blog/launcher-and-extended-backends/" ++++ + +- A new [launcher app](https://github.com/mudler/LocalAI/pull/6127) for macOS and Linux, so LocalAI can be started and managed without the terminal. +- Extended backend support for Mac and Nvidia L4T. +- MLX-Audio. +- WAN 2.2. + +Released as part of [v3.5.0](https://github.com/mudler/LocalAI/releases/tag/v3.5.0). diff --git a/docs/content/blog/2025-10-05-mcp-support.md b/docs/content/blog/2025-10-05-mcp-support.md new file mode 100644 index 000000000..3f8f4aaed --- /dev/null +++ b/docs/content/blog/2025-10-05-mcp-support.md @@ -0,0 +1,10 @@ ++++ +title = "Model Context Protocol (MCP) support" +date = 2025-10-05 +description = "Agentic capabilities through MCP, with a new chat/completion endpoint that can call MCP tools." +url = "/blog/mcp-support/" ++++ + +LocalAI gains [Model Context Protocol](https://modelcontextprotocol.io) support for agentic capabilities, through [a new chat/completion endpoint](https://github.com/mudler/LocalAI/pull/6381) that can reach MCP tools, plus [a UI toggle to enable it](https://github.com/mudler/LocalAI/pull/6400). + +See [MCP]({{% relref "features/mcp" %}}). diff --git a/docs/content/blog/2025-11-24-import-models-via-url-and-chat-history.md b/docs/content/blog/2025-11-24-import-models-via-url-and-chat-history.md new file mode 100644 index 000000000..637062741 --- /dev/null +++ b/docs/content/blog/2025-11-24-import-models-via-url-and-chat-history.md @@ -0,0 +1,11 @@ ++++ +title = "Import models via URL, multiple chats and history" +date = 2025-11-24 +description = "Point LocalAI at a model URL to import it, and keep several chat threads with their history in the UI." +url = "/blog/import-models-via-url-and-chat-history/" ++++ + +Two usability changes: + +- [Import models via URL](https://github.com/mudler/LocalAI/pull/7245). Paste a model URL and LocalAI handles the download and configuration. +- [Multiple chats and history](https://github.com/mudler/LocalAI/pull/7325) in the UI, so conversations persist and can run in parallel. diff --git a/docs/content/blog/2025-12-16-memory-reclaimer-and-multi-gpu-fitting.md b/docs/content/blog/2025-12-16-memory-reclaimer-and-multi-gpu-fitting.md new file mode 100644 index 000000000..780c4073e --- /dev/null +++ b/docs/content/blog/2025-12-16-memory-reclaimer-and-multi-gpu-fitting.md @@ -0,0 +1,12 @@ ++++ +title = "Dynamic memory reclaimer, multi-GPU fitting and Vibevoice" +date = 2025-12-16 +description = "Reclaim GPU memory from idle models, fit llama.cpp models across multiple GPUs automatically, and generate long-form speech with Vibevoice." +url = "/blog/memory-reclaimer-and-multi-gpu-fitting/" ++++ + +Three additions this month: + +- [A dynamic memory resource reclaimer](https://github.com/mudler/LocalAI/pull/7583), which frees GPU memory held by idle models. +- [Automatic multi-GPU model fitting for llama.cpp](https://github.com/mudler/LocalAI/pull/7584), so a model too large for one device is split across several without hand-tuning. +- [The Vibevoice backend](https://github.com/mudler/LocalAI/pull/7494) for long-form speech. diff --git a/docs/content/blog/2026-01-18-localai-3-10-0.md b/docs/content/blog/2026-01-18-localai-3-10-0.md new file mode 100644 index 000000000..617a3ab16 --- /dev/null +++ b/docs/content/blog/2026-01-18-localai-3-10-0.md @@ -0,0 +1,17 @@ ++++ +title = "LocalAI 3.10.0" +date = 2026-01-18 +description = "Anthropic API support, the Open Responses API, video and image generation with LTX-2, unified GPU backends, tool streaming, Moonshine and Pocket-TTS." +url = "/blog/localai-3-10-0/" ++++ + +LocalAI 3.10.0 is out. + +- Anthropic API support. +- The Open Responses API. +- Video and image generation with LTX-2. +- Unified GPU backends. +- Tool streaming. +- Moonshine and Pocket-TTS. + +[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v3.10.0). diff --git a/docs/content/blog/2026-02-05-realtime-api-and-ace-step.md b/docs/content/blog/2026-02-05-realtime-api-and-ace-step.md new file mode 100644 index 000000000..0deaca1c6 --- /dev/null +++ b/docs/content/blog/2026-02-05-realtime-api-and-ace-step.md @@ -0,0 +1,11 @@ ++++ +title = "Realtime API and ACE-Step 1.5" +date = 2026-02-05 +description = "Audio-to-audio with tool calling through the Realtime API, plus ACE-Step 1.5 music generation." +url = "/blog/realtime-api-and-ace-step/" ++++ + +Two additions this month: + +- [The Realtime API for audio-to-audio with tool calling](https://github.com/mudler/LocalAI/pull/6245). See [Realtime API]({{% relref "features/openai-realtime" %}}). +- [ACE-Step 1.5 support](https://github.com/mudler/LocalAI/pull/8396) for music generation. diff --git a/docs/content/blog/2026-03-14-localai-4-0-0.md b/docs/content/blog/2026-03-14-localai-4-0-0.md new file mode 100644 index 000000000..c830e69ee --- /dev/null +++ b/docs/content/blog/2026-03-14-localai-4-0-0.md @@ -0,0 +1,16 @@ ++++ +title = "LocalAI 4.0.0: native agentic orchestration" +date = 2026-03-14 +description = "The Agenthub community hub, a full React UI rewrite with Canvas mode, MCP Apps with tool streaming, WebRTC realtime audio, and MLX-distributed." +url = "/blog/localai-4-0-0/" ++++ + +LocalAI 4.0.0 brings agentic orchestration into the core. + +- Native agentic orchestration, with the new [Agenthub](https://agenthub.localai.io) community hub. +- A full React UI rewrite, including Canvas mode. +- [MCP Apps and client-side MCP](https://github.com/mudler/LocalAI/pull/8947) with tool streaming. +- [WebRTC realtime audio](https://github.com/mudler/LocalAI/pull/8790). +- [MLX-distributed](https://github.com/mudler/LocalAI/pull/8801). + +[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.0.0). diff --git a/docs/content/blog/2026-04-02-localai-4-1-0.md b/docs/content/blog/2026-04-02-localai-4-1-0.md new file mode 100644 index 000000000..97745866f --- /dev/null +++ b/docs/content/blog/2026-04-02-localai-4-1-0.md @@ -0,0 +1,17 @@ ++++ +title = "LocalAI 4.1.0: LocalAI becomes a control tower" +date = 2026-04-02 +description = "Distributed cluster mode with VRAM-aware routing and autoscaling, a multi-user platform with OIDC, per-user quotas, in-UI fine-tuning, and a visual pipeline editor." +url = "/blog/localai-4-1-0/" ++++ + +LocalAI 4.1.0 turns LocalAI into a control tower rather than a single inference server. + +- Distributed cluster mode, with VRAM-aware smart routing and autoscaling. +- A multi-user platform with OIDC and API keys. +- Per-user quotas with predictive analytics. +- In-UI fine-tuning with TRL, including automatic export to GGUF. +- An on-the-fly quantization backend. +- A visual pipeline editor. + +[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.1.0). diff --git a/docs/content/blog/2026-04-22-face-recognition-backend.md b/docs/content/blog/2026-04-22-face-recognition-backend.md new file mode 100644 index 000000000..0074f0e9f --- /dev/null +++ b/docs/content/blog/2026-04-22-face-recognition-backend.md @@ -0,0 +1,20 @@ ++++ +title = "Face recognition backend" +date = 2026-04-22 +description = "insightface-powered 1:1 verification, 1:N identification, face embedding, detection and demographic analysis." +url = "/blog/face-recognition-backend/" ++++ + +A new face recognition backend, powered by `insightface`, covering: + +- 1:1 verification +- 1:N identification +- Face embedding +- Face detection +- Demographic analysis + +It ships with two model options: the non-commercial `buffalo_l`, and an Apache 2.0 alternative from the OpenCV Zoo. + +See [Face recognition]({{% relref "features/face-recognition" %}}). Shipped in [PR #9480](https://github.com/mudler/LocalAI/pull/9480). + +The engine was later rewritten from scratch in C++/ggml: see [Native biometric backends]({{% relref "blog/2026-06-28-native-biometric-backends" %}}). diff --git a/docs/content/blog/2026-05-04-audio-transform.md b/docs/content/blog/2026-05-04-audio-transform.md new file mode 100644 index 000000000..f417cafaf --- /dev/null +++ b/docs/content/blog/2026-05-04-audio-transform.md @@ -0,0 +1,19 @@ ++++ +title = "Audio Transform" +date = 2026-05-04 +description = "A generic audio-in / audio-out endpoint with an optional reference signal. First implementation: LocalVQE, a joint AEC, noise suppression and dereverberation engine." +url = "/blog/audio-transform/" ++++ + +Audio Transform is a generic audio-in / audio-out endpoint, with an optional reference signal for tasks that need one. + +The first implementation is [LocalVQE](https://github.com/localai-org/LocalVQE), a C++ backend doing joint acoustic echo cancellation, noise suppression and dereverberation in a DeepVQE-style model. + +Both call styles are supported: + +- Batch, via `POST /audio/transformations`. +- Bidirectional streaming, via the `/audio/transformations/stream` WebSocket. + +Studio gains a "Transform" tab with synchronized waveform players for the input, reference and output signals. + +See [Audio transform]({{% relref "features/audio-transform" %}}). Shipped in [PR #9640](https://github.com/mudler/LocalAI/pull/9640). diff --git a/docs/content/blog/2026-05-05-speaker-diarization.md b/docs/content/blog/2026-05-05-speaker-diarization.md new file mode 100644 index 000000000..14df31e39 --- /dev/null +++ b/docs/content/blog/2026-05-05-speaker-diarization.md @@ -0,0 +1,17 @@ ++++ +title = "Speaker diarization" +date = 2026-05-05 +description = "A /v1/audio/diarization endpoint returning who spoke when, backed by sherpa-onnx and vibevoice-cpp." +url = "/blog/speaker-diarization/" ++++ + +`POST /v1/audio/diarization` is a new endpoint that returns "who spoke when" as a list of segments. + +Two backends serve it: + +- `sherpa-onnx` for pure diarization, combining pyannote-3.0, speaker embeddings and clustering. +- `vibevoice-cpp` for diarization bundled with long-form ASR. + +Responses are available as `json`, `verbose_json` or `rttm`. + +See [Audio diarization]({{% relref "features/audio-diarization" %}}). Shipped in [PR #9654](https://github.com/mudler/LocalAI/pull/9654). diff --git a/docs/content/blog/2026-05-11-localai-4-2-0.md b/docs/content/blog/2026-05-11-localai-4-2-0.md new file mode 100644 index 000000000..d373cc33d --- /dev/null +++ b/docs/content/blog/2026-05-11-localai-4-2-0.md @@ -0,0 +1,120 @@ ++++ +title = "LocalAI 4.2.0: who spoke when, and whose face is that" +date = 2026-05-11 +description = "Speaker diarization, voice and face recognition, and an Ollama-compatible API." +url = "/blog/localai-4-2-0/" ++++ + +![Diarization: segment, embed, and cluster into speaker-labelled segments](/images/diagrams/diarization-pipeline.png) + +You record an hour of standup, run it through Whisper, and get back one long wall of text. Every word is correct. You still have no idea who said any of them, so you end up scrubbing through the audio with the transcript open in another window, guessing at voices. + +4.2.0 is mostly about that class of problem. Audio and images carry more than "here are the words" or "here is a picture", and until now LocalAI had nowhere to put the rest of it. + +Enough chitchat, let's look at what's in it. + +## Who spoke when + +There's a new `/v1/audio/diarization` endpoint, shaped like `/v1/audio/transcriptions` so your existing multipart code mostly carries over: + +```bash +curl http://localhost:8080/v1/audio/diarization \ + -H "Content-Type: multipart/form-data" \ + -F file="@meeting.wav" \ + -F model="vibevoice-cpp-asr" \ + -F num_speakers=3 +``` + +```json +{ + "task": "diarize", + "duration": 12.34, + "num_speakers": 2, + "segments": [ + {"id": 0, "speaker": "SPEAKER_00", "label": "0", "start": 0.00, "end": 2.34}, + {"id": 1, "speaker": "SPEAKER_01", "label": "1", "start": 2.34, "end": 4.10} + ] +} +``` + +Two backends serve it. [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) does pure diarization (pyannote-3.0 segmentation, a speaker-embedding extractor, then clustering) and never transcribes, so you don't pay for ASR you didn't ask for. `vibevoice-cpp` emits speaker-labelled segments as a by-product of its long-form ASR pass, so with `include_text=true` you get a transcript per segment for free! `response_format` gives you `json`, `verbose_json`, or `rttm` if you want to feed the output to `dscore`. + +One thing to know before you build on it: `SPEAKER_00` is local to a single request. Run the same meeting twice and the numbering can come out differently, and nothing promises that `SPEAKER_00` in Monday's recording is the same human as `SPEAKER_00` in Tuesday's. If you need identity across files, pair it with `/v1/voice/embed` and keep your own embedding store. Which brings me to.. + +## Voices and faces + +`/v1/voice/*` is new: verify (are these two clips the same person?), identify (which of my enrolled speakers is this?), embed (give me the vector, I'll do the rest myself), and analyze (age, gender, emotion). + +```bash +local-ai models install speechbrain-ecapa-tdnn + +curl -sX POST http://localhost:8080/v1/voice/verify \ + -H "Content-Type: application/json" \ + -d '{ + "model": "speechbrain-ecapa-tdnn", + "audio1": "https://example.com/alice_1.wav", + "audio2": "https://example.com/alice_2.wav" + }' +``` + +```json +{"verified": true, "distance": 0.18, "threshold": 0.25} +``` + +The default threshold is around 0.25 for ECAPA-TDNN, and it moves per engine, so pass `threshold` explicitly if you swap the model out. + +`/v1/face/*` does the same thing for faces, plus detection and demographics, and 4.2.0 adds antispoofing. Holding a printed photo or a phone screen up to the camera is the oldest attack on face auth there is, and the liveness check rejects it. + +Some honest limits. Liveness is an arms race and this is not bank-grade. The demographic heads emit confident-looking numbers for age and emotion that you should read as a rough signal and not as a fact about a person. And the default `insightface` buffalo packs are released for non-commercial research use only, so if you're shipping this in a product, pick the OpenCV Zoo entry instead. That's in the docs, but people skip docs, so it's here too. + +The samples never leave your machine, which is the part I actually care about. They go from your process to the backend running next to it and nowhere else. Doing biometrics against somebody else's cloud API always felt like the worst possible trade. + +## Point your ollama client at LocalAI + +```sh +OLLAMA_HOST=http://localhost:8080 ollama run qwen3 +``` + +LocalAI answers the Ollama API now, so a tool that only ever learned to talk to Ollama keeps working with no code change on your side. `/api/chat`, `/api/generate`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps` and `/api/version` all land on the engine you were already running, and your existing `/v1/*` clients are untouched. + +There's no `/api/pull` in there. Models come from the LocalAI gallery or from a URL you hand it, so `ollama run` against something you haven't installed yet won't go and fetch it for you. + +## Video, and a UI repaint + +`stable-diffusion.ggml` generates video now! There are gallery entries for Wan 2.1 FLF2V 14B 720P and Wan i2v 720p, including first-last-frame interpolation. + +The React UI got a long cycle of work. The chat is redesigned, the palette moved to Nord, and there's i18n across English, Italiano, Español, Deutsch and 简体中文. You can brand your instance too - name, tagline, logo, favicon - and the login page, sidebar, footer and browser tab all pick it up. Handy if you run LocalAI for a team and would rather it didn't look like somebody's side project. + +The model config editor is interactive now, with autocomplete over known fields and live validation, and it renames the file on save so you stop accumulating three copies of the same config. + +## Eleven new backends + +sglang, ik-llama.cpp, TurboQuant, sam.cpp, Kokoros, qwen3tts.cpp, tinygrad-multimodal (experimental, don't build anything load-bearing on it yet), vibevoice.cpp, LocalVQE, insightface, and voice-rec. + +vLLM reached feature parity with llama.cpp in this cycle. The full `AsyncEngineArgs` surface is exposed as a generic YAML map, and tensor-parallel distributed workers let a single model span nodes. There are CUDA 13 builds for vLLM, vLLM-omni and sglang, plus L4T arm64 for Jetson-class boards. + +## The unglamorous half + +Most of the 279 pull requests here are not features. A sample of what actually went in: + +- llama.cpp renamed its `common` target to `llama-common`, which broke the TurboQuant build until the detection was fixed. +- ik-llama.cpp needed a patch to `clip.cpp` for the new `ggml_quantize_chunk` signature, plus adapting to the `common_grammar` struct in `sampling.h`. +- `mlx-vlm` is pinned to v0.4.4 to unblock CUDA builds. +- vLLM dropped the flash-attn wheel to avoid a torch 2.10 ABI mismatch. +- Whisper transcriptions can be cancelled by the client, through the ggml `abort_callback`, so aborting a request frees the GPU instead of letting it run to completion in the background. +- faster-whisper emits word-level timestamps. +- gfx1151 (Strix Halo / Ryzen AI MAX) works, with `AMDGPU_TARGETS` exposed as a build-arg. + +On the security side: an unsafe `sprintf()` came out of the C++ grpc-server, env-supplied API keys are stripped from Settings API requests before they get persisted so they can't leak back out through the config, and deleting a user on PostgreSQL cascades across everything they owned instead of leaving orphaned rows behind. + +Distributed mode got a hardening pass. Round-robin across replicas of the same model, "Upgrade All" scoped to the nodes that actually have the backend installed, NATS `backend.upgrade` split off from install, and correct VRAM/RAM reporting on NVIDIA unified-memory hosts. + +## Thanks! + +This one had a lot of hands on it. Thanks to @richiejp for the model config editor, Kokoros and a pile of build fixes, @Anai-Guo, @russell, @leinasi2014, @keithmattix for gfx1151, @orbisai0security and @SAY-5 for the security work, @walcz-de, @thelittlefireman, @sec171, @pjbrzozowski, @mvanhorn, @arteven, @Dennisadira, @eglia, @arbrick, @neurocis and @ER-EPR. + +If you're wiring up diarization or the voice endpoints and get stuck, open an issue or reach out, I'm genuinely happy to help you get it working. And if LocalAI is useful to you, consider [donating](https://github.com/sponsors/mudler) or just telling somebody about it. The more the merrier! + +[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.2.0). See [Speaker diarization]({{% relref "features/audio-diarization" %}}), [Voice recognition]({{% relref "features/voice-recognition" %}}) and [Face recognition]({{% relref "features/face-recognition" %}}). + +Cheers! diff --git a/docs/content/blog/2026-05-24-localai-4-3-0.md b/docs/content/blog/2026-05-24-localai-4-3-0.md new file mode 100644 index 000000000..7d013653d --- /dev/null +++ b/docs/content/blog/2026-05-24-localai-4-3-0.md @@ -0,0 +1,15 @@ ++++ +title = "LocalAI 4.3.0" +date = 2026-05-24 +description = "llama.cpp prompt cache on by default, keyless cosign signing of backend images, per-key and per-user usage attribution, and Distributed v3." +url = "/blog/localai-4-3-0/" ++++ + +LocalAI 4.3.0 is out. + +- [Prompt cache on by default for llama.cpp](https://github.com/mudler/LocalAI/pull/9925). Repeated system prompts collapse from minutes to seconds. +- [Keyless cosign signing of backend OCI images](https://github.com/mudler/LocalAI/pull/9823). +- [Per-API-key and per-user usage attribution](https://github.com/mudler/LocalAI/pull/9920). +- Distributed v3, with [per-request replica routing](https://github.com/mudler/LocalAI/pull/9968). + +[Full release notes](https://github.com/mudler/LocalAI/releases/tag/v4.3.0). diff --git a/docs/content/blog/2026-06-11-realtime-voice-assistant-demo.md b/docs/content/blog/2026-06-11-realtime-voice-assistant-demo.md new file mode 100644 index 000000000..02e3e0905 --- /dev/null +++ b/docs/content/blog/2026-06-11-realtime-voice-assistant-demo.md @@ -0,0 +1,12 @@ ++++ +title = "Realtime voice assistant demo and pipeline streaming" +date = 2026-06-11 +description = "A tiny Go client for the Realtime API with a full talk-back loop and tool calling, plus streaming of the realtime pipeline stages." +url = "/blog/realtime-voice-assistant-demo/" ++++ + +The new [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) is a small Go client for the Realtime API with a complete talk-back voice loop and tool calling. It is intended as a reference you can read end to end. + +On the server side, two supporting changes landed: [streaming of the realtime LLM, TTS and transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176), and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231). + +See [Realtime API]({{% relref "features/openai-realtime" %}}). diff --git a/docs/content/blog/2026-06-12-distributed-mode-hardening.md b/docs/content/blog/2026-06-12-distributed-mode-hardening.md new file mode 100644 index 000000000..b9835b7c8 --- /dev/null +++ b/docs/content/blog/2026-06-12-distributed-mode-hardening.md @@ -0,0 +1,16 @@ ++++ +title = "Distributed mode hardening" +date = 2026-06-12 +description = "Prefix-cache-aware routing, a production-ready request router, ds4 layer-split inference, NATS JWT auth with TLS/mTLS, and resumable uploads." +url = "/blog/distributed-mode-hardening/" ++++ + +Distributed mode picked up a round of production hardening: + +- [Prefix-cache-aware routing](https://github.com/mudler/LocalAI/pull/10071), so requests sharing a prompt prefix land on the replica that already holds it. +- [A production-ready request router with auto-sized embedding and rerank batches](https://github.com/mudler/LocalAI/pull/10104). +- [ds4 layer-split distributed inference](https://github.com/mudler/LocalAI/pull/10098). +- [NATS JWT auth plus TLS/mTLS](https://github.com/mudler/LocalAI/pull/10159). +- [Resumable file uploads](https://github.com/mudler/LocalAI/pull/10109). + +See [Distributed inferencing]({{% relref "features/distributed_inferencing" %}}). diff --git a/docs/content/blog/2026-06-12-new-backends-and-models.md b/docs/content/blog/2026-06-12-new-backends-and-models.md new file mode 100644 index 000000000..054e6bc6c --- /dev/null +++ b/docs/content/blog/2026-06-12-new-backends-and-models.md @@ -0,0 +1,15 @@ ++++ +title = "New backends and models: locate-anything.cpp, Ideogram4, Gemma 4" +date = 2026-06-12 +description = "Open-vocabulary object detection via ggml, Ideogram4 image generation, llama.cpp video input, and the Gemma 4 QAT family with MTP pairs." +url = "/blog/new-backends-and-models-june-2026/" ++++ + +A batch of new capability this month: + +- [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml. +- [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in `stablediffusion-ggml`. +- [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216). +- [The Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). + +Plus two usability additions: an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228). diff --git a/docs/content/blog/2026-06-13-speech-push-parakeet-crispasr-piper.md b/docs/content/blog/2026-06-13-speech-push-parakeet-crispasr-piper.md new file mode 100644 index 000000000..e0ecb643f --- /dev/null +++ b/docs/content/blog/2026-06-13-speech-push-parakeet-crispasr-piper.md @@ -0,0 +1,17 @@ ++++ +title = "A big speech push: parakeet.cpp, CrispASR and 60 Piper voices" +date = 2026-06-13 +description = "Segment timestamps, multilingual streaming, dynamic batching and CUDA graphs for parakeet.cpp, plus a new ASR/TTS backend and a large Piper voice drop." +url = "/blog/speech-push-parakeet-crispasr-piper/" ++++ + +A concentrated round of speech work landed this month. + +[parakeet.cpp](https://github.com/mudler/parakeet.cpp), our ASR engine, gained: + +- [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207) +- [a multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199) +- [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) +- [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273) + +Alongside it, the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR and TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery, together with [per-request TTS instructions and parameters](https://github.com/mudler/LocalAI/pull/10172). diff --git a/docs/content/blog/2026-06-18-pii-analyze-redact-api.md b/docs/content/blog/2026-06-18-pii-analyze-redact-api.md new file mode 100644 index 000000000..7de2f749a --- /dev/null +++ b/docs/content/blog/2026-06-18-pii-analyze-redact-api.md @@ -0,0 +1,15 @@ ++++ +title = "PII analyze and redact API" +date = 2026-06-18 +description = "The PII detection pipeline becomes a standalone service, callable without routing a chat request through the middleware." +url = "/blog/pii-analyze-redact-api/" ++++ + +The PII detection pipeline (NER plus restricted-regex pattern tiers) is now reachable directly, without routing a chat request through the middleware: + +- `POST /api/pii/analyze` returns the detected entity spans. +- `POST /api/pii/redact` returns the sanitised text, or `400 pii_blocked`. + +Events also gain an `origin` field (`middleware`, `proxy`, `pii_analyze`, `pii_redact`), so `/api/pii/events` can be filtered by which surface produced them. + +See [Middleware]({{% relref "operations/middleware" %}}#analyze--redact-api). Shipped in [PR #10360](https://github.com/mudler/LocalAI/pull/10360). diff --git a/docs/content/blog/2026-06-22-sound-classification.md b/docs/content/blog/2026-06-22-sound-classification.md new file mode 100644 index 000000000..48dc7c375 --- /dev/null +++ b/docs/content/blog/2026-06-22-sound-classification.md @@ -0,0 +1,12 @@ ++++ +title = "Sound classification with ced.cpp" +date = 2026-06-22 +description = "A new /v1/audio/classification endpoint for audio tagging, returning scored AudioSet labels." +url = "/blog/sound-classification/" ++++ + +`POST /v1/audio/classification` is a new endpoint for audio tagging and sound-event classification. It returns scored [AudioSet](https://research.google.com/audioset/) labels: baby cry, glass breaking, alarms, and several hundred others. + +It is backed by [ced.cpp](https://github.com/localai-org/ced.cpp), a 527-class AudioSet tagger ported to ggml by the LocalAI team. + +See [Audio classification]({{% relref "features/audio-classification" %}}). Shipped in [PR #10425](https://github.com/mudler/LocalAI/pull/10425). diff --git a/docs/content/blog/2026-06-28-native-biometric-backends.md b/docs/content/blog/2026-06-28-native-biometric-backends.md new file mode 100644 index 000000000..fe55fbd6e --- /dev/null +++ b/docs/content/blog/2026-06-28-native-biometric-backends.md @@ -0,0 +1,15 @@ ++++ +title = "Native biometric backends: voice-detect.cpp and face-detect.cpp" +date = 2026-06-28 +description = "Two from-scratch C++/ggml engines replace the heavier Python insightface and speaker-recognition backends." +url = "/blog/native-biometric-backends/" ++++ + +Two new biometric engines built by the LocalAI team, both from-scratch C++/ggml implementations with no Python and no onnxruntime at inference time: + +- [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) for speaker recognition and voice analysis: ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, and wav2vec2 age/gender/emotion. +- [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing: SCRFD/ArcFace and YuNet/SFace. + +Both ship self-contained GGUF weights, hold bit-exact parity with the reference implementations, and reach cuDNN parity on GPU. They replace the heavier Python `insightface` and `speaker-recognition` backends. + +Shipped in [PR #10441](https://github.com/mudler/LocalAI/pull/10441). diff --git a/docs/content/blog/2026-06-30-concurrent-scoring-and-pii-ner.md b/docs/content/blog/2026-06-30-concurrent-scoring-and-pii-ner.md new file mode 100644 index 000000000..3145b8b88 --- /dev/null +++ b/docs/content/blog/2026-06-30-concurrent-scoring-and-pii-ner.md @@ -0,0 +1,15 @@ ++++ +title = "Concurrent scoring and PII NER on llama.cpp" +date = 2026-06-30 +description = "Score and TokenClassify now ride llama.cpp's server task queue instead of locking the context, so they run alongside chat traffic." +url = "/blog/concurrent-scoring-and-pii-ner/" ++++ + +The `Score` primitive (used by the router classifier) and `TokenClassify` (used by the PII NER tier) previously locked the llama.cpp context for the duration of the call. They now ride llama.cpp's server task queue instead. + +What changes as a result: + +- Scoring and token classification run concurrently with chat, completion and embedding traffic, and with each other. +- The `known_usecases` restriction that forced dedicated scorer and NER model configs on `llama-cpp` is lifted. +- Repeated scoring calls reuse the prompt KV cache across candidates. +- Scoring inputs are no longer capped by the physical batch size. diff --git a/docs/content/blog/2026-07-05-model-capabilities-endpoint.md b/docs/content/blog/2026-07-05-model-capabilities-endpoint.md new file mode 100644 index 000000000..a1564e7d5 --- /dev/null +++ b/docs/content/blog/2026-07-05-model-capabilities-endpoint.md @@ -0,0 +1,14 @@ ++++ +title = "Model capabilities endpoint" +date = 2026-07-05 +description = "GET /v1/models/capabilities reports what each model can do and which modalities it accepts, so clients stop guessing from backend names." +url = "/blog/model-capabilities-endpoint/" ++++ + +`GET /v1/models/capabilities` is a new endpoint: an additive superset of `/v1/models` that reports each model's `capabilities` alongside its `input_modalities` and `output_modalities` (`text`, `image`, `audio`, `video`). + +The practical effect is that a client can decide where to send an attachment by asking the server, instead of pattern-matching on backend names. Modalities are either inferred by LocalAI or declared explicitly in the model config. + +Because the endpoint is additive, existing `/v1/models` consumers are unaffected. + +See [API discovery]({{% relref "features/api-discovery" %}}#model-capabilities). Shipped in [PR #10687](https://github.com/mudler/LocalAI/pull/10687). diff --git a/docs/content/blog/2026-07-12-longcat-video-and-avatar-generation.md b/docs/content/blog/2026-07-12-longcat-video-and-avatar-generation.md new file mode 100644 index 000000000..cb70c0568 --- /dev/null +++ b/docs/content/blog/2026-07-12-longcat-video-and-avatar-generation.md @@ -0,0 +1,16 @@ ++++ +title = "LongCat video and avatar generation" +date = 2026-07-12 +description = "A dedicated CUDA backend for LongCat-Video text/image-to-video and LongCat-Video-Avatar-1.5 speech-driven avatars." +url = "/blog/longcat-video-and-avatar-generation/" ++++ + +LocalAI gains a dedicated CUDA backend for the LongCat family: `LongCat-Video` for text-to-video and image-to-video, and `LongCat-Video-Avatar-1.5` for speech-driven avatars. + +Highlights: + +- Multi-segment continuation, so a clip can be extended beyond a single generation window. +- Portrait and recorded-audio inputs wired into Studio. +- An SDPA CUDA 13 ARM64 build, which makes the backend usable on DGX Spark. + +See [Video generation]({{% relref "features/video-generation" %}}) for configuration and the available model entries. Shipped in [PR #10792](https://github.com/mudler/LocalAI/pull/10792). diff --git a/docs/content/blog/_index.md b/docs/content/blog/_index.md new file mode 100644 index 000000000..c6e630373 --- /dev/null +++ b/docs/content/blog/_index.md @@ -0,0 +1,17 @@ ++++ +title = "Blog" +weight = 10 +icon = "newspaper" +alwaysopen = false +aliases = ["/basics/news/", "/whats-new/"] ++++ + +Announcements, release write-ups and feature notes from the LocalAI team. + +Full changelogs for every version live on [GitHub Releases](https://github.com/mudler/LocalAI/releases). This page is the narrative archive: what shipped, and why it matters. + +{{% notice tip %}} +Prefer a feed reader? Subscribe to [/blog/index.xml](/blog/index.xml). +{{% /notice %}} + +{{< postlist >}}