mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 01:48:06 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs: onboarding overhaul, dedup, and error docs (#7711) (#10895)

Browse Source 
* docs: fix CPU image tag (latest, not latest-cpu)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: use canonical localai/localai registry in models guide

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: replace dead llama-stable backend with llama-cpp

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: correct mitm-proxy intercept config and redaction tier

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fix text-to-audio endpoint and broken notice block

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fix VAD example, stale FAQ, broken link, CLI list, whats-new dump

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: render advanced/reference section indexes (consolidate _index)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove duplicate getting-started build/kubernetes pages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fold container image reference into installation/containers

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove stale advanced fine-tuning page (superseded by features/fine-tuning)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: fold distribution/longcat/sound pages into their parents

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: make getting-started index accurate and complete

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: carry one concrete model through the getting-started path

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add end-to-end 'build your first agent' walkthrough

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add runtime errors reference; consolidate troubleshooting from FAQ

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add agent actions catalog

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: agent-scoped MCP, skills walkthrough, agentic disambiguation

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add concrete gallery install lines to media feature pages

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: merge installation into getting-started (URLs preserved via aliases)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add Operations section; move operator pages and P2P API reference

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: journey-ordered top nav and grouped feature sections

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: add docs-with-code process gate (PR template + agent instructions)

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: remove em/en dashes from documentation prose

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]
2026-07-17 22:08:20 +02:00
committed by GitHub
parent d3ea65a112
commit 40d35c0385
86 changed files with 1832 additions and 2114 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.agents/coding-style.md
View File

@@ -65,6 +65,7 @@ This is enforced by `forbidigo` (see `.golangci.yml`): `http.DefaultClient` and
The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
- **Docs-with-code rule**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. The PR template carries a checklist item for this.
- **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.

1
.github/PULL_REQUEST_TEMPLATE.md vendored
View File

@@ -7,6 +7,7 @@ This PR fixes #
**[Signed commits](../CONTRIBUTING.md#signing-off-on-commits-developer-certificate-of-origin)**
- [ ] Yes, I signed my commits.
- [ ] Documentation updated (docs/content/) for user-facing changes, or not applicable
<!--
Thank you for contributing to LocalAI!

2
AGENTS.md
View File

@@ -39,7 +39,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
- **Logging**: Use `github.com/mudler/xlog` (same API as slog)
- **Go style**: Prefer `any` over `interface{}`
- **Comments**: Explain *why*, not *what*
- **Docs**: Update `docs/content/` when adding features or changing config
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds

4
docs/content/_index.md
View File

@@ -9,7 +9,7 @@ type = "home"
<div class="lai-hero__copy">
<p class="lai-signal"><span></span> Open source · MIT licensed</p>
<h1>One runtime.<br><strong>Every kind of AI.</strong><br>Your hardware.</h1>
<p class="lai-hero__lede">LocalAI runs text, vision, speech, sound, images, video, embeddings, reranking, and autonomous agents behind one modular stack—from a CPU laptop to a distributed GPU cluster.</p>
<p class="lai-hero__lede">LocalAI runs text, vision, speech, sound, images, video, embeddings, reranking, and autonomous agents behind one modular stack-from a CPU laptop to a distributed GPU cluster.</p>
<div class="lai-actions">
<a class="lai-button" href="/installation/">Install LocalAI <b>→</b></a>
<a class="lai-link" href="https://github.com/mudler/LocalAI">View on GitHub ↗</a>
@@ -33,7 +33,7 @@ type = "home"
<div class="lai-architecture__copy">
<p>A small core, not a giant bundle.</p>
<h2>Backends arrive when the model needs them.</h2>
<p>LocalAI keeps the core lean. Each backend wraps a best-in-class engine—llama.cpp, vLLM, SGLang, MLX, whisper.cpp, diffusion engines, and many more—as an isolated service pulled on demand.</p>
<p>LocalAI keeps the core lean. Each backend wraps a best-in-class engine-llama.cpp, vLLM, SGLang, MLX, whisper.cpp, diffusion engines, and many more-as an isolated service pulled on demand.</p>
<ul><li>Install, update, or remove engines independently.</li><li>Mix CPU, NVIDIA, AMD, Intel, Apple Silicon, Vulkan, and Jetson.</li><li>Build your own backend in any language through an open gRPC contract.</li></ul>
<a href="/reference/architecture/">Explore the architecture →</a>
</div>

105
docs/content/advanced/_index.en.md
View File

@@ -1,5 +1,5 @@
---
weight: 20
weight: 5
title: "Advanced"
description: "Advanced usage"
type: chapter
@@ -9,4 +9,105 @@ date: 2020-10-06T08:49:15+00:00
lastmod: 2020-10-06T08:49:15+00:00
draft: false
images: []
---
---
## Overview
The **Advanced** section covers in-depth topics for users who want to fully leverage LocalAI's capabilities beyond basic usage. These pages are designed for developers, DevOps engineers, and power users who need fine-grained control over model configuration, system resources, and deployment infrastructure.
### Who Should Read This Section
- **Developers** integrating LocalAI into applications
- **DevOps Engineers** deploying LocalAI in production
- **ML Engineers** optimizing model performance
- **System Administrators** managing multi-user installations
---
## Topics
### 🚀 [Advanced Usage](advanced-usage.md)
Comprehensive guide to advanced LocalAI features including multi-modal inference, custom backends, and extended API capabilities.
**Key topics:**
- Multi-modal model support
- Custom backend integration
- Advanced API endpoints
- Request/response customization
**Recommended for:** Developers extending LocalAI functionality
---
### 🎯 [Model Configuration](model-configuration.md)
Complete reference for model configuration files, parameters, and optimization settings.
**Key topics:**
- Configuration file format
- Model-specific parameters
- Quantization settings
- Performance tuning
**Recommended for:** ML engineers optimizing model behavior
---
### 🔒 [Reverse Proxy & TLS](reverse-proxy-tls.md)
Complete guide to securing LocalAI deployments with reverse proxies and TLS certificates.
**Key topics:**
- Nginx/Apache configuration
- TLS certificate setup
- Authentication layers
- Production hardening
**Recommended for:** DevOps engineers deploying to production
---
### 💾 [VRAM Management](vram-management.md)
Advanced techniques for managing GPU memory and optimizing parallel inference.
**Key topics:**
- GPU memory allocation
- Multi-model loading
- Batch processing
- Resource scheduling
**Recommended for:** Users running multiple models on limited hardware
---
## Quick Links
| Task | Documentation |
|------|---------------|
| Configure a model | [Model Configuration](model-configuration.md) |
| Deploy securely | [Reverse Proxy & TLS](reverse-proxy-tls.md) |
| Optimize VRAM usage | [VRAM Management](vram-management.md) |
| Extend functionality | [Advanced Usage](advanced-usage.md) |
---
## Prerequisites
Before diving into advanced topics, ensure you have:
1. ✅ Completed the [Getting Started](../getting-started/) guide
2. ✅ Successfully run LocalAI with a basic model
3. ✅ Basic understanding of command-line interfaces
4. ✅ Familiarity with YAML configuration (for most topics)
---
## Related Sections
- 📚 [Reference](../reference/) - API documentation and command reference
- 🔌 [Installation](../installation/) - Deployment options and requirements
- ⭐ [Features](../features/) - Overview of LocalAI capabilities
---
## Navigation
← [Getting Started](../getting-started/) | [Reference](../reference/) →

126
docs/content/advanced/_index.md
View File

@@ -1,126 +0,0 @@
---
weight: 20
title: "Advanced"
description: "Advanced usage and configuration"
type: chapter
icon: settings
lead: "Deep dive into LocalAI's advanced features, configuration options, and optimization techniques."
date: 2020-10-06T08:49:15+00:00
lastmod: 2026-03-09T00:00:00+00:00
draft: false
images: []
---
## Overview
The **Advanced** section covers in-depth topics for users who want to fully leverage LocalAI's capabilities beyond basic usage. These pages are designed for developers, DevOps engineers, and power users who need fine-grained control over model configuration, system resources, and deployment infrastructure.
### Who Should Read This Section
- **Developers** integrating LocalAI into applications
- **DevOps Engineers** deploying LocalAI in production
- **ML Engineers** optimizing model performance
- **System Administrators** managing multi-user installations
---
## Topics
### 🚀 [Advanced Usage](advanced-usage.md)
Comprehensive guide to advanced LocalAI features including multi-modal inference, custom backends, and extended API capabilities.
**Key topics:**
- Multi-modal model support
- Custom backend integration
- Advanced API endpoints
- Request/response customization
**Recommended for:** Developers extending LocalAI functionality
---
### 🎯 [Model Configuration](model-configuration.md)
Complete reference for model configuration files, parameters, and optimization settings.
**Key topics:**
- Configuration file format
- Model-specific parameters
- Quantization settings
- Performance tuning
**Recommended for:** ML engineers optimizing model behavior
---
### 🔧 [Fine-Tuning](fine-tuning.md)
Guide to fine-tuning models with LocalAI for domain-specific applications.
**Key topics:**
- Training data preparation
- Fine-tuning parameters
- Evaluation metrics
- Best practices
**Recommended for:** Users creating specialized models
---
### 🔒 [Reverse Proxy & TLS](reverse-proxy-tls.md)
Complete guide to securing LocalAI deployments with reverse proxies and TLS certificates.
**Key topics:**
- Nginx/Apache configuration
- TLS certificate setup
- Authentication layers
- Production hardening
**Recommended for:** DevOps engineers deploying to production
---
### 💾 [VRAM Management](vram-management.md)
Advanced techniques for managing GPU memory and optimizing parallel inference.
**Key topics:**
- GPU memory allocation
- Multi-model loading
- Batch processing
- Resource scheduling
**Recommended for:** Users running multiple models on limited hardware
---
## Quick Links
| Task | Documentation |
|------|---------------|
| Configure a model | [Model Configuration](model-configuration.md) |
| Deploy securely | [Reverse Proxy & TLS](reverse-proxy-tls.md) |
| Optimize VRAM usage | [VRAM Management](vram-management.md) |
| Extend functionality | [Advanced Usage](advanced-usage.md) |
---
## Prerequisites
Before diving into advanced topics, ensure you have:
1. ✅ Completed the [Getting Started](../getting-started/) guide
2. ✅ Successfully run LocalAI with a basic model
3. ✅ Basic understanding of command-line interfaces
4. ✅ Familiarity with YAML configuration (for most topics)
---
## Related Sections
- 📚 [Reference](../reference/) - API documentation and command reference
- 🔌 [Installation](../installation/) - Deployment options and requirements
- ⭐ [Features](../features/) - Overview of LocalAI capabilities
---
## Navigation
← [Getting Started](../getting-started/) | [Reference](../reference/) →

4
docs/content/advanced/advanced-usage.md
View File

@@ -20,7 +20,7 @@ parameters:
context_size: 512
threads: 10
backend: llama-stable
backend: llama-cpp
template:
completion: completion
@@ -140,7 +140,7 @@ parameters:
# Relative to the models path
model: ...
backend: llama-stable
backend: llama-cpp
```
### Connect external backends

126
docs/content/advanced/fine-tuning.md
View File

@@ -1,126 +0,0 @@
+++
disableToc = false
title = "Fine-tuning LLMs for text generation"
weight = 22
+++
![Fine-tuning recipe: from dataset to a servable GGUF via LoRA fine-tune and merge](/images/diagrams/finetune-recipe.png)
{{% notice note %}}
Section under construction
{{% /notice %}}
This section covers how to fine-tune a language model for text generation and consume it in LocalAI.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mudler/LocalAI/blob/master/examples/e2e-fine-tuning/notebook.ipynb)
## Requirements
For this example you will need at least a 12GB VRAM of GPU and a Linux box.
## Fine-tuning
Fine-tuning a language model is a process that requires a lot of computational power and time.
Currently LocalAI doesn't support the fine-tuning endpoint as LocalAI but there are are [plans](https://github.com/mudler/LocalAI/issues/596) to support that. For the time being a guide is proposed here to give a simple starting point on how to fine-tune a model and use it with LocalAI (but also with llama.cpp).
There is an e2e example of fine-tuning a LLM model to use with [LocalAI](https://github.com/mudler/LocalAI) written by [@mudler](https://github.com/mudler) available [here](https://github.com/mudler/LocalAI/tree/master/examples/e2e-fine-tuning/).
The steps involved are:
- Preparing a dataset
- Prepare the environment and install dependencies
- Fine-tune the model
- Merge the Lora base with the model
- Convert the model to gguf
- Use the model with LocalAI
## Dataset preparation
We are going to need a dataset or a set of datasets.
Axolotl supports a variety of formats, in the notebook and in this example we are aiming for a very simple dataset and build that manually, so we are going to use the `completion` format which requires the full text to be used for fine-tuning.
A dataset for an instructor model (like Alpaca) can look like the following:
```json
[
{
"text": "As an AI language model you are trained to reply to an instruction. Try to be as much polite as possible\n\n## Instruction\n\nWrite a poem about a tree.\n\n## Response\n\nTrees are beautiful, ...",
},
{
"text": "As an AI language model you are trained to reply to an instruction. Try to be as much polite as possible\n\n## Instruction\n\nWrite a poem about a tree.\n\n## Response\n\nTrees are beautiful, ...",
}
]
```
Every block in the text is the whole text that is used to fine-tune. For example, for an instructor model it follows the following format (more or less):
```
<System prompt>
## Instruction
<Question, instruction>
## Response
<Expected response from the LLM>
```
The instruction format works such as when we are going to inference with the model, we are going to feed it only the first part up to the `## Instruction` block, and the model is going to complete the text with the `## Response` block.
Prepare a dataset, and upload it to your Google Drive in case you are using the Google colab. Otherwise place it next the `axolotl.yaml` file as `dataset.json`.
### Install dependencies
```bash
git clone https://github.com/OpenAccess-AI-Collective/axolotl && pushd axolotl && git checkout 797f3dd1de8fd8c0eafbd1c9fdb172abd9ff840a && popd #0.3.0
pip install packaging
pushd axolotl && pip install -e '.[flash-attn,deepspeed]' && popd
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.3.0/flash_attn-2.3.0+cu117torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
```
Configure accelerate:
```bash
accelerate config default
```
## Fine-tuning
We will need to configure axolotl. In this example is provided a file to use `axolotl.yaml` that uses openllama-3b for fine-tuning. Copy the `axolotl.yaml` file and edit it to your needs. The dataset needs to be next to it as `dataset.json`. You can find the axolotl.yaml file [here](https://github.com/mudler/LocalAI/tree/master/examples/e2e-fine-tuning/).
If you have a big dataset, you can pre-tokenize it to speedup the fine-tuning process:
```bash
python -m axolotl.cli.preprocess axolotl.yaml
```
Now we are ready to start the fine-tuning process:
```bash
accelerate launch -m axolotl.cli.train axolotl.yaml
```
After we have finished the fine-tuning, we merge the Lora base with the model:
```bash
python3 -m axolotl.cli.merge_lora axolotl.yaml --lora_model_dir="./qlora-out" --load_in_8bit=False --load_in_4bit=False
```
And we convert it to the gguf format that LocalAI can consume:
```bash
git clone https://github.com/ggerganov/llama.cpp.git
pushd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release && popd
pushd llama.cpp && python3 convert_hf_to_gguf.py ../qlora-out/merged && popd
pushd llama.cpp/build/bin && ./llama-quantize ../../../qlora-out/merged/Merged-33B-F16.gguf \
../../../custom-model-q4_0.gguf q4_0
```
Now you should have ended up with a `custom-model-q4_0.gguf` file that you can copy in the LocalAI models directory and use it with LocalAI.

28
docs/content/advanced/model-configuration.md
View File

@@ -35,7 +35,7 @@ parameters:
context_size: 512
threads: 10
backend: llama-stable
backend: llama-cpp
template:
completion: completion
@@ -51,13 +51,13 @@ When using `--models-config-file`, you can define multiple models as a list:
parameters:
model: model1.bin
context_size: 512
backend: llama-stable
backend: llama-cpp
- name: model2
parameters:
model: model2.bin
context_size: 1024
backend: llama-stable
backend: llama-cpp
```
## Core Configuration Fields
@@ -269,7 +269,7 @@ YARN (Yet Another RoPE extensioN) settings for context extension:
### Speculative Decoding
Speculative decoding speeds up text generation by predicting multiple tokens ahead and verifying them in a single forward pass. The output is identical to normal decoding — only faster. This feature is only available with the `llama-cpp` backend.
Speculative decoding speeds up text generation by predicting multiple tokens ahead and verifying them in a single forward pass. The output is identical to normal decoding - only faster. This feature is only available with the `llama-cpp` backend.
There are two approaches:
@@ -291,7 +291,7 @@ options:
#### N-gram Self-Speculative Decoding
Uses patterns from the token history to predict future tokens — no extra model required. Works well for repetitive or structured output (code, JSON, lists).
Uses patterns from the token history to predict future tokens - no extra model required. Works well for repetitive or structured output (code, JSON, lists).
```yaml
name: my-model
@@ -476,7 +476,7 @@ This is the load-time reasoning configuration. The orthogonal per-request `enabl
#### `reasoning_effort` as a chat-template kwarg
`reasoning_effort` is also forwarded to the backend as a `chat_template_kwarg`, so models whose **jinja chat template** keys on it — e.g. gpt-oss (Harmony) or LFM2.5 — honor the **level**, not just the on/off `enable_thinking` flag. This matters for models that ignore `enable_thinking` entirely (LFM2.5 keeps emitting `<think>` for `enable_thinking=false`, but respects `reasoning_effort`).
`reasoning_effort` is also forwarded to the backend as a `chat_template_kwarg`, so models whose **jinja chat template** keys on it - e.g. gpt-oss (Harmony) or LFM2.5 - honor the **level**, not just the on/off `enable_thinking` flag. This matters for models that ignore `enable_thinking` entirely (LFM2.5 keeps emitting `<think>` for `enable_thinking=false`, but respects `reasoning_effort`).
Set a per-model default in the config so every request inherits it (a per-request `reasoning_effort` still overrides):
@@ -629,7 +629,7 @@ These options apply when using the `vllm` backend:
| `disable_log_stats` | bool | Disable logging statistics |
| `dtype` | string | Data type (e.g., `float16`, `bfloat16`) |
| `flash_attention` | string | Flash attention configuration |
| `cache_type_k` | string | Key cache quantization type. Maps to llama.cpp's `-ctk`. Accepted values for llama.cpp-family backends (`llama-cpp`, `ik-llama-cpp`, `turboquant`): `f16`, `f32`, `q8_0`, `q4_0`, `q4_1`, `q5_0`, `q5_1`. The `turboquant` backend additionally accepts `turbo2`, `turbo3`, `turbo4` — the fork's TurboQuant KV-cache schemes. `turbo3`/`turbo4` auto-enable flash_attention. |
| `cache_type_k` | string | Key cache quantization type. Maps to llama.cpp's `-ctk`. Accepted values for llama.cpp-family backends (`llama-cpp`, `ik-llama-cpp`, `turboquant`): `f16`, `f32`, `q8_0`, `q4_0`, `q4_1`, `q5_0`, `q5_1`. The `turboquant` backend additionally accepts `turbo2`, `turbo3`, `turbo4` - the fork's TurboQuant KV-cache schemes. `turbo3`/`turbo4` auto-enable flash_attention. |
| `cache_type_v` | string | Value cache quantization type. Maps to llama.cpp's `-ctv`. Same accepted values as `cache_type_k`. Note: any quantized V cache requires flash_attention to be enabled. |
| `limit_mm_per_prompt` | object | Limit multimodal content per prompt: `{image: int, video: int, audio: int}` |
@@ -915,7 +915,7 @@ Define pipelines for audio-to-audio processing and the [Realtime API]({{%relref
## gRPC Configuration
Backend gRPC communication settings. These control the readiness handshake
between LocalAI and a freshly spawned backend process — LocalAI polls the
between LocalAI and a freshly spawned backend process - LocalAI polls the
backend's `Health` gRPC method up to `grpc.attempts` times, sleeping
`grpc.attempts_sleep_time` seconds between polls, before giving up and
terminating the backend as unresponsive.
@@ -928,7 +928,7 @@ terminating the backend as unresponsive.
**Total load window ≈ `grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)`.**
The default of `20 × 2 s ≈ 40 s` is fine for typical backends but is too
short for large models that need substantial time to become gRPC-ready
after the process starts — for example NVFP4 / FP8 models whose shard
after the process starts - for example NVFP4 / FP8 models whose shard
loading and CUDA-graph capture can take several minutes, or slow storage
backends. If the backend keeps getting killed while still legitimately
loading (visible as `exitCode=120` + `rpc error: code = Canceled desc =
@@ -946,7 +946,7 @@ grpc:
This gives a ~700 s window while keeping health-check polling frequent
enough to detect real backend crashes quickly. The values only affect
the initial readiness handshake — inference-request timeouts and the
the initial readiness handshake - inference-request timeouts and the
watchdog are unchanged.
## Overrides
@@ -974,7 +974,7 @@ known_usecases:
Available flags: `chat`, `completion`, `edit`, `embeddings`, `rerank`, `image`, `transcript`, `tts`, `sound_generation`, `tokenize`, `vad`, `video`, `detection`, `llm` (combination of CHAT, COMPLETION, EDIT).
`token_classify` marks a model as a token-classification (NER) provider for the PII filter (e.g. an `openai-privacy-filter` GGUF). Declare it explicitly together with `embeddings: true` (the classifier loads via TOKEN_CLS pooling). It runs on the dedicated `privacy-filter` backend (`backend/cpp/privacy-filter`), a standalone GGML engine for the `openai-privacy-filter` family — separate from `llama-cpp`, which no longer carries the token-classification path.
`token_classify` marks a model as a token-classification (NER) provider for the PII filter (e.g. an `openai-privacy-filter` GGUF). Declare it explicitly together with `embeddings: true` (the classifier loads via TOKEN_CLS pooling). It runs on the dedicated `privacy-filter` backend (`backend/cpp/privacy-filter`), a standalone GGML engine for the `openai-privacy-filter` family - separate from `llama-cpp`, which no longer carries the token-classification path.
### Known input and output modalities
@@ -1007,7 +1007,7 @@ PII redaction is NER-based and runs on the **request** (input) side. It has two
- token_classify
pii_detection:
min_score: 0.5 # drop detections below this confidence
default_action: mask # mask | block | allow — applied to any detected
default_action: mask # mask | block | allow - applied to any detected
# group with no explicit entry (empty = mask)
entity_actions: # which PII to block vs mask vs allow-log
PASSWORD: block
@@ -1015,7 +1015,7 @@ PII redaction is NER-based and runs on the **request** (input) side. It has two
EMAIL: mask
```
- **Consuming models** opt in and reference one or more detectors by name — no per-consumer policy:
- **Consuming models** opt in and reference one or more detectors by name - no per-consumer policy:
```yaml
name: my-assistant
@@ -1036,7 +1036,7 @@ Here's a comprehensive example combining many options:
```yaml
name: my-llm-model
description: A high-performance LLM model
backend: llama-stable
backend: llama-cpp
parameters:
model: my-model.gguf

16
docs/content/advanced/vram-management.md
View File

@@ -141,7 +141,7 @@ LOCALAI_SINGLE_ACTIVE_BACKEND=true ./local-ai
## Solution 1b: Concurrency Groups (per-model anti-affinity)
`--max-active-backends` is a global count — three loaded models is fine, but it
`--max-active-backends` is a global count - three loaded models is fine, but it
doesn't know that two of them are 120B and shouldn't share a GPU.
**Concurrency groups** give per-model rules: any two models that share a group
name are mutually exclusive on the same node. Loading one evicts the others.
@@ -154,7 +154,7 @@ This addresses [issue #9659](https://github.com/mudler/LocalAI/issues/9659):
### Configuration
Declare groups per model in the YAML config — no CLI flag, no env var:
Declare groups per model in the YAML config - no CLI flag, no env var:
```yaml
# llama-120b-a.yaml
@@ -175,7 +175,7 @@ concurrency_groups: ["vram-heavy"]
```
```yaml
# zed-predict.yaml — no groups, runs alongside anything
# zed-predict.yaml - no groups, runs alongside anything
name: zed-predict
backend: llama-cpp
parameters:
@@ -190,7 +190,7 @@ With this configuration:
`vram-heavy`); `zed-predict` stays loaded.
A model can declare multiple groups; two models conflict if they share **any**
group name. Group names are arbitrary strings — pick names that make sense for
group name. Group names are arbitrary strings - pick names that make sense for
your hardware (`vram-heavy`, `gpu-1`, `large-context`, ...).
### Interaction with other knobs
@@ -198,7 +198,7 @@ your hardware (`vram-heavy`, `gpu-1`, `large-context`, ...).
- **`--max-active-backends`**: groups are checked *before* the LRU cap. Group
evictions may already make room; LRU then enforces the global count.
- **`pinned: true`**: a pinned model is never evicted, including by a group
conflict. The new request is loaded with a warning logged — pinning two
conflict. The new request is loaded with a warning logged - pinning two
models in the same group is a configuration mismatch.
- **`--force-eviction-when-busy`**: same retry semantics as LRU. A busy
conflict is skipped and retried (`--lru-eviction-max-retries`,
@@ -207,7 +207,7 @@ your hardware (`vram-heavy`, `gpu-1`, `large-context`, ...).
### Distributed mode
`concurrency_groups` is enforced **per node**, not cluster-wide — VRAM is a
`concurrency_groups` is enforced **per node**, not cluster-wide - VRAM is a
node-local resource, so two heavy models on different nodes is fine. The
distributed scheduler additionally uses the rule as a placement hint: when
choosing where to load a new model, it prefers nodes that don't already host a
@@ -215,7 +215,7 @@ same-group model, falling back to eviction only if every candidate has a
conflict.
`concurrency_groups` composes with `NodeSelector` (which decides *which
nodes* a model is eligible for) — the two filters apply in sequence. Use
nodes* a model is eligible for) - the two filters apply in sequence. Use
`NodeSelector` to target hardware classes; use `concurrency_groups` to keep
specific models from co-residing on whichever node hosts them.
@@ -424,7 +424,7 @@ curl -X POST http://localhost:8080/backend/shutdown \
To stop all models, you'll need to call the endpoint for each loaded model individually, or use the web UI to stop all models at once.
Conversely, you can pre-load a model into memory ahead of its first request with `POST /backend/load` (the inverse of shutdown) — see [Backend Monitor]({{%relref "features/backend-monitor" %}}).
Conversely, you can pre-load a model into memory ahead of its first request with `POST /backend/load` (the inverse of shutdown) - see [Backend Monitor]({{%relref "operations/backend-monitor" %}}).
### Best Practices

32
docs/content/faq.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "FAQ"
weight = 24
weight = 9
icon = "quiz"
url = "/faq/"
+++
@@ -30,7 +30,7 @@ Yes. You can load any compatible model, not just the ones in the gallery. And be
### How do I get models?
Most gguf-based models should work, but newer models may require additions to the API. If a model doesn't work, please feel free to open up issues. However, be cautious about downloading models from the internet and directly onto your machine, as there may be security vulnerabilities in lama.cpp or ggml that could be maliciously exploited. Some models can be found on Hugging Face: https://huggingface.co/models?search=gguf, or models from gpt4all are compatible too: https://github.com/nomic-ai/gpt4all.
The easiest way is the built-in model gallery: open the web interface at `http://localhost:8080`, go to the Models page, and install a model by name. You can also install from the command line with `local-ai models install <model-name>`. Most gguf-based models work, and you can point LocalAI at any compatible GGUF from Hugging Face (https://huggingface.co/models?search=gguf). Be cautious about downloading models from untrusted sources, as there may be security vulnerabilities in llama.cpp or ggml that could be maliciously exploited.
### Where are models stored?
@@ -64,17 +64,9 @@ Model sizes vary significantly depending on the model and quantization level:
LocalAI applies a set of defaults when loading models with the llama.cpp backend, one of these is mirostat sampling - while it achieves better results, it slows down the inference. You can disable this by setting `mirostat: 0` in the model config file. See also the advanced section ({{%relref "advanced/advanced-usage" %}}) for more information and [this issue](https://github.com/mudler/LocalAI/issues/2780).
### What's the difference with Serge, or XXX?
LocalAI is a multi-model solution that doesn't focus on a specific model type (e.g., llama.cpp or alpaca.cpp), and it handles all of these internally for faster inference, easy to set up locally and deploy to Kubernetes.
### Everything is slow, how is it possible?
There are few situation why this could occur. Some tips are:
- Don't use HDD to store your models. Prefer SSD over HDD. In case you are stuck with HDD, disable `mmap` in the model config file so it loads everything in memory.
- Watch out CPU overbooking. Ideally the `--threads` should match the number of physical cores. For instance if your CPU has 4 cores, you would ideally allocate `<= 4` threads to a model.
- Run LocalAI with `DEBUG=true`. This gives more information, including stats on the token inference speed.
- Check that you are actually getting an output: run a simple curl request with `"stream": true` to see how fast the model is responding.
See the performance section of the runtime errors reference: {{% relref "reference/runtime-errors" %}}.
### Can I use it with a Discord bot, or XXX?
@@ -84,23 +76,15 @@ Yes! If the client uses OpenAI and supports setting a different base URL to send
There is GPU support, see {{%relref "features/GPU-acceleration" %}}.
### Where is the webUI?
### Where is the WebUI?
There is the availability of localai-webui and chatbot-ui in the examples section and can be setup as per the instructions. However as LocalAI is an API you can already plug it into existing projects that provides are UI interfaces to OpenAI's APIs. There are several already on Github, and should be compatible with LocalAI already (as it mimics the OpenAI API)
### Does it work with AutoGPT?
Yes, see the [examples](https://github.com/mudler/LocalAI-examples)!
LocalAI ships with a built-in web interface. Once LocalAI is running, open `http://localhost:8080` in your browser to manage models, chat, and configure the server. There is nothing extra to install. Because LocalAI also exposes an OpenAI-compatible API, you can additionally plug it into any existing application that talks to the OpenAI API by pointing that application at the LocalAI endpoint.
### How can I troubleshoot when something is wrong?
Enable the debug mode by setting `DEBUG=true` in the environment variables. This will give you more information on what's going on.
You can also specify `--debug` in the command line.
You can also specify `--debug` in the command line. For matching a specific runtime or backend error message to its cause and fix, see the runtime errors reference: {{% relref "reference/runtime-errors" %}}.
### I'm getting 'invalid pitch' error when running with CUDA, what's wrong?
### I'm getting 'invalid pitch', 'SIGILL', a CUDA out-of-memory error, or a model that will not load
This typically happens when your prompt exceeds the context size. Try to reduce the prompt size, or increase the context size.
### I'm getting a 'SIGILL' error, what's wrong?
Your CPU probably does not have support for certain instructions that are compiled by default in the pre-built binaries. If you are running in a container, try setting `REBUILD=true` and disable the CPU instructions that are not compatible with your CPU. For instance: `CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make build`
These runtime and backend failures, and how to read the real error behind an HTTP `500`, are documented in the runtime errors reference: {{% relref "reference/runtime-errors" %}}.

6
docs/content/features/GPU-acceleration.md
View File

@@ -1,11 +1,11 @@
+++
disableToc = false
title = "GPU Acceleration"
weight = 9
weight = 73
url = "/features/gpu-acceleration/"
+++
This page covers how to use LocalAI with GPU acceleration across different hardware vendors. For container image tags and registry details, see [Container Images]({{%relref "getting-started/container-images" %}}). For memory management with multiple GPU-accelerated models, see [VRAM Management]({{%relref "advanced/vram-management" %}}).
This page covers how to use LocalAI with GPU acceleration across different hardware vendors. For container image tags and registry details, see [Container Images]({{%relref "getting-started/containers" %}}). For memory management with multiple GPU-accelerated models, see [VRAM Management]({{%relref "advanced/vram-management" %}}).
## Automatic Backend Detection
@@ -153,7 +153,7 @@ llama_init_from_file: kv self size = 512.00 MB
There are a limited number of tested configurations for ROCm systems however most newer dedicated GPU consumer grade devices seem to be supported under the current ROCm 7 implementation.
Due to the nature of ROCm it is best to run all implementations in containers as this limits the number of packages required for installation on host system, compatibility and package versions for dependencies across all variations of OS must be tested independently if desired, please refer to the [build]({{%relref "installation/build#Acceleration" %}}) documentation.
Due to the nature of ROCm it is best to run all implementations in containers as this limits the number of packages required for installation on host system, compatibility and package versions for dependencies across all variations of OS must be tested independently if desired, please refer to the [build]({{%relref "getting-started/build#Acceleration" %}}) documentation.
### Requirements

90
docs/content/features/_index.en.md
View File

@@ -1,48 +1,76 @@
+++
disableToc = false
title = "Features"
weight = 8
weight = 4
icon = "lightbulb"
type = "chapter"
url = "/features/"
+++
LocalAI provides a comprehensive set of features for running AI models locally. This section covers all the capabilities and functionalities available in LocalAI.
LocalAI provides a comprehensive set of features for running AI models locally. The pages in this section are grouped by capability, and the left navigation is ordered to match these groups.
## Core Features
## Text
- **[Text Generation](text-generation/)** - Generate text with GPT-compatible models using various backends
- **[Image Generation](image-generation/)** - Create images with Stable Diffusion and other diffusion models
- **[Audio Processing](audio-to-text/)** - Transcribe audio to text and generate speech from text
- **[Text to Audio](text-to-audio/)** - Generate speech from text with TTS models
- **[Sound Generation](sound-generation/)** - Generate music and sound effects from text descriptions
- **[Voice Activity Detection](voice-activity-detection/)** - Detect speech segments in audio data
- **[Video Generation](video-generation/)** - Generate videos from text prompts with optional image or audio conditioning
- **[LongCat Video and Avatar](longcat-video/)** - Run text/image-to-video and audio-driven avatars on NVIDIA CUDA and DGX Spark
- **[Embeddings](embeddings/)** - Generate vector embeddings for semantic search and RAG applications
- **[GPT Vision](gpt-vision/)** - Analyze and understand images with vision-language models
- **[Text Generation]({{% relref "features/text-generation" %}})** - Generate text with GPT-compatible models using various backends.
- **[OpenAI Functions and tools]({{% relref "features/openai-functions" %}})** - Use function calling and the tools API with local models.
- **[Constrained Grammars]({{% relref "features/constrained_grammars" %}})** - Control model output format with BNF grammars.
- **[Interleaved thinking]({{% relref "features/interleaved-thinking" %}})** - Reasoning models that interleave thought and output.
- **[Model aliases]({{% relref "features/model-aliases" %}})** - Expose one model under several names.
- **[API discovery]({{% relref "features/api-discovery" %}})** - How LocalAI advertises its capability surface.
## Advanced Features
## Agents
- **[OpenAI Functions](openai-functions/)** - Use function calling and tools API with local models
- **[Realtime API](openai-realtime/)** - Low-latency multi-modal conversations (voice+text) over WebSocket
- **[Constrained Grammars](constrained_grammars/)** - Control model output format with BNF grammars
- **[GPU Acceleration](GPU-acceleration/)** - Optimize performance with GPU support
- **[Distribution](distribution/)** - Scale inference across multiple nodes (P2P federation or production distributed mode)
- **[P2P API](p2p/)** - Monitor and manage P2P worker and federated nodes
- **[Model Context Protocol (MCP)](mcp/)** - Enable agentic capabilities with MCP integration
- **[Agents](agents/)** - Autonomous AI agents with tools, knowledge base, and skills
- **[Agents]({{% relref "features/agents" %}})** - Autonomous AI agents with tools, knowledge base, and skills.
- **[Agent Actions]({{% relref "features/agent-actions" %}})** - The catalog of actions an agent can run.
- **[Model Context Protocol (MCP)]({{% relref "features/mcp" %}})** - Give a model external tools over MCP.
- **[LocalAI Assistant]({{% relref "features/localai-assistant" %}})** - Chat to administer your LocalAI instance.
## Specialized Features
## Audio
- **[Object Detection](object-detection/)** - Detect and locate objects in images
- **[Reranker](reranker/)** - Improve retrieval accuracy with cross-encoder models
- **[Stores](stores/)** - Vector similarity search for embeddings
- **[Model Gallery](model-gallery/)** - Browse and install pre-configured models
- **[Backends](backends/)** - Learn about available backends and how to manage them
- **[Backend Monitor](backend-monitor/)** - Monitor backend status and resource usage
- **[Runtime Settings](runtime-settings/)** - Configure application settings via web UI without restarting
- **[Audio to text]({{% relref "features/audio-to-text" %}})** - Transcribe audio to text.
- **[Text to audio]({{% relref "features/text-to-audio" %}})** - Generate speech, music, and sound effects from text.
- **[Audio classification]({{% relref "features/audio-classification" %}})** - Classify sounds and audio events.
- **[Audio diarization]({{% relref "features/audio-diarization" %}})** - Separate speakers in an audio stream.
- **[Audio transform]({{% relref "features/audio-transform" %}})** - Transform and process audio.
- **[Voice Activity Detection]({{% relref "features/voice-activity-detection" %}})** - Detect speech segments in audio data.
- **[Voice recognition]({{% relref "features/voice-recognition" %}})** - Identify speakers by voice.
- **[Realtime API]({{% relref "features/openai-realtime" %}})** - Low-latency multi-modal conversations (voice and text) over WebSocket.
## Vision
- **[GPT Vision]({{% relref "features/gpt-vision" %}})** - Analyze and understand images with vision-language models.
- **[Object detection]({{% relref "features/object-detection" %}})** - Detect and locate objects in images.
- **[Face recognition]({{% relref "features/face-recognition" %}})** - Recognize faces in images.
## Image and Video
- **[Image Generation]({{% relref "features/image-generation" %}})** - Create images with Stable Diffusion and other diffusion models.
- **[Video Generation]({{% relref "features/video-generation" %}})** - Generate videos from text, image, or audio conditioning, including LongCat and avatar workflows.
## Retrieval
- **[Embeddings]({{% relref "features/embeddings" %}})** - Generate vector embeddings for semantic search and RAG applications.
- **[Reranker]({{% relref "features/reranker" %}})** - Improve retrieval accuracy with cross-encoder models.
- **[Stores]({{% relref "features/stores" %}})** - Vector similarity search for embeddings.
## Distributed and acceleration
- **[Distributed inference]({{% relref "features/distributed_inferencing" %}})** - Scale inference across multiple nodes (P2P federation or production distributed mode).
- **[Distributed mode]({{% relref "features/distributed-mode" %}})** - The production distributed deployment.
- **[MLX distributed]({{% relref "features/mlx-distributed" %}})** - Distributed inference on Apple Silicon with MLX.
- **[GPU Acceleration]({{% relref "features/GPU-acceleration" %}})** - Optimize performance with GPU support.
## Platform and model management
- **[Backends]({{% relref "features/backends" %}})** - Available backends and how to manage them.
- **[Model Gallery]({{% relref "features/model-gallery" %}})** - Browse and install pre-configured models.
- **[Runtime Settings]({{% relref "features/runtime-settings" %}})** - Configure application settings via the web UI without restarting.
- **[Quantization]({{% relref "features/quantization" %}})** - Quantize models for smaller memory footprint.
- **[Fine-tuning]({{% relref "features/fine-tuning" %}})** - Fine-tune models on your own data.
- **[Authentication]({{% relref "features/authentication" %}})** - Protect the API with keys.
For operator-facing runtime, proxy, and monitoring concerns (middleware, cloud and MITM proxies, backend monitor), see the [Operations]({{% relref "operations" %}}) section.
## Getting Started
To start using these features, make sure you have [LocalAI installed](/installation/) and have [downloaded some models](/getting-started/models/). Then explore the feature pages above to learn how to use each capability.
To start using these features, make sure you have [LocalAI installed]({{% relref "getting-started/install" %}}) and have [downloaded some models]({{% relref "getting-started/models" %}}). Then explore the feature pages above to learn how to use each capability.

129
docs/content/features/agent-actions.md Normal file
View File

@@ -0,0 +1,129 @@
+++
disableToc = false
title = "Agent actions"
weight = 21
url = '/features/agent-actions'
+++
Actions are the tools an agent can call while it reasons: search the web, open a browser, read or write a GitHub repository, generate an image, remember a fact, and so on. This page lists the actions that ship with LocalAI (through the embedded [LocalAGI](https://github.com/mudler/LocalAGI) library) and explains how to enable one on an agent.
For an end-to-end first run, see {{% relref "getting-started/first-agent" %}}. For the agent platform overview, see {{% relref "features/agents" %}}.
## Enabling an action on an agent
Actions are attached per agent, not globally.
- **In the web UI:** open the Agents page, edit an agent, and add actions in its configuration. Each action exposes its own configuration fields (for example an API token or a target repository) that you fill in when you add it.
- **In an agent config (JSON):** an agent's exported configuration lists its enabled actions and their settings, so an imported agent carries its actions with it.
Some actions need credentials or external configuration to work (a GitHub token, SMTP server details, a Telegram bot token). An action that is enabled but missing its required configuration will fail at call time, not when you add it. If an imported agent will not run, a missing action credential is one of the things to check (see the checklist in {{% relref "getting-started/first-agent" %}}).
## Discovering the current catalog at runtime
The set of built-in actions is served by the API, so it always reflects the version you are running:
```bash
curl http://localhost:8080/api/agents/actions
```
This returns the list of available action names. Two related endpoints describe and run a single action:
- `POST /api/agents/actions/{name}/definition` returns the action's parameter/configuration schema (send a JSON body with an optional `config` object).
- `POST /api/agents/actions/{name}/run` executes the action directly (JSON body with `config` and `params`), which is useful for testing an action's configuration outside an agent conversation.
## Custom actions
You can add your own actions without rebuilding LocalAI by pointing it at a directory of custom action definitions:
```bash
LOCALAI_AGENT_POOL_CUSTOM_ACTIONS_DIR=/path/to/custom-actions local-ai run
```
Actions found in that directory become available to agents alongside the built-in ones.
## Built-in actions
The actions below ship with LocalAI. The exact catalog can change between releases, so treat `GET /api/agents/actions` (and the Agents UI action picker) as the source of truth for the version you are running.
### Web and knowledge
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `search` | Web search. | Search provider configuration. |
| `browse` | Browse a web page (fetch and read its content). | None. |
| `scraper` | Scrape structured content from a page. | None. |
| `wikipedia` | Look up an article on Wikipedia. | None. |
### Content generation
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `generate_image` | Generate an image (through LocalAI's own image endpoint). | An installed image-generation model. |
| `generate_song` | Generate audio/music. | An installed audio-generation model. |
| `generate_pdf` | Produce a PDF document. | None. |
### Memory
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `add_to_memory` | Store a fact in the agent's memory. | Agent memory/knowledge base enabled. |
| `list_memory` | List stored memories. | Agent memory enabled. |
| `search_memory` | Search stored memories. | Agent memory enabled. |
| `remove_from_memory` | Remove a stored memory. | Agent memory enabled. |
### Reminders
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `set_reminder` | Set a reminder. | None. |
| `list_reminders` | List reminders. | None. |
| `remove_reminder` | Remove a reminder. | None. |
### Messaging and notifications
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `send-mail` | Send an email. | SMTP server and credentials. |
| `send-telegram-message` | Send a Telegram message. | A Telegram bot token. |
| `twitter-post` | Post to X/Twitter. | X/Twitter API credentials. |
| `webhook` | Call an outbound webhook. | The target webhook URL. |
### GitHub
All GitHub actions authenticate with a GitHub token and (for most) a target owner/repository.
| Action | What it does |
|--------|--------------|
| `github-issue-opener` | Open an issue. |
| `github-issue-editor` | Edit an issue. |
| `github-issue-closer` | Close an issue. |
| `github-issue-reader` | Read an issue. |
| `github-issue-commenter` | Comment on an issue. |
| `github-issue-searcher` | Search issues. |
| `github-issue-labeler` | Label an issue. |
| `github-pr-reader` | Read a pull request. |
| `github-pr-commenter` | Comment on a pull request. |
| `github-pr-reviewer` | Review a pull request. |
| `github-pr-creator` | Create a pull request. |
| `github-readme` | Read a repository README. |
| `github-repository-get-content` | Get a file's content. |
| `github-get-all-repository-content` | Get all repository content. |
| `github-repository-list-files` | List files in a repository. |
| `github-repository-search-files` | Search files in a repository. |
| `github-repository-create-or-update-content` | Create or update a file. |
### Other
| Action | What it does | Typically requires |
|--------|--------------|--------------------|
| `shell-command` | Run a shell command. | Enable with care; grants command execution. |
| `call_agents` | Delegate to another agent. | Another configured agent. |
| `pikvm_power_control` | Control power through a PiKVM device. | PiKVM endpoint and credentials. |
| `counter` | A simple counter (example/utility action). | None. |
| `custom` | A user-defined custom action. | See [Custom actions](#custom-actions). |
## See also
- {{% relref "features/agents" %}} for the agent platform overview.
- {{% relref "getting-started/first-agent" %}} for a full walkthrough, including the "will not run" checklist.
- {{% relref "features/mcp" %}} for attaching external MCP tool servers to an agent (an alternative to built-in actions).

115
docs/content/features/agents.md
View File

@@ -1,26 +1,40 @@
+++
disableToc = false
title = "Agents"
weight = 21
weight = 20
url = '/features/agents'
+++
![The in-process agent loop: agents call LocalAI's own chat API in a loop, streaming progress over SSE](/images/diagrams/agents-loop.png)
LocalAI includes a built-in agent platform powered by [LocalAGI](https://github.com/mudler/LocalAGI). Agents are autonomous AI entities that can reason, use tools, maintain memory, and interact with external services — all running locally as part of the LocalAI process.
LocalAI includes a built-in agent platform powered by [LocalAGI](https://github.com/mudler/LocalAGI). Agents are autonomous AI entities that can reason, use tools, maintain memory, and interact with external services, all running locally as part of the LocalAI process.
LocalAGI is embedded in LocalAI. There is nothing separate to install or run.
{{% notice info %}}
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
- **Agents** (this page): autonomous agents you build that reason, use tools, and act on their own. Start here if you want to create an agent.
- **LocalAI Assistant** ({{% relref "features/localai-assistant" %}}): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
- **MCP** ({{% relref "features/mcp" %}}): a way to give a model external tools through the Model Context Protocol.
{{% /notice %}}
{{% notice tip %}}
New to agents? The [Build your first agent]({{% relref "getting-started/first-agent" %}}) walkthrough takes you from an empty Agents page to an agent that answers a message and uses one tool.
{{% /notice %}}
## Overview
The agent system provides:
- **Autonomous agents** with configurable goals, personalities, and capabilities
- **Tool/Action support** — agents can execute actions (web search, code execution, API calls, etc.)
- **Knowledge base (RAG)** — per-agent collections with document upload, chunking, and semantic search
- **Skills system** — reusable skill definitions that agents can leverage, with git-based skill repositories
- **SSE streaming** — real-time chat with agents via Server-Sent Events
- **Import/Export** — share agent configurations as JSON files
- **Agent Hub** — browse and download ready-made agents from [agenthub.localai.io](https://agenthub.localai.io)
- **Web UI** — full management interface for creating, editing, chatting with, and monitoring agents
- **Tool/Action support** - agents can execute actions (web search, code execution, API calls, etc.)
- **Knowledge base (RAG)** - per-agent collections with document upload, chunking, and semantic search
- **Skills system** - reusable skill definitions that agents can leverage, with git-based skill repositories
- **SSE streaming** - real-time chat with agents via Server-Sent Events
- **Import/Export** - share agent configurations as JSON files
- **Agent Hub** - browse and download ready-made agents from [agenthub.localai.io](https://agenthub.localai.io)
- **Web UI** - full management interface for creating, editing, chatting with, and monitoring agents
## Getting Started
@@ -43,7 +57,7 @@ You can import agent configurations from JSON files:
1. Download an agent configuration from the [Agent Hub](https://agenthub.localai.io) or export one from another LocalAI instance
2. On the **Agents** page, click **Import**
3. Select the JSON file — you'll be taken to the edit form to review and adjust the configuration before saving
3. Select the JSON file - you'll be taken to the edit form to review and adjust the configuration before saving
4. Click **Create Agent** to finalize the import
## Configuration
@@ -77,7 +91,7 @@ All agent-related settings can be configured via environment variables:
### Knowledge Base Storage
By default, the knowledge base uses **chromem** — an in-process vector store that requires no external dependencies. For production deployments with larger knowledge bases, you can switch to **PostgreSQL** with pgvector support:
By default, the knowledge base uses **chromem** - an in-process vector store that requires no external dependencies. For production deployments with larger knowledge bases, you can switch to **PostgreSQL** with pgvector support:
```bash
LOCALAI_AGENT_POOL_VECTOR_ENGINE=postgres
@@ -88,7 +102,7 @@ The PostgreSQL image `quay.io/mudler/localrecall:v0.5.2-postgresql` is pre-confi
#### Connection safety timeouts (PostgreSQL only)
The embedded vector store sets per-connection timeouts so a single stuck or corrupt index can never hold a lock indefinitely and stall every other collection operation. Safe defaults are applied automatically — you only need to set these to override them:
The embedded vector store sets per-connection timeouts so a single stuck or corrupt index can never hold a lock indefinitely and stall every other collection operation. Safe defaults are applied automatically - you only need to set these to override them:
| Variable | Default | Description |
|----------|---------|-------------|
@@ -173,16 +187,67 @@ volumes:
Each agent has its own configuration that controls its behavior. Key settings include:
- **Name** — unique identifier for the agent
- **Model** — the LLM model the agent uses for reasoning
- **System Prompt** — defines the agent's personality and instructions
- **Actions** — tools the agent can use (web search, code execution, etc.)
- **Connectors** — external integrations (Slack, Discord, etc.)
- **Knowledge Base** — collections of documents for RAG
- **MCP Servers** — Model Context Protocol servers for additional tool access
- **Name** - unique identifier for the agent
- **Model** - the LLM model the agent uses for reasoning
- **System Prompt** - defines the agent's personality and instructions
- **Actions** - tools the agent can use (web search, code execution, etc.). See the {{% relref "features/agent-actions" %}} for the full catalog of built-in actions and how to configure each one.
- **Connectors** - external integrations (Slack, Discord, etc.)
- **Knowledge Base** - collections of documents for RAG
- **MCP Servers** - Model Context Protocol servers for additional tool access
The pool-level defaults (API URL, API key, models) can be set via environment variables. Individual agents can further override these in their configuration, allowing them to use different LLM providers (OpenAI, other LocalAI instances, etc.) on a per-agent basis.
## Skills
Skills are reusable instruction sets (a name, a description, and the skill's content, optionally with attached resource files) that an agent can draw on while it works. They can be authored directly or imported from git-based skill repositories, so a set of skills can be shared across agents and machines.
{{% notice warning %}}
Skills are **disabled by default**. The skills service only runs when you start LocalAI with `LOCALAI_AGENT_POOL_ENABLE_SKILLS=true`. With the default `LOCALAI_AGENT_POOL_ENABLE_SKILLS=false`, the Skills UI and the `/api/agents/skills` endpoints are inactive, and an imported agent that expects a skill will not find it.
{{% /notice %}}
### Enable skills
Start LocalAI with the skills service turned on:
```bash
LOCALAI_AGENT_POOL_ENABLE_SKILLS=true local-ai run
```
In Docker, add the same variable to the container environment.
### Create a skill
With skills enabled, open the **Agents** section in the web interface and go to the Skills area. Create a skill by giving it:
- a **name** (used to reference the skill),
- a **description** (what the skill is for),
- the skill **content** (the instructions the agent follows when the skill applies),
- optionally, resource files the skill needs.
The same operation is available over REST:
```bash
curl http://localhost:8080/api/agents/skills \
-H "Content-Type: application/json" \
-d '{
"name": "changelog-writer",
"description": "Write a release changelog from a list of merged PRs",
"content": "When asked for a changelog, group the PRs by type (feature, fix, docs) and write one concise bullet per PR."
}'
```
You can also import a skill archive with `POST /api/agents/skills/import`, or add a git skill repository so its skills are pulled in.
### Use a skill
Once a skill exists and the skills service is enabled, agents can use it as part of their reasoning. List the skills currently available with:
```bash
curl http://localhost:8080/api/agents/skills
```
If a skill you expect is missing, confirm LocalAI was started with `LOCALAI_AGENT_POOL_ENABLE_SKILLS=true`.
## API Endpoints
All agent endpoints are grouped under `/api/agents/`:
@@ -310,10 +375,10 @@ curl -N http://localhost:8080/api/agents/my-agent/sse
The SSE stream emits the following event types:
- `json_message` — agent/user messages
- `json_message_status` — processing status updates (`processing` / `completed`)
- `status` — system messages (reasoning steps, action results)
- `json_error` — error notifications
- `json_message` - agent/user messages
- `json_message_status` - processing status updates (`processing` / `completed`)
- `status` - system messages (reasoning steps, action results)
- `json_error` - error notifications
## Generated Files and Outputs
@@ -328,7 +393,7 @@ Some agent actions (image generation, PDF creation, audio synthesis) produce fil
This design ensures that:
- Actions can write files to any directory they need
- The file-serving endpoint is confined to a single trusted directory — no arbitrary filesystem access
- The file-serving endpoint is confined to a single trusted directory - no arbitrary filesystem access
- Symlink traversal is blocked via `filepath.EvalSymlinks` validation
### Accessing Generated Files
@@ -375,7 +440,7 @@ This returns a JSON object including the `outputsDir` field.
Agents run in-process within LocalAI. By default, each agent calls back into LocalAI's own API (`http://127.0.0.1:<port>/v1/chat/completions`) for LLM inference. This means:
- No external dependencies — everything runs in a single binary
- No external dependencies - everything runs in a single binary
- Agents use the same models loaded in LocalAI
- Per-agent overrides allow pointing individual agents to external providers
- Agent state is persisted to disk and restored on restart

14
docs/content/features/api-discovery.md
View File

@@ -1,13 +1,13 @@
+++
title = "API Discovery & Instructions"
weight = 27
weight = 15
toc = true
description = "Programmatic API discovery for agents, tools, and automation"
tags = ["API", "Agents", "Instructions", "Configuration", "Advanced"]
categories = ["Features"]
+++
LocalAI exposes a set of discovery endpoints that let external agents, coding assistants, and automation tools programmatically learn what the instance can do and how to control it — without reading documentation ahead of time.
LocalAI exposes a set of discovery endpoints that let external agents, coding assistants, and automation tools programmatically learn what the instance can do and how to control it - without reading documentation ahead of time.
## Quick start
@@ -58,7 +58,7 @@ Returns the instance version, all available endpoint URLs (flat and categorized)
}
```
The `capabilities` object reflects the current runtime configuration — for example, `mcp` is only `true` if MCP is enabled, and `agents` is `true` only if the agent pool is running.
The `capabilities` object reflects the current runtime configuration - for example, `mcp` is only `true` if MCP is enabled, and `agents` is `true` only if the agent pool is running.
## Instructions API
@@ -104,7 +104,7 @@ Returns a compact list of instruction areas:
| `model-management` | Browse gallery, install, delete, manage models and backends |
| `config-management` | Discover, read, and modify model config fields with VRAM estimation |
| `monitoring` | System metrics, backend status, system information |
| `mcp` | Model Context Protocol — tool-augmented chat with MCP servers |
| `mcp` | Model Context Protocol - tool-augmented chat with MCP servers |
| `agents` | Agent task and job management |
| `video` | Video generation from text prompts |
@@ -128,7 +128,7 @@ curl http://localhost:8080/api/instructions/config-management?format=json
`GET /v1/models/capabilities`
An additive, LocalAI-specific superset of `/v1/models`. It returns the same set of models but enriches each entry with the **capabilities** the model supports and the **input/output modalities** it accepts and produces. Use it to decide, before sending a request, whether a given model can take an image, audio, or video attachment directly — or whether the input needs converting/transcribing first.
An additive, LocalAI-specific superset of `/v1/models`. It returns the same set of models but enriches each entry with the **capabilities** the model supports and the **input/output modalities** it accepts and produces. Use it to decide, before sending a request, whether a given model can take an image, audio, or video attachment directly - or whether the input needs converting/transcribing first.
Because it is purely additive, clients that only understand `/v1/models` keep working unchanged; they simply never call this route.
@@ -158,8 +158,8 @@ curl http://localhost:8080/v1/models/capabilities
}
```
- **`capabilities`** — canonical usecase strings (e.g. `chat`, `vision`, `transcript`, `tts`, `embeddings`, `image`, `video`) plus the modifiers `tools` and `thinking`.
- **`input_modalities` / `output_modalities`** — the subsets of `{text, image, audio, video}` the model accepts and produces. LocalAI combines usecase-based inference, backend settings such as vLLM `limit_mm_per_prompt`, and explicit model-level `known_input_modalities` / `known_output_modalities`. The explicit fields cover distinctions a usecase cannot express, such as a video model that also accepts speech.
- **`capabilities`** - canonical usecase strings (e.g. `chat`, `vision`, `transcript`, `tts`, `embeddings`, `image`, `video`) plus the modifiers `tools` and `thinking`.
- **`input_modalities` / `output_modalities`** - the subsets of `{text, image, audio, video}` the model accepts and produces. LocalAI combines usecase-based inference, backend settings such as vLLM `limit_mm_per_prompt`, and explicit model-level `known_input_modalities` / `known_output_modalities`. The explicit fields cover distinctions a usecase cannot express, such as a video model that also accepts speech.
The same query parameters as `/v1/models` are honored (`filter`, `excludeConfigured`), and the same per-user model allowlist is applied when authentication is enabled.

14
docs/content/features/audio-classification.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Sound Classification"
weight = 18
weight = 32
url = "/features/audio-classification/"
+++
@@ -21,7 +21,7 @@ Content-Type: multipart/form-data
| Field | Type | Description |
|-------|------|-------------|
| `file` | file (required) | audio file in any format `ffmpeg` accepts |
| `model` | string (required) | name of the sound-classification-capable model (e.g. `ced-base`) |
| `model` | string (required) | name of the sound-classification-capable model (e.g. `ced-base-f16`) |
| `top_k` | int | number of top tags to return (0 = backend default) |
| `threshold` | float | drop tags scoring below this value |
@@ -29,7 +29,7 @@ Content-Type: multipart/form-data
```json
{
"model": "ced-base",
"model": "ced-base-f16",
"detections": [
{"index": 23, "label": "Baby cry, infant cry", "score": 0.87},
{"index": 22, "label": "Crying, sobbing", "score": 0.41}
@@ -41,11 +41,17 @@ Detections are returned in score-descending order. Scores are per-class probabil
## Example
First install a classification model from the gallery (the example below uses `ced-base-f16`):
```bash
local-ai run ced-base-f16
```
```bash
curl http://localhost:8080/v1/audio/classification \
-H "Content-Type: multipart/form-data" \
-F file="@/path/to/clip.wav" \
-F model="ced-base" \
-F model="ced-base-f16" \
-F top_k=10
```

30
docs/content/features/audio-diarization.md
View File

@@ -1,18 +1,18 @@
+++
disableToc = false
title = "Speaker Diarization"
weight = 17
weight = 33
url = "/features/audio-diarization/"
+++
![Diarization: segment, embed, and cluster (or a single ASR pass) into speaker-labelled segments](/images/diagrams/diarization-pipeline.png)
Speaker diarization answers the question **"who spoke when?"** — given an audio clip with multiple speakers, it returns time-stamped segments labelled with a stable speaker ID (`SPEAKER_00`, `SPEAKER_01`, …).
Speaker diarization answers the question **"who spoke when?"** - given an audio clip with multiple speakers, it returns time-stamped segments labelled with a stable speaker ID (`SPEAKER_00`, `SPEAKER_01`, …).
LocalAI exposes this through the `/v1/audio/diarization` endpoint, modelled after `/v1/audio/transcriptions`. Two backends are supported today:
- **[sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)** — pyannote-3.0 segmentation + a speaker-embedding extractor (3D-Speaker, NeMo, WeSpeaker) + fast clustering. Pure diarization — no transcription cost. Recommended when you only need speaker turns.
- **[vibevoice.cpp](https://github.com/microsoft/VibeVoice)** — produces speaker-labelled segments as a by-product of its long-form ASR pass, so you can optionally get a transcript per segment for free.
- **[sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx)** - pyannote-3.0 segmentation + a speaker-embedding extractor (3D-Speaker, NeMo, WeSpeaker) + fast clustering. Pure diarization - no transcription cost. Recommended when you only need speaker turns.
- **[vibevoice.cpp](https://github.com/microsoft/VibeVoice)** - produces speaker-labelled segments as a by-product of its long-form ASR pass, so you can optionally get a transcript per segment for free.
Because diarization is exposed as a regular OpenAI-compatible endpoint, any HTTP client works. There is no Python dependency on pyannote or NeMo on the consumer side.
@@ -37,7 +37,7 @@ Content-Type: multipart/form-data
| `include_text` | bool | when the backend can emit per-segment transcript for free, populate it |
| `response_format` | string | `json` (default), `verbose_json`, or `rttm` |
### Response — `json` (default)
### Response - `json` (default)
Compact payload, no transcription, no per-speaker summary:
@@ -55,7 +55,7 @@ Compact payload, no transcription, no per-speaker summary:
`speaker` is the normalized, zero-padded label clients should display. `label` preserves the raw backend-emitted ID for clients that maintain their own speaker dictionary.
### Response — `verbose_json`
### Response - `verbose_json`
Adds per-speaker totals and (when the backend supports it and `include_text=true`) the per-segment transcript:
@@ -76,7 +76,7 @@ Adds per-speaker totals and (when the backend supports it and `include_text=true
}
```
### Response — `rttm`
### Response - `rttm`
NIST RTTM, the standard interchange format used by `pyannote.metrics` / `dscore`:
@@ -89,15 +89,23 @@ Returned as `Content-Type: text/plain; charset=utf-8`.
## Quick start
First install a diarization-capable model from the gallery. The example below uses `vibevoice-cpp-asr`, which serves the vibevoice.cpp backend and returns speaker-labelled segments (and, optionally, a transcript):
```bash
local-ai run vibevoice-cpp-asr
```
```bash
curl http://localhost:8080/v1/audio/diarization \
-H "Content-Type: multipart/form-data" \
-F file="@meeting.wav" \
-F model="pyannote-diarization" \
-F model="vibevoice-cpp-asr" \
-F num_speakers=3
```
## Backend setup — sherpa-onnx (pure diarization)
The sections below show how to configure the two supported backends by hand when you want full control over the segmentation and embedding models.
## Backend setup - sherpa-onnx (pure diarization)
Sherpa-onnx needs two ONNX models: pyannote segmentation and a speaker-embedding extractor. Place them under your LocalAI models directory and reference them from the YAML:
@@ -119,7 +127,7 @@ known_usecases:
Both `model:` and `diarize.embedding_model=` are resolved relative to the LocalAI models directory.
## Backend setup — vibevoice.cpp (diarization + ASR)
## Backend setup - vibevoice.cpp (diarization + ASR)
vibevoice.cpp's ASR mode emits `[{Start, End, Speaker, Content}]` natively, so a single pass gives both diarization and transcription:
@@ -150,7 +158,7 @@ curl http://localhost:8080/v1/audio/diarization \
## Notes
- **Speaker identity across files**: speaker IDs (`SPEAKER_00`, `SPEAKER_01`, …) are local to each request. To track the same person across multiple recordings, combine `/v1/audio/diarization` with `/v1/voice/embed` (speaker embedding) and maintain your own embedding store.
- **Hints vs. forces**: `num_speakers` overrides clustering when set; `min_speakers` / `max_speakers` are advisory and only honored by backends that expose a range hint. vibevoice.cpp ignores them — its model picks the count itself.
- **Hints vs. forces**: `num_speakers` overrides clustering when set; `min_speakers` / `max_speakers` are advisory and only honored by backends that expose a range hint. vibevoice.cpp ignores them - its model picks the count itself.
- **Sample rate**: input is automatically converted to 16 kHz mono via ffmpeg before the backend sees it; sherpa-onnx pyannote-3.0 requires 16 kHz.
## See also

12
docs/content/features/audio-to-text.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Audio to Text"
weight = 16
weight = 30
url = "/features/audio-to-text/"
+++
@@ -12,12 +12,12 @@ The transcription endpoint allows to convert audio files to text. The endpoint s
- **moonshine**: Ultra-fast transcription engine optimized for low-end devices
- **faster-whisper**: Fast Whisper implementation with CTranslate2
- **[parakeet-cpp](https://github.com/mudler/parakeet.cpp)**: A C++/ggml port of NVIDIA NeMo Parakeet (FastConformer TDT/CTC/RNNT/hybrid). Runs quantized GGUFs on CPU or GPU, emits word-level timestamps, and supports cache-aware streaming (the `realtime_eou` model surfaces end-of-utterance events).
- **llama-cpp**: Route transcription to any multimodal-audio GGUF model served by the `llama-cpp` backend (e.g. [Qwen3-ASR](https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF), Voxtral, Qwen2-Audio). Under the hood the request is converted into a chat completion with the audio attached via the model's audio encoder — the same path the upstream llama.cpp server uses. Set `backend: llama-cpp` in the model YAML and point `mmproj` at the matching audio encoder.
- **llama-cpp**: Route transcription to any multimodal-audio GGUF model served by the `llama-cpp` backend (e.g. [Qwen3-ASR](https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF), Voxtral, Qwen2-Audio). Under the hood the request is converted into a chat completion with the audio attached via the model's audio encoder - the same path the upstream llama.cpp server uses. Set `backend: llama-cpp` in the model YAML and point `mmproj` at the matching audio encoder.
- **voxtral**: Voxtral-family models served by a dedicated backend
The endpoint input supports all the audio formats supported by `ffmpeg`.
> Looking for **"who spoke when"** instead of a flat transcript? See [Speaker Diarization](/features/audio-diarization/) — `/v1/audio/diarization` returns time-stamped speaker segments and supports the `rttm` format used by `pyannote.metrics`.
> Looking for **"who spoke when"** instead of a flat transcript? See [Speaker Diarization](/features/audio-diarization/) - `/v1/audio/diarization` returns time-stamped speaker segments and supports the `rttm` format used by `pyannote.metrics`.
## Usage
@@ -107,7 +107,7 @@ In addition to `file` and `model`, the endpoint accepts the following multipart
| `timestamp_granularities[]` | Multi-value form field: `word` and/or `segment`. Honored when the backend produces the requested granularity. |
| `response_format` | One of `json` (default for backwards-compat), `verbose_json`, `text`, `srt`, `vtt`, `lrc`. |
| `stream` | When `true`, the endpoint emits an SSE stream of `transcript.text.delta` events followed by a final `transcript.text.done` event. |
| `diarize` | LocalAI extension — speaker diarization (whisper.cpp only). |
| `diarize` | LocalAI extension - speaker diarization (whisper.cpp only). |
The response body for `verbose_json` includes `text`, `language`, `duration`, and `segments[]` (with `speaker` populated when diarization is enabled).
@@ -133,7 +133,7 @@ data: {"type":"transcript.text.done","text":"And so, my fellow Americans..."}
data: [DONE]
```
Backends that do not natively stream tokens fall back to emitting one delta plus a done event with the full text — the SSE contract is identical either way.
Backends that do not natively stream tokens fall back to emitting one delta plus a done event with the full text - the SSE contract is identical either way.
## Using the llama-cpp backend with an audio-capable model
@@ -220,4 +220,4 @@ By default each request runs on its own. Raise `batch_max_size` (for example 4 t
## See also
- [Audio Transform]({{< relref "audio-transform.md" >}}) — clean up the audio (echo cancellation, noise suppression, dereverberation) before passing it to a transcription model.
- [Audio Transform]({{< relref "audio-transform.md" >}}) - clean up the audio (echo cancellation, noise suppression, dereverberation) before passing it to a transcription model.

30
docs/content/features/audio-transform.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Audio Transform"
weight = 17
weight = 34
url = "/features/audio-transform/"
+++
@@ -9,7 +9,7 @@ url = "/features/audio-transform/"
The audio-transform endpoints take **audio in** and emit **audio out**, optionally
conditioned on a second reference audio signal. The category is generic by
design — concrete operations include joint **acoustic echo cancellation +
design - concrete operations include joint **acoustic echo cancellation +
noise suppression + dereverberation** (LocalVQE), voice conversion (reference
= target speaker), pitch shifting, audio super-resolution, and so on.
@@ -22,14 +22,14 @@ is a derivative of the Microsoft DeepVQE paper.
Every audio-transform request carries:
- **`audio`** — the primary input file (required).
- **`reference`** — an auxiliary signal whose meaning is backend-specific (optional).
- **`audio`** - the primary input file (required).
- **`reference`** - an auxiliary signal whose meaning is backend-specific (optional).
- For echo cancellation: the loopback / far-end signal played through the speakers.
- For voice conversion: the target speaker's reference clip.
- For pitch / style transfer: a tonal or style reference.
- When omitted, the backend treats it as silence and degrades gracefully (LocalVQE,
for example, does denoise + dereverb only when ref is empty).
- **`params`** — a generic `key=value` map forwarded to the backend.
- **`params`** - a generic `key=value` map forwarded to the backend.
- LocalVQE keys: `noise_gate=true|false`, `noise_gate_threshold_dbfs=<float>`.
This shape mirrors WebRTC's `ProcessStream(near)` / `ProcessReverseStream(far)`
@@ -38,23 +38,29 @@ AEC challenge 2-channel WAV convention.
## Batch endpoint
`POST /audio/transformations` (alias `POST /audio/transform`) — multipart
`POST /audio/transformations` (alias `POST /audio/transform`) - multipart
form-data, returns audio bytes.
| Field | Type | Required | Notes |
|---|---|---|---|
| `model` | string | yes | Audio-transform model id (e.g. `localvqe`) |
| `model` | string | yes | Audio-transform model id (e.g. `localvqe-v1.3-4.8m`) |
| `audio` | file | yes | Primary input audio |
| `reference` | file | no | Optional auxiliary signal |
| `response_format` | string | no | `wav` (default), `mp3`, `ogg`, `flac` |
| `sample_rate` | int | no | Desired output sample rate |
| `params[<key>]` | string | no | Repeated; forwarded to backend |
First install an audio-transform model from the gallery (the examples below use `localvqe-v1.3-4.8m`):
```bash
local-ai run localvqe-v1.3-4.8m
```
Example (LocalVQE: cancel echo, suppress noise, gate residual):
```bash
curl -X POST http://localhost:8080/audio/transformations \
-F model=localvqe \
-F model=localvqe-v1.3-4.8m \
-F audio=@mic.wav \
-F reference=@loopback.wav \
-F 'params[noise_gate]=true' \
@@ -67,7 +73,7 @@ the operation reduces to noise suppression + dereverberation.
## Streaming endpoint
`GET /audio/transformations/stream` — bidirectional WebSocket. The first
`GET /audio/transformations/stream` - bidirectional WebSocket. The first
client message is a JSON envelope; subsequent client messages are binary
PCM frames; server emits binary PCM frames at the same cadence.
@@ -78,7 +84,7 @@ PCM frames; server emits binary PCM frames at the same cadence.
```json
{
"type": "session.update",
"model": "localvqe",
"model": "localvqe-v1.3-4.8m",
"sample_format": "S16_LE",
"sample_rate": 16000,
"frame_samples": 256,
@@ -125,8 +131,8 @@ A reasonable starting point is `-50` dBFS.
## Configuring a model
LocalVQE ships several weight releases in the gallery: `localvqe-v1.3-4.8m`
(current default — best quality), `localvqe-v1.2-1.3m` and `localvqe-v1.1-1.3m`
(compact, ~¼ the per-hop cost — good for low-core or power-constrained hosts).
(current default - best quality), `localvqe-v1.2-1.3m` and `localvqe-v1.1-1.3m`
(compact, ~¼ the per-hop cost - good for low-core or power-constrained hosts).
All share the same backend and request API; only the `model` filename differs.
```yaml

28
docs/content/features/authentication.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Authentication & Authorization"
weight = 26
weight = 85
url = '/features/authentication'
+++
@@ -26,7 +26,7 @@ Clients provide the key via any of these methods:
- `xi-api-key: <key>` header
- `token` cookie
Legacy API keys grant **full admin access** — there is no role separation. For multi-user deployments with role-based access, use the user authentication system instead.
Legacy API keys grant **full admin access** - there is no role separation. For multi-user deployments with role-based access, use the user authentication system instead.
API keys can also be managed at runtime through the [Runtime Settings]({{%relref "features/runtime-settings" %}}) interface.
@@ -39,7 +39,7 @@ The user authentication system provides:
- **Session-based authentication** with secure cookies
- **OAuth login** (GitHub) and **OIDC single sign-on** (Keycloak, Google, Okta, Authentik, etc.)
- **Per-user API keys** for programmatic access
- **Admin route gating** — management endpoints are restricted to admins
- **Admin route gating** - management endpoints are restricted to admins
- **Per-user usage tracking** with token consumption metrics
### Enabling Authentication
@@ -74,7 +74,7 @@ localai run
| Environment Variable | Default | Description |
|---|---|---|
| `LOCALAI_AUTH` | `false` | Enable user authentication and authorization |
| `LOCALAI_AUTH_DATABASE_URL` | `{DataPath}/database.db` | Database URL — `postgres://...` for PostgreSQL, or a file path for SQLite |
| `LOCALAI_AUTH_DATABASE_URL` | `{DataPath}/database.db` | Database URL - `postgres://...` for PostgreSQL, or a file path for SQLite |
| `GITHUB_CLIENT_ID` | | GitHub OAuth App Client ID (auto-enables auth when set) |
| `GITHUB_CLIENT_SECRET` | | GitHub OAuth App Client Secret |
| `LOCALAI_OIDC_ISSUER` | | OIDC issuer URL for auto-discovery (e.g. `https://accounts.google.com`) |
@@ -111,7 +111,7 @@ When disabled:
There are two roles:
- **Admin**: Full access to all endpoints, including model management, backend configuration, system settings, traces, agents, and user management.
- **User**: Access to inference endpoints only — chat completions, embeddings, image/video/audio generation, TTS, MCP chat, and their own usage statistics.
- **User**: Access to inference endpoints only - chat completions, embeddings, image/video/audio generation, TTS, MCP chat, and their own usage statistics.
The **first user** to sign in is automatically assigned the admin role. Additional users can be promoted to admin via the admin user management API or by setting `LOCALAI_ADMIN_EMAIL` to their email address.
@@ -146,7 +146,7 @@ curl -X DELETE http://localhost:8080/api/auth/admin/invites/<invite-id> \
curl http://localhost:8080/api/auth/invite/<code>/check
```
Share the invite URL (`/invite/<code>`) with the user. When they open it, the registration form is pre-filled with the invite code. Invite codes are single-use — once consumed, they cannot be reused. Expired or used invites are rejected.
Share the invite URL (`/invite/<code>`) with the user. When they open it, the registration form is pre-filled with the invite code. Invite codes are single-use - once consumed, they cannot be reused. Expired or used invites are rejected.
For GitHub OAuth, the invite code is passed as a query parameter to the login URL (`/api/auth/github/login?invite_code=<code>`) and stored in a cookie during the OAuth flow.
@@ -191,7 +191,7 @@ When auth is enabled, the React UI sidebar dynamically shows/hides sections base
- **All users see**: Home, Chat, Images, Video, TTS, Sound, Talk, Usage, API docs link
- **Admins also see**: Install Models, Agents section (Agents, Skills, Memory, MCP CI Jobs), System section (Backends, Traces, Swarm, System, Settings)
Admin-only pages are also protected at the router level — navigating directly to an admin URL redirects non-admin users to the home page.
Admin-only pages are also protected at the router level - navigating directly to an admin URL redirects non-admin users to the home page.
### GitHub OAuth Setup
@@ -228,7 +228,7 @@ LOCALAI_OIDC_ISSUER=https://authentik.example.com/application/o/localai/
LOCALAI_OIDC_ISSUER=https://your-org.okta.com
```
For OIDC, invite codes work the same way as GitHub OAuth — the invite code is passed as a query parameter to the login URL (`/api/auth/oidc/login?invite_code=<code>`) and stored in a cookie during the OAuth flow.
For OIDC, invite codes work the same way as GitHub OAuth - the invite code is passed as a query parameter to the login URL (`/api/auth/oidc/login?invite_code=<code>`) and stored in a cookie during the OAuth flow.
### User API Keys
@@ -298,10 +298,10 @@ curl "http://localhost:8080/api/auth/admin/usage?period=month&user_id=<user-id>"
```
**Period values:**
- `day` — last 24 hours, bucketed by hour
- `week` — last 7 days, bucketed by day
- `month` — last 30 days, bucketed by day (default)
- `all` — all time, bucketed by month
- `day` - last 24 hours, bucketed by hour
- `week` - last 7 days, bucketed by day
- `month` - last 30 days, bucketed by day (default)
- `all` - all time, bucketed by month
**Response format:**
@@ -410,7 +410,7 @@ Usage rows recorded before this feature have no `source` column. On startup, `In
Legacy API keys and user authentication can be used simultaneously. When both are configured:
1. User sessions and user API keys are checked first
2. Legacy API keys are checked as fallback — they grant **admin-level access**
2. Legacy API keys are checked as fallback - they grant **admin-level access**
3. This allows a gradual migration from shared API keys to per-user accounts
## Build Requirements
@@ -425,4 +425,4 @@ GO_TAGS=auth make build
go build -tags auth ./...
```
The default Dockerfile includes `GO_TAGS="auth"`, so all Docker images ship with auth support. When building from source without the `auth` tag, setting `LOCALAI_AUTH=true` has no effect — the system operates without authentication.
The default Dockerfile includes `GO_TAGS="auth"`, so all Docker images ship with auth support. When building from source without the `auth` tag, setting `LOCALAI_AUTH=true` has no effect - the system operates without authentication.

4
docs/content/features/backends.md
View File

@@ -1,7 +1,7 @@
---
title: "Backends"
description: "Learn how to use, manage, and develop backends in LocalAI"
weight: 4
weight: 80
url: "/backends/"
---
@@ -129,7 +129,7 @@ LocalAI supports various types of backends:
- **Text-to-Speech Backends**: For speech synthesis (e.g., piper, Kokoro, VibeVoice, Qwen3-TTS)
- **Sound Generation Backends**: For music and audio generation (e.g., ACE-Step)
- **Sound Classification Backends**: For sound-event classification / audio tagging - identifying everyday sounds like baby cry, glass breaking, alarms (e.g., ced.cpp)
- **Image & Video Generation Backends**: For diffusion and audio-conditioned avatar models (e.g., stable-diffusion.cpp, diffusers, vLLM-Omni, [LongCat-Video]({{%relref "features/longcat-video" %}}))
- **Image & Video Generation Backends**: For diffusion and audio-conditioned avatar models (e.g., stable-diffusion.cpp, diffusers, vLLM-Omni, [LongCat-Video]({{%relref "features/video-generation" %}}))
- **Vision & Detection Backends**: For object detection, segmentation, depth, and face/voice recognition (e.g., rf-detr.cpp, locate-anything.cpp, sam3.cpp, insightface)
- **Audio Processing Backends**: For voice activity detection and audio enhancement (e.g., Silero VAD, LocalVQE)
- **Utility Backends**: For reranking, PII/NER token classification, fine-tuning, quantization, and vector storage (e.g., rerankers, privacy-filter.cpp, TRL, local-store)

2
docs/content/features/constrained_grammars.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Constrained Grammars"
weight = 15
weight = 12
url = "/features/constrained_grammars/"
+++

38
docs/content/features/distributed-mode.md
View File

@@ -1,14 +1,14 @@
+++
disableToc = false
title = "Distributed Mode"
weight = 14
weight = 71
url = "/features/distributed-mode/"
+++
Distributed mode enables horizontal scaling of LocalAI across multiple machines using **PostgreSQL** for state and node registry, and **NATS** for real-time coordination. Unlike the [P2P/federation approach]({{% relref "features/distributed_inferencing" %}}), distributed mode is designed for production deployments and Kubernetes environments where you need centralized management, health monitoring, and deterministic routing.
{{% notice note %}}
Distributed mode requires authentication enabled with a **PostgreSQL** database — SQLite is not supported. This is because the node registry, job store, and other distributed state are stored in PostgreSQL tables.
Distributed mode requires authentication enabled with a **PostgreSQL** database - SQLite is not supported. This is because the node registry, job store, and other distributed state are stored in PostgreSQL tables.
{{% /notice %}}
## Architecture Overview
@@ -17,7 +17,7 @@ Distributed mode requires authentication enabled with a **PostgreSQL** database
**Frontends** are stateless LocalAI instances that receive API requests and route them to worker nodes via the **SmartRouter**. All frontends share state through PostgreSQL and coordinate via NATS.
**Workers** are generic processes that self-register with a frontend. They don't have a fixed backend type — the SmartRouter dynamically installs the required backend via NATS `backend.install` events when a model request arrives.
**Workers** are generic processes that self-register with a frontend. They don't have a fixed backend type - the SmartRouter dynamically installs the required backend via NATS `backend.install` events when a model request arrives.
### Scheduling Algorithm
@@ -36,8 +36,8 @@ Each model gets its own gRPC backend process, so a single worker can serve multi
## Prerequisites
- **PostgreSQL** (with pgvector extension recommended for RAG) — used for node registry, job store, auth, and shared state
- **NATS** server — used for real-time backend lifecycle events and file staging
- **PostgreSQL** (with pgvector extension recommended for RAG) - used for node registry, job store, auth, and shared state
- **NATS** server - used for real-time backend lifecycle events and file staging
- All services must be on the same network (or reachable via configured URLs)
## Quick Start with Docker Compose
@@ -65,7 +65,7 @@ The frontend is a standard LocalAI instance with distributed mode enabled. These
| `--nats-url` | `LOCALAI_NATS_URL` | *(required)* | NATS server URL (e.g., `nats://localhost:4222`) |
| `--registration-token` | `LOCALAI_REGISTRATION_TOKEN` | *(empty)* | Token that workers must provide to register |
| `--registration-require-auth` | `LOCALAI_REGISTRATION_REQUIRE_AUTH` | `false` | Fail startup when distributed mode is enabled but the registration token is empty (node endpoints and worker file-transfer would otherwise be unauthenticated) |
| `--distributed-require-auth` | `LOCALAI_DISTRIBUTED_REQUIRE_AUTH` | `false` | **Umbrella switch.** Implies both `--nats-require-auth` and `--registration-require-auth` — one knob to lock down the NATS bus *and* the registration/file-transfer layer. Set this in production instead of the two granular flags. |
| `--distributed-require-auth` | `LOCALAI_DISTRIBUTED_REQUIRE_AUTH` | `false` | **Umbrella switch.** Implies both `--nats-require-auth` and `--registration-require-auth` - one knob to lock down the NATS bus *and* the registration/file-transfer layer. Set this in production instead of the two granular flags. |
| `--auto-approve-nodes` | `LOCALAI_AUTO_APPROVE_NODES` | `false` | Auto-approve new worker nodes (skip admin approval) |
| `--distributed-shared-models` | `LOCALAI_DISTRIBUTED_SHARED_MODELS` | `false` | Assert that every node mounts the **same** models directory at the **same** path (a shared volume). When `true`, the router skips file staging entirely and workers load models directly from the shared path instead of re-downloading them. See [Shared models directory](#shared-models-directory). |
| `--auth` | `LOCALAI_AUTH` | `false` | **Must be `true`** for distributed mode |
@@ -110,7 +110,7 @@ The same env vars apply to backend workers and `local-ai agent-worker`. If the s
Workers connect with that JWT and seed automatically (shown once; store securely). Override with `LOCALAI_NATS_JWT` / `LOCALAI_NATS_USER_SEED` if needed. Set `LOCALAI_NATS_REQUIRE_AUTH=true` on workers when the bus requires credentials.
When `LOCALAI_NATS_REQUIRE_AUTH=true` and no static credentials are provided, a worker that registers while still **pending admin approval** keeps re-registering (with backoff) until an admin approves it and the frontend mints its JWT — it does not start unauthenticated. This retry is **bounded**: if the node is never approved (or no credentials are minted) after a large number of attempts, the worker exits non-zero so the failure is visible (a crash-looping or failed worker) rather than hanging silently. Minted worker JWTs are also **refreshed automatically** before they expire (the worker re-registers at ~75% of the JWT lifetime), so long-running workers survive past `LOCALAI_NATS_WORKER_JWT_TTL`; the NATS connection picks up the new JWT on its next reconnect. If refresh fails persistently, the worker exits (to restart and re-acquire) rather than drifting toward an expired, unrenewable JWT. Statically configured (`LOCALAI_NATS_JWT`) and service (`LOCALAI_NATS_SERVICE_JWT`) credentials are used as-is and not refreshed.
When `LOCALAI_NATS_REQUIRE_AUTH=true` and no static credentials are provided, a worker that registers while still **pending admin approval** keeps re-registering (with backoff) until an admin approves it and the frontend mints its JWT - it does not start unauthenticated. This retry is **bounded**: if the node is never approved (or no credentials are minted) after a large number of attempts, the worker exits non-zero so the failure is visible (a crash-looping or failed worker) rather than hanging silently. Minted worker JWTs are also **refreshed automatically** before they expire (the worker re-registers at ~75% of the JWT lifetime), so long-running workers survive past `LOCALAI_NATS_WORKER_JWT_TTL`; the NATS connection picks up the new JWT on its next reconnect. If refresh fails persistently, the worker exits (to restart and re-acquire) rather than drifting toward an expired, unrenewable JWT. Statically configured (`LOCALAI_NATS_JWT`) and service (`LOCALAI_NATS_SERVICE_JWT`) credentials are used as-is and not refreshed.
Generate operator/account material with [`scripts/nats-auth-setup.sh`](https://github.com/mudler/LocalAI/blob/master/scripts/nats-auth-setup.sh) (requires [nsc](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nsc)). Configure the NATS server with account resolver JWTs before enabling `LOCALAI_NATS_REQUIRE_AUTH`.
@@ -130,7 +130,7 @@ For multi-host deployments where workers don't share a filesystem, S3-compatible
| `--storage-access-key` | `LOCALAI_STORAGE_ACCESS_KEY` | *(empty)* | S3 access key |
| `--storage-secret-key` | `LOCALAI_STORAGE_SECRET_KEY` | *(empty)* | S3 secret key |
When S3 is not configured, model files are transferred directly from the frontend to workers via **HTTP** — no shared filesystem needed. Each worker runs a small HTTP file transfer server alongside the gRPC backend process. This is the default and works out of the box.
When S3 is not configured, model files are transferred directly from the frontend to workers via **HTTP** - no shared filesystem needed. Each worker runs a small HTTP file transfer server alongside the gRPC backend process. This is the default and works out of the box.
For high-throughput or very large model files, S3 can be more efficient since it avoids streaming through the frontend.
@@ -160,7 +160,7 @@ during installation as well as the committed snapshot.
{{% /notice %}}
{{% notice warning %}}
The worker HTTP file transfer server is authenticated by `LOCALAI_REGISTRATION_TOKEN`. If the token is **empty**, the server **fails open** — anyone who can reach the port gets read/write access to the worker's models/staging/data directories (a remote model-poisoning / exfiltration vector). The worker logs a loud warning at startup in this case. Always set `LOCALAI_REGISTRATION_TOKEN` in distributed mode, and set `LOCALAI_DISTRIBUTED_REQUIRE_AUTH=true` (frontend **and** workers) to make a missing token *or* missing NATS credentials a hard startup error rather than a silent fail-open. Firewall the file-transfer port (gRPC base − 1) so only the frontend can reach it.
The worker HTTP file transfer server is authenticated by `LOCALAI_REGISTRATION_TOKEN`. If the token is **empty**, the server **fails open** - anyone who can reach the port gets read/write access to the worker's models/staging/data directories (a remote model-poisoning / exfiltration vector). The worker logs a loud warning at startup in this case. Always set `LOCALAI_REGISTRATION_TOKEN` in distributed mode, and set `LOCALAI_DISTRIBUTED_REQUIRE_AUTH=true` (frontend **and** workers) to make a missing token *or* missing NATS credentials a hard startup error rather than a silent fail-open. Firewall the file-transfer port (gRPC base − 1) so only the frontend can reach it.
{{% /notice %}}
### Watching Backend Installs
@@ -190,7 +190,7 @@ progress, its row in the breakdown will still show terminal status
## Worker Configuration
Workers are started with the `worker` subcommand. Each worker is generic — it doesn't need a backend type at startup:
Workers are started with the `worker` subcommand. Each worker is generic - it doesn't need a backend type at startup:
```bash
local-ai worker \
@@ -301,7 +301,7 @@ Workers start as generic processes with no backend installed. When the SmartRout
3. Replies with the allocated gRPC address
4. The SmartRouter calls `LoadModel` via direct gRPC to that address
Workers can run **multiple models concurrently** — each model gets its own gRPC process on a separate port. For example, an embedding model on port 50051 and a chat model on port 50052 can run simultaneously on the same worker.
Workers can run **multiple models concurrently** - each model gets its own gRPC process on a separate port. For example, an embedding model on port 50051 and a chat model on port 50052 can run simultaneously on the same worker.
When the SmartRouter needs to free capacity, it can unload models with zero in-flight requests without affecting other models on the same worker.
@@ -309,7 +309,7 @@ When the SmartRouter needs to free capacity, it can unload models with zero in-f
The API is split into two prefixes with distinct auth:
### `/api/node/` — Node self-service
### `/api/node/` - Node self-service
Used by workers themselves (registration, heartbeat, etc.). Authenticated via the registration token, exempt from global auth.
@@ -321,7 +321,7 @@ Used by workers themselves (registration, heartbeat, etc.). Authenticated via th
| `GET` | `/api/node/:id/models` | Query own loaded models |
| `DELETE` | `/api/node/:id` | Deregister self |
### `/api/nodes/` — Admin management
### `/api/nodes/` - Admin management
Used by the WebUI and admin API consumers. Requires admin authentication.
@@ -396,7 +396,7 @@ To skip manual approval and let nodes join immediately, set `--auto-approve-node
| `healthy` | Node is active and responding to heartbeats |
| `unhealthy` | Node has missed heartbeats beyond the threshold (detected by the HealthMonitor) |
| `offline` | Node is temporarily offline (graceful shutdown or stale heartbeat). The node row is preserved so re-registration restores the previous approval status without requiring re-approval |
| `draining` | Node is shutting down gracefully — no new requests are routed to it, existing in-flight requests are allowed to complete |
| `draining` | Node is shutting down gracefully - no new requests are routed to it, existing in-flight requests are allowed to complete |
## Agent Workers
@@ -436,7 +436,7 @@ MCP servers configured in model configs work in distributed mode. The frontend r
A single vLLM model can span multiple GPU nodes via data parallelism: the head node serves the OpenAI API and runs the local DP ranks, follower nodes run vanilla `vllm serve --headless` and speak ZMQ directly to the head. LocalAI's role is starting the follower processes and surfacing them in the admin UI; the cross-rank tensor traffic is vLLM's own.
This mode is **operator-launched** — the head config and each follower's invocation must agree on the topology (`data_parallel_size`, `data_parallel_size_local`, `data_parallel_address`, `data_parallel_rpc_port`). The SmartRouter does not place follower ranks automatically.
This mode is **operator-launched** - the head config and each follower's invocation must agree on the topology (`data_parallel_size`, `data_parallel_size_local`, `data_parallel_address`, `data_parallel_rpc_port`). The SmartRouter does not place follower ranks automatically.
### Head node configuration
@@ -472,7 +472,7 @@ local-ai p2p-worker vllm \
--registration-token changeme
```
`--register-to` is optional but recommended — it makes the follower visible in the admin UI as an `agent`-type node tagged with `node.role=vllm-follower`. Without it the worker just runs vLLM and exits silently when vLLM does. The role label discourages SmartRouter from placing other models on the follower; pair it with model selectors like `{"!node.role":"vllm-follower"}` if you also run regular LocalAI models on the same fleet.
`--register-to` is optional but recommended - it makes the follower visible in the admin UI as an `agent`-type node tagged with `node.role=vllm-follower`. Without it the worker just runs vLLM and exits silently when vLLM does. The role label discourages SmartRouter from placing other models on the follower; pair it with model selectors like `{"!node.role":"vllm-follower"}` if you also run regular LocalAI models on the same fleet.
### Worked example: 2-node Kimi-K2.6 deployment
@@ -505,7 +505,7 @@ A `curl http://10.0.0.1:8080/v1/chat/completions ...` against the head will then
### Intel Arc / XPU notes
vLLM XPU supports DP (`vllm/platforms/xpu.py:198` handles `world_size_across_dp > 1`; ranks bind to `xpu:{local_rank}` in `xpu_worker.py:62`, with xccl as the collective backend). Each rank still needs a distinct discrete GPU — the iGPU on a hybrid host is not a viable second device.
vLLM XPU supports DP (`vllm/platforms/xpu.py:198` handles `world_size_across_dp > 1`; ranks bind to `xpu:{local_rank}` in `xpu_worker.py:62`, with xccl as the collective backend). Each rank still needs a distinct discrete GPU - the iGPU on a hybrid host is not a viable second device.
Older XE-HPG GPUs (e.g. Arc A770) need to bypass the cutlass attention path:
@@ -514,7 +514,7 @@ engine_args:
attention_backend: TRITON_ATTN
```
`docker-compose.vllm-multinode.intel.yaml` at the repo root is the Intel equivalent of `docker-compose.vllm-multinode.yaml` — uses `/dev/dri` passthrough, `ZE_AFFINITY_MASK` to pin each rank to one device, and `latest-gpu-intel` images. Run via `./tests/e2e/vllm-multinode/smoke.sh --intel`.
`docker-compose.vllm-multinode.intel.yaml` at the repo root is the Intel equivalent of `docker-compose.vllm-multinode.yaml` - uses `/dev/dri` passthrough, `ZE_AFFINITY_MASK` to pin each rank to one device, and `latest-gpu-intel` images. Run via `./tests/e2e/vllm-multinode/smoke.sh --intel`.
### Caveats
@@ -596,7 +596,7 @@ ds4 layer-split inference is **manual setup** in this release (Phase 1): you pla
**Adding worker capacity:** Start additional `worker` instances pointing to the same frontend. They self-register automatically:
```bash
# Additional workers — no backend type needed
# Additional workers - no backend type needed
local-ai worker \
--register-to http://frontend:8080 \
--node-name worker-2 \

15
docs/content/features/distributed_inferencing.md
View File

@@ -1,8 +1,9 @@
+++
disableToc = false
title = "P2P / Federated Inference"
weight = 15
weight = 70
url = "/features/distribute/"
aliases = ["/features/distribution/"]
+++
![Federated vs worker mode: federated routes a whole request to one node; worker shards one model across nodes](/images/diagrams/federated-vs-worker.png)
@@ -11,6 +12,18 @@ url = "/features/distribute/"
Looking for production-grade horizontal scaling with PostgreSQL and NATS? See [Distributed Mode]({{% relref "features/distributed-mode" %}}).
{{% /notice %}}
## Choosing a distributed mode
LocalAI can spread inference across multiple machines in three ways. Pick the one that matches your setup:
| Mode | Best for | Guide |
|------|----------|-------|
| **P2P / Federated inference** | Ad-hoc clusters, community sharing, quick experimentation. Nodes discover each other via a shared libp2p token, with no central server. | This page |
| **Distributed Mode (PostgreSQL + NATS)** | Production deployments, Kubernetes, and managed infrastructure. Stateless frontends behind a load balancer, workers self-register, and state lives in PostgreSQL. | [Distributed Mode]({{% relref "features/distributed-mode" %}}) |
| **MLX distributed** | Apple Silicon clusters running MLX models over the MLX distributed runtime. | [MLX Distributed]({{% relref "features/mlx-distributed" %}}) |
For the low-level protocol and endpoints used by P2P workers, see the [P2P API reference]({{% relref "reference/p2p-api" %}}).
This functionality enables LocalAI to distribute inference requests across multiple worker nodes, improving efficiency and performance. Nodes are automatically discovered and connect via p2p by using a shared token which makes sure the communication is secure and private between the nodes of the network.
LocalAI supports two modes of distributed inferencing via p2p:

34
docs/content/features/distribution.md
View File

@@ -1,34 +0,0 @@
+++
disableToc = false
title = "Distribution"
weight = 13
url = "/features/distribution/"
+++
LocalAI supports distributing inference workloads across multiple machines. There are two approaches, each suited to different use cases:
## Distributed Mode (PostgreSQL + NATS)
Production-grade horizontal scaling with centralized management. Frontends are stateless LocalAI instances behind a load balancer; workers self-register and receive backends dynamically via NATS. State lives in PostgreSQL.
**Best for:** production deployments, Kubernetes, managed infrastructure.
[Read more]({{% relref "features/distributed-mode" %}})
## P2P / Federated Inference
Peer-to-peer networking via libp2p. Share a token to form a cluster with automatic discovery — no central server required. Supports federated load balancing and worker-mode weight sharding.
**Best for:** ad-hoc clusters, community sharing, quick experimentation.
[Read more]({{% relref "features/distributed_inferencing" %}})
## Quick Comparison
| | P2P / Federation | Distributed Mode |
|---|---|---|
| **Discovery** | Automatic via libp2p token | Self-registration to frontend URL |
| **State storage** | In-memory / ledger | PostgreSQL |
| **Coordination** | Gossip protocol | NATS messaging |
| **Node management** | Automatic | REST API + WebUI |
| **Setup complexity** | Minimal (share a token) | Requires PostgreSQL + NATS |

4
docs/content/features/embeddings.md
View File

@@ -1,14 +1,14 @@
+++
disableToc = false
title = "Embeddings"
weight = 13
weight = 60
url = "/features/embeddings/"
+++
LocalAI supports generating embeddings for text or list of tokens.
For face embeddings specifically, see the
[Face Recognition](/features/face-recognition/) feature — it produces
[Face Recognition](/features/face-recognition/) feature - it produces
512-d L2-normalized vectors tuned for face similarity.
For the API documentation you can refer to the OpenAI docs: https://platform.openai.com/docs/api-reference/embeddings

32
docs/content/features/face-recognition.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Face Recognition"
weight = 14
weight = 42
url = "/features/face-recognition/"
+++
@@ -99,7 +99,7 @@ by license and accuracy needs.
|---|---|---|---|
| `insightface-buffalo-l` | SCRFD-10GF + ArcFace R50 + GenderAge | ~326 MB | **Non-commercial research only** (upstream insightface weights) |
| `insightface-buffalo-s` | SCRFD-500MF + MBF + GenderAge | ~159 MB | **Non-commercial research only** |
| `insightface-opencv` | YuNet + SFace | ~40 MB | **Apache 2.0 — commercial-safe** |
| `insightface-opencv` | YuNet + SFace | ~40 MB | **Apache 2.0 - commercial-safe** |
The `insightface` Python library itself is MIT, but the pretrained model
packs (buffalo_l, buffalo_s, antelopev2) are released by the upstream
@@ -144,7 +144,7 @@ Response:
## 1:N identification workflow (register → identify → forget)
This is the primary "face recognition" flow. Under the hood it uses
LocalAI's built-in in-memory vector store — no external database to
LocalAI's built-in in-memory vector store - no external database to
stand up.
1. Register known faces:
@@ -184,7 +184,7 @@ stand up.
{{% notice warning %}}
**Storage caveat.** The default vector store is in-memory. All
registered faces are lost when LocalAI restarts. Persistent storage
(pgvector) is a tracked future enhancement — the face-recognition HTTP
(pgvector) is a tracked future enhancement - the face-recognition HTTP
API is designed to swap the backing store without changing the wire
format.
{{% /notice %}}
@@ -198,7 +198,7 @@ format.
| `model` | string | gallery entry name (e.g. `insightface-buffalo-l`) |
| `img1`, `img2` | string | URL, base64, or data-URI |
| `threshold` | float, optional | cosine-distance cutoff; default depends on engine |
| `anti_spoofing` | bool, optional | also run MiniFASNet liveness on each image — see [Antispoofing](#antispoofing-liveness-detection) |
| `anti_spoofing` | bool, optional | also run MiniFASNet liveness on each image - see [Antispoofing](#antispoofing-liveness-detection) |
Returns `verified`, `distance`, `threshold`, `confidence`, `model`,
`img1_area`, `img2_area`, and `processing_time_ms`. When
@@ -219,7 +219,7 @@ Returns demographic attributes for every detected face:
Only `insightface-buffalo-l` / `insightface-buffalo-s` populate age and
gender (genderage head). `insightface-opencv` returns face regions with
empty attributes — SFace has no demographic classifier. Emotion and
empty attributes - SFace has no demographic classifier. Emotion and
race are always empty in the current release.
### `POST /v1/face/register` (1:N enrollment)
@@ -273,12 +273,12 @@ SFace.
> **Note:** the OpenAI-compatible `/v1/embeddings` endpoint is
> intentionally text-only by contract (`input` is a string or list of
> strings of TEXT to embed) — passing an image data-URI there does
> strings of TEXT to embed) - passing an image data-URI there does
> nothing useful. Use `/v1/face/embed` for image inputs.
### Reused endpoint
- `POST /v1/detection` — returns face bounding boxes with
- `POST /v1/detection` - returns face bounding boxes with
`class_name: "face"`; works for both engines.
## Antispoofing (liveness detection)
@@ -288,7 +288,7 @@ MiniFASNetV2 + MiniFASNetV1SE ensemble (Apache 2.0, ~4 MB total, CPU-only)
alongside the face recognition weights. Set `anti_spoofing: true` on
`/v1/face/verify` or `/v1/face/analyze` to run liveness on each detected
face. The two models look at different crop scales and their softmax
outputs are averaged before argmax — the upstream-recommended setup.
outputs are averaged before argmax - the upstream-recommended setup.
`/v1/face/verify` with liveness gating:
@@ -323,7 +323,7 @@ Response (fields added when `anti_spoofing` is enabled):
```
If either image fails liveness (`is_real=false`), `verified` is forced
to `false` — similarity alone is not enough.
to `false` - similarity alone is not enough.
`/v1/face/analyze` reports per-face `is_real` and `antispoof_score`
when the flag is set.
@@ -331,14 +331,14 @@ when the flag is set.
**Fail-loud semantics.** If `anti_spoofing: true` is sent against a
model installed without the MiniFASNet files (e.g. a custom entry that
only listed the face recognition weights), the request returns a gRPC
`FAILED_PRECONDITION` error — the endpoint will never silently return
`FAILED_PRECONDITION` error - the endpoint will never silently return
`is_real=false`. Re-install the gallery entry or point the backend at a
model that bundles the MiniFASNet ONNX files.
{{% notice info %}}
The MiniFASNet score is best at catching **printed photos and screen
replays**. Deepfake videos and high-quality prosthetics are out of
scope — liveness here is a low-cost first line of defence, not a
scope - liveness here is a low-cost first line of defence, not a
guarantee. For higher assurance, combine with challenge-response (e.g.
ask the user to turn their head).
{{% /notice %}}
@@ -360,14 +360,14 @@ The recommended default `threshold` for `/v1/face/verify` and
| MBF (`buffalo_s`) | ~0.40 |
| SFace (`opencv`) | ~0.50 |
Pass `threshold` explicitly when switching engines — the per-engine
Pass `threshold` explicitly when switching engines - the per-engine
default only fires when the field is omitted.
## Related features
- [Object Detection](/features/object-detection/) — generic bounding-box
- [Object Detection](/features/object-detection/) - generic bounding-box
detection; `/v1/detection` works with the insightface backend too.
- [Embeddings](/features/embeddings/) — raw vector extraction; face
- [Embeddings](/features/embeddings/) - raw vector extraction; face
embeddings live in the same endpoint under the hood.
- [Stores](/features/stores/) — the generic vector store powering the
- [Stores](/features/stores/) - the generic vector store powering the
1:N recognition pipeline.

21
docs/content/features/fine-tuning.md
View File

@@ -1,8 +1,9 @@
+++
disableToc = false
title = "Fine-Tuning"
weight = 18
weight = 84
url = '/features/fine-tuning/'
aliases = ['/advanced/fine-tuning/']
+++
![The fine-tune job lifecycle: create, train with SSE progress, then export to LoRA, merged, or GGUF](/images/diagrams/finetune-job-lifecycle.png)
@@ -139,12 +140,12 @@ GRPO training requires reward functions to evaluate model completions. Specify t
| Name | Description | Parameters |
|------|-------------|-----------|
| `format_reward` | Checks `<think>...</think>` then answer format (1.0/0.0) | — |
| `reasoning_accuracy_reward` | Extracts `<answer>` content, compares to dataset's `answer` column | — |
| `format_reward` | Checks `<think>...</think>` then answer format (1.0/0.0) | - |
| `reasoning_accuracy_reward` | Extracts `<answer>` content, compares to dataset's `answer` column | - |
| `length_reward` | Score based on proximity to target length [0, 1] | `target_length` (default: 200) |
| `xml_tag_reward` | Scores properly opened/closed `<think>` and `<answer>` tags | — |
| `no_repetition_reward` | Penalizes n-gram repetition [0, 1] | — |
| `code_execution_reward` | Checks Python code block syntax validity (1.0/0.0) | — |
| `xml_tag_reward` | Scores properly opened/closed `<think>` and `<answer>` tags | - |
| `no_repetition_reward` | Penalizes n-gram repetition [0, 1] | - |
| `code_execution_reward` | Checks Python code block syntax validity (1.0/0.0) | - |
#### Inline Custom Reward Functions
@@ -200,10 +201,10 @@ curl -X POST http://localhost:8080/api/fine-tuning/jobs \
When fine-tuning is enabled, a "Fine-Tune" page appears in the sidebar under the Agents section. The UI provides:
1. **Job Configuration** — Select backend, model, training method, adapter type, and hyperparameters
2. **Dataset Upload** — Upload local datasets or reference HuggingFace datasets
3. **Training Monitor** — Real-time loss chart, progress bar, metrics display
4. **Export** — Export trained models in various formats
1. **Job Configuration** - Select backend, model, training method, adapter type, and hyperparameters
2. **Dataset Upload** - Upload local datasets or reference HuggingFace datasets
3. **Training Monitor** - Real-time loss chart, progress bar, metrics display
4. **Export** - Export trained models in various formats
## Dataset Formats

20
docs/content/features/gpt-vision.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "GPT Vision"
weight = 14
weight = 40
url = "/features/gpt-vision/"
+++
@@ -14,11 +14,17 @@ LocalAI supports understanding images by using [LLaVA](https://llava.hliu.cc/),
OpenAI docs: https://platform.openai.com/docs/guides/vision
First install a vision-capable model from the gallery (the examples below use `moondream2-20250414`, a small vision model):
```bash
local-ai run moondream2-20250414
```
To let LocalAI understand and reply with what sees in the image, use the `/v1/chat/completions` endpoint, for example with curl:
```bash
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llava",
"model": "moondream2-20250414",
"messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'
```
@@ -26,10 +32,16 @@ Grammars and function tools can be used as well in conjunction with vision APIs:
```bash
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "llava", "grammar": "root ::= (\"yes\" | \"no\")",
"model": "moondream2-20250414", "grammar": "root ::= (\"yes\" | \"no\")",
"messages": [{"role": "user", "content": [{"type":"text", "text": "Is there some grass in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}'
```
### Setup
To setup the LLaVa models, follow the full example in the [configuration examples](https://github.com/mudler/LocalAI-examples/blob/main/configurations/llava/llava.yaml).
Install a vision-capable model from the gallery, either from the **Models** page in the web UI or from the CLI:
```bash
local-ai run moondream2-20250414
```
Other vision models are available in the gallery (for example `smolvlm-instruct` and `smolvlm2-2.2b-instruct`). Browse them on the **Models** page or see the {{% relref "features/model-gallery" %}}.

4
docs/content/features/image-generation.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Image Generation"
weight = 12
weight = 50
url = "/features/image-generation/"
+++
@@ -156,7 +156,7 @@ diffusers:
#### Dependencies
This is an extra backend - in the container is already available and there is nothing to do for the setup. Do not use *core* images (ending with `-core`). If you are building manually, see the [build instructions]({{%relref "installation/build" %}}).
This is an extra backend - in the container is already available and there is nothing to do for the setup. Do not use *core* images (ending with `-core`). If you are building manually, see the [build instructions]({{%relref "getting-started/build" %}}).
#### Model setup

2
docs/content/features/interleaved-thinking.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Interleaved Thinking with Tool Calls"
weight = 17
weight = 13
url = "/features/interleaved-thinking/"
+++

18
docs/content/features/localai-assistant.md
View File

@@ -1,17 +1,25 @@
+++
disableToc = false
title = "LocalAI Assistant"
weight = 27
weight = 23
url = '/features/localai-assistant'
+++
LocalAI Assistant is an admin-only chat modality. When enabled on a chat session, the conversation is wired to an in-process MCP server that exposes LocalAI's own admin/management surface as tools. You can install models, manage backends, edit model configs and check system status by chatting — no REST calls or YAML edits.
{{% notice info %}}
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
- **LocalAI Assistant** (this page): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
- **Agents** ({{% relref "features/agents" %}}): autonomous agents you build that reason, use tools, and act on their own.
- **MCP** ({{% relref "features/mcp" %}}): a way to give a model external tools through the Model Context Protocol.
{{% /notice %}}
LocalAI Assistant is an admin-only chat modality. When enabled on a chat session, the conversation is wired to an in-process MCP server that exposes LocalAI's own admin/management surface as tools. You can install models, manage backends, edit model configs and check system status by chatting, with no REST calls or YAML edits.
The same MCP server is published as a Go package and can also be served over **stdio** to control a remote LocalAI instance from outside (e.g. from a desktop MCP host, Cursor, or `mcphost`).
## Enabling the assistant in chat
Open the chat UI as an **admin** user and pick a chat-capable model in the model selector. The header shows a **Manage** toggle — flip it on, and a `Manage mode` badge appears next to the chat title. Starter chips ("What is installed?", "Install a chat model", "Show system status", "Update a backend") help you get going.
Open the chat UI as an **admin** user and pick a chat-capable model in the model selector. The header shows a **Manage** toggle - flip it on, and a `Manage mode` badge appears next to the chat title. Starter chips ("What is installed?", "Install a chat model", "Show system status", "Update a backend") help you get going.
The home page also exposes a **Manage by chat** CTA that opens a fresh chat already in Manage mode.
@@ -35,7 +43,7 @@ When disabled, the chat handler refuses requests with `metadata.localai_assistan
- The chat toggle is hidden for non-admin users.
- The chat handler re-checks admin role at request time even when auth is configured to skip the assistant feature gate (defense in depth).
- The MCP server itself is in-process — there is no localhost loopback, no synthetic API key, and no extra TCP socket.
- The MCP server itself is in-process - there is no localhost loopback, no synthetic API key, and no extra TCP socket.
- Mutating tools (`install_model`, `delete_model`, `edit_model_config`, `upgrade_backend`, …) are guarded by a system-prompt rule that requires the LLM to confirm the action with the user before calling them. There is no separate code-side preview/apply step.
## Standalone stdio MCP server
@@ -44,7 +52,7 @@ You can run the same admin tool surface as a stdio MCP server pointed at any Loc
```bash
local-ai mcp-server --target http://remote.localai:8080 --api-key <admin-key>
# read-only mode — skips registration of every mutating tool
# read-only mode - skips registration of every mutating tool
local-ai mcp-server --target http://remote.localai:8080 --read-only
```

169
docs/content/features/longcat-video.md
View File

@@ -1,169 +0,0 @@
+++
disableToc = false
title = "LongCat Video and Avatar"
weight = 19
url = "/features/longcat-video/"
+++
LocalAI's `longcat-video` backend serves Meituan's official LongCat video-generation models through the `/video` API and the Studio **Video** page.
| Gallery model | Upstream checkpoint | Inputs | Output |
|---------------|---------------------|--------|--------|
| `longcat-video` | `meituan-longcat/LongCat-Video` | text, optional start image | video |
| `longcat-video-avatar-1.5` | `meituan-longcat/LongCat-Video-Avatar-1.5` | text, audio, optional portrait | video with the source audio |
The base checkpoint supports text-to-video and image-to-video. Avatar 1.5 adds audio-driven character animation, optional portrait conditioning, and continuation segments for longer speech.
{{% notice warning %}}
LongCat is a large, CUDA-only model family. LocalAI publishes this backend for Linux with NVIDIA CUDA 12 or CUDA 13 on x86_64 and CUDA 13 on ARM64. CPU, ROCm, and macOS images are not available. Avatar 1.5 also loads components from the base checkpoint, so reserve substantial disk and GPU or unified memory.
{{% /notice %}}
## Install from the Model Gallery
Install one or both recipes from **Models** in the web UI, or use the CLI:
```bash
local-ai models install longcat-video
local-ai models install longcat-video-avatar-1.5
```
You can also import either official Hugging Face URL. The importer recognizes the two repositories and writes a `longcat-video` model config with the appropriate use case and input/output modalities.
The required OCI backend is installed automatically when LocalAI first loads the model. The hardware detector selects the CUDA 12, CUDA 13, or CUDA 13 ARM64 variant.
### DGX Spark and NVIDIA ARM64
Use a LocalAI CUDA 13 ARM64 image as described in [GPU acceleration]({{%relref "features/GPU-acceleration" %}}). The backend defaults to PyTorch SDPA, avoiding the FlashAttention dependency that is commonly unavailable on Blackwell ARM64 systems.
For unified-memory systems, start with BF16 (`use_int8:false`, the default). INT8 lowers steady-state DiT memory but can have a higher load-time peak because the full model is materialized before the quantized weights are applied.
## Generate in Studio
1. Open **Studio**, then choose **Video**.
2. Select `longcat-video` or `longcat-video-avatar-1.5`.
3. Enter a prompt and choose `832x480` or `1280x720`.
4. Expand **Reference media** to upload a start image. For Avatar 1.5, upload or record the speech under **Avatar audio**.
5. Select **Generate**.
The base model can run without a reference image for text-to-video. Avatar 1.5 requires audio; the portrait is optional.
## API examples
### Text-to-video
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d '{
"model": "longcat-video",
"prompt": "A cinematic tracking shot through a misty redwood forest",
"width": 832,
"height": 480,
"num_frames": 93,
"fps": 15
}'
```
### Image-to-video
`start_image` accepts raw base64, a browser-style data URI, or a public HTTP(S) URL:
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d "{
\"model\": \"longcat-video\",
\"prompt\": \"The subject turns toward the camera as leaves move in the breeze\",
\"start_image\": \"$(base64 --wrap=0 portrait.png)\",
\"params\": {
\"resolution\": \"480p\"
}
}"
```
### Avatar from speech and a portrait
`audio` accepts raw base64, a data URI, or a public HTTP(S) URL. Each staged image or audio input is limited to 128 MiB.
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d "{
\"model\": \"longcat-video-avatar-1.5\",
\"prompt\": \"A friendly presenter speaking naturally to camera\",
\"start_image\": \"$(base64 --wrap=0 portrait.png)\",
\"audio\": \"$(base64 --wrap=0 speech.wav)\",
\"width\": 832,
\"height\": 480,
\"params\": {
\"offload_kv_cache\": \"true\"
}
}"
```
Avatar output is generated at 25 FPS and is muxed with the submitted audio. When neither `num_frames` nor `params.num_segments` is provided, LocalAI derives the continuation count from the audio duration, up to the model's `max_segments` setting.
## Model configuration
The gallery and importer make each model self-describing. A manual Avatar 1.5 config looks like this:
```yaml
name: longcat-video-avatar-1.5
backend: longcat-video
known_usecases:
- video
known_input_modalities:
- text
- image
- audio
known_output_modalities:
- video
options:
- attention_backend:sdpa
- use_distill:true
- max_segments:8
parameters:
model: meituan-longcat/LongCat-Video-Avatar-1.5
```
The explicit modality declarations are used by `GET /v1/models/capabilities` and attachment-aware clients. They avoid inferring model behavior from backend or checkpoint names.
### Load options
Model load options use `key:value` entries in `options`:
| Option | Default | Description |
|--------|---------|-------------|
| `attention_backend` | `sdpa` | `sdpa`, `auto`, `flash2`, `flash3`, or `xformers`; packaged images guarantee `sdpa` |
| `use_distill` | Avatar: `true`; base: `false` | Use the checkpoint's accelerated distillation path |
| `use_int8` | `false` | Use Avatar 1.5's INT8 DiT; unsupported by the base model |
| `base_model` | `meituan-longcat/LongCat-Video` | Base tokenizer, text encoder, and VAE used by Avatar 1.5 |
| `max_segments` | `8` | Maximum continuation segments accepted for one request |
| `resolution` | `480p` | Default image-conditioned resolution: `480p` or `720p` |
The initial backend supports one GPU per process. Tensor or context parallel sizes above one are rejected.
### Per-request parameters
The `/video` request's `params` object accepts string values:
| Parameter | Description |
|-----------|-------------|
| `num_segments` | Explicit number of Avatar continuation segments |
| `audio_guidance_scale` | Audio classifier-free guidance when distillation is disabled |
| `offload_kv_cache` | Offload continuation KV cache (`true` or `false`) |
| `ref_img_index` | Reference-frame index used during continuation |
| `mask_frame_range` | Number of frames blended around continuation boundaries |
| `resolution` | Per-request image-conditioned resolution (`480p` or `720p`) |
With distillation enabled, Avatar uses eight inference steps and fixed text/audio guidance of `1.0`. Disable `use_distill` in the model config before tuning `step`, `cfg_scale`, or `audio_guidance_scale`.
## Troubleshooting
- **HTTP 400, audio is required**: Avatar 1.5 was selected without `audio`.
- **HTTP 400, request needs too many segments**: trim the audio or raise `max_segments` in the model options.
- **HTTP 412**: the installed LocalAI runtime cannot select a compatible NVIDIA backend image.
- **Out of memory while loading**: use BF16 on unified-memory hardware, close other GPU workloads, or reduce model concurrency. INT8 is not guaranteed to reduce peak load memory.
- **Slow first request**: the backend and checkpoints are downloaded and loaded on demand; subsequent requests reuse the loaded pipeline.
See the general [`/video` API reference]({{%relref "features/video-generation" %}}) for the complete request and response schema.

25
docs/content/features/mcp.md
View File

@@ -1,6 +1,6 @@
+++
title = "Model Context Protocol (MCP)"
weight = 20
weight = 22
toc = true
description = "Agentic capabilities with Model Context Protocol integration"
tags = ["MCP", "Agents", "Tools", "Advanced"]
@@ -12,6 +12,14 @@ categories = ["Features"]
LocalAI now supports the **Model Context Protocol (MCP)**, enabling powerful agentic capabilities by connecting AI models to external tools and services. This feature allows your LocalAI models to interact with various MCP servers, providing access to real-time data, APIs, and specialized tools.
{{% notice info %}}
**Looking for something else?** LocalAI has three related agentic features that are easy to confuse:
- **MCP** (this page): a way to give a model external tools through the Model Context Protocol.
- **Agents** ({{% relref "features/agents" %}}): autonomous agents you build that reason, use tools, and act on their own.
- **LocalAI Assistant** ({{% relref "features/localai-assistant" %}}): an admin chat modality for administering LocalAI itself (install models, manage backends) by chatting.
{{% /notice %}}
## What is MCP?
The Model Context Protocol is a standard for connecting AI models to external tools and data sources. It enables AI agents to:
@@ -102,6 +110,19 @@ Configure local command-based MCP servers:
- **`max_iterations`**: Maximum number of MCP tool execution loop iterations (default: 10). Each iteration allows the model to call tools and receive results before generating the next response.
## Agent-scoped MCP servers
The `mcp:` block above attaches MCP servers to a **model**: every request that uses that model config can reach those tools. LocalAI's [agents]({{% relref "features/agents" %}}) can also attach MCP servers to a single **agent**, independent of the model it runs on. This is the mechanism the Agents feature advertises as "MCP Servers".
Use model-scoped MCP (the `mcp:` block in a model YAML) when the tools belong to the model itself and every caller of that model should get them. Use agent-scoped MCP when only one agent should have a given tool, or when different agents sharing the same model need different tools.
Agent-scoped MCP servers are configured per agent, not in a model YAML:
- **In the web UI:** open the **Agents** page, edit an agent, and fill in its **MCP Servers** (remote HTTP servers) and **MCP STDIO Servers** (local command servers) fields. The JSON shape is the same `mcpServers` map used above.
- **In an agent config (JSON):** the agent stores remote servers under `mcp_servers` and local command servers under `mcp_stdio_servers`, so an exported/imported agent carries its MCP servers with it.
An agent-scoped MCP server that is unreachable is a common reason an imported agent's tool "times out or is unavailable"; see the checklist in {{% relref "getting-started/first-agent" %}}.
## Usage
### Selecting MCP Servers via `metadata`
@@ -384,7 +405,7 @@ MCP sessions are automatically managed by LocalAI:
- **Cleanup on model unload**: When a model is unloaded (idle watchdog eviction, manual stop, or shutdown), all associated MCP sessions are closed and resources freed
- **Graceful shutdown**: All MCP sessions are closed when LocalAI shuts down
This means you don't need to manually manage MCP connections — they follow the model's lifecycle automatically.
This means you don't need to manually manage MCP connections - they follow the model's lifecycle automatically.
## Supported MCP Servers

18
docs/content/features/mlx-distributed.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "(experimental) MLX Distributed Inference"
weight = 18
weight = 72
url = '/features/mlx-distributed/'
+++
@@ -77,16 +77,16 @@ options:
The **hostfile** is a JSON array where entry `i` is the `"ip:port"` that **rank `i` listens on** for ring communication. All ranks must use the same hostfile so they know how to reach each other.
**Example:** Two Macs — Mac A (`192.168.1.10`) and Mac B (`192.168.1.11`):
**Example:** Two Macs - Mac A (`192.168.1.10`) and Mac B (`192.168.1.11`):
```json
["192.168.1.10:5555", "192.168.1.11:5555"]
```
- Entry 0 (`192.168.1.10:5555`) — the address rank 0 (Mac A) listens on for ring communication
- Entry 1 (`192.168.1.11:5555`) — the address rank 1 (Mac B) listens on for ring communication
- Entry 0 (`192.168.1.10:5555`) - the address rank 0 (Mac A) listens on for ring communication
- Entry 1 (`192.168.1.11:5555`) - the address rank 1 (Mac B) listens on for ring communication
Port 5555 is arbitrary — use any available port, but it must be open in your firewall.
Port 5555 is arbitrary - use any available port, but it must be open in your firewall.
### JACCL Backend (RDMA/Thunderbolt)
@@ -109,7 +109,7 @@ The **device matrix** is a JSON 2D array describing the RDMA device name between
]
```
JACCL requires a **coordinator** — a TCP service that helps all ranks establish RDMA connections. Rank 0 (the LocalAI machine) is always the coordinator. Workers are told the coordinator address via their `--coordinator` CLI flag (see [Starting Workers](#jaccl-workers) below).
JACCL requires a **coordinator** - a TCP service that helps all ranks establish RDMA connections. Rank 0 (the LocalAI machine) is always the coordinator. Workers are told the coordinator address via their `--coordinator` CLI flag (see [Starting Workers](#jaccl-workers) below).
### Without hostfile (single-node)
@@ -130,7 +130,7 @@ These can also be set via environment variables (`MLX_DISTRIBUTED_HOSTFILE`, `ML
## Starting Workers
LocalAI starts the rank 0 process (gRPC server) automatically when the model is loaded. But you still need to start **worker processes** (ranks 1, 2, ...) on the other machines. These workers participate in every forward pass but don't serve any API — they wait for commands from rank 0.
LocalAI starts the rank 0 process (gRPC server) automatically when the model is loaded. But you still need to start **worker processes** (ranks 1, 2, ...) on the other machines. These workers participate in every forward pass but don't serve any API - they wait for commands from rank 0.
### Ring Workers
@@ -190,11 +190,11 @@ Starts a worker or manual rank 0 process.
| `--rank` | `MLX_RANK` | *(required)* | Rank of this process (0 = gRPC server + ring participant, >0 = worker only) |
| `--backend` | `MLX_DISTRIBUTED_BACKEND` | `ring` | `ring` (TCP pipeline parallelism) or `jaccl` (RDMA tensor parallelism) |
| `--addr` | `MLX_DISTRIBUTED_ADDR` | `localhost:50051` | gRPC API listen address (rank 0 only, for LocalAI or external access) |
| `--coordinator` | `MLX_JACCL_COORDINATOR` | | JACCL coordinator `ip:port` — rank 0's address for RDMA setup (all ranks must use the same value) |
| `--coordinator` | `MLX_JACCL_COORDINATOR` | | JACCL coordinator `ip:port` - rank 0's address for RDMA setup (all ranks must use the same value) |
### `worker p2p-mlx`
P2P mode — auto-discovers peers and generates hostfile.
P2P mode - auto-discovers peers and generates hostfile.
| Flag | Env | Default | Description |
|------|-----|---------|-------------|

4
docs/content/features/model-aliases.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Model Aliases"
weight = 24
weight = 14
url = "/features/model-aliases/"
+++
@@ -78,4 +78,4 @@ its own, so swapping its target stays a valid pure redirect.
Aliases are a static 1:1 redirect. For classifier-based or load-balanced
selection across several downstream models, use the intelligent router in the
[Middleware]({{%relref "features/middleware" %}}) feature instead.
[Middleware]({{%relref "operations/middleware" %}}) feature instead.

2
docs/content/features/model-gallery.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Model Gallery"
weight = 18
weight = 81
url = '/models'
+++

8
docs/content/features/object-detection.md
View File

@@ -1,14 +1,14 @@
+++
disableToc = false
title = "Object Detection"
weight = 13
weight = 41
url = "/features/object-detection/"
+++
LocalAI supports object detection and image segmentation through various backends. This feature allows you to identify and locate objects within images with high accuracy and real-time performance. Available backends include [RF-DETR](https://github.com/roboflow/rf-detr) (Python) and [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) (native C++/ggml) for object detection and segmentation, and [sam3.cpp](https://github.com/PABannier/sam3.cpp) for image segmentation (SAM 3/2/EdgeTAM).
For detecting **faces** specifically, see the dedicated
[Face Recognition](/features/face-recognition/) feature — its
[Face Recognition](/features/face-recognition/) feature - its
`/v1/detection` support is tuned for face bounding boxes and ships
with commercially-safe model options.
@@ -140,7 +140,7 @@ You can browse and install this model through the LocalAI web interface or using
The `rfdetr-cpp` backend is a native C++/ggml implementation of RF-DETR
inference based on [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp). It
runs as a Go gRPC service that dlopens a per-CPU-variant shared library, so
there is no Python runtime on the inference path — startup is fast and the
there is no Python runtime on the inference path - startup is fast and the
binary is self-contained.
Compared to the Python `rfdetr` backend, the native backend:
@@ -193,7 +193,7 @@ Compared to the Python `rfdetr` backend, the native backend:
Pre-quantized GGUFs are published under
[`mudler/rfdetr-cpp-*`](https://huggingface.co/mudler?search_models=rfdetr-cpp)
on Hugging Face. Each repo carries the F32/F16/Q8_0/Q4_K quants — F16 is
on Hugging Face. Each repo carries the F32/F16/Q8_0/Q4_K quants - F16 is
the recommended default (matches F32 accuracy, ~1.86x smaller).
#### Segmentation Output

18
docs/content/features/openai-functions.md
View File

@@ -2,13 +2,13 @@
+++
disableToc = false
title = "OpenAI Functions and Tools"
weight = 17
weight = 11
url = "/features/openai-functions/"
+++
![Function calling: one tool-call request shape, each backend's native parser extracts the calls](/images/diagrams/tool-call-parsers.png)
LocalAI supports running the OpenAI [functions and tools API](https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools) across multiple backends. The OpenAI request shape is the same regardless of which backend runs your model — LocalAI is responsible for extracting structured tool calls from the model's output before returning the response.
LocalAI supports running the OpenAI [functions and tools API](https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools) across multiple backends. The OpenAI request shape is the same regardless of which backend runs your model - LocalAI is responsible for extracting structured tool calls from the model's output before returning the response.
![localai-functions-1](https://github.com/ggerganov/llama.cpp/assets/2420543/5bd15da2-78c1-4625-be90-1e938e6823f1)
@@ -23,10 +23,10 @@ LocalAI also supports [JSON mode](https://platform.openai.com/docs/guides/text-g
| Backend | How tool calls are extracted |
|---------|------------------------------|
| `llama.cpp` | C++ incremental parser; any `ggml`/`gguf` model works out of the box, no configuration needed |
| `vllm` | vLLM's native `ToolParserManager` — select a parser with `tool_parser:<name>` in the model `options`. Auto-set by the gallery importer for known families |
| `vllm` | vLLM's native `ToolParserManager` - select a parser with `tool_parser:<name>` in the model `options`. Auto-set by the gallery importer for known families |
| `vllm-omni` | Same as vLLM |
| `mlx` | `mlx_lm.tool_parsers` — **auto-detected from the chat template**, no configuration needed |
| `mlx-vlm` | `mlx_vlm.tool_parsers` (with fallback to mlx-lm parsers) — **auto-detected from the chat template**, no configuration needed |
| `mlx` | `mlx_lm.tool_parsers` - **auto-detected from the chat template**, no configuration needed |
| `mlx-vlm` | `mlx_vlm.tool_parsers` (with fallback to mlx-lm parsers) - **auto-detected from the chat template**, no configuration needed |
Reasoning content (`<think>...</think>` blocks from DeepSeek R1, Qwen3, Gemma 4, etc.) is returned in the OpenAI `reasoning_content` field on the same backends. When a model both reasons and calls a tool in the same turn, see [Interleaved Thinking with Tool Calls]({{%relref "features/interleaved-thinking" %}}) for how the reasoning survives the tool-result round trip.
@@ -34,7 +34,7 @@ Reasoning content (`<think>...</think>` blocks from DeepSeek R1, Qwen3, Gemma 4,
### llama.cpp
No configuration required — the autoparser detects the tool call format for any `ggml`/`gguf` model that was trained with tool support.
No configuration required - the autoparser detects the tool call format for any `ggml`/`gguf` model that was trained with tool support.
### vLLM / vLLM Omni
@@ -52,13 +52,13 @@ template:
use_tokenizer_template: true
```
When you import a vLLM model through the LocalAI gallery, the importer looks up the model family and pre-fills `tool_parser:` and `reasoning_parser:` for you — you only need to override them for non-standard model names.
When you import a vLLM model through the LocalAI gallery, the importer looks up the model family and pre-fills `tool_parser:` and `reasoning_parser:` for you - you only need to override them for non-standard model names.
Available tool parsers include `hermes`, `llama3_json`, `llama4_pythonic`, `mistral`, `qwen3_xml`, `deepseek_v3`, `granite4`, `kimi_k2`, `glm45`, and more. Available reasoning parsers include `deepseek_r1`, `qwen3`, `mistral`, `gemma4`, `granite`. See the upstream vLLM documentation for the full list.
### MLX / MLX-VLM
MLX backends **auto-detect** the right tool parser by inspecting the model's chat template — you don't need to set anything. Just load an MLX-quantized model that was trained with tool support:
MLX backends **auto-detect** the right tool parser by inspecting the model's chat template - you don't need to set anything. Just load an MLX-quantized model that was trained with tool support:
```yaml
name: qwen2.5-0.5b-mlx
@@ -69,7 +69,7 @@ template:
use_tokenizer_template: true
```
The gallery importer will still append `tool_parser:` and `reasoning_parser:` entries to the YAML for visibility and consistency with the other backends, but those are informational — the runtime auto-detection in the MLX backend ignores them and uses the parser matched to the chat template.
The gallery importer will still append `tool_parser:` and `reasoning_parser:` entries to the YAML for visibility and consistency with the other backends, but those are informational - the runtime auto-detection in the MLX backend ignores them and uses the parser matched to the chat template.
Supported parser families: `hermes`/`json_tools`, `mistral`, `gemma4`, `glm47`, `kimi_k2`, `longcat`, `minimax_m2`, `pythonic`, `qwen3_coder`, `function_gemma`.

34
docs/content/features/openai-realtime.md
View File

@@ -1,7 +1,7 @@
---
title: "Realtime API"
weight: 60
weight: 37
---
![The realtime voice loop: VAD to STT to LLM to TTS, over WebSocket or WebRTC](/images/diagrams/realtime-pipeline.png)
@@ -51,21 +51,21 @@ pipeline:
- **streaming.tts**: emit a `response.output_audio.delta` per audio chunk the TTS backend produces (requires a backend that supports streaming synthesis), instead of one delta for the whole utterance. Falls back to a single unary delta otherwise.
- **streaming.transcription**: stream `conversation.item.input_audio_transcription.delta` events as the transcript is produced (requires a transcription backend that supports streaming).
- **streaming.llm**: stream the LLM reply token-by-token as `response.output_audio_transcript.delta` events. The full reply is buffered and synthesized once it is complete — streamed as audio chunks when `streaming.tts` is enabled (and the TTS backend supports it), otherwise as a single unary delta. Reasoning/thinking is always stripped from the spoken transcript. Tool calls are supported while streaming when the LLM uses its tokenizer template (`use_tokenizer_template: true`): the backend's autoparser then delivers content and tool calls separately, so the spoken transcript never leaks tool-call tokens. Grammar-based function calling keeps the buffered path.
- **streaming.clause_chunking**: instead of buffering the whole reply before TTS, split it into speakable clauses and synthesize each as soon as it completes, lowering the time-to-first-audio. The splitter is script-aware: it uses Unicode sentence segmentation (so it handles CJK `。！？` with no whitespace), CJK clause punctuation (`，、；：`), and Thai/Lao spaces — it does **not** rely on whitespace sentence boundaries, so it works for languages such as Chinese, Japanese and Thai where the old per-sentence approach degraded to whole-message buffering. Requires `streaming.llm`; scripts that genuinely need a dictionary (e.g. Khmer, Burmese) simply stay buffered until a space or end-of-message. Off by default.
- **streaming.llm**: stream the LLM reply token-by-token as `response.output_audio_transcript.delta` events. The full reply is buffered and synthesized once it is complete - streamed as audio chunks when `streaming.tts` is enabled (and the TTS backend supports it), otherwise as a single unary delta. Reasoning/thinking is always stripped from the spoken transcript. Tool calls are supported while streaming when the LLM uses its tokenizer template (`use_tokenizer_template: true`): the backend's autoparser then delivers content and tool calls separately, so the spoken transcript never leaks tool-call tokens. Grammar-based function calling keeps the buffered path.
- **streaming.clause_chunking**: instead of buffering the whole reply before TTS, split it into speakable clauses and synthesize each as soon as it completes, lowering the time-to-first-audio. The splitter is script-aware: it uses Unicode sentence segmentation (so it handles CJK `。！？` with no whitespace), CJK clause punctuation (`，、；：`), and Thai/Lao spaces - it does **not** rely on whitespace sentence boundaries, so it works for languages such as Chinese, Japanese and Thai where the old per-sentence approach degraded to whole-message buffering. Requires `streaming.llm`; scripts that genuinely need a dictionary (e.g. Khmer, Burmese) simply stay buffered until a space or end-of-message. Off by default.
All streaming flags are off by default, so existing pipelines are unaffected.
### Model warm-up (cold start)
Without warm-up the pipeline's models are loaded into memory only on first use *within* a session: the VAD on the first audio chunk, transcription at the first end-of-speech, the LLM on the first reply, and TTS on the first spoken output. On a cold session this staggers a load delay across those first few interactions — and a model that fails to load (missing weights, wrong backend, out of memory) only fails part-way through the first turn.
Without warm-up the pipeline's models are loaded into memory only on first use *within* a session: the VAD on the first audio chunk, transcription at the first end-of-speech, the LLM on the first reply, and TTS on the first spoken output. On a cold session this staggers a load delay across those first few interactions - and a model that fails to load (missing weights, wrong backend, out of memory) only fails part-way through the first turn.
To avoid that, LocalAI **warms the pipeline by default**: it loads the VAD, transcription, LLM and TTS backends into memory *before* the session is announced, and the session start **blocks until they are all ready**. The loads run concurrently, so the wait is the slowest single model, not the sum. This means:
- The first turn pays no cold-start cost — every backend is already resident.
- The first turn pays no cold-start cost - every backend is already resident.
- **Model-load errors surface at session start.** If any stage fails to load, the session is not started and the client receives a `model_load_error` instead of `session.created`, so a broken pipeline fails fast and visibly rather than mid-call.
Set `disable_warmup: true` to restore the lazy "load on first use" behavior — session start no longer waits on loading and load errors surface on the first turn instead. Useful if you want idle sessions to avoid holding model memory they may never use:
Set `disable_warmup: true` to restore the lazy "load on first use" behavior - session start no longer waits on loading and load errors surface on the first turn instead. Useful if you want idle sessions to avoid holding model memory they may never use:
```yaml
name: gpt-realtime
@@ -79,7 +79,7 @@ pipeline:
#### Pre-loading a pipeline on demand
Warm-up only fires when a realtime session opens. To load a pipeline into memory ahead of time — e.g. to warm it right after boot, or when running with `disable_warmup: true` — POST the model name to the admin-only `/backend/load` endpoint. For a pipeline model it loads every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) concurrently:
Warm-up only fires when a realtime session opens. To load a pipeline into memory ahead of time - e.g. to warm it right after boot, or when running with `disable_warmup: true` - POST the model name to the admin-only `/backend/load` endpoint. For a pipeline model it loads every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) concurrently:
```bash
curl -X POST http://localhost:8080/backend/load \
@@ -87,16 +87,16 @@ curl -X POST http://localhost:8080/backend/load \
-d '{"model": "gpt-realtime"}'
```
The endpoint is not realtime-specific — it pre-loads any model. See [Backend Monitor]({{%relref "features/backend-monitor" %}}) for the full request/response reference (it is the inverse of `/backend/shutdown`).
The endpoint is not realtime-specific - it pre-loads any model. See [Backend Monitor]({{%relref "operations/backend-monitor" %}}) for the full request/response reference (it is the inverse of `/backend/shutdown`).
### Turn detection
Turn detection decides when the user has finished speaking and the pipeline should respond. Two modes are supported, matching the OpenAI session schema:
- **`server_vad`** (default): silence-based. The VAD model watches the audio and the turn commits after `silence_duration_ms` (default 500 ms) of silence. Simple and model-agnostic, but a fixed silence window must trade interrupting mid-sentence pauses against sluggish responses.
- **`semantic_vad`**: model-driven. The transcription model itself signals end-of-utterance and the silence window becomes dynamic: short right after the model emits its end-of-utterance token, much longer when it does not — so pausing to think no longer gets cut off, while finished sentences get a fast response.
- **`semantic_vad`**: model-driven. The transcription model itself signals end-of-utterance and the silence window becomes dynamic: short right after the model emits its end-of-utterance token, much longer when it does not - so pausing to think no longer gets cut off, while finished sentences get a fast response.
`semantic_vad` requires a transcription model that emits an end-of-utterance token over a cache-aware streaming decode — currently `parakeet-cpp-realtime_eou_120m-v1` (the model is trained to distinguish "paused, expecting a reply" from "paused mid-thought"). The realtime pipeline feeds it the microphone audio live while the user speaks. With any other transcription backend the session degrades gracefully to silence-only detection using the eagerness timeout below (a warning is logged once). The model also emits a distinct end-of-backchannel token (`<EOB>`) for short acknowledgments like "uh-huh": those are transcribed but never treated as the user yielding the turn.
`semantic_vad` requires a transcription model that emits an end-of-utterance token over a cache-aware streaming decode - currently `parakeet-cpp-realtime_eou_120m-v1` (the model is trained to distinguish "paused, expecting a reply" from "paused mid-thought"). The realtime pipeline feeds it the microphone audio live while the user speaks. With any other transcription backend the session degrades gracefully to silence-only detection using the eagerness timeout below (a warning is logged once). The model also emits a distinct end-of-backchannel token (`<EOB>`) for short acknowledgments like "uh-huh": those are transcribed but never treated as the user yielding the turn.
Sessions can opt in via `session.update` (`turn_detection: {"type": "semantic_vad", "eagerness": "medium"}`), or the pipeline can set a server-side default so clients need no changes:
@@ -115,14 +115,14 @@ pipeline:
A client `session.update` still overrides `type` and `eagerness` per session.
**Eagerness** sets the fallback silence window used when no end-of-utterance token was seen (the model missed it, or the user genuinely trails off): `low` waits 8 s, `medium`/`auto` 4 s, `high` 2 s — the same max-timeout semantics OpenAI documents. After the token is seen, the turn commits on the next VAD tick (~300 ms).
**Eagerness** sets the fallback silence window used when no end-of-utterance token was seen (the model missed it, or the user genuinely trails off): `low` waits 8 s, `medium`/`auto` 4 s, `high` 2 s - the same max-timeout semantics OpenAI documents. After the token is seen, the turn commits on the next VAD tick (~300 ms).
**Live captions**: while the user speaks, `semantic_vad` streams `conversation.item.input_audio_transcription.delta` events under the item id the commit will later reuse, so clients can render the words as they are recognized. The `completed` event at commit carries the authoritative transcript and replaces the partial text (with `retranscribe: true` it may differ from the captions); a turn discarded before commit emits `conversation.item.input_audio_transcription.failed` so clients can retract its captions.
**`retranscribe`** (server-side only, semantic_vad only) cross-checks the streaming decode against a batch decode at commit time:
- `false` (default): the transcript accumulated from the live stream is used as-is — the model runs once per utterance and the LLM starts immediately at commit.
- `true`: the committed audio is re-transcribed offline. If the batch decode also ends with the end-of-utterance token the turn proceeds (using the batch transcript); if it does **not**, the commit is cancelled and the session keeps listening — treating the streaming token as a false positive. Both transcripts are compared and logged, which makes this mode a useful diagnostic for how well the streaming and batch decodes align, at the cost of one extra decode per turn.
- `false` (default): the transcript accumulated from the live stream is used as-is - the model runs once per utterance and the LLM starts immediately at commit.
- `true`: the committed audio is re-transcribed offline. If the batch decode also ends with the end-of-utterance token the turn proceeds (using the batch transcript); if it does **not**, the commit is cancelled and the session keeps listening - treating the streaming token as a false positive. Both transcripts are compared and logged, which makes this mode a useful diagnostic for how well the streaming and batch decodes align, at the cost of one extra decode per turn.
### Disabling thinking
@@ -142,14 +142,14 @@ By default a realtime session feeds only the last `max_history_items` turns to t
Compaction works with two numbers:
- **`max_history_items`** is the *live window* — the recent turns kept verbatim in the prompt.
- **`compaction.trigger_items`** is the *high-water mark* — let the buffer grow to here, then summarize the overflow (everything above `max_history_items`) into a rolling memory and evict it. It must be greater than `max_history_items`; if it is not, it is clamped up.
- **`max_history_items`** is the *live window* - the recent turns kept verbatim in the prompt.
- **`compaction.trigger_items`** is the *high-water mark* - let the buffer grow to here, then summarize the overflow (everything above `max_history_items`) into a rolling memory and evict it. It must be greater than `max_history_items`; if it is not, it is clamped up.
The gap between the two controls how often summarization runs: a summary call fires roughly every `(trigger_items - max_history_items)` turns (here, about every 6 turns).
```yaml
pipeline:
max_history_items: 6 # live window — recent turns kept verbatim
max_history_items: 6 # live window - recent turns kept verbatim
compaction:
enabled: true
trigger_items: 12 # summarize overflow back down to max_history_items
@@ -204,7 +204,7 @@ Or set the `EXTERNAL_GRPC_BACKENDS` environment variable if running a local buil
EXTERNAL_GRPC_BACKENDS=opus:/path/to/backend/go/opus/opus
```
The opus backend is loaded automatically when a WebRTC session starts. It does not require any model configuration file — just the backend binary.
The opus backend is loaded automatically when a WebRTC session starts. It does not require any model configuration file - just the backend binary.
#### WebRTC behind Docker host networking or NAT

10
docs/content/features/quantization.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Model Quantization"
weight = 19
weight = 83
url = '/features/quantization/'
+++
@@ -144,10 +144,10 @@ The UI also supports entering a custom quantization type string for any format s
A "Quantize" page appears in the sidebar under the Tools section. The UI provides:
1. **Job Configuration** — Select model, quantization type (dropdown with presets or custom input), backend, and HuggingFace token
2. **Progress Monitor** — Real-time progress bar and log output via SSE
3. **Jobs List** — View all quantization jobs with status, stop/delete actions
4. **Output** — Download the quantized GGUF file or import it directly into LocalAI for immediate use
1. **Job Configuration** - Select model, quantization type (dropdown with presets or custom input), backend, and HuggingFace token
2. **Progress Monitor** - Real-time progress bar and log output via SSE
3. **Jobs List** - View all quantization jobs with status, stop/delete actions
4. **Output** - Download the quantized GGUF file or import it directly into LocalAI for immediate use
## Architecture

2
docs/content/features/reranker.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Reranker"
weight = 11
weight = 61
url = "/features/reranker/"
+++

2
docs/content/features/runtime-settings.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Runtime Settings"
weight = 25
weight = 82
url = '/features/runtime-settings'
+++

104
docs/content/features/sound-generation.md
View File

@@ -1,104 +0,0 @@
+++
disableToc = false
title = "Sound Generation"
weight = 19
url = "/features/sound-generation/"
+++
LocalAI supports generating audio from text descriptions via the `/v1/sound-generation` endpoint. This endpoint is compatible with the [ElevenLabs sound generation API](https://elevenlabs.io/docs/api-reference/sound-generation) and can produce music, sound effects, and other audio content.
## API
- **Method:** `POST`
- **Endpoint:** `/v1/sound-generation`
### Request
The request body is JSON. There are two usage modes: simple and advanced.
#### Simple mode
| Parameter | Type | Required | Description |
|------------------|----------|----------|----------------------------------------------|
| `model_id` | `string` | Yes | Model identifier |
| `text` | `string` | Yes | Audio description or prompt |
| `instrumental` | `bool` | No | Generate instrumental audio (no vocals) |
| `vocal_language` | `string` | No | Language code for vocals (e.g. `bn`, `ja`) |
#### Advanced mode
| Parameter | Type | Required | Description |
|---------------------|----------|----------|-------------------------------------------------|
| `model_id` | `string` | Yes | Model identifier |
| `text` | `string` | Yes | Text prompt or description |
| `duration_seconds` | `float` | No | Target duration in seconds |
| `prompt_influence` | `float` | No | Temperature / prompt influence parameter |
| `do_sample` | `bool` | No | Enable sampling |
| `think` | `bool` | No | Enable extended thinking for generation |
| `caption` | `string` | No | Caption describing the audio |
| `lyrics` | `string` | No | Lyrics for the generated audio |
| `bpm` | `int` | No | Beats per minute |
| `keyscale` | `string` | No | Musical key/scale (e.g. `Ab major`) |
| `language` | `string` | No | Language code |
| `vocal_language` | `string` | No | Vocal language (fallback if `language` is empty) |
| `timesignature` | `string` | No | Time signature (e.g. `4`) |
| `instrumental` | `bool` | No | Generate instrumental audio (no vocals) |
### Response
Returns a binary audio file with the appropriate `Content-Type` header (e.g. `audio/wav`, `audio/mpeg`, `audio/flac`, `audio/ogg`).
## Usage
### Generate a sound effect
```bash
curl http://localhost:8080/v1/sound-generation \
-H "Content-Type: application/json" \
-d '{
"model_id": "sound-model",
"text": "rain falling on a tin roof"
}' \
--output rain.wav
```
### Generate a song with vocals
```bash
curl http://localhost:8080/v1/sound-generation \
-H "Content-Type: application/json" \
-d '{
"model_id": "sound-model",
"text": "a soft Bengali love song for a quiet evening",
"instrumental": false,
"vocal_language": "bn"
}' \
--output song.wav
```
### Generate music with advanced parameters
```bash
curl http://localhost:8080/v1/sound-generation \
-H "Content-Type: application/json" \
-d '{
"model_id": "sound-model",
"text": "upbeat pop",
"caption": "A funky Japanese disco track",
"lyrics": "[Verse 1]\nDancing in the neon lights",
"think": true,
"bpm": 120,
"duration_seconds": 225,
"keyscale": "Ab major",
"language": "ja",
"timesignature": "4"
}' \
--output disco.wav
```
## Error Responses
| Status Code | Description |
|-------------|--------------------------------------------------|
| 400 | Missing or invalid model or request parameters |
| 500 | Backend error during sound generation |

4
docs/content/features/stores.md
View File

@@ -2,7 +2,7 @@
+++
disableToc = false
title = "Stores"
weight = 18
weight = 62
url = '/stores'
+++
@@ -12,7 +12,7 @@ a low level API that consists of only `get`, `set`, `delete` and `find`.
{{% notice tip %}}
**Face recognition uses this store.** The 1:N face identification flow
(`/v1/face/register`, `/v1/face/identify`, `/v1/face/forget`) is built
on top of the generic store — see
on top of the generic store - see
[Face Recognition](/features/face-recognition/) for the face-oriented
API.
{{% /notice %}}

28
docs/content/features/text-generation.md
View File

@@ -519,7 +519,7 @@ The `llama.cpp` backend supports additional configuration options that can be sp
| `cache_idle_slots` or `idle_slots_cache` | boolean | On a new task, save the previous slot's KV state into the prompt cache (and clear the slot) so a later request with the same prefix can warm-load it. Default: `true`. Auto-disabled by the server if `kv_unified=false` or `cache_ram=0`. | `cache_idle_slots:false` |
| `n_ctx_checkpoints` or `ctx_checkpoints` | integer | Maximum number of context checkpoints per slot (used for partial-prefix recovery, e.g. SWA). Default: `32`. | `ctx_checkpoints:16` |
| `checkpoint_min_step` or `checkpoint_min_spacing` (aliases: `checkpoint_every_nt`, `checkpoint_every_n_tokens`) | integer | Minimum spacing in tokens between context checkpoints. `0` disables the minimum-spacing gate. Default: `256`. (Renamed upstream from `checkpoint_every_nt`; semantics shifted from a fixed cadence to a minimum spacing.) | `checkpoint_min_step:1024` |
| `split_mode` or `sm` | string | How to split the model across multiple GPUs: `none` (single GPU only), `layer` (default — split layers and KV across GPUs), `row` (split rows across GPUs), `tensor` (experimental tensor parallelism, requires `flash_attention: true`, manually set `context_size`, and a llama.cpp build that includes [#19378](https://github.com/ggml-org/llama.cpp/pull/19378); it historically also required KV-cache quantization to be disabled, but [#23792](https://github.com/ggml-org/llama.cpp/pull/23792) lifts that restriction so `cache_type_k`/`cache_type_v` quantization can be combined with tensor parallelism on builds that include it). | `split_mode:tensor` |
| `split_mode` or `sm` | string | How to split the model across multiple GPUs: `none` (single GPU only), `layer` (default - split layers and KV across GPUs), `row` (split rows across GPUs), `tensor` (experimental tensor parallelism, requires `flash_attention: true`, manually set `context_size`, and a llama.cpp build that includes [#19378](https://github.com/ggml-org/llama.cpp/pull/19378); it historically also required KV-cache quantization to be disabled, but [#23792](https://github.com/ggml-org/llama.cpp/pull/23792) lifts that restriction so `cache_type_k`/`cache_type_v` quantization can be combined with tensor parallelism on builds that include it). | `split_mode:tensor` |
**Example configuration with options:**
@@ -543,7 +543,7 @@ options:
##### Hardware auto-tuning (and how to override it)
On a detected GPU, LocalAI fills a few performance-relevant defaults the model config leaves unset — a larger physical batch on NVIDIA Blackwell, and a VRAM-scaled `parallel` slot count for concurrent serving. Both are gated on **per-device** VRAM at the model's context: when a large context already fills a single card (e.g. a 27B model with a 200k context across 2×16 GiB), the batch boost and the extra parallel slots are suppressed so they can't tip the tighter GPU into CUDA out-of-memory.
On a detected GPU, LocalAI fills a few performance-relevant defaults the model config leaves unset - a larger physical batch on NVIDIA Blackwell, and a VRAM-scaled `parallel` slot count for concurrent serving. Both are gated on **per-device** VRAM at the model's context: when a large context already fills a single card (e.g. a 27B model with a 200k context across 2×16 GiB), the batch boost and the extra parallel slots are suppressed so they can't tip the tighter GPU into CUDA out-of-memory.
Anything you set explicitly in the model YAML always wins, so to pin a value just set it (e.g. `batch: 512` or `options: ["parallel:1"]`). The effective values are logged at `INFO` when a model loads (`effective runtime tuning …`). To turn the hardware auto-tuning off entirely and run llama.cpp's stock behavior, set:
@@ -579,7 +579,7 @@ Set `cache_ram:0` to opt out of the prompt cache entirely (saves host RAM at the
### ik_llama.cpp
[ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) is a hard fork of `llama.cpp` by Iwan Kawrakow that focuses on superior CPU and hybrid GPU/CPU performance. It ships additional quantization types (IQK quants), custom quantization mixes, Multi-head Latent Attention (MLA) for DeepSeek models, and fine-grained tensor offload controls — particularly useful for running very large models on commodity CPU hardware.
[ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) is a hard fork of `llama.cpp` by Iwan Kawrakow that focuses on superior CPU and hybrid GPU/CPU performance. It ships additional quantization types (IQK quants), custom quantization mixes, Multi-head Latent Attention (MLA) for DeepSeek models, and fine-grained tensor offload controls - particularly useful for running very large models on commodity CPU hardware.
{{% notice note %}}
@@ -597,7 +597,7 @@ The `ik-llama-cpp` backend supports the following features:
#### Setup
The backend is distributed as a separate container image and can be installed from the LocalAI backend gallery, or specified directly in a model configuration. GGUF models loaded with this backend benefit from ik_llama.cpp's optimized CPU kernels — especially useful for MoE models and large quantized models that would otherwise be GPU-bound.
The backend is distributed as a separate container image and can be installed from the LocalAI backend gallery, or specified directly in a model configuration. GGUF models loaded with this backend benefit from ik_llama.cpp's optimized CPU kernels - especially useful for MoE models and large quantized models that would otherwise be GPU-bound.
#### YAML configuration
@@ -620,7 +620,7 @@ The aliases `ik-llama` and `ik_llama` are also accepted.
### turboquant (llama.cpp fork with TurboQuant KV-cache)
[llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant) is a `llama.cpp` fork that adds the **TurboQuant KV-cache** quantization scheme. It reuses the upstream `llama.cpp` codebase and ships as a drop-in alternative backend inside LocalAI, sharing the same gRPC server sources as the stock `llama-cpp` backend — so any GGUF model that runs on `llama-cpp` also runs on `turboquant`.
[llama-cpp-turboquant](https://github.com/TheTom/llama-cpp-turboquant) is a `llama.cpp` fork that adds the **TurboQuant KV-cache** quantization scheme. It reuses the upstream `llama.cpp` codebase and ships as a drop-in alternative backend inside LocalAI, sharing the same gRPC server sources as the stock `llama-cpp` backend - so any GGUF model that runs on `llama-cpp` also runs on `turboquant`.
You would pick `turboquant` when you want **smaller KV-cache memory pressure** (longer contexts on the same VRAM) or to experiment with the fork's quantized KV representations on top of the standard `cache_type_k` / `cache_type_v` knobs already supported by upstream `llama.cpp`.
@@ -661,7 +661,7 @@ cache_type_v: turbo3
context_size: 8192
```
The `cache_type_k` / `cache_type_v` fields map to llama.cpp's `-ctk` / `-ctv` flags. The stock `llama-cpp` backend only accepts the standard llama.cpp types — to use `turbo2` / `turbo3` / `turbo4` you need this `turboquant` backend, which is where the fork's TurboQuant code paths actually take effect. Pick `q8_0` here and you're just running stock llama.cpp KV quantization; pick `turbo*` and you're running TurboQuant.
The `cache_type_k` / `cache_type_v` fields map to llama.cpp's `-ctk` / `-ctv` flags. The stock `llama-cpp` backend only accepts the standard llama.cpp types - to use `turbo2` / `turbo3` / `turbo4` you need this `turboquant` backend, which is where the fork's TurboQuant code paths actually take effect. Pick `q8_0` here and you're just running stock llama.cpp KV quantization; pick `turbo*` and you're running TurboQuant.
#### Reference
@@ -736,13 +736,13 @@ engine_args:
The shape of `speculative_config` follows vLLM's
[`SpeculativeConfig`](https://docs.vllm.ai/en/latest/api/vllm/config/speculative.html)
— `method` picks the algorithm, the remaining keys are method-specific.
- `method` picks the algorithm, the remaining keys are method-specific.
Drafters from [z-lab](https://huggingface.co/z-lab) are paired with
specific target models; pick the one that matches your target. The
drafter loads in its native precision regardless of the target's
`quantization:` setting.
Another example — picking a non-default attention backend (e.g. on
Another example - picking a non-default attention backend (e.g. on
hardware where the default cutlass kernels aren't supported):
```yaml
@@ -784,7 +784,7 @@ The backend will pull the model from HuggingFace on first load.
The same `engine_args:` map that the vLLM backend accepts is also
honoured by the SGLang backend. Keys are validated against
[`ServerArgs`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py)
— SGLang's central configuration dataclass — and forwarded verbatim to
- SGLang's central configuration dataclass - and forwarded verbatim to
`Engine(**kwargs)`. Unknown keys fail at load time with the closest
valid name as a hint. Unlike vLLM, `ServerArgs` is flat: speculative
decoding fields are top-level (`speculative_algorithm`,
@@ -805,7 +805,7 @@ candidate tokens per target step, which SGLang then verifies in
parallel. Flags below are transcribed verbatim from the
[SGLang Gemma 4 cookbook](https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4#speculative-decoding-mtp-server-commands).
For consumer GPUs in the 16–24 GB range, use **E4B** (8 B total /
For consumer GPUs in the 16-24 GB range, use **E4B** (8 B total /
4 B effective parameters):
```yaml
@@ -828,12 +828,12 @@ engine_args:
speculative_eagle_topk: 1
```
For smaller cards (8–12 GB), drop to **E2B** (5 B total / 2 B effective)
For smaller cards (8-12 GB), drop to **E2B** (5 B total / 2 B effective)
by swapping the model paths to `google/gemma-4-E2B-it` and
`google/gemma-4-E2B-it-assistant`; the rest of the flags stay the same.
`NEXTN` is normalised to `EAGLE` inside `ServerArgs.__post_init__`, so
either value works — the cookbook uses `NEXTN`. `mem_fraction_static`
either value works - the cookbook uses `NEXTN`. `mem_fraction_static`
is the share of GPU memory SGLang reserves for the model + KV pool;
0.85 is the cookbook's default and adapts to whatever single GPU the
backend is running on.
@@ -854,14 +854,14 @@ single-GPU recipes.
EAGLE-style draft head), `DFLASH` (block-diffusion drafters from
[z-lab](https://huggingface.co/z-lab) for the Qwen3 family), `STANDALONE`
(a smaller draft LLM verifying a larger target), and `NGRAM` (no draft
model — pure prefix-history speculation). See SGLang's
model - pure prefix-history speculation). See SGLang's
[speculative-decoding docs](https://docs.sglang.io/advanced_features/speculative_decoding.html)
for the full algorithm matrix.
#### Tool calling and reasoning parsers
SGLang's native parsers stream `tool_calls` and `reasoning_content`
inside `ChatDelta` — the LocalAI Python backend wires them up
inside `ChatDelta` - the LocalAI Python backend wires them up
per-request rather than via `engine_args:`. Pick a parser by name:
```yaml

53
docs/content/features/text-to-audio.md
View File

@@ -2,8 +2,9 @@
+++
disableToc = false
title = "Text to Audio (TTS)"
weight = 11
weight = 31
url = "/features/text-to-audio/"
aliases = ["/features/sound-generation/"]
+++
## API Compatibility
@@ -38,7 +39,7 @@ Administrators can manage reusable voice-cloning references from **Operate → V
3. Confirm that you have permission to clone the voice, then save the profile.
4. Open **Text to Speech**, choose a model marked **Cloning ready**, and select the saved voice.
The browser converts uploads and recordings to mono, 24 kHz, 16-bit PCM WAV so the same profile works across compatible backends. Clips must be between 1 and 120 seconds and no larger than 50 MiB; 6–30 seconds of clean, single-speaker audio is recommended. Profile audio is private biometric source material: LocalAI stores it below its configured data path, serves previews only to authenticated TTS users, and never returns its filesystem path.
The browser converts uploads and recordings to mono, 24 kHz, 16-bit PCM WAV so the same profile works across compatible backends. Clips must be between 1 and 120 seconds and no larger than 50 MiB; 6-30 seconds of clean, single-speaker audio is recommended. Profile audio is private biometric source material: LocalAI stores it below its configured data path, serves previews only to authenticated TTS users, and never returns its filesystem path.
### Voice profile API
@@ -253,9 +254,9 @@ curl http://localhost:8080/v1/audio/speech -H "Content-Type: application/json" -
}' --output music.flac
```
**Advanced mode** (using the `/sound` endpoint):
**Advanced mode** (using the `/v1/sound-generation` endpoint):
```bash
curl http://localhost:8080/sound -H "Content-Type: application/json" -d '{
curl http://localhost:8080/v1/sound-generation -H "Content-Type: application/json" -d '{
"model": "ace-step-turbo",
"caption": "A funky Japanese disco track",
"lyrics": "[Verse 1]\n...",
@@ -266,6 +267,40 @@ curl http://localhost:8080/sound -H "Content-Type: application/json" -d '{
}' --output music.flac
```
#### Music and sound generation API (`/v1/sound-generation`)
The `/v1/sound-generation` endpoint is compatible with the [ElevenLabs sound generation API](https://elevenlabs.io/docs/api-reference/sound-generation) and can produce music, sound effects, and other audio content. It responds with a binary audio file and the appropriate `Content-Type` header (for example `audio/wav`, `audio/mpeg`, `audio/flac`, or `audio/ogg`). The request body is JSON and supports two usage modes.
**Simple mode:**
| Parameter | Type | Required | Description |
|------------------|----------|----------|----------------------------------------------|
| `model_id` | `string` | Yes | Model identifier (for example `ace-step-turbo`) |
| `text` | `string` | Yes | Audio description or prompt |
| `instrumental` | `bool` | No | Generate instrumental audio (no vocals) |
| `vocal_language` | `string` | No | Language code for vocals (for example `bn`, `ja`) |
**Advanced mode:**
| Parameter | Type | Required | Description |
|---------------------|----------|----------|-------------------------------------------------|
| `model_id` | `string` | Yes | Model identifier (for example `ace-step-turbo`) |
| `text` | `string` | Yes | Text prompt or description |
| `duration_seconds` | `float` | No | Target duration in seconds |
| `prompt_influence` | `float` | No | Temperature / prompt influence parameter |
| `do_sample` | `bool` | No | Enable sampling |
| `think` | `bool` | No | Enable extended thinking for generation |
| `caption` | `string` | No | Caption describing the audio |
| `lyrics` | `string` | No | Lyrics for the generated audio |
| `bpm` | `int` | No | Beats per minute |
| `keyscale` | `string` | No | Musical key/scale (for example `Ab major`) |
| `language` | `string` | No | Language code |
| `vocal_language` | `string` | No | Vocal language (fallback if `language` is empty) |
| `timesignature` | `string` | No | Time signature (for example `4`) |
| `instrumental` | `bool` | No | Generate instrumental audio (no vocals) |
Error responses: `400` for a missing or invalid model or request parameters, and `500` for a backend error during sound generation.
#### Configuration
You can configure ACE-Step models with various options:
@@ -323,14 +358,14 @@ tts:
# Available English voices: Carter, Davis, Emma, Frank, Grace, Mike
```
Then you can use the model:
```
{{% notice note %}}
The realtime 0.5B preset model is not advertised to the Voice Library because it does not accept a raw reference WAV per request. For Voice Library profiles, use a `vibevoice-cpp` 1.5B reference-WAV model; LocalAI detects the 1.5B variant automatically, or a custom name can set `tts.voice_cloning: true`.
{{% /notice %}}
curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
Then you can use the model:
```bash
curl http://localhost:8080/tts -H "Content-Type: application/json" -d '{
"model": "vibevoice",
"input":"Hello!"
}' | aplay

176
docs/content/features/video-generation.md
View File

@@ -1,8 +1,9 @@
+++
disableToc = false
title = "Video Generation"
weight = 18
weight = 51
url = "/features/video-generation/"
aliases = ["/features/longcat-video/"]
+++
LocalAI can generate videos from text prompts and optional image or audio conditioning via the `/video` endpoint. Supported backends include `diffusers`, `stablediffusion`, `vllm-omni`, and the dedicated `longcat-video` backend.
@@ -51,13 +52,19 @@ Returns an OpenAI-compatible JSON response:
## Usage
First install a video-generation model from the gallery (the examples below use `longcat-video`):
```bash
local-ai run longcat-video
```
### Generate a video from a text prompt
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d '{
"model": "video-model",
"model": "longcat-video",
"prompt": "A cat playing in a garden on a sunny day",
"width": 512,
"height": 512,
@@ -86,7 +93,7 @@ curl http://localhost:8080/video \
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d '{
"model": "video-model",
"model": "longcat-video",
"prompt": "A timelapse of flowers blooming",
"start_image": "https://example.com/flowers.jpg",
"num_frames": 24,
@@ -103,7 +110,7 @@ curl http://localhost:8080/video \
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d '{
"model": "video-model",
"model": "longcat-video",
"prompt": "Ocean waves on a beach",
"response_format": "b64_json"
}'
@@ -111,7 +118,166 @@ curl http://localhost:8080/video \
## LongCat-Video and Avatar 1.5
The dedicated `longcat-video` backend serves the official base and Avatar 1.5 checkpoints, including CUDA 13 ARM64 systems such as DGX Spark. See [LongCat Video and Avatar]({{%relref "features/longcat-video" %}}) for gallery installation, Studio instructions, complete model YAML, API examples, tuning options, and hardware requirements.
LocalAI's `longcat-video` backend serves Meituan's official LongCat video-generation models through the `/video` API and the Studio **Video** page.
| Gallery model | Upstream checkpoint | Inputs | Output |
|---------------|---------------------|--------|--------|
| `longcat-video` | `meituan-longcat/LongCat-Video` | text, optional start image | video |
| `longcat-video-avatar-1.5` | `meituan-longcat/LongCat-Video-Avatar-1.5` | text, audio, optional portrait | video with the source audio |
The base checkpoint supports text-to-video and image-to-video. Avatar 1.5 adds audio-driven character animation, optional portrait conditioning, and continuation segments for longer speech.
{{% notice warning %}}
LongCat is a large, CUDA-only model family. LocalAI publishes this backend for Linux with NVIDIA CUDA 12 or CUDA 13 on x86_64 and CUDA 13 on ARM64. CPU, ROCm, and macOS images are not available. Avatar 1.5 also loads components from the base checkpoint, so reserve substantial disk and GPU or unified memory.
{{% /notice %}}
### Install from the Model Gallery
Install one or both recipes from **Models** in the web UI, or use the CLI:
```bash
local-ai models install longcat-video
local-ai models install longcat-video-avatar-1.5
```
You can also import either official Hugging Face URL. The importer recognizes the two repositories and writes a `longcat-video` model config with the appropriate use case and input/output modalities.
The required OCI backend is installed automatically when LocalAI first loads the model. The hardware detector selects the CUDA 12, CUDA 13, or CUDA 13 ARM64 variant.
#### DGX Spark and NVIDIA ARM64
Use a LocalAI CUDA 13 ARM64 image as described in [GPU acceleration]({{%relref "features/GPU-acceleration" %}}). The backend defaults to PyTorch SDPA, avoiding the FlashAttention dependency that is commonly unavailable on Blackwell ARM64 systems.
For unified-memory systems, start with BF16 (`use_int8:false`, the default). INT8 lowers steady-state DiT memory but can have a higher load-time peak because the full model is materialized before the quantized weights are applied.
### Generate in Studio
1. Open **Studio**, then choose **Video**.
2. Select `longcat-video` or `longcat-video-avatar-1.5`.
3. Enter a prompt and choose `832x480` or `1280x720`.
4. Expand **Reference media** to upload a start image. For Avatar 1.5, upload or record the speech under **Avatar audio**.
5. Select **Generate**.
The base model can run without a reference image for text-to-video. Avatar 1.5 requires audio; the portrait is optional.
### LongCat API examples
#### Text-to-video
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d '{
"model": "longcat-video",
"prompt": "A cinematic tracking shot through a misty redwood forest",
"width": 832,
"height": 480,
"num_frames": 93,
"fps": 15
}'
```
#### Image-to-video
`start_image` accepts raw base64, a browser-style data URI, or a public HTTP(S) URL:
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d "{
\"model\": \"longcat-video\",
\"prompt\": \"The subject turns toward the camera as leaves move in the breeze\",
\"start_image\": \"$(base64 --wrap=0 portrait.png)\",
\"params\": {
\"resolution\": \"480p\"
}
}"
```
#### Avatar from speech and a portrait
`audio` accepts raw base64, a data URI, or a public HTTP(S) URL. Each staged image or audio input is limited to 128 MiB.
```bash
curl http://localhost:8080/video \
-H "Content-Type: application/json" \
-d "{
\"model\": \"longcat-video-avatar-1.5\",
\"prompt\": \"A friendly presenter speaking naturally to camera\",
\"start_image\": \"$(base64 --wrap=0 portrait.png)\",
\"audio\": \"$(base64 --wrap=0 speech.wav)\",
\"width\": 832,
\"height\": 480,
\"params\": {
\"offload_kv_cache\": \"true\"
}
}"
```
Avatar output is generated at 25 FPS and is muxed with the submitted audio. When neither `num_frames` nor `params.num_segments` is provided, LocalAI derives the continuation count from the audio duration, up to the model's `max_segments` setting.
### LongCat model configuration
The gallery and importer make each model self-describing. A manual Avatar 1.5 config looks like this:
```yaml
name: longcat-video-avatar-1.5
backend: longcat-video
known_usecases:
- video
known_input_modalities:
- text
- image
- audio
known_output_modalities:
- video
options:
- attention_backend:sdpa
- use_distill:true
- max_segments:8
parameters:
model: meituan-longcat/LongCat-Video-Avatar-1.5
```
The explicit modality declarations are used by `GET /v1/models/capabilities` and attachment-aware clients. They avoid inferring model behavior from backend or checkpoint names.
#### Load options
Model load options use `key:value` entries in `options`:
| Option | Default | Description |
|--------|---------|-------------|
| `attention_backend` | `sdpa` | `sdpa`, `auto`, `flash2`, `flash3`, or `xformers`; packaged images guarantee `sdpa` |
| `use_distill` | Avatar: `true`; base: `false` | Use the checkpoint's accelerated distillation path |
| `use_int8` | `false` | Use Avatar 1.5's INT8 DiT; unsupported by the base model |
| `base_model` | `meituan-longcat/LongCat-Video` | Base tokenizer, text encoder, and VAE used by Avatar 1.5 |
| `max_segments` | `8` | Maximum continuation segments accepted for one request |
| `resolution` | `480p` | Default image-conditioned resolution: `480p` or `720p` |
The initial backend supports one GPU per process. Tensor or context parallel sizes above one are rejected.
#### Per-request parameters
The `/video` request's `params` object accepts string values:
| Parameter | Description |
|-----------|-------------|
| `num_segments` | Explicit number of Avatar continuation segments |
| `audio_guidance_scale` | Audio classifier-free guidance when distillation is disabled |
| `offload_kv_cache` | Offload continuation KV cache (`true` or `false`) |
| `ref_img_index` | Reference-frame index used during continuation |
| `mask_frame_range` | Number of frames blended around continuation boundaries |
| `resolution` | Per-request image-conditioned resolution (`480p` or `720p`) |
With distillation enabled, Avatar uses eight inference steps and fixed text/audio guidance of `1.0`. Disable `use_distill` in the model config before tuning `step`, `cfg_scale`, or `audio_guidance_scale`.
### LongCat troubleshooting
- **HTTP 400, audio is required**: Avatar 1.5 was selected without `audio`.
- **HTTP 400, request needs too many segments**: trim the audio or raise `max_segments` in the model options.
- **HTTP 412**: the installed LocalAI runtime cannot select a compatible NVIDIA backend image.
- **Out of memory while loading**: use BF16 on unified-memory hardware, close other GPU workloads, or reduce model concurrency. INT8 is not guaranteed to reduce peak load memory.
- **Slow first request**: the backend and checkpoints are downloaded and loaded on demand; subsequent requests reuse the loaded pipeline.
## Error Responses

35
docs/content/features/voice-activity-detection.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Voice Activity Detection (VAD)"
weight = 17
weight = 35
url = "/features/voice-activity-detection/"
+++
@@ -35,13 +35,34 @@ Returns a JSON object with detected speech segments:
### Example request
The `/v1/vad` endpoint expects the `audio` field to be an array of raw
16kHz mono PCM samples as `float32` values, so the request body is usually
built from a real audio file rather than typed by hand.
First convert any audio file to 16kHz mono with ffmpeg:
```bash
curl http://localhost:8080/v1/vad \
-H "Content-Type: application/json" \
-d '{
"model": "silero-vad",
"audio": [0.0012, -0.0045, 0.0053, -0.0021, ...]
}'
ffmpeg -i input.mp3 -ar 16000 -ac 1 -f wav speech.wav
```
Then load the samples and POST them (this snippet needs
`pip install soundfile numpy requests`):
```python
import soundfile as sf
import numpy as np
import requests
audio, sample_rate = sf.read("speech.wav")
if audio.ndim > 1:
audio = audio.mean(axis=1) # downmix to mono
samples = audio.astype(np.float32).tolist()
response = requests.post(
"http://localhost:8080/v1/vad",
json={"model": "silero-vad", "audio": samples},
)
print(response.json())
```
### Example response

30
docs/content/features/voice-recognition.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Voice Recognition"
weight = 15
weight = 36
url = "/features/voice-recognition/"
+++
@@ -96,11 +96,11 @@ under one image.
| Gallery entry | Model | Size | License |
|---|---|---|---|
| `speechbrain-ecapa-tdnn` | ECAPA-TDNN on VoxCeleb (SpeechBrain) | ~17 MB | **Apache 2.0 — commercial-safe** |
| `wespeaker-resnet34` | WeSpeaker ResNet34 ONNX | ~26 MB | **Apache 2.0 — commercial-safe** |
| `speechbrain-ecapa-tdnn` | ECAPA-TDNN on VoxCeleb (SpeechBrain) | ~17 MB | **Apache 2.0 - commercial-safe** |
| `wespeaker-resnet34` | WeSpeaker ResNet34 ONNX | ~26 MB | **Apache 2.0 - commercial-safe** |
Both entries are commercial-safe Apache-2.0. SpeechBrain is the
default — it's a lightweight pure-PyTorch checkpoint that auto-
default - it's a lightweight pure-PyTorch checkpoint that auto-
downloads on first use. The `wespeaker-resnet34` entry wires the
direct-ONNX path for CPU-only deployments that don't want the torch
runtime.
@@ -181,7 +181,7 @@ hood.
**Storage caveat.** The default vector store is in-memory. All
registered speakers are lost when LocalAI restarts. Persistent storage
(pgvector) is a tracked future enhancement shared with face
recognition — the voice-recognition HTTP API is designed to swap the
recognition - the voice-recognition HTTP API is designed to swap the
backing store without changing the wire format.
{{% /notice %}}
@@ -194,7 +194,7 @@ backing store without changing the wire format.
| `model` | string | gallery entry name (e.g. `speechbrain-ecapa-tdnn`) |
| `audio1`, `audio2` | string | URL, base64, or data-URI of an audio file |
| `threshold` | float, optional | cosine-distance cutoff; default 0.25 for ECAPA-TDNN |
| `anti_spoofing` | bool, optional | reserved — unused in the current release |
| `anti_spoofing` | bool, optional | reserved - unused in the current release |
Returns `verified`, `distance`, `threshold`, `confidence`, `model`,
and `processing_time_ms`.
@@ -211,7 +211,7 @@ speech:
| `actions` | string[] | subset of `["age","gender","emotion"]`; empty = all supported |
Emotion is inferred from the SUPERB emotion-recognition checkpoint
(`superb/wav2vec2-base-superb-er`, Apache 2.0) — 4-way categorical
(`superb/wav2vec2-base-superb-er`, Apache 2.0) - 4-way categorical
neutral / happy / angry / sad. The model auto-downloads on the first
analyze call.
@@ -232,7 +232,7 @@ attributes it could compute. When nothing can be computed the backend
returns `501 Unimplemented`.
Analyze is supported by both `speechbrain-ecapa-tdnn` and
`wespeaker-resnet34` — the speaker recognizer and the analysis head
`wespeaker-resnet34` - the speaker recognizer and the analysis head
are independent.
### `POST /v1/voice/register` (1:N enrollment)
@@ -285,7 +285,7 @@ depends on the recognizer: 192 for ECAPA-TDNN, 256 for WeSpeaker
ResNet34.
> **Note:** the OpenAI-compatible `/v1/embeddings` endpoint is
> intentionally text-only — it does nothing useful with audio input.
> intentionally text-only - it does nothing useful with audio input.
> Use `/v1/voice/embed` for audio.
## Audio input
@@ -298,7 +298,7 @@ before the gRPC call. All audio fields accept:
- Raw base64 (no prefix).
- Data URIs (`data:audio/wav;base64,...`).
The backend itself always receives a filesystem path — the same
The backend itself always receives a filesystem path - the same
convention the Whisper / Voxtral transcription backends use.
## Threshold reference
@@ -309,17 +309,17 @@ convention the Whisper / Voxtral transcription backends use.
| WeSpeaker ResNet34 | ~0.30 |
| 3D-Speaker ERes2Net | ~0.28 |
Pass `threshold` explicitly when switching recognizers — the per-model
Pass `threshold` explicitly when switching recognizers - the per-model
default only applies when omitted.
## Related features
- [Face Recognition](/features/face-recognition/) — the image analog;
- [Face Recognition](/features/face-recognition/) - the image analog;
the two share a registry design.
- [Audio to Text](/features/audio-to-text/) — transcription (Whisper,
- [Audio to Text](/features/audio-to-text/) - transcription (Whisper,
Voxtral, faster-whisper). Runs in addition to, not instead of,
voice recognition.
- [Stores](/features/stores/) — the generic vector store powering
- [Stores](/features/stores/) - the generic vector store powering
both the face and voice 1:N recognition pipelines.
- [Embeddings](/features/embeddings/) — text-only OpenAI-compatible
- [Embeddings](/features/embeddings/) - text-only OpenAI-compatible
embedding endpoint; for audio embeddings use `/v1/voice/embed`.

21
docs/content/getting-started/_index.en.md
View File

@@ -2,23 +2,26 @@
+++
disableToc = false
title = "Getting started"
weight = 3
weight = 2
icon = "rocket_launch"
type = "chapter"
+++
Welcome to LocalAI! This section covers everything you need to know **after installation** to start using LocalAI effectively.
Welcome to LocalAI! This section takes you from a fresh install to a working chat, a working API call, and your first customized model. Follow the pages in order the first time through; each one builds on the previous.
{{% notice tip %}}
**Haven't installed LocalAI yet?**
See the [Installation guide](/installation/) to install LocalAI first. **Docker is the recommended installation method** for most users.
See the [Installation guide]({{% relref "getting-started/install" %}}) first. **Docker is the recommended installation method** for most users.
{{% /notice %}}
## What's in This Section
## The path
- **[Quickstart Guide](quickstart/)** - Get started quickly with your first API calls and model downloads
- **[Install and Run Models](models/)** - Learn how to install, configure, and run AI models
- **[Customize Models](customize-model/)** - Customize model configurations and prompt templates
- **[Container Images Reference](container-images/)** - Complete reference for available Docker images
- **[Try It Out](try-it-out/)** - Explore examples and use cases
1. **Install** LocalAI (see the [Installation guide]({{% relref "getting-started/install" %}})).
2. **[Quickstart]({{% relref "getting-started/quickstart" %}})** - start LocalAI, install your first model, and get your first chat reply from both the web interface and the API.
3. **[Install and run models]({{% relref "getting-started/models" %}})** - the different ways to install models (gallery, Hugging Face, OCI, local files) and run them.
4. **[Try it out]({{% relref "getting-started/try-it-out" %}})** - copy-paste examples for every OpenAI-compatible endpoint (chat, vision, functions, audio, images, embeddings).
5. **[Customize models]({{% relref "getting-started/customize-model" %}})** - edit model configuration and prompt templates to tune behavior.
6. **[Troubleshooting]({{% relref "getting-started/troubleshooting" %}})** - fixes for the most common install and startup problems.
Once you have a model answering, the [Build your first agent]({{% relref "getting-started/first-agent" %}}) walkthrough shows how to turn that model into an autonomous agent that can use tools.

183
docs/content/getting-started/build.md
View File

@@ -1,12 +1,183 @@
+++
disableToc = false
title = "Build LocalAI from source"
weight = 6
title = "Build LocalAI"
icon = "model_training"
weight = 12
url = '/basics/build/'
ico = "rocket_launch"
+++
Building LocalAI from source is an installation method that allows you to compile LocalAI yourself, which is useful for custom configurations, development, or when you need specific build options.
For complete build instructions, see the [Build from Source](/installation/build/) documentation in the Installation section.
### Build
LocalAI can be built as a container image or as a single, portable binary. Note that some model architectures might require Python libraries, which are not included in the binary.
LocalAI's extensible architecture allows you to add your own backends, which can be written in any language, and as such the container images contains also the Python dependencies to run all the available backends (for example, in order to run backends like __Diffusers__ that allows to generate images and videos from text).
This section contains instructions on how to build LocalAI from source.
#### Build LocalAI locally
##### Requirements
In order to build LocalAI locally, you need the following requirements:
- Golang >= 1.21
- GCC
- GRPC
To install the dependencies follow the instructions below:
{{< tabs >}}
{{% tab title="Apple" %}}
Install `xcode` from the App Store
```bash
brew install go protobuf protoc-gen-go protoc-gen-go-grpc wget
```
{{% /tab %}}
{{% tab title="Debian" %}}
```bash
apt install golang make protobuf-compiler-grpc
```
After you have golang installed and working, you can install the required binaries for compiling the golang protobuf components via the following commands
```bash
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
```
{{% /tab %}}
{{% tab title="From source" %}}
```bash
make build
```
{{% /tab %}}
{{< /tabs >}}
##### Build
To build LocalAI with `make`:
```
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
make build
```
This should produce the binary `local-ai`
#### Container image
Requirements:
- Docker or podman, or a container engine
In order to build the `LocalAI` container image locally you can use `docker`, for example:
```
docker build -t localai .
docker run localai
```
### Example: Build on mac
Building on Mac (M1, M2 or M3) works, but you may need to install some prerequisites using `brew`.
The below has been tested by one mac user and found to work. Note that this doesn't use Docker to run the server:
Install `xcode` from the Apps Store (needed for metalkit)
```
brew install abseil cmake go grpc protobuf wget protoc-gen-go protoc-gen-go-grpc
git clone https://github.com/go-skynet/LocalAI.git
cd LocalAI
make build
wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf -O models/phi-2.Q2_K
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/phi-2.Q2_K.tmpl
./local-ai backends install llama-cpp
./local-ai --models-path=./models/ --debug=true
curl http://localhost:8080/v1/models
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2.Q2_K",
"messages": [{"role": "user", "content": "How are you?"}],
"temperature": 0.9
}'
```
#### Troubleshooting mac
- If you encounter errors regarding a missing utility metal, install `Xcode` from the App Store.
- After the installation of Xcode, if you receive a xcrun error `'xcrun: error: unable to find utility "metal", not a developer tool or in PATH'`. You might have installed the Xcode command line tools before installing Xcode, the former one is pointing to an incomplete SDK.
```
xcode-select --print-path
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
```
- If completions are slow, ensure that `gpu-layers` in your model yaml matches the number of layers from the model in use (or simply use a high number such as 256).
- If you get a compile error: `error: only virtual member functions can be marked 'final'`, reinstall all the necessary brew packages, clean the build, and try again.
```
brew reinstall go grpc protobuf wget
make clean
make build
```
## Build backends
LocalAI have several backends available for installation in the backend gallery. The backends can be also built by source. As backends might vary from language and dependencies that they require, the documentation will provide generic guidance for few of the backends, which can be applied with some slight modifications also to the others.
### Manually
Typically each backend include a Makefile which allow to package the backend.
In the LocalAI repository, for instance you can build a backend by doing:
```
git clone https://github.com/go-skynet/LocalAI.git
make -C LocalAI/backend/python/vllm
```
### With Docker
Building with docker is simpler as abstracts away all the requirement, and focuses on building the final OCI images that are available in the gallery. This allows for instance also to build locally a backend and install it with LocalAI. You can refer to [Backends](https://localai.io/backends/) for general guidance on how to install and develop backends.
In the LocalAI repository, you can build a backend by doing:
```
git clone https://github.com/go-skynet/LocalAI.git
make docker-build-<backend-name>
```
Note that `make` is only by convenience, in reality it just runs a simple `docker` command as:
```bash
docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:<backend-name> -f LocalAI/backend/Dockerfile.golang --build-arg BACKEND=<backend-name> .
```
Note:
- BUILD_TYPE can be either: `cublas`, `hipblas`, `sycl_f16`, `sycl_f32`, `metal`.
- BASE_IMAGE is tested on `ubuntu:24.04` (and defaults to it) and `quay.io/go-skynet/intel-oneapi-base:latest` for intel/sycl

126
docs/content/getting-started/container-images.md
View File

@@ -1,126 +0,0 @@
+++
disableToc = false
title = "Run with container images"
weight = 6
url = '/basics/container/'
ico = "rocket_launch"
+++
LocalAI provides a variety of images to support different environments. These images are available on [quay.io](https://quay.io/repository/go-skynet/local-ai?tab=tags) and [Docker Hub](https://hub.docker.com/r/localai/localai).
For GPU Acceleration support for Nvidia video graphic cards, use the Nvidia/CUDA images, if you don't have a GPU, use the CPU images. If you have AMD or Mac Silicon, see the [build section]({{%relref "installation/build" %}}).
{{% notice tip %}}
**Available Images Types**:
- Images ending with `-core` are smaller images without predownload python dependencies. Use these images if you plan to use `llama.cpp`, `stablediffusion-ncn` or `rwkv` backends - if you are not sure which one to use, do **not** use these images.
{{% /notice %}}
#### Prerequisites
Before you begin, ensure you have a container engine installed if you are not using the binaries. Suitable options include Docker or Podman. For installation instructions, refer to the following guides:
- [Install Docker Desktop (Mac, Windows, Linux)](https://docs.docker.com/get-docker/)
- [Install Podman (Linux)](https://podman.io/getting-started/installation)
- [Install Docker engine (Servers)](https://docs.docker.com/engine/install/#get-started)
{{% notice tip %}}
**Hardware Requirements:** The hardware requirements for LocalAI vary based on the model size and quantization method used. For performance benchmarks with different backends, such as `llama.cpp`, visit [this link](https://github.com/ggerganov/llama.cpp#memorydisk-requirements). The `rwkv` backend is noted for its lower resource consumption.
{{% /notice %}}
## Standard container images
Standard container images do not have pre-installed models. Use these if you want to configure models manually.
{{< tabs >}}
{{% tab title="Vanilla / CPU Images" %}}
| Description | Quay | Docker Hub |
| --- | --- |-----------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master` | `localai/localai:master` |
| Latest tag | `quay.io/go-skynet/local-ai:latest` | `localai/localai:latest` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}` | `localai/localai:{{< version >}}` |
{{% /tab %}}
{{% tab title="GPU Images CUDA 12" %}}
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-12` | `localai/localai:master-gpu-nvidia-cuda-12` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-nvidia-cuda-12` | `localai/localai:latest-gpu-nvidia-cuda-12` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-nvidia-cuda-12` | `localai/localai:{{< version >}}-gpu-nvidia-cuda-12` |
{{% /tab %}}
{{% tab title="GPU Images CUDA 13" %}}
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-13` | `localai/localai:master-gpu-nvidia-cuda-13` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-nvidia-cuda-13` | `localai/localai:latest-gpu-nvidia-cuda-13` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-nvidia-cuda-13` | `localai/localai:{{< version >}}-gpu-nvidia-cuda-13` |
{{% /tab %}}
{{% tab title="Intel GPU" %}}
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-intel` | `localai/localai:master-gpu-intel` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-intel` | `localai/localai:latest-gpu-intel` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-intel` | `localai/localai:{{< version >}}-gpu-intel` |
{{% /tab %}}
{{% tab title="AMD GPU" %}}
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-hipblas` | `localai/localai:master-gpu-hipblas` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-hipblas` | `localai/localai:latest-gpu-hipblas` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-hipblas` | `localai/localai:{{< version >}}-gpu-hipblas` |
{{% /tab %}}
{{% tab title="Vulkan Images" %}}
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-vulkan` | `localai/localai:master-vulkan` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-vulkan` | `localai/localai:latest-gpu-vulkan` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-vulkan` | `localai/localai:{{< version >}}-vulkan` |
{{% /tab %}}
{{% tab title="Nvidia Linux for tegra (CUDA 12)" %}}
These images are compatible with Nvidia ARM64 devices with CUDA 12, such as the Jetson Nano, Jetson Xavier NX, and Jetson AGX Orin. For more information, see the [Nvidia L4T guide]({{%relref "reference/nvidia-l4t" %}}).
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64` | `localai/localai:master-nvidia-l4t-arm64` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64` | `localai/localai:latest-nvidia-l4t-arm64` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-nvidia-l4t-arm64` | `localai/localai:{{< version >}}-nvidia-l4t-arm64` |
{{% /tab %}}
{{% tab title="Nvidia Linux for tegra (CUDA 13)" %}}
These images are compatible with Nvidia ARM64 devices with CUDA 13, such as the Nvidia DGX Spark. For more information, see the [Nvidia L4T guide]({{%relref "reference/nvidia-l4t" %}}).
| Description | Quay | Docker Hub |
| --- | --- |-------------------------------------------------------------|
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-cuda-13` | `localai/localai:master-nvidia-l4t-arm64-cuda-13` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13` | `localai/localai:latest-nvidia-l4t-arm64-cuda-13` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-nvidia-l4t-arm64-cuda-13` | `localai/localai:{{< version >}}-nvidia-l4t-arm64-cuda-13` |
{{% /tab %}}
{{< /tabs >}}
## See Also
- [GPU acceleration]({{%relref "features/gpu-acceleration" %}})

99
docs/content/installation/containers.md → docs/content/getting-started/containers.md
View File

@@ -1,8 +1,10 @@
---
title: Containers
description: Install and use LocalAI with container engines (Docker, Podman)
weight: 1
weight: 8
url: '/installation/containers/'
aliases:
- '/basics/container/'
---
LocalAI supports Docker, Podman, and other OCI-compatible container engines. This guide covers the common aspects of running LocalAI in containers.
@@ -231,7 +233,7 @@ After installation:
### GPU not detected
- Ensure Docker has GPU access: `docker run --rm --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi`
- For Podman, see the [Podman installation guide](/installation/podman/#gpu-not-detected)
- For Podman, pass the GPU with the `--device` flags shown in the GPU sections above (for example `--device nvidia.com/gpu=all`)
- For NVIDIA: Install [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)
- For AMD: Ensure devices are accessible: `ls -la /dev/kfd /dev/dri`
@@ -270,9 +272,100 @@ This indicates a Docker/NVIDIA Container Toolkit configuration issue. The contai
- Verify disk space: `df -h`
- Check container logs for errors: `docker logs local-ai` or `podman logs local-ai`
## Full image reference
The quick-start examples above use the Docker Hub image names. Every image is published to both [Docker Hub](https://hub.docker.com/r/localai/localai) and [Quay](https://quay.io/repository/go-skynet/local-ai?tab=tags). The tables below map the Docker Hub tag to its Quay equivalent for each variant. Replace `{{< version >}}` with a released version to pin a specific build.
{{< tabs >}}
{{% tab title="Vanilla / CPU Images" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master` | `localai/localai:master` |
| Latest tag | `quay.io/go-skynet/local-ai:latest` | `localai/localai:latest` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}` | `localai/localai:{{< version >}}` |
{{% /tab %}}
{{% tab title="GPU Images CUDA 12" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-12` | `localai/localai:master-gpu-nvidia-cuda-12` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-nvidia-cuda-12` | `localai/localai:latest-gpu-nvidia-cuda-12` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-nvidia-cuda-12` | `localai/localai:{{< version >}}-gpu-nvidia-cuda-12` |
{{% /tab %}}
{{% tab title="GPU Images CUDA 13" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-13` | `localai/localai:master-gpu-nvidia-cuda-13` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-nvidia-cuda-13` | `localai/localai:latest-gpu-nvidia-cuda-13` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-nvidia-cuda-13` | `localai/localai:{{< version >}}-gpu-nvidia-cuda-13` |
{{% /tab %}}
{{% tab title="Intel GPU" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-intel` | `localai/localai:master-gpu-intel` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-intel` | `localai/localai:latest-gpu-intel` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-intel` | `localai/localai:{{< version >}}-gpu-intel` |
{{% /tab %}}
{{% tab title="AMD GPU" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-hipblas` | `localai/localai:master-gpu-hipblas` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-hipblas` | `localai/localai:latest-gpu-hipblas` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-hipblas` | `localai/localai:{{< version >}}-gpu-hipblas` |
{{% /tab %}}
{{% tab title="Vulkan Images" %}}
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-gpu-vulkan` | `localai/localai:master-gpu-vulkan` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-gpu-vulkan` | `localai/localai:latest-gpu-vulkan` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-gpu-vulkan` | `localai/localai:{{< version >}}-gpu-vulkan` |
{{% /tab %}}
{{% tab title="Nvidia Linux for tegra (CUDA 12)" %}}
These images are compatible with Nvidia ARM64 devices with CUDA 12, such as the Jetson Nano, Jetson Xavier NX, and Jetson AGX Orin. For more information, see the [Nvidia L4T guide]({{%relref "reference/nvidia-l4t" %}}).
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64` | `localai/localai:master-nvidia-l4t-arm64` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64` | `localai/localai:latest-nvidia-l4t-arm64` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-nvidia-l4t-arm64` | `localai/localai:{{< version >}}-nvidia-l4t-arm64` |
{{% /tab %}}
{{% tab title="Nvidia Linux for tegra (CUDA 13)" %}}
These images are compatible with Nvidia ARM64 devices with CUDA 13, such as the Nvidia DGX Spark. For more information, see the [Nvidia L4T guide]({{%relref "reference/nvidia-l4t" %}}).
| Description | Quay | Docker Hub |
| --- | --- | --- |
| Latest images from the branch (development) | `quay.io/go-skynet/local-ai:master-nvidia-l4t-arm64-cuda-13` | `localai/localai:master-nvidia-l4t-arm64-cuda-13` |
| Latest tag | `quay.io/go-skynet/local-ai:latest-nvidia-l4t-arm64-cuda-13` | `localai/localai:latest-nvidia-l4t-arm64-cuda-13` |
| Versioned image | `quay.io/go-skynet/local-ai:{{< version >}}-nvidia-l4t-arm64-cuda-13` | `localai/localai:{{< version >}}-nvidia-l4t-arm64-cuda-13` |
{{% /tab %}}
{{< /tabs >}}
## See Also
- [Container Images Reference](/getting-started/container-images/) - Complete image reference
- [Full image reference](#full-image-reference) - Complete Quay and Docker Hub image matrix
- [Install Models](/getting-started/models/) - Install and configure models
- [GPU Acceleration](/features/gpu-acceleration/) - GPU setup and optimization
- [Kubernetes Installation](/installation/kubernetes/) - Deploy on Kubernetes

2
docs/content/getting-started/customize-model.md
View File

@@ -69,4 +69,4 @@ docker run -p 8080:8080 localai/localai:{{< version >}} https://gist.githubuserc
## Next Steps
- Visit the [advanced section]({{%relref "advanced" %}}) for more insights on prompt templates and configuration files.
- To learn about fine-tuning an LLM model, check out the [fine-tuning section]({{%relref "advanced/fine-tuning" %}}).
- To learn about fine-tuning an LLM model, check out the [fine-tuning section]({{%relref "features/fine-tuning" %}}).

2
docs/content/installation/docker.md → docs/content/getting-started/docker.md
View File

@@ -1,7 +1,7 @@
---
title: "Docker Installation"
description: "Install LocalAI using Docker containers - the recommended installation method"
weight: 2
weight: 13
url: '/installation/docker/'
redirectURI: '/installation/containers/'
---

70
docs/content/getting-started/first-agent.md Normal file
View File

@@ -0,0 +1,70 @@
+++
disableToc = false
title = "Build your first agent"
weight = 7
icon = "smart_toy"
+++
LocalAGI is embedded in LocalAI. There is nothing separate to install or run.
The agent platform ships inside the LocalAI binary and container image, and it is enabled by default. If you already have LocalAI running and a model installed, you have everything you need to build an agent. This page walks you from an empty Agents page to an agent that answers a message and uses one tool.
## Before you start: install a tool-calling model
An agent is a loop around a chat model, so it needs a model that supports tool (function) calling. This guide uses `qwen3-4b`, a small CPU-friendly Qwen3 model that supports tool calling. It is the same model used in the [Quickstart]({{% relref "getting-started/quickstart" %}}), so if you followed that page you already have it.
Install it either from the web interface or from the CLI:
- **Web interface:** open the **Models** page at `http://localhost:8080`, search for `qwen3-4b`, and click **Install**.
- **CLI:**
```bash
local-ai run qwen3-4b
```
For other ways to install models (Hugging Face, OCI, local files), see the [model gallery]({{% relref "features/model-gallery" %}}).
## Give the agent a model
An agent with no model set cannot answer. The agent has nothing to send your message to, so it will fail to respond until you assign it a model. You can set the model in two ways:
- **Per agent:** choose `qwen3-4b` in the agent's **Model** field when you create or edit it (covered below). This is the usual choice.
- **As a default for every new agent:** start LocalAI with an environment variable so new agents are created with that model already selected:
```bash
LOCALAI_AGENT_POOL_DEFAULT_MODEL=qwen3-4b
```
Setting a per-agent model always overrides the default.
## Create the agent
1. Open the **Agents** page in the web interface.
2. Click **Create Agent**.
3. Fill in the form:
- **Name:** for example `helper`.
- **Model:** select `qwen3-4b`.
- **System prompt:** a short instruction that sets the agent's behavior, for example `You are a concise, helpful assistant.`
- **Action:** add one simple action so the agent has a tool to call. A search action is a good first choice. Some actions need credentials (for example an API key); pick one whose requirements you can satisfy, or start with an action that needs none.
4. Save the agent.
## Send a message
Open the new agent from the Agents page and type a message in its chat box, for example `Hello, what can you do?`. The agent replies in the chat panel within a few seconds. When the agent decides to use the action you configured, you will see the tool call and its result appear inline before the final answer, streamed live as the agent works.
That is a complete agent: a model, a system prompt, and one tool, all running inside your LocalAI process.
## If your imported agent will not run
If you imported an agent from the [Agent Hub](https://agenthub.localai.io) or a JSON file and it does not respond, work through this checklist. Each symptom maps to a fix:
- **The agent does not answer at all:** the model it references is not installed. Open the **Models** page and install the model named in the agent's configuration (or change the agent's model to one you have, such as `qwen3-4b`).
- **An action always fails:** the action is missing its API keys or other credentials. Open the agent's action configuration and supply the required keys.
- **A tool times out or is unavailable:** the MCP server that provides it is unreachable. Confirm the MCP server is running and that the agent points at the correct address.
- **A skill the agent expects is not available:** skills are disabled by default. Start LocalAI with `LOCALAI_AGENT_POOL_ENABLE_SKILLS=true` to turn the skills service on (the default is `LOCALAI_AGENT_POOL_ENABLE_SKILLS=false`).
## Next steps
- [Model gallery]({{% relref "features/model-gallery" %}}) - install more models, including larger tool-calling models for more capable agents.
- [Agent actions catalog]({{% relref "features/agent-actions" %}}) - the full list of built-in actions an agent can use and how to configure them.
- [Agent-scoped MCP]({{% relref "features/mcp" %}}) - connect an agent to external Model Context Protocol servers to give it more tools.

43
docs/content/getting-started/install.md Normal file
View File

@@ -0,0 +1,43 @@
---
weight: 1
title: "Install LocalAI"
description: "How to install LocalAI"
icon: download
url: '/installation/'
---
LocalAI can be installed in multiple ways depending on your platform and preferences.
## Video Walkthrough
[![Installation Video](https://img.youtube.com/vi/cMVNnlqwfw4/maxresdefault.jpg)](https://www.youtube.com/watch?v=cMVNnlqwfw4)
## Installation Methods
Choose the installation method that best suits your needs:
1. **[Containers]({{% relref "getting-started/containers" %}})** ⭐ **Recommended** - Works on all platforms, supports Docker and Podman
2. **[macOS]({{% relref "getting-started/macos" %}})** - Download and install the DMG application
3. **[Linux]({{% relref "getting-started/linux" %}})** - Install on Linux using binaries
4. **[Kubernetes]({{% relref "getting-started/kubernetes" %}})** - Deploy LocalAI on Kubernetes clusters
5. **[Build from Source]({{% relref "getting-started/build" %}})** - Build LocalAI from source code
## Quick Start
**Recommended: Containers (Docker or Podman)**
```bash
# With Docker
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
# Or with Podman
podman run -p 8080:8080 --name local-ai -ti localai/localai:latest
```
This will start LocalAI. The API will be available at `http://localhost:8080`.
For other platforms:
- **macOS**: Download the [DMG]({{% relref "getting-started/macos" %}})
- **Linux**: See the [Linux installation guide]({{% relref "getting-started/linux" %}}) for binary installation.
For detailed instructions, see the [Containers installation guide]({{% relref "getting-started/containers" %}}).

2
docs/content/getting-started/kubernetes.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Run with Kubernetes"
weight = 6
weight = 11
url = '/basics/kubernetes/'
ico = "rocket_launch"
+++

19
docs/content/installation/linux.md → docs/content/getting-started/linux.md
View File

@@ -1,7 +1,7 @@
---
title: "Linux Installation"
description: "Install LocalAI on Linux using binaries"
weight: 3
weight: 9
url: '/installation/linux/'
---
@@ -25,6 +25,23 @@ chmod +x local-ai-*
./local-ai-*
```
### Run your first model
Starting the binary on its own gives you an empty server. To get a working chat right away, run LocalAI with a model name and it will download and serve it from the gallery:
```bash
./local-ai-* run qwen3-4b
```
Once it is ready, open the WebUI at `http://localhost:8080` or send a request to the API:
```bash
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "qwen3-4b",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
### System Requirements
Hardware requirements vary based on:

2
docs/content/installation/macos.md → docs/content/getting-started/macos.md
View File

@@ -1,7 +1,7 @@
---
title: "macOS Installation"
description: "Install LocalAI on macOS using the DMG application"
weight: 1
weight: 10
---

14
docs/content/getting-started/models.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Setting Up Models"
weight = 2
weight = 3
icon = "hub"
description = "Learn how to install, configure, and manage models in LocalAI"
+++
@@ -207,7 +207,7 @@ mkdir models
cp your-model.gguf models/
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:latest --models-path /models --context-size 700 --threads 4
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4
curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d '{
"model": "your-model.gguf",
@@ -219,7 +219,7 @@ curl http://localhost:8080/v1/completions -H "Content-Type: application/json" -d
{{% notice tip %}}
**Other Docker Images**:
For other Docker images, please refer to the table in [the container images section]({{% relref "getting-started/container-images" %}}).
For other Docker images, please refer to the table in [the container images section]({{% relref "getting-started/containers" %}}).
{{% /notice %}}
### Example:
@@ -231,7 +231,7 @@ wget https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGUF/resolve/main
cp -rf prompt-templates/getting_started.tmpl models/luna-ai-llama2.tmpl
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm quay.io/go-skynet/local-ai:latest --models-path /models --context-size 700 --threads 4
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4
curl http://localhost:8080/v1/models
@@ -243,7 +243,7 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso
```
{{% notice note %}}
- If running on Apple Silicon (ARM), it is **not** recommended to run on Docker due to emulation. Follow the [build instructions]({{% relref "installation/build" %}}) to use Metal acceleration for full GPU support.
- If running on Apple Silicon (ARM), it is **not** recommended to run on Docker due to emulation. Follow the [build instructions]({{% relref "getting-started/build" %}}) to use Metal acceleration for full GPU support.
- If you are running on Apple x86_64, you can use Docker without additional gain from building it from source.
{{% /notice %}}
@@ -279,7 +279,7 @@ Note: If you are on Windows, ensure the project is on the Linux filesystem to av
{{% /tab %}}
{{% tab title="Kubernetes" %}}
For Kubernetes deployment, see the [Kubernetes installation guide]({{% relref "installation/kubernetes" %}}).
For Kubernetes deployment, see the [Kubernetes installation guide]({{% relref "getting-started/kubernetes" %}}).
{{% /tab %}}
{{% tab title="From Binary" %}}
@@ -306,7 +306,7 @@ Press "Allow Anyway."
{{% /tab %}}
{{% tab title="From Source" %}}
For instructions on building LocalAI from source, see the [Build from Source guide]({{% relref "installation/build" %}}).
For instructions on building LocalAI from source, see the [Build from Source guide]({{% relref "getting-started/build" %}}).
{{% /tab %}}
{{< /tabs >}}

33
docs/content/getting-started/quickstart.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Quickstart"
weight = 1
weight = 2
url = '/basics/getting_started/'
icon = "rocket_launch"
+++
@@ -10,7 +10,7 @@ icon = "rocket_launch"
**LocalAI** is a free, open-source alternative to OpenAI (Anthropic, etc.), functioning as a drop-in replacement REST API for local inferencing. It allows you to run [LLMs]({{% relref "features/text-generation" %}}), generate images, and produce audio, all locally or on-premises with consumer-grade hardware, supporting multiple model families and architectures.
LocalAI comes with a **built-in web interface** for chatting with models, managing installations, configuring AI agents, and more — no extra tools needed.
LocalAI comes with a **built-in web interface** for chatting with models, managing installations, configuring AI agents, and more, with no extra tools needed.
{{% notice tip %}}
@@ -32,20 +32,20 @@ This guide assumes you have already [installed LocalAI](/installation/). If you
Once installed, start LocalAI. For Docker installations:
```bash
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-cpu
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
```
For GPU acceleration, choose the image that matches your hardware:
| Hardware | Docker image |
|----------|-------------|
| CPU only | `localai/localai:latest-cpu` |
| CPU only | `localai/localai:latest` |
| NVIDIA CUDA | `localai/localai:latest-gpu-nvidia-cuda-12` |
| AMD (ROCm) | `localai/localai:latest-gpu-hipblas` |
| Intel GPU | `localai/localai:latest-gpu-intel` |
| Vulkan | `localai/localai:latest-gpu-vulkan` |
For NVIDIA GPUs, add `--gpus all`. For AMD/Intel/Vulkan, add the appropriate `--device` flags. See [Container images]({{% relref "getting-started/container-images" %}}) for the full reference.
For NVIDIA GPUs, add `--gpus all`. For AMD/Intel/Vulkan, add the appropriate `--device` flags. See [Container images]({{% relref "getting-started/containers" %}}) for the full reference.
### Using the Web Interface
@@ -58,14 +58,17 @@ Open **http://localhost:8080** in your browser. The web interface lets you:
- **Monitor system resources** and loaded models
- **Configure settings** including GPU acceleration
To get started, navigate to the **Models** page, browse the gallery, and install a model. Once installed, head to the **Chat** page to start a conversation.
To get your first chat working:
1. Open the **Models** page and search for `qwen3-4b`. Click **Install** on the `qwen3-4b` entry and wait for the download to finish. (`qwen3-4b` is a small, CPU-friendly Qwen3 model that also supports tool calling, so you can reuse it later in the [Build your first agent]({{% relref "getting-started/first-agent" %}}) walkthrough.)
2. Open the **Chat** page, select `qwen3-4b` from the model dropdown, type a message, and send it. You should get a reply within a few seconds.
### Downloading models from the CLI
When starting LocalAI (either via Docker or via CLI) you can specify as argument a list of models to install automatically before starting the API, for example:
```bash
local-ai run llama-3.2-1b-instruct:q4_k_m
local-ai run qwen3-4b
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
@@ -83,7 +86,7 @@ local-ai models install <name> # Install a model
**Automatic Backend Detection**: When you install models from the gallery or YAML files, LocalAI automatically detects your system's GPU capabilities (NVIDIA, AMD, Intel) and downloads the appropriate backend. For advanced configuration options, see [GPU Acceleration]({{% relref "features/gpu-acceleration#automatic-backend-detection" %}}).
{{% /notice %}}
For a full list of options, you can run LocalAI with `--help` or refer to the [Linux Installation guide]({{% relref "installation/linux" %}}) for installer configuration options.
For a full list of options, run LocalAI with `--help`, or see the [Linux Installation guide]({{% relref "getting-started/linux" %}}).
### Using the API
@@ -93,7 +96,7 @@ LocalAI exposes an OpenAI-compatible API. You can use it with any OpenAI SDK or
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama-3.2-1b-instruct:q4_k_m",
"model": "qwen3-4b",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
@@ -102,16 +105,16 @@ LocalAI also supports the **Anthropic Messages API**, the **Open Responses API**
## Built-in AI Agents
LocalAI includes a built-in AI agent platform with support for the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). You can create agents that use tools, browse the web, execute code, and interact with external services — all from the web interface.
LocalAI includes a built-in AI agent platform with support for the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). You can create agents that use tools, browse the web, execute code, and interact with external services, all from the web interface.
To get started with agents:
1. Install a model that supports tool calling (most modern LLMs do)
2. Navigate to the **Agents** page in the web interface
3. Create a new agent, configure its tools and system prompt
4. Start chatting — the agent will use tools autonomously
4. Start chatting; the agent will use tools autonomously
No separate installation required — agents are part of LocalAI.
No separate installation required: agents are part of LocalAI. For a full step-by-step walkthrough, see [Build your first agent]({{% relref "getting-started/first-agent" %}}).
## Scaling with Distributed Mode
@@ -121,16 +124,16 @@ For production deployments or when you need more compute, LocalAI supports distr
- **P2P federation**: Connect multiple LocalAI instances for load-balanced inference
- **Model sharding**: Split large models across multiple machines
See the **Nodes** page in the web interface or the [Distribution docs]({{% relref "features/distribution" %}}) for setup instructions.
See the **Nodes** page in the web interface or the [Distributed inference docs]({{% relref "features/distributed_inferencing" %}}) for setup instructions.
## What's Next?
There is much more to explore! LocalAI supports video generation, voice cloning, embeddings, image understanding, and more. Check out:
- [Container images reference]({{% relref "getting-started/container-images" %}})
- [Container images reference]({{% relref "getting-started/containers" %}})
- [Try the API endpoints]({{% relref "getting-started/try-it-out" %}})
- [All features]({{% relref "features" %}})
- [Model gallery](https://models.localai.io)
- [Run models manually]({{% relref "getting-started/models" %}})
- [Build from source]({{% relref "installation/build" %}})
- [Build from source]({{% relref "getting-started/build" %}})
- [Examples](https://github.com/mudler/LocalAI/tree/master/examples#examples)

6
docs/content/getting-started/troubleshooting.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Troubleshooting"
weight = 9
weight = 6
url = '/basics/troubleshooting/'
icon = "build"
+++
@@ -172,7 +172,7 @@ If your GPU is not in the default target list, open up an Issue. Supported targe
docker run --device /dev/dri ...
```
Use container images with `gpu-intel` in the tag. **Known issue:** SYCL hangs when `mmap: true` is set — disable it in your model config:
Use container images with `gpu-intel` in the tag. **Known issue:** SYCL hangs when `mmap: true` is set - disable it in your model config:
```yaml
mmap: false
@@ -429,7 +429,7 @@ LOCALAI_P2P_DISABLE_DHT=true local-ai run
### P2P Limitations
- Only a single model is currently supported for distributed inference
- Workers must be detected before inference starts — you cannot add workers mid-inference
- Workers must be detected before inference starts - you cannot add workers mid-inference
- Workers mode supports llama-cpp compatible models only
See the [Distributed Inferencing guide](/features/distributed-inferencing/) for full setup instructions.

18
docs/content/getting-started/try-it-out.md
View File

@@ -27,22 +27,26 @@ You can test chat models from the CLI without keeping a separate `curl` command
local-ai run
# Terminal 2
local-ai chat --model gpt-4
local-ai chat --model qwen3-4b
```
`local-ai chat` connects to a running LocalAI server, opens an interactive chat prompt, and exits when you type `/exit`, `/quit`, or `/bye`. Use `/models` to list installed models, `/model <name>` to switch models, and `/clear` to reset the current conversation. If the server exposes exactly one model, LocalAI uses that model automatically:
```bash
# Terminal 1
local-ai run llama-3.2-1b-instruct:q4_k_m
local-ai run qwen3-4b
# Terminal 2
local-ai chat
```
When more than one model is configured, pass `--model` with the installed model name to avoid ambiguity. Use `--endpoint` to connect to a non-default server, for example `local-ai chat --endpoint http://127.0.0.1:8081 --model gpt-4`.
When more than one model is configured, pass `--model` with the installed model name to avoid ambiguity. Use `--endpoint` to connect to a non-default server, for example `local-ai chat --endpoint http://127.0.0.1:8081 --model qwen3-4b`.
You can also test out the API endpoints using `curl`, few examples are listed below. The models we are referring here (`gpt-4`, `gpt-4-vision-preview`, `tts-1`, `whisper-1`) are examples - replace them with the model names you have installed.
You can also test out the API endpoints using `curl`. A few examples are listed below.
{{% notice note %}}
Each example assumes you have already installed the model it names. The chat and function-calling examples below use `qwen3-4b` (install it from the Models page or with `local-ai run qwen3-4b`). The other examples name a model for the task they show (`gpt-4-vision-preview` for vision, `tts-1` for text to speech, `whisper-1` for transcription, `text-embedding-ada-002` for embeddings); replace each with the name of a model you have installed for that task.
{{% /notice %}}
### Text Generation
@@ -53,7 +57,7 @@ Creates a model response for the given chat conversation. [OpenAI documentation]
```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{ "model": "gpt-4", "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}] }'
-d '{ "model": "qwen3-4b", "messages": [{"role": "user", "content": "How are you doing?", "temperature": 0.1}] }'
```
</details>
@@ -98,7 +102,7 @@ Call functions
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4",
"model": "qwen3-4b",
"messages": [
{
"role": "user",
@@ -273,6 +277,6 @@ curl http://localhost:8080/embeddings \
Don't use the model file as `model` in the request unless you want to handle the prompt template for yourself.
Use the model names like you would do with OpenAI like in the examples below. For instance `gpt-4-vision-preview`, or `gpt-4`.
Use the installed model's own name as the `model` value, the same way you would pass a model name to OpenAI. For instance `qwen3-4b` for chat, or `gpt-4-vision-preview` for a vision model you have installed under that name.
{{% /notice %}}

43
docs/content/installation/_index.en.md
View File

@@ -1,43 +0,0 @@
---
weight: 2
title: "Installation"
description: "How to install LocalAI"
type: chapter
icon: download
---
LocalAI can be installed in multiple ways depending on your platform and preferences.
## Video Walkthrough
[![Installation Video](https://img.youtube.com/vi/cMVNnlqwfw4/maxresdefault.jpg)](https://www.youtube.com/watch?v=cMVNnlqwfw4)
## Installation Methods
Choose the installation method that best suits your needs:
1. **[Containers](containers/)** ⭐ **Recommended** - Works on all platforms, supports Docker and Podman
2. **[macOS](macos/)** - Download and install the DMG application
3. **[Linux](linux/)** - Install on Linux using binaries
4. **[Kubernetes](kubernetes/)** - Deploy LocalAI on Kubernetes clusters
5. **[Build from Source](build/)** - Build LocalAI from source code
## Quick Start
**Recommended: Containers (Docker or Podman)**
```bash
# With Docker
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
# Or with Podman
podman run -p 8080:8080 --name local-ai -ti localai/localai:latest
```
This will start LocalAI. The API will be available at `http://localhost:8080`.
For other platforms:
- **macOS**: Download the [DMG](macos/)
- **Linux**: See the [Linux installation guide](linux/) for binary installation.
For detailed instructions, see the [Containers installation guide](containers/).

183
docs/content/installation/build.md
View File

@@ -1,183 +0,0 @@
+++
disableToc = false
title = "Build LocalAI"
icon = "model_training"
weight = 5
url = '/basics/build/'
+++
### Build
LocalAI can be built as a container image or as a single, portable binary. Note that some model architectures might require Python libraries, which are not included in the binary.
LocalAI's extensible architecture allows you to add your own backends, which can be written in any language, and as such the container images contains also the Python dependencies to run all the available backends (for example, in order to run backends like __Diffusers__ that allows to generate images and videos from text).
This section contains instructions on how to build LocalAI from source.
#### Build LocalAI locally
##### Requirements
In order to build LocalAI locally, you need the following requirements:
- Golang >= 1.21
- GCC
- GRPC
To install the dependencies follow the instructions below:
{{< tabs >}}
{{% tab title="Apple" %}}
Install `xcode` from the App Store
```bash
brew install go protobuf protoc-gen-go protoc-gen-go-grpc wget
```
{{% /tab %}}
{{% tab title="Debian" %}}
```bash
apt install golang make protobuf-compiler-grpc
```
After you have golang installed and working, you can install the required binaries for compiling the golang protobuf components via the following commands
```bash
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
```
{{% /tab %}}
{{% tab title="From source" %}}
```bash
make build
```
{{% /tab %}}
{{< /tabs >}}
##### Build
To build LocalAI with `make`:
```
git clone https://github.com/go-skynet/LocalAI
cd LocalAI
make build
```
This should produce the binary `local-ai`
#### Container image
Requirements:
- Docker or podman, or a container engine
In order to build the `LocalAI` container image locally you can use `docker`, for example:
```
docker build -t localai .
docker run localai
```
### Example: Build on mac
Building on Mac (M1, M2 or M3) works, but you may need to install some prerequisites using `brew`.
The below has been tested by one mac user and found to work. Note that this doesn't use Docker to run the server:
Install `xcode` from the Apps Store (needed for metalkit)
```
brew install abseil cmake go grpc protobuf wget protoc-gen-go protoc-gen-go-grpc
git clone https://github.com/go-skynet/LocalAI.git
cd LocalAI
make build
wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q2_K.gguf -O models/phi-2.Q2_K
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/phi-2.Q2_K.tmpl
./local-ai backends install llama-cpp
./local-ai --models-path=./models/ --debug=true
curl http://localhost:8080/v1/models
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "phi-2.Q2_K",
"messages": [{"role": "user", "content": "How are you?"}],
"temperature": 0.9
}'
```
#### Troubleshooting mac
- If you encounter errors regarding a missing utility metal, install `Xcode` from the App Store.
- After the installation of Xcode, if you receive a xcrun error `'xcrun: error: unable to find utility "metal", not a developer tool or in PATH'`. You might have installed the Xcode command line tools before installing Xcode, the former one is pointing to an incomplete SDK.
```
xcode-select --print-path
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
```
- If completions are slow, ensure that `gpu-layers` in your model yaml matches the number of layers from the model in use (or simply use a high number such as 256).
- If you get a compile error: `error: only virtual member functions can be marked 'final'`, reinstall all the necessary brew packages, clean the build, and try again.
```
brew reinstall go grpc protobuf wget
make clean
make build
```
## Build backends
LocalAI have several backends available for installation in the backend gallery. The backends can be also built by source. As backends might vary from language and dependencies that they require, the documentation will provide generic guidance for few of the backends, which can be applied with some slight modifications also to the others.
### Manually
Typically each backend include a Makefile which allow to package the backend.
In the LocalAI repository, for instance you can build a backend by doing:
```
git clone https://github.com/go-skynet/LocalAI.git
make -C LocalAI/backend/python/vllm
```
### With Docker
Building with docker is simpler as abstracts away all the requirement, and focuses on building the final OCI images that are available in the gallery. This allows for instance also to build locally a backend and install it with LocalAI. You can refer to [Backends](https://localai.io/backends/) for general guidance on how to install and develop backends.
In the LocalAI repository, you can build a backend by doing:
```
git clone https://github.com/go-skynet/LocalAI.git
make docker-build-<backend-name>
```
Note that `make` is only by convenience, in reality it just runs a simple `docker` command as:
```bash
docker build --build-arg BUILD_TYPE=$(BUILD_TYPE) --build-arg BASE_IMAGE=$(BASE_IMAGE) -t local-ai-backend:<backend-name> -f LocalAI/backend/Dockerfile.golang --build-arg BACKEND=<backend-name> .
```
Note:
- BUILD_TYPE can be either: `cublas`, `hipblas`, `sycl_f16`, `sycl_f32`, `metal`.
- BASE_IMAGE is tested on `ubuntu:24.04` (and defaults to it) and `quay.io/go-skynet/intel-oneapi-base:latest` for intel/sycl

31
docs/content/installation/kubernetes.md
View File

@@ -1,31 +0,0 @@
+++
disableToc = false
title = "Run with Kubernetes"
weight = 4
url = '/basics/kubernetes/'
ico = "rocket_launch"
+++
For installing LocalAI in Kubernetes, the deployment file from the `examples` can be used and customized as preferred:
```
kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment.yaml
```
For Nvidia GPUs:
```
kubectl apply -f https://raw.githubusercontent.com/mudler/LocalAI-examples/refs/heads/main/kubernetes/deployment-nvidia.yaml
```
Alternatively, the [helm chart](https://github.com/go-skynet/helm-charts) can be used as well:
```bash
helm repo add go-skynet https://go-skynet.github.io/helm-charts/
helm repo update
helm show values go-skynet/local-ai > values.yaml
helm install local-ai go-skynet/local-ai -f values.yaml
```

56
docs/content/integrations.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "Integrations"
weight = 19
weight = 8
icon = "sync"
+++
@@ -14,54 +14,54 @@ Feel free to open up a Pull request (by clicking at the "Edit page" below) to ge
### Build & Deploy
- [aikit](https://github.com/sozercan/aikit) — Build and deploy custom LocalAI containers
- [Helm chart](https://github.com/go-skynet/helm-charts) — Deploy LocalAI on Kubernetes
- [GitHub Actions](https://github.com/marketplace/actions/start-localai) — Use LocalAI in CI/CD workflows
- [aikit](https://github.com/sozercan/aikit) - Build and deploy custom LocalAI containers
- [Helm chart](https://github.com/go-skynet/helm-charts) - Deploy LocalAI on Kubernetes
- [GitHub Actions](https://github.com/marketplace/actions/start-localai) - Use LocalAI in CI/CD workflows
### Web UIs
- [localai-admin](https://github.com/Jirubizu/localai-admin)
- [LocalAI-frontend](https://github.com/go-skynet/LocalAI-frontend)
- [QA-Pilot](https://github.com/reid41/QA-Pilot) — Interactive chat for navigating GitHub code repositories
- [Big AGI](https://github.com/enricoros/big-agi) — Powerful web interface running entirely in the browser
- [QA-Pilot](https://github.com/reid41/QA-Pilot) - Interactive chat for navigating GitHub code repositories
- [Big AGI](https://github.com/enricoros/big-agi) - Powerful web interface running entirely in the browser
### Agentic Libraries & Assistants
- [cogito](https://github.com/mudler/cogito) — Agentic library for Go
- [LocalAGI](https://github.com/mudler/LocalAGI) — Local smart assistant with autonomous agents
- [cogito](https://github.com/mudler/cogito) - Agentic library for Go
- [LocalAGI](https://github.com/mudler/LocalAGI) - Local smart assistant with autonomous agents
### MCP Servers
- [MCPs](https://github.com/mudler/MCPs) — Model Context Protocol servers
- [MCPs](https://github.com/mudler/MCPs) - Model Context Protocol servers
### OS Assistants
- [Keygeist](https://github.com/mudler/Keygeist) — AI-powered keyboard operator for Linux
- [Keygeist](https://github.com/mudler/Keygeist) - AI-powered keyboard operator for Linux
### Voice
- [VoxInput](https://github.com/richiejp/VoxInput) — Use voice to control your desktop
- [VoxInput](https://github.com/richiejp/VoxInput) - Use voice to control your desktop
### IDE & Editor Plugins
- [VSCode extension](https://github.com/badgooooor/localai-vscode-plugin)
- [GPTLocalhost (Word Add-in)](https://gptlocalhost.com/demo#LocalAI) — Run LocalAI in Microsoft Word locally
- [GPTLocalhost (Word Add-in)](https://gptlocalhost.com/demo#LocalAI) - Run LocalAI in Microsoft Word locally
### Framework Integrations
- [Langchain (Python)](https://python.langchain.com/docs/integrations/providers/localai/) — [pypi](https://pypi.org/project/langchain-localai/)
- [langchain4j](https://github.com/langchain4j/langchain4j) — Java LangChain
- [lingoose](https://github.com/henomis/lingoose) — Go framework for LLM apps
- [LLPhant](https://github.com/theodo-group/LLPhant) — PHP library for LLMs and vector databases
- [FlowiseAI](https://github.com/FlowiseAI/Flowise) — Low-code LLM app builder
- [Langchain (Python)](https://python.langchain.com/docs/integrations/providers/localai/) - [pypi](https://pypi.org/project/langchain-localai/)
- [langchain4j](https://github.com/langchain4j/langchain4j) - Java LangChain
- [lingoose](https://github.com/henomis/lingoose) - Go framework for LLM apps
- [LLPhant](https://github.com/theodo-group/LLPhant) - PHP library for LLMs and vector databases
- [FlowiseAI](https://github.com/FlowiseAI/Flowise) - Low-code LLM app builder
- [LLMStack](https://github.com/trypromptly/LLMStack)
- [Midori AI Subsystem Manager](https://io.midori-ai.xyz/subsystem/manager/)
### Terminal Tools
- [ShellOracle](https://github.com/djcopley/ShellOracle) — Terminal utility
- [Shell-Pilot](https://github.com/reid41/shell-pilot) — Interact with LLMs via pure shell scripts
- [Mods](https://github.com/charmbracelet/mods) — AI on the command line
- [ShellOracle](https://github.com/djcopley/ShellOracle) - Terminal utility
- [Shell-Pilot](https://github.com/reid41/shell-pilot) - Interact with LLMs via pure shell scripts
- [Mods](https://github.com/charmbracelet/mods) - AI on the command line
### Chat Bots
@@ -72,24 +72,24 @@ Feel free to open up a Pull request (by clicking at the "Edit page" below) to ge
### Home Automation
- [Extended OpenAI Conversation](https://github.com/jekalmin/extended_openai_conversation) — Conversation agent for Home Assistant that supports a custom OpenAI endpoint
- [LLM Vision](https://github.com/valentinfrlch/ha-llmvision) — Image & video feed analysis for Home Assistant
- [Extended OpenAI Conversation](https://github.com/jekalmin/extended_openai_conversation) - Conversation agent for Home Assistant that supports a custom OpenAI endpoint
- [LLM Vision](https://github.com/valentinfrlch/ha-llmvision) - Image & video feed analysis for Home Assistant
- [OpenAI TTS Speech Service](https://github.com/sfortis/openai_tts) - OpenAI TTS custom component for Home Assistant
- [LocalAI Monitor](https://github.com/loryanstrant/HA-LocalAI-Monitor) — Monitor & control of LocalAI from Home Assistant
- [LocalAI Monitor](https://github.com/loryanstrant/HA-LocalAI-Monitor) - Monitor & control of LocalAI from Home Assistant
- Nextcloud [integration plugin](https://apps.nextcloud.com/apps/integration_openai) and [AI assistant](https://apps.nextcloud.com/apps/assistant)
### Automation & DevOps
- [Reflexia](https://github.com/JackBekket/Reflexia) — Auto-documentation
- [GitHelper](https://github.com/JackBekket/GitHelper) — GitHub bot for issues with code and documentation context
- [kairos](https://github.com/kairos-io/kairos) — Immutable Linux OS
- [Reflexia](https://github.com/JackBekket/Reflexia) - Auto-documentation
- [GitHelper](https://github.com/JackBekket/GitHelper) - GitHub bot for issues with code and documentation context
- [kairos](https://github.com/kairos-io/kairos) - Immutable Linux OS
### Other Integrations
- [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm)
- [Logseq GPT3 OpenAI plugin](https://github.com/briansunter/logseq-plugin-gpt3-openai)
- [CodeGPT (JetBrains)](https://plugins.jetbrains.com/plugin/21056-codegpt) — Custom OpenAI-compatible endpoints
- [Wave Terminal](https://docs.waveterm.dev/features/supportedLLMs/localai) — Native LocalAI support
- [CodeGPT (JetBrains)](https://plugins.jetbrains.com/plugin/21056-codegpt) - Custom OpenAI-compatible endpoints
- [Wave Terminal](https://docs.waveterm.dev/features/supportedLLMs/localai) - Native LocalAI support
- [Obsidian BMO Chatbot](https://github.com/longy2k/obsidian-bmo-chatbot)
- [spark](https://github.com/cedriking/spark)
- [openops (Mattermost)](https://github.com/mattermost/openops)

16
docs/content/operations/_index.en.md Normal file
View File

@@ -0,0 +1,16 @@
---
weight: 6
title: "Operations"
description: "Operator-facing runtime, proxy, and monitoring concerns"
type: chapter
icon: manage_accounts
---
This section collects the operator-facing concerns for running LocalAI in production: request middleware, cloud and MITM proxies, and backend monitoring. These pages are about running and governing a LocalAI instance rather than about a specific inference feature.
## Pages
- [Middleware: PII filtering and intelligent routing]({{% relref "operations/middleware" %}}) - per-model PII redaction and policy-based request routing.
- [Cloud passthrough proxy]({{% relref "operations/cloud-proxy" %}}) - forward requests to OpenAI, Anthropic, or any compatible provider.
- [MITM proxy for Claude Code / Codex CLI]({{% relref "operations/mitm-proxy" %}}) - redact PII from cloud-AI traffic without LocalAI holding API keys.
- [Backend Monitor]({{% relref "operations/backend-monitor" %}}) - monitor, pre-load, and shut down running model backends.

0
docs/content/features/backend-monitor.md → docs/content/operations/backend-monitor.md
View File

19
docs/content/features/cloud-proxy.md → docs/content/operations/cloud-proxy.md
View File

@@ -2,6 +2,7 @@
title = "Cloud passthrough proxy"
weight = 28
toc = true
url = "/features/cloud-proxy/"
description = "Forward requests to OpenAI, Anthropic, or any compatible provider"
tags = ["Proxy", "Cloud", "Routing", "Advanced"]
categories = ["Features"]
@@ -13,7 +14,7 @@ LocalAI can forward chat-completion and Anthropic Messages requests to an
external provider instead of running them through the local gRPC backend
pipeline. Configure a model with `backend: cloud-proxy` and a `proxy.upstream_url`,
and LocalAI bypasses templating, MCP injection, and the local model loader
entirely — the upstream sees the body the client sent (with only the top-level
entirely - the upstream sees the body the client sent (with only the top-level
`model` field optionally rewritten).
The streaming PII filter still runs over the upstream's SSE stream, so cloud
@@ -21,7 +22,7 @@ egress remains subject to the same redaction rules a local model would apply.
## When to use this
- Mix local and cloud models in the same LocalAI instance — clients hit one
- Mix local and cloud models in the same LocalAI instance - clients hit one
endpoint, LocalAI dispatches per model.
- Apply LocalAI's auth, usage tracking, and PII redaction to cloud traffic
before the body leaves the network.
@@ -41,15 +42,15 @@ egress remains subject to the same redaction rules a local model would apply.
6. The streaming PII filter rewrites per-token text in flight; the upstream's
event names and metadata pass through unchanged.
Passthrough mode is **wire-format-faithful** — it does not translate request
Passthrough mode is **wire-format-faithful** - it does not translate request
shapes between providers. A client posting an OpenAI-shaped body to an
Anthropic upstream will get a confused upstream. Use the matching wire format,
or switch to translate mode (below).
## Configuration
The cloud-proxy backend has one knob — the provider it should authenticate
against — and two modes:
The cloud-proxy backend has one knob - the provider it should authenticate
against - and two modes:
| `proxy.mode` | What it does | When to use |
|---|---|---|
@@ -112,7 +113,7 @@ proxy:
pii:
enabled: true
# Block — not just mask — leaked credentials before they reach the upstream.
# Block - not just mask - leaked credentials before they reach the upstream.
patterns:
- id: api_key_prefix
action: block
@@ -158,7 +159,7 @@ proxy:
upstream_model: claude-3-5-sonnet-20241022
```
Translate mode currently routes only pure-text completions — tool calls,
Translate mode currently routes only pure-text completions - tool calls,
image blocks, and per-request usage tokens are dropped through the
internal `Predict()` signature. Use passthrough mode when your clients need
the upstream's full feature set.
@@ -225,10 +226,10 @@ for the full router and PII-filter reference.
## Operational notes
- Cloud-proxy backends load like any other gRPC backend — they consume one
- Cloud-proxy backends load like any other gRPC backend - they consume one
process per loaded model and appear in the backend management view, but
they hold no GPU memory.
- Usage stats and the trace log capture cloud-proxy requests like any other
request. Token counts come from the upstream's `usage` field when present.
- Set `request_timeout_seconds` defensively — a hung upstream otherwise ties
- Set `request_timeout_seconds` defensively - a hung upstream otherwise ties
up an HTTP handler until the client disconnects.

109
docs/content/features/middleware.md → docs/content/operations/middleware.md
View File

@@ -2,6 +2,7 @@
title = "Middleware: PII filtering and intelligent routing"
weight = 27
toc = true
url = "/features/middleware/"
description = "Per-model PII redaction and policy-based request routing"
tags = ["Routing", "Privacy", "PII", "Middleware", "Advanced"]
categories = ["Features"]
@@ -31,7 +32,7 @@ client ── auth ── route-model ── per-model PII ── backend ──
The router runs first (it picks the target model so per-model PII has
something to gate on), per-model PII runs next (gated by the resolved
config), and the backend executes. Filtering is **request-side only** —
config), and the backend executes. Filtering is **request-side only** -
the request body is scanned and rewritten before forwarding; the response
is not touched (NER over a streamed response is left as a follow-up). Each
subsystem writes to its own admin-visible log: `/api/router/decisions` for
@@ -74,7 +75,7 @@ field in place and events carry message-local offsets.
A **detector** is a `token_classify` model (e.g. an `openai-privacy-filter`
GGUF) that carries the detection *policy* in a top-level `pii_detection:`
block — defined once, on the model itself:
block - defined once, on the model itself:
```yaml
name: privacy-filter-multilingual
@@ -100,13 +101,13 @@ emits (the privacy-filter family uses uppercase names like `EMAIL`,
### Pattern detector tier
NER is the wrong tool for high-entropy, highly-regular **secrets** — API keys,
NER is the wrong tool for high-entropy, highly-regular **secrets** - API keys,
tokens, private-key blocks. A trained NER model has no "API key" class, so it
fragments a key into the nearest categories it *does* know and can leave the
secret part exposed. Those secrets are exactly what a regex catches cheaply.
A **pattern detector** is a detector model (`backend: pattern`) that matches
secrets with a **restricted regex subset** compiled to Go's RE2 engine —
secrets with a **restricted regex subset** compiled to Go's RE2 engine -
linear-time, no backtracking, no ReDoS. It runs entirely in-process: no model
download, no backend, zero VRAM. Install the gallery's **`secret-filter`** for a
ready-made set, or define your own:
@@ -133,14 +134,14 @@ pii_detection:
A match is reported under its group (built-in group name, or the pattern
`name`), so `entity_actions` / `default_action` apply exactly as for NER.
**The restricted grammar** (validated at load — an invalid pattern is rejected,
**The restricted grammar** (validated at load - an invalid pattern is rejected,
not silently ignored):
- Allowed: literals, character classes `[…]` and `\w \d \s`, alternation,
anchors `^ $ \b`, and quantifiers `? * + {m,n}`.
- Rejected: `.` (any-char), capturing groups, and `{n,m}` bounds over 4096.
- **Required anchor**: every pattern must contain a fixed literal run of at
least 3 characters (e.g. `sk-ant-`, `ghp_`, `AKIA`). This admits real key
shapes but rejects open-ended ones — an email or a bare `\w+` has no such
shapes but rejects open-ended ones - an email or a bare `\w+` has no such
anchor and belongs to the [NER tier](#detector-models).
Use both tiers together: reference an NER detector *and* a pattern detector in a
@@ -149,7 +150,7 @@ model's `pii.detectors` (or as instance defaults); their hits union, and a
### Consuming models
Any model opts in by enabling PII and referencing one or more detectors —
Any model opts in by enabling PII and referencing one or more detectors -
no per-consumer policy:
```yaml
@@ -179,7 +180,7 @@ scope for now.
The **Detector models** table on the Middleware → Filtering page lists every
`token_classify` detector model (neural NER models and in-process pattern
matchers alike) and exposes a per-row **Default** toggle. Toggling a detector
on adds it to the instance-wide default detector set — one or more models
on adds it to the instance-wide default detector set - one or more models
applied to any PII-enabled model that names none of its own `pii.detectors`.
It is persisted through `POST /api/settings` and read live, so a change takes
effect on the next request without a restart. A default that names a model no
@@ -200,31 +201,31 @@ cloud-proxy traffic is scanned with no per-model config.
Resolution precedence (the single decision point is `ResolvePIIPolicy`,
shared by the chat middleware and the MITM listener so both agree):
1. An explicit `pii.enabled` on the model wins — `true` or `false`.
1. An explicit `pii.enabled` on the model wins - `true` or `false`.
2. Otherwise PII is on if the backend defaults it on (`cloud-proxy`).
3. Detectors are the model's own `pii.detectors`; if it lists none, the
instance-wide default detector(s) are used.
A model that resolves enabled but ends up with no detector at all (a
cloud-proxy model with no model detectors and no instance default) scans
nothing — set a default detector to close that gap.
nothing - set a default detector to close that gap.
### Admin page
The `/app/middleware` page (admin role only) has four tabs — **Filtering**,
The `/app/middleware` page (admin role only) has four tabs - **Filtering**,
**Routing**, **MITM Proxy** (see the [MITM doc]({{< relref "mitm-proxy.md" >}})),
and **Events**. The Filtering tab has a **Detector models** table (every
`token_classify` filter model, with the per-row Default toggle above and an
edit link to each detector's config, plus an *Add detector model* button) and
a per-model table listing only the models PII can actually apply to — chat /
a per-model table listing only the models PII can actually apply to - chat /
completion / embeddings / edit consumers and cloud-proxy models, not
VAD/STT/image models or the detector models themselves. Each row reports the
**effective** `enabled` state as an inline **toggle** — flipping it writes an
**effective** `enabled` state as an inline **toggle** - flipping it writes an
explicit `pii.enabled` to that model's YAML (a server-side deep-merge that
preserves `pii.detectors` and every other field), so a cloud-proxy model shown
on by backend default can be turned off, and vice-versa — plus the
resolved detector(s) — with a *(default)* marker when they come from the
instance-wide default rather than the model's YAML — why it is on (`YAML` /
on by backend default can be turned off, and vice-versa - plus the
resolved detector(s) - with a *(default)* marker when they come from the
instance-wide default rather than the model's YAML - why it is on (`YAML` /
`backend default`), and the recent event count. Detection *policy*
(entity→action, min score) is still edited on each detector model's config
(Models → edit → PII), not globally.
@@ -234,22 +235,22 @@ instance-wide default rather than the model's YAML — why it is on (`YAML` /
The same detection pipeline is also exposed as a standalone service, so a
client can scan or sanitise a string **without** routing a full chat request
through it (the inline path above). Two endpoints, both requiring a normal API
key (the `pii_filter` feature — not admin):
key (the `pii_filter` feature - not admin):
- `POST /api/pii/analyze` — detect only. Returns the matched entity spans
- `POST /api/pii/analyze` - detect only. Returns the matched entity spans
(`entity_type`, `source` `ner`|`pattern`, `start`/`end`, `score`, `action`)
and a `blocked` flag, **without modifying the text**.
- `POST /api/pii/redact` — apply the configured policy. Returns `redacted_text`
- `POST /api/pii/redact` - apply the configured policy. Returns `redacted_text`
(with masked spans replaced by `[REDACTED:<id>]`) and `masked`; when a `block`
action fires it returns `400` with `type: pii_blocked` and the offending
entities — never a redacted body.
entities - never a redacted body.
Both take the same request: `text` plus a detector selection — either explicit
Both take the same request: `text` plus a detector selection - either explicit
detector model names in `detectors`, or a consuming `model` whose **effective**
policy is used: the model's own `pii.detectors`, else the
[instance-wide default detectors](#instance-wide-default-detector), exactly as
the inline filter resolves them. A `model` with PII disabled — or enabled but
with no detector anywhere — is a `400`: the inline filter would scan nothing
the inline filter resolves them. A `model` with PII disabled - or enabled but
with no detector anywhere - is a `400`: the inline filter would scan nothing
for it, and the API says so rather than implying a clean scan. The detection
policy lives on the detector models exactly as for the inline filter. The raw
matched value is never returned (an admin may pass `reveal: true` to include
@@ -257,7 +258,7 @@ the audit `hash_prefix`).
`text` is scanned as a single document. To reproduce the inline filter's
conversation-context behaviour for multi-message content, join the messages
with blank lines into one `text` — NER detection quality depends on that
with blank lines into one `text` - NER detection quality depends on that
context (a bare `4421` is nothing; after "what are the last four digits of
your card?" it is a PIN).
@@ -286,7 +287,7 @@ the redact-API rows.
|---|---|---|---|
| POST | `/api/pii/analyze` | api key (`pii_filter`) | Detect PII in a string; returns entity spans, no mutation. |
| POST | `/api/pii/redact` | api key (`pii_filter`) | Redact a string per policy; returns `redacted_text` or `400 pii_blocked`. |
| GET | `/api/pii/events` | admin | Recent middleware events — PII redactions, MITM connect/traffic, admission denials. Filterable by `correlation_id`, `user_id`, `pattern_id` (e.g. `ner:EMAIL`), `kind`, `origin`. |
| GET | `/api/pii/events` | admin | Recent middleware events - PII redactions, MITM connect/traffic, admission denials. Filterable by `correlation_id`, `user_id`, `pattern_id` (e.g. `ner:EMAIL`), `kind`, `origin`. |
| GET | `/api/middleware/status` | admin | Aggregated dashboard data: per-model PII state + detectors + router status + MITM status + admission status. One round-trip for the UI. |
### MCP tools
@@ -296,7 +297,7 @@ The same surface is mirrored through the LocalAI Assistant MCP server:
| Tool | Read/Write | Purpose |
|---|---|---|
| `get_pii_events` | read | Recent redaction / block events with optional filters. |
| `get_middleware_status` | read | Aggregator — the same payload as `GET /api/middleware/status`. |
| `get_middleware_status` | read | Aggregator - the same payload as `GET /api/middleware/status`. |
Detection policy is part of a detector model's config, so it is managed
through the model-config tools (`edit_model_config`), not a dedicated PII
@@ -311,7 +312,7 @@ a client addresses it (`"model": "smart-router"`), the middleware
classifies the prompt, picks a downstream candidate model, rewrites
`input.Model` to the candidate, and the standard model-resolution path
runs against that resolved target. ACL checks, disabled-state, and
per-model PII all apply to the resolved model — the router does
per-model PII all apply to the resolved model - the router does
*model selection only*.
#### Depth-1 invariant
@@ -329,7 +330,7 @@ predictable.
If no candidate's label set covers the active label set from the classifier,
or the classifier errors out, the router uses `cfg.Router.Fallback`.
An empty `fallback` causes the dispatch to fail with HTTP 500 rather
than silently routing somewhere unintended — fail-fast, not
than silently routing somewhere unintended - fail-fast, not
silent-bypass.
### Available classifiers
@@ -368,7 +369,7 @@ The `score` classifier works like this:
activates multiple labels falls to a candidate that covers them all.
This is the Arch-Router approach extended for multi-label. The
distribution carries more signal than the argmax — reading off the
distribution carries more signal than the argmax - reading off the
spread lets one prompt activate multiple policies and route to a model
capable of all of them.
@@ -388,7 +389,7 @@ probability distributions which translate to a higher
On llama-cpp, scoring rides the server's task queue alongside
generation and embeddings, so the classifier may share a model config
with `chat`/`completion`/`embeddings` — a dedicated scorer model is no
with `chat`/`completion`/`embeddings` - a dedicated scorer model is no
longer required. Repeated calls with the same prompt also reuse the
prompt's KV cache across candidates.
@@ -416,12 +417,12 @@ The reranker scores the *description* (natural English) rather than
asking a small LM to score the *label* as a next-token continuation,
so it tends to be more robust when policy labels are abstract slugs
(`compliance-review`, `tier-2-support`). The trade-off is one
reranker round-trip per request — bge-m3 in ColBERT mode is fast
reranker round-trip per request - bge-m3 in ColBERT mode is fast
enough on GPU that this is comparable to the Score path for most
workloads. The `embedding_cache` block applies identically.
The reranker model's `type:` (in the model YAML) selects which
underlying scoring head loads — `colbert` for late-interaction MaxSim,
underlying scoring head loads - `colbert` for late-interaction MaxSim,
`cross-encoder` for cross-attention scoring. The classifier itself is
indifferent; pick the head that fits your latency / quality budget.
@@ -441,7 +442,7 @@ router:
# canonical choice.
classifier_model: arch-router-1.5b
# Bounded LRU keyed on (case-folded, whitespace-trimmed) prompt — prompts
# Bounded LRU keyed on (case-folded, whitespace-trimmed) prompt - prompts
# repeat in agent loops; the cache amortises the classifier round-trip
# across them. 0 here means "use the default" (1024); the cache cannot be
# disabled from YAML today.
@@ -449,7 +450,7 @@ router:
# Softmax probability floor a label must clear to join the active label set.
# 0 = use the package default (0.15). 0.40 is a better empirical
# starting point on Arch-Router-1.5B — see the tuning note below.
# starting point on Arch-Router-1.5B - see the tuning note below.
activation_threshold: 0.40
# Used when no candidate covers the active label set, or the classifier
@@ -457,7 +458,7 @@ router:
fallback: qwen3-0.6b
# The label vocabulary. Descriptions are fed verbatim into the
# classifier's system prompt — short, action-oriented sentences work
# classifier's system prompt - short, action-oriented sentences work
# best ("writing or debugging code", "small talk").
policies:
- label: code-generation
@@ -467,7 +468,7 @@ router:
- label: math-reasoning
description: arithmetic, equations, percentage calculations, or step-by-step word problems
# Routing table — order matters (smallest → largest). See "Score
# Routing table - order matters (smallest → largest). See "Score
# classifier" above for the matching rule.
candidates:
- model: qwen3-0.6b
@@ -492,14 +493,14 @@ three-policy setup above, sweeping the threshold over a hand-labeled
| 0.50 | 67% | 97% |
The classifier's argmax matches the dominant label 93% of the time on
this corpus — what the threshold controls is how much secondary-label
this corpus - what the threshold controls is how much secondary-label
noise leaks into the active label set. Low thresholds push single-label
queries to multi-label-capable (larger) candidates unnecessarily; 0.40
keeps the dominant label dominant without losing genuine compound
activations.
Re-tune per (classifier-model, policy-set) pair. The `/api/score`
endpoint (see below) is the convenient probe — it returns the raw
endpoint (see below) is the convenient probe - it returns the raw
length-normalized log-probabilities so you can sweep thresholds offline
without driving real chat completions.
@@ -508,7 +509,7 @@ without driving real chat completions.
Classification is the most expensive thing the middleware does. The
score classifier already memo-caches verbatim repeats (case- and
whitespace-folded prompt → decision); the **embedding cache** is the
L2 tier that catches *semantically similar* prompts — "How do I exit
L2 tier that catches *semantically similar* prompts - "How do I exit
vim?" and "i need to quit vim" can share a decision instead of running
the classifier twice.
@@ -535,7 +536,7 @@ router:
```
Omit the block entirely to disable. The cache adds two new failure modes
(embedder unavailable, store unavailable) — both fall through to the
(embedder unavailable, store unavailable) - both fall through to the
inner classifier so routing keeps working.
#### How it works
@@ -552,41 +553,41 @@ For each request:
paraphrases.
The local-store collection is named `router-cache-<router-model-name>` by
default — each router gets its own collection so two routers can't
default - each router gets its own collection so two routers can't
cross-contaminate. Collections persist on disk (local-store is the
canonical persistent vector backend), so the cache survives restarts.
#### Tuning notes
- **Similarity threshold**: 0.80 is the package default — re-tune
- **Similarity threshold**: 0.80 is the package default - re-tune
per (embedding model, corpus). The histogram on the Routing tab
shows where the cosine distribution actually sits; pick a
threshold above the cross-intent cluster and below the paraphrase
cluster.
- **Confidence threshold**: 0.60 corresponds roughly to "the
classifier is committed to a top label." Don't lower this — caching
classifier is committed to a top label." Don't lower this - caching
unsure decisions propagates the uncertainty.
- **Cache flush**: invalidates automatically when the router YAML
changes (the classifier cache is fingerprinted by `yaml.Marshal`),
but the underlying local-store collection still holds the old
payloads. Manual flush via local-store admin or by renaming
`store_name` if you need a hard reset.
- **Latency budget**: an embedding round-trip (typically 30–80ms for
- **Latency budget**: an embedding round-trip (typically 30-80ms for
small embedding models) plus KNN search (~5ms) is added to every
*miss* on top of the classifier latency. Cache hits skip the
classifier entirely. Break-even is around 7–10% hit rate; agent
classifier entirely. Break-even is around 7-10% hit rate; agent
loops with repeated phrasing easily exceed this.
### Admin page
The `/app/middleware` page has a **Routing** tab listing every router
model's classifier, policies, candidates, and fallback. The **Events**
tab shows the decision log — one row per classified request with
tab shows the decision log - one row per classified request with
correlation ID, requested model, served model, classifier name, active
labels, top-label score, and latency.
Routing decisions are stored in an in-process ring buffer (default
capacity 5,000). The decision log is for audit and tuning — the
capacity 5,000). The decision log is for audit and tuning - the
canonical usage log lives in `/api/usage` and correlates by request ID.
### REST surface
@@ -595,7 +596,7 @@ canonical usage log lives in `/api/usage` and correlates by request ID.
|---|---|---|---|
| GET | `/api/router/status` | any | Router configuration: each router model's classifier, policies, candidates. |
| GET | `/api/router/decisions` | admin | Decision log with optional filters (`correlation_id`, `user_id`, `router_model`, `limit`). |
| POST | `/api/score` | admin | Direct access to the `Score` gRPC primitive — useful for offline threshold tuning. Body: `{"model": "<classifier-model>", "prompt": "<chatml-prompt>", "candidates": ["label-a", ...], "length_normalize": true}`. The llama-cpp and vLLM backends implement Score; other backends return `UNIMPLEMENTED`. |
| POST | `/api/score` | admin | Direct access to the `Score` gRPC primitive - useful for offline threshold tuning. Body: `{"model": "<classifier-model>", "prompt": "<chatml-prompt>", "candidates": ["label-a", ...], "length_normalize": true}`. The llama-cpp and vLLM backends implement Score; other backends return `UNIMPLEMENTED`. |
### MCP tools
@@ -604,8 +605,8 @@ canonical usage log lives in `/api/usage` and correlates by request ID.
| `get_router_decisions` | read | Recent decision log with optional filters. |
| `get_middleware_status` | read | Includes the router section listing configured router models. |
Mutating routing config — adding a candidate, changing the classifier
model — is YAML-only today; reload with `POST /models/reload` to pick
Mutating routing config - adding a candidate, changing the classifier
model - is YAML-only today; reload with `POST /models/reload` to pick
up edits without restarting.
### Operational notes
@@ -623,20 +624,20 @@ up edits without restarting.
`classifier_cache_size` is the highest-leverage knob for repeat-query
workloads (agent loops).
- **Decision log size**: 5,000-entry ring buffer per process. The
log is in-process and not persisted — pair with the usage log for
log is in-process and not persisted - pair with the usage log for
long-horizon audit.
---
## Related features
- [Cloud passthrough proxy]({{< relref "cloud-proxy.md" >}}) — combine
- [Cloud passthrough proxy]({{< relref "cloud-proxy.md" >}}) - combine
the router with `proxy-*` backends to send simple prompts to local
models and complex ones to cloud providers.
- [MITM proxy]({{< relref "mitm-proxy.md" >}}) — apply the same PII
- [MITM proxy]({{< relref "mitm-proxy.md" >}}) - apply the same PII
filter to Claude Code, Codex CLI, and any HTTPS client without
LocalAI holding their API keys.
- [Authentication]({{< relref "authentication.md" >}}) — admin role is
- [Authentication]({{< relref "authentication.md" >}}) - admin role is
required for mutating endpoints and the `/app/middleware` page; in
no-auth single-user mode the synthetic local user has admin role
automatically.

51
docs/content/features/mitm-proxy.md → docs/content/operations/mitm-proxy.md
View File

@@ -2,6 +2,7 @@
title = "MITM proxy for Claude Code / Codex CLI"
weight = 29
toc = true
url = "/features/mitm-proxy/"
description = "Redact PII from cloud-AI traffic without LocalAI holding API keys"
tags = ["Proxy", "MITM", "Privacy", "Routing", "Advanced"]
categories = ["Features"]
@@ -12,8 +13,8 @@ categories = ["Features"]
LocalAI can act as a local HTTPS proxy that **redacts PII from your Claude
Code, OpenAI Codex CLI, or any HTTPS client** without holding their API keys.
The proxy intercepts only the LLM API endpoints you allowlist (default:
`api.anthropic.com`, `api.openai.com`); everything else — OAuth, telemetry,
package fetches — passes through as a plain TCP tunnel.
`api.anthropic.com`, `api.openai.com`); everything else - OAuth, telemetry,
package fetches - passes through as a plain TCP tunnel.
Use this when:
@@ -39,11 +40,11 @@ and distributing the generated CA cert.
to the real upstream over its own TLS connection.
5. The streaming SSE response runs through the same `pii.StreamFilter`
the cloud-proxy backend uses.
6. For non-allowlisted hosts, the proxy is a plain CONNECT tunnel — no
6. For non-allowlisted hosts, the proxy is a plain CONNECT tunnel - no
TLS termination, no inspection, no CA trust required.
The CLI authenticates with its own subscription / API key as it normally
would. LocalAI never holds the credential — it just observes and rewrites
would. LocalAI never holds the credential - it just observes and rewrites
the request body.
## Quick start
@@ -77,32 +78,46 @@ PII the model emits in its streaming response is masked before reaching
your terminal. Events appear in the LocalAI middleware admin page under
**Filtering → Recent events**.
The same works for Codex CLI — set `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS`
The same works for Codex CLI - set `HTTPS_PROXY` and `NODE_EXTRA_CA_CERTS`
and run `codex`.
## Configuration
The proxy is enabled with two startup settings:
| Flag / env | Default | Purpose |
|---|---|---|
| `--mitm-listen` / `LOCALAI_MITM_LISTEN` | empty (disabled) | Address to bind the proxy listener on |
| `--mitm-ca-dir` / `LOCALAI_MITM_CA_DIR` | `<data-path>/mitm-ca` | Where to persist the CA cert + key |
| `--mitm-intercept-hosts` / `LOCALAI_MITM_INTERCEPT_HOSTS` | `api.anthropic.com,api.openai.com` | Hosts to terminate TLS for; everything else tunnels |
There is no global intercept-hosts flag. The hosts whose TLS is terminated
and scanned are declared **per model**, in the model YAML `mitm.hosts:`
block. Each model that names one or more hosts owns those hosts; everything
not listed by any model tunnels through untouched. A cloud-proxy model that
should intercept Anthropic traffic looks like:
```yaml
name: claude
backend: cloud-proxy
mitm:
hosts:
- api.anthropic.com
```
Hostnames are case-insensitive. Add custom upstreams (e.g. an
OpenAI-compatible third-party provider) by extending the allowlist and
ensuring their endpoint paths match `/v1/chat/completions` or
`/v1/messages`.
OpenAI-compatible third-party provider) by adding their hostname to a
model's `mitm.hosts:` list and ensuring their endpoint paths match
`/v1/chat/completions` or `/v1/messages`. You can create these models from
the Add Model UI.
## What gets redacted
Same patterns the regular request middleware uses:
- Email addresses → masked
- Phone numbers → masked
- US Social Security Numbers → masked
- Credit card numbers (Luhn-verified) → masked
- IPv4 addresses → masked
- API key prefixes (`sk-`, `pk-`, `ghp_`, `github_pat_`, `xoxb-`) → **blocked**
The MITM proxy runs the same PII detection as the regular request
middleware. Detection is **NER-based** (a token-classification detector
model), not a fixed regex list: the older pattern tier has been removed.
See {{% relref "operations/middleware" %}} for how detector models, entity
groups, and the `mask` / `block` actions are configured, and for the
instance-wide default detector.
A `block` action returns HTTP 400 with `error.type=pii_blocked` to the
client. The CLI sees the rejection and shows it as a request error.
@@ -116,7 +131,7 @@ include MITM events alongside direct-API events.
- **The CA private key is the master credential.** Anyone with read
access to `<data-path>/mitm-ca/ca.key` can forge TLS for any host the
proxy could intercept. The file is mode 0600; keep it that way.
- The proxy listener accepts plaintext HTTP `CONNECT` requests — bind it
- The proxy listener accepts plaintext HTTP `CONNECT` requests - bind it
to localhost (`--mitm-listen 127.0.0.1:8443`) unless you've added auth
in front of the listener. There is no built-in API-key check on this
port.

22
docs/content/overview.md
View File

@@ -29,15 +29,15 @@ In today's AI landscape, privacy, control, and flexibility are paramount. LocalA
The LocalAI core is a single small binary (or container). It gives you everything you need to serve models, and pulls each model backend on demand, so you install only what you use:
- **OpenAI-compatible API** — Drop-in replacement for OpenAI, Anthropic, and Open Responses APIs
- **Built-in Web Interface** — Chat, model management, agent creation, image generation, and system monitoring
- **AI Agents** — Create autonomous agents with MCP (Model Context Protocol) tool support, directly from the UI
- **OpenAI-compatible API** - Drop-in replacement for OpenAI, Anthropic, and Open Responses APIs
- **Built-in Web Interface** - Chat, model management, agent creation, image generation, and system monitoring
- **AI Agents** - Create autonomous agents with MCP (Model Context Protocol) tool support, directly from the UI
- **Any Model, Any Modality**: LLMs, image and video, text-to-speech, speech-to-text, vision, and embeddings, each on its own backend, pulled automatically when you load a model
- **GPU Acceleration** — Automatic detection and support for NVIDIA, AMD, Intel, and Vulkan GPUs
- **Distributed Mode** — Scale horizontally with worker nodes, P2P federation, and model sharding
- **No GPU Required** — Runs on CPU with consumer-grade hardware
- **GPU Acceleration** - Automatic detection and support for NVIDIA, AMD, Intel, and Vulkan GPUs
- **Distributed Mode** - Scale horizontally with worker nodes, P2P federation, and model sharding
- **No GPU Required** - Runs on CPU with consumer-grade hardware
LocalAI integrates [LocalAGI](https://github.com/mudler/LocalAGI) (agent platform) and [LocalRecall](https://github.com/mudler/LocalRecall) (semantic memory) as built-in libraries — no separate installation needed.
LocalAI integrates [LocalAGI](https://github.com/mudler/LocalAGI) (agent platform) and [LocalRecall](https://github.com/mudler/LocalRecall) (semantic memory) as built-in libraries - no separate installation needed.
Each backend is a dedicated gRPC service that LocalAI builds around a best-in-class engine (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX, and more), exposing it through the unified API. Backends ship as standard OCI images and run as isolated processes, so each one can be installed, upgraded, or removed without touching the core, can even run on a separate machine, and a fault in one never brings down the rest.
@@ -52,12 +52,12 @@ LocalAI can be installed in several ways. **Docker is the recommended installati
The quickest way to get started with LocalAI is using Docker:
```bash
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest-cpu
docker run -p 8080:8080 --name local-ai -ti localai/localai:latest
```
Then open **http://localhost:8080** to access the web interface, install models, and start chatting.
For GPU support, see the [Container images reference]({{% relref "getting-started/container-images" %}}) or the [Quickstart guide]({{% relref "getting-started/quickstart" %}}).
For GPU support, see the [Container images reference]({{% relref "getting-started/containers" %}}) or the [Quickstart guide]({{% relref "getting-started/quickstart" %}}).
For complete installation instructions including Docker, macOS, Linux, Kubernetes, and building from source, see the [Installation guide](/installation/).
@@ -97,8 +97,8 @@ Ready to dive in? Here are some recommended next steps:
LocalAI is created by [Ettore Di Giacinto](https://github.com/mudler) and maintained by the LocalAI team:
- **[Ettore Di Giacinto](https://github.com/mudler)** — original author and project lead
- **[Richard Palethorpe](https://github.com/richiejp)** — maintainer
- **[Ettore Di Giacinto](https://github.com/mudler)** - original author and project lead
- **[Richard Palethorpe](https://github.com/richiejp)** - maintainer
LocalAI is helped by the wider community of contributors. See the full [contributors list](https://github.com/mudler/LocalAI/graphs/contributors).

170
docs/content/reference/_index.en.md
View File

@@ -1,5 +1,5 @@
---
weight: 23
weight: 7
title: "References"
description: "Reference"
type: chapter
@@ -9,4 +9,170 @@ date: 2020-10-06T08:49:15+00:00
lastmod: 2020-10-06T08:49:15+00:00
draft: false
images: []
---
---
## Overview
The **References** section provides comprehensive technical documentation for developers and system administrators working with LocalAI. This includes API specifications, command-line reference, architecture details, and compatibility information.
### Who Should Read This Section
- **API Developers** integrating LocalAI into applications
- **System Administrators** configuring and maintaining installations
- **DevOps Engineers** understanding architecture and deployment
- **Troubleshooters** diagnosing issues and errors
---
## Topics
### 🔌 [API Errors](api-errors.md)
Complete reference of all API error codes, messages, and troubleshooting guidance.
**Key topics:**
- Error code enumeration
- Error message formats
- Common error scenarios
- Resolution strategies
**Recommended for:** Developers handling API responses and errors
---
### 🏗️ [Architecture](architecture.md)
Overview of LocalAI's system architecture, components, and data flow.
**Key topics:**
- System components
- Data flow diagrams
- Component interactions
- Design principles
**Recommended for:** System architects and advanced users
---
### 📦 [Binaries](binaries.md)
Reference documentation for LocalAI binary files and their purposes.
**Key topics:**
- Binary file descriptions
- Usage instructions
- File locations
- Version information
**Recommended for:** Users understanding installation structure
---
### 💻 [CLI Reference](cli-reference.md)
Complete command-line interface reference with all commands, flags, and examples.
**Key topics:**
- Command enumeration
- Flag documentation
- Usage examples
- Environment variables
**Recommended for:** Command-line users and script developers
---
### 🔄 [Compatibility Table](compatibility-table.md)
Comprehensive compatibility matrix for models, backends, and platforms.
**Key topics:**
- Model compatibility
- Backend support matrix
- Platform requirements
- Version compatibility
**Recommended for:** Users selecting models and configurations
---
### 🎮 [NVIDIA L4T](nvidia-l4t.md)
Documentation for NVIDIA Linux for Tegra (L4T) platform support.
**Key topics:**
- L4T platform requirements
- Installation procedures
- Configuration options
- Known issues
**Recommended for:** NVIDIA Jetson/Tegra users
---
### 🔤 [Shell Completion](shell-completion.md)
Guide to enabling shell completion for LocalAI CLI commands.
**Key topics:**
- Bash completion setup
- Zsh completion setup
- Fish completion setup
- Customization options
**Recommended for:** Command-line power users
---
### 📊 [System Info](system-info.md)
Reference for system information commands and diagnostics.
**Key topics:**
- System info commands
- Diagnostic output
- Hardware detection
- Performance metrics
**Recommended for:** System administrators and troubleshooters
---
### 🤖 [AI Coding Assistants](ai-coding-assistants.md)
Policy for AI-assisted contributions: licensing, DCO, and attribution.
**Key topics:**
- Aligned with the Linux kernel's AI assistants policy
- Signed-off-by and DCO rules
- `Assisted-by` commit trailer format
- Scope and responsibility of the human submitter
**Recommended for:** Contributors using AI coding assistants (Claude, Copilot, Cursor, Codex, etc.)
---
## Quick Links
| Task | Documentation |
|------|---------------|
| API integration | [API Errors](api-errors.md) |
| CLI commands | [CLI Reference](cli-reference.md) |
| Check compatibility | [Compatibility Table](compatibility-table.md) |
| System diagnostics | [System Info](system-info.md) |
| Contribute with AI assistance | [AI Coding Assistants](ai-coding-assistants.md) |
---
## Prerequisites
Before using reference documentation, ensure you have:
1. ✅ LocalAI installed and running
2. ✅ Basic understanding of APIs and command-line interfaces
3. ✅ Familiarity with your target platform (Linux, macOS, Windows)
---
## Related Sections
- 📖 [Advanced](../advanced/) - Deep dive into configuration and optimization
- 🔌 [Installation](../installation/) - Setup and deployment
- ⭐ [Features](../features/) - Feature overview
---
## Navigation
← [Advanced](../advanced/) | [FAQ](../faq.md) →

178
docs/content/reference/_index.md
View File

@@ -1,178 +0,0 @@
---
weight: 23
title: "References"
description: "Complete reference documentation"
type: chapter
icon: menu_book
lead: "API documentation, command reference, and technical specifications."
date: 2020-10-06T08:49:15+00:00
lastmod: 2026-03-09T00:00:00+00:00
draft: false
images: []
---
## Overview
The **References** section provides comprehensive technical documentation for developers and system administrators working with LocalAI. This includes API specifications, command-line reference, architecture details, and compatibility information.
### Who Should Read This Section
- **API Developers** integrating LocalAI into applications
- **System Administrators** configuring and maintaining installations
- **DevOps Engineers** understanding architecture and deployment
- **Troubleshooters** diagnosing issues and errors
---
## Topics
### 🔌 [API Errors](api-errors.md)
Complete reference of all API error codes, messages, and troubleshooting guidance.
**Key topics:**
- Error code enumeration
- Error message formats
- Common error scenarios
- Resolution strategies
**Recommended for:** Developers handling API responses and errors
---
### 🏗️ [Architecture](architecture.md)
Overview of LocalAI's system architecture, components, and data flow.
**Key topics:**
- System components
- Data flow diagrams
- Component interactions
- Design principles
**Recommended for:** System architects and advanced users
---
### 📦 [Binaries](binaries.md)
Reference documentation for LocalAI binary files and their purposes.
**Key topics:**
- Binary file descriptions
- Usage instructions
- File locations
- Version information
**Recommended for:** Users understanding installation structure
---
### 💻 [CLI Reference](cli-reference.md)
Complete command-line interface reference with all commands, flags, and examples.
**Key topics:**
- Command enumeration
- Flag documentation
- Usage examples
- Environment variables
**Recommended for:** Command-line users and script developers
---
### 🔄 [Compatibility Table](compatibility-table.md)
Comprehensive compatibility matrix for models, backends, and platforms.
**Key topics:**
- Model compatibility
- Backend support matrix
- Platform requirements
- Version compatibility
**Recommended for:** Users selecting models and configurations
---
### 🎮 [NVIDIA L4T](nvidia-l4t.md)
Documentation for NVIDIA Linux for Tegra (L4T) platform support.
**Key topics:**
- L4T platform requirements
- Installation procedures
- Configuration options
- Known issues
**Recommended for:** NVIDIA Jetson/Tegra users
---
### 🔤 [Shell Completion](shell-completion.md)
Guide to enabling shell completion for LocalAI CLI commands.
**Key topics:**
- Bash completion setup
- Zsh completion setup
- Fish completion setup
- Customization options
**Recommended for:** Command-line power users
---
### 📊 [System Info](system-info.md)
Reference for system information commands and diagnostics.
**Key topics:**
- System info commands
- Diagnostic output
- Hardware detection
- Performance metrics
**Recommended for:** System administrators and troubleshooters
---
### 🤖 [AI Coding Assistants](ai-coding-assistants.md)
Policy for AI-assisted contributions — licensing, DCO, and attribution.
**Key topics:**
- Aligned with the Linux kernel's AI assistants policy
- Signed-off-by and DCO rules
- `Assisted-by` commit trailer format
- Scope and responsibility of the human submitter
**Recommended for:** Contributors using AI coding assistants (Claude, Copilot, Cursor, Codex, etc.)
---
## Quick Links
| Task | Documentation |
|------|---------------|
| API integration | [API Errors](api-errors.md) |
| CLI commands | [CLI Reference](cli-reference.md) |
| Check compatibility | [Compatibility Table](compatibility-table.md) |
| System diagnostics | [System Info](system-info.md) |
| Contribute with AI assistance | [AI Coding Assistants](ai-coding-assistants.md) |
---
## Prerequisites
Before using reference documentation, ensure you have:
1. ✅ LocalAI installed and running
2. ✅ Basic understanding of APIs and command-line interfaces
3. ✅ Familiarity with your target platform (Linux, macOS, Windows)
---
## Related Sections
- 📖 [Advanced](../advanced/) - Deep dive into configuration and optimization
- 🔌 [Installation](../installation/) - Setup and deployment
- ⭐ [Features](../features/) - Feature overview
---
## Navigation
← [Advanced](../advanced/) | [FAQ](../faq.md) →

12
docs/content/reference/ai-coding-assistants.md
View File

@@ -11,9 +11,9 @@ This document provides guidance for AI tools and developers using AI assistance
AI tools helping with LocalAI development should follow the standard project development process:
- [CONTRIBUTING.md](https://github.com/mudler/LocalAI/blob/master/CONTRIBUTING.md) — development workflow, commit conventions, and PR guidelines
- [AGENTS.md](https://github.com/mudler/LocalAI/blob/master/AGENTS.md) — the agent entry point with links to all detailed topic guides
- [.agents/ai-coding-assistants.md](https://github.com/mudler/LocalAI/blob/master/.agents/ai-coding-assistants.md) — the full policy source of truth
- [CONTRIBUTING.md](https://github.com/mudler/LocalAI/blob/master/CONTRIBUTING.md) - development workflow, commit conventions, and PR guidelines
- [AGENTS.md](https://github.com/mudler/LocalAI/blob/master/AGENTS.md) - the agent entry point with links to all detailed topic guides
- [.agents/ai-coding-assistants.md](https://github.com/mudler/LocalAI/blob/master/.agents/ai-coding-assistants.md) - the full policy source of truth
## Licensing and Legal Requirements
@@ -44,9 +44,9 @@ Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
Where:
- `AGENT_NAME` — name of the AI tool or framework (e.g., `Claude`, `Copilot`, `Cursor`)
- `MODEL_VERSION` — specific model version used (e.g., `claude-opus-4-7`, `gpt-5`)
- `[TOOL1] [TOOL2]` — optional specialized analysis tools invoked by the agent (e.g., `golangci-lint`, `staticcheck`, `go vet`)
- `AGENT_NAME` - name of the AI tool or framework (e.g., `Claude`, `Copilot`, `Cursor`)
- `MODEL_VERSION` - specific model version used (e.g., `claude-opus-4-7`, `gpt-5`)
- `[TOOL1] [TOOL2]` - optional specialized analysis tools invoked by the agent (e.g., `golangci-lint`, `staticcheck`, `go vet`)
Basic development tools (git, go, make, editors) should **not** be listed.

44
docs/content/reference/api-errors.md
View File

@@ -7,6 +7,10 @@ weight = 26
This page documents the error responses returned by the LocalAI API. LocalAI supports multiple API formats (OpenAI, Anthropic, Open Responses), each with its own error structure.
{{% notice note %}}
For backend and runtime failure messages (a model that will not load, `could not load model`, `grpc service not ready`, `SIGILL`, VRAM out of memory, the model-load cooldown `503`), rather than API-envelope errors, see {{% relref "reference/runtime-errors" %}}.
{{% /notice %}}
## Error Response Formats
### OpenAI-Compatible Format
@@ -90,7 +94,7 @@ The `/v1/responses` endpoint returns errors with this structure:
### Authentication Errors (401)
When authentication is enabled — either via API keys (`LOCALAI_API_KEY`) or the user auth system (`LOCALAI_AUTH=true`) — API requests must include valid credentials. Credentials can be provided through:
When authentication is enabled - either via API keys (`LOCALAI_API_KEY`) or the user auth system (`LOCALAI_AUTH=true`) - API requests must include valid credentials. Credentials can be provided through:
- `Authorization: Bearer <key>` header (API key, user API key, or session ID)
- `x-api-key: <key>` header
@@ -156,7 +160,7 @@ When `LOCALAI_OPAQUE_ERRORS=true` is set, all error responses return an empty bo
## Per-Endpoint Error Scenarios
### Chat Completions — `POST /v1/chat/completions`
### Chat Completions - `POST /v1/chat/completions`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -173,14 +177,14 @@ curl http://localhost:8080/v1/chat/completions \
See also: [Text Generation]({{%relref "features/text-generation" %}})
### Completions — `POST /v1/completions`
### Completions - `POST /v1/completions`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
| 400 | Invalid or malformed request body | `Bad Request` |
| 500 | Backend inference failure | `Internal Server Error` |
### Embeddings — `POST /v1/embeddings`
### Embeddings - `POST /v1/embeddings`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -190,7 +194,7 @@ See also: [Text Generation]({{%relref "features/text-generation" %}})
See also: [Embeddings]({{%relref "features/embeddings" %}})
### Image Generation — `POST /v1/images/generations`
### Image Generation - `POST /v1/images/generations`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -200,7 +204,7 @@ See also: [Embeddings]({{%relref "features/embeddings" %}})
See also: [Image Generation]({{%relref "features/image-generation" %}})
### Image Editing (Inpainting) — `POST /v1/images/edits`
### Image Editing (Inpainting) - `POST /v1/images/edits`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -209,7 +213,7 @@ See also: [Image Generation]({{%relref "features/image-generation" %}})
| 400 | Missing `mask` file | `missing mask file` |
| 500 | Storage preparation failure | `failed to prepare storage` |
### Audio Transcription — `POST /v1/audio/transcriptions`
### Audio Transcription - `POST /v1/audio/transcriptions`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -219,7 +223,7 @@ See also: [Image Generation]({{%relref "features/image-generation" %}})
See also: [Audio to Text]({{%relref "features/audio-to-text" %}})
### Text to Speech — `POST /v1/audio/speech`, `POST /tts`
### Text to Speech - `POST /v1/audio/speech`, `POST /tts`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -229,7 +233,7 @@ See also: [Audio to Text]({{%relref "features/audio-to-text" %}})
See also: [Text to Audio]({{%relref "features/text-to-audio" %}})
### ElevenLabs TTS — `POST /v1/text-to-speech/:voice-id`
### ElevenLabs TTS - `POST /v1/text-to-speech/:voice-id`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -237,7 +241,7 @@ See also: [Text to Audio]({{%relref "features/text-to-audio" %}})
| 400 | Model not found in configuration | `Bad Request` |
| 500 | Backend inference failure | `Internal Server Error` |
### ElevenLabs Sound Generation — `POST /v1/sound-generation`
### ElevenLabs Sound Generation - `POST /v1/sound-generation`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -245,7 +249,7 @@ See also: [Text to Audio]({{%relref "features/text-to-audio" %}})
| 400 | Model not found in configuration | `Bad Request` |
| 500 | Backend inference failure | `Internal Server Error` |
### Reranking — `POST /v1/rerank`, `POST /jina/v1/rerank`
### Reranking - `POST /v1/rerank`, `POST /jina/v1/rerank`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -255,7 +259,7 @@ See also: [Text to Audio]({{%relref "features/text-to-audio" %}})
See also: [Reranker]({{%relref "features/reranker" %}})
### Anthropic Messages — `POST /v1/messages`
### Anthropic Messages - `POST /v1/messages`
| Status | Cause | Error Type | Example Message |
|--------|----------------------------------|-------------------------|-------------------------------------------------|
@@ -282,7 +286,7 @@ curl http://localhost:8080/v1/messages \
}
```
### Open Responses — `POST /v1/responses`
### Open Responses - `POST /v1/responses`
| Status | Cause | Error Type | Example Message |
|--------|-------------------------------------|-------------------------|--------------------------------------------------------|
@@ -315,14 +319,14 @@ curl http://localhost:8080/v1/responses \
}
```
### Open Responses — `GET /v1/responses/:id`
### Open Responses - `GET /v1/responses/:id`
| Status | Cause | Error Type | Example Message |
|--------|---------------------------|-------------------------|----------------------------------|
| 400 | Missing response ID | `invalid_request_error` | `response ID is required` |
| 404 | Response not found | `not_found` | `response not found: <id>` |
### Open Responses Events — `GET /v1/responses/:id/events`
### Open Responses Events - `GET /v1/responses/:id/events`
| Status | Cause | Error Type | Example Message |
|--------|--------------------------------------|-------------------------|---------------------------------------------------------|
@@ -332,7 +336,7 @@ curl http://localhost:8080/v1/responses \
| 404 | Response not found | `not_found` | `response not found: <id>` |
| 500 | Failed to retrieve events | `server_error` | `failed to get events: <details>` |
### Object Detection — `POST /v1/detection`
### Object Detection - `POST /v1/detection`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -342,7 +346,7 @@ curl http://localhost:8080/v1/responses \
See also: [Object Detection]({{%relref "features/object-detection" %}})
### Video Generation — `POST /v1/video/generations`
### Video Generation - `POST /v1/video/generations`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -350,7 +354,7 @@ See also: [Object Detection]({{%relref "features/object-detection" %}})
| 400 | Model not found in configuration | `Bad Request` |
| 500 | Backend inference failure | `Internal Server Error` |
### Voice Activity Detection — `POST /v1/audio/vad`
### Voice Activity Detection - `POST /v1/audio/vad`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
@@ -358,14 +362,14 @@ See also: [Object Detection]({{%relref "features/object-detection" %}})
| 400 | Model not found in configuration | `Bad Request` |
| 500 | Backend inference failure | `Internal Server Error` |
### Tokenize — `POST /v1/tokenize`
### Tokenize - `POST /v1/tokenize`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|
| 400 | Invalid or malformed request body | `Bad Request` |
| 400 | Model not found in configuration | `Bad Request` |
### Models — `GET /v1/models`, `GET /models`
### Models - `GET /v1/models`, `GET /models`
| Status | Cause | Example Message |
|--------|--------------------------------|---------------------------------------|

2
docs/content/reference/architecture.md
View File

@@ -22,4 +22,4 @@ Now, why did I choose to go with C++ bindings, you ask? Well, I wanted to keep L
As if that wasn't exciting enough, as the project gained traction, [mkellerman](https://github.com/mkellerman) and [Aisuko](https://github.com/Aisuko) jumped in to lend a hand. mkellerman helped set up some killer examples, while Aisuko is becoming our community maestro. The community now is growing even more with new contributors and users, and I couldn't be happier about it!
Oh, and let's not forget the real MVP here—[llama.cpp](https://github.com/ggerganov/llama.cpp). Without this extraordinary piece of software, LocalAI wouldn't even exist. So, a big shoutout to the community for making this magic happen!
Oh, and let's not forget the real MVP here-[llama.cpp](https://github.com/ggerganov/llama.cpp). Without this extraordinary piece of software, LocalAI wouldn't even exist. So, a big shoutout to the community for making this magic happen!

10
docs/content/reference/cli-reference.md
View File

@@ -106,7 +106,7 @@ For more information on VRAM management, see [VRAM and Memory Management]({{%rel
| Parameter | Default | Description | Environment Variable |
|-----------|---------|-------------|----------------------|
| `--auth-enabled` | `false` | Enable user authentication and authorization | `$LOCALAI_AUTH` |
| `--auth-database-url` | `{DataPath}/database.db` | Database URL for auth — `postgres://...` for PostgreSQL, or a file path for SQLite | `$LOCALAI_AUTH_DATABASE_URL`, `$DATABASE_URL` |
| `--auth-database-url` | `{DataPath}/database.db` | Database URL for auth - `postgres://...` for PostgreSQL, or a file path for SQLite | `$LOCALAI_AUTH_DATABASE_URL`, `$DATABASE_URL` |
| `--github-client-id` | | GitHub OAuth App Client ID (auto-enables auth when set) | `$GITHUB_CLIENT_ID` |
| `--github-client-secret` | | GitHub OAuth App Client Secret | `$GITHUB_CLIENT_SECRET` |
| `--oidc-issuer` | | OIDC issuer URL for auto-discovery | `$LOCALAI_OIDC_ISSUER` |
@@ -151,13 +151,19 @@ LocalAI supports several subcommands beyond `run`:
- `local-ai models` - Manage LocalAI models and definitions
- `local-ai backends` - Manage LocalAI backends and definitions
- `local-ai chat` - Open an interactive chat session against a running LocalAI server
- `local-ai tts` - Convert text to speech
- `local-ai sound-generation` - Generate audio files from text or audio
- `local-ai transcript` - Convert audio to text
- `local-ai worker` - Run workers to distribute workload (llama.cpp-only)
- `local-ai agent` - Run agents standalone without the full LocalAI server
- `local-ai mcp-server` - Run the LocalAI admin tool surface as a stdio MCP server (controls a remote LocalAI instance over HTTP)
- `local-ai worker` - Start a worker for distributed mode (generic, backend-agnostic)
- `local-ai p2p-worker` - Run workers to distribute workload via p2p (llama.cpp-only)
- `local-ai agent-worker` - Start an agent worker for distributed mode (executes agent chats via NATS)
- `local-ai util` - Utility commands
- `local-ai explorer` - Run P2P explorer
- `local-ai federated` - Run LocalAI in federated mode
- `local-ai completion` - Generate shell completion scripts for bash, zsh, or fish
Use `local-ai <command> --help` for more information on each command.

2
docs/content/reference/nvidia-l4t.md
View File

@@ -91,7 +91,7 @@ page in the frontend shows the node as fully used, check two things:
to your `docker run`, or `capabilities: [gpu, utility]` in compose /
Kubernetes device reservations.
2. Pass `--init` to `docker run` (or `init: true` in compose) so the
container has a proper PID 1 reaper — otherwise short-lived child
container has a proper PID 1 reaper - otherwise short-lived child
processes like `nvidia-smi` can intermittently fail with
`waitid: no child processes`.

2
docs/content/features/p2p.md → docs/content/reference/p2p-api.md
View File

@@ -1,6 +1,6 @@
+++
disableToc = false
title = "P2P API"
title = "P2P API reference"
weight = 22
url = "/features/p2p/"
+++

63
docs/content/reference/runtime-errors.md Normal file
View File

@@ -0,0 +1,63 @@
+++
disableToc = false
title = "Runtime errors and troubleshooting"
weight = 26
+++
This page maps the runtime and backend error messages you actually see in the logs (or in an API response) to their likely cause and fix. It covers failures that happen while a model is loading or running, as opposed to API-envelope validation errors (bad request shape, unknown field, wrong content type), which are documented in {{% relref "reference/api-errors" %}}.
If you only have an HTTP `500` and no message, read [How to read the real error](#how-to-read-the-real-error) first: the useful text is almost always in the server log, not in the HTTP body.
## Symptom, cause, fix
The left column is the literal string as it appears in the LocalAI server log (or, for the cooldown case, in the HTTP response). Match on the string, then read across.
| Error string / symptom | Likely cause | Fix |
|------------------------|--------------|-----|
| `could not load model: ...` | The selected backend started but rejected the model (bad path, corrupt or truncated GGUF, wrong architecture, unsupported quantization). The `...` is the backend's own message. | Read the wrapped backend message. Re-download the model if it is truncated. Confirm the backend matches the model (for example a GGUF needs `llama-cpp`). Run with `DEBUG=true` to see the full backend output. |
| `could not load model (no success): ...` | The backend replied to the load request but reported failure without a fatal error. | Same as above. The trailing message is the backend's status text; check it for the concrete reason (out of memory, unsupported option, missing file). |
| `could not load model - all backends returned error: ...` | LocalAI tried every candidate backend for the model and each one failed. Usually the backend for this model type is not installed, or the model file is unusable. | Install the correct backend with `local-ai backends install <backend>` (or from the Backends page). Confirm the model config `backend:` field names an installed backend. Inspect the concatenated per-backend messages for the real cause. |
| `grpc service not ready` | The backend process was spawned but its gRPC server did not become healthy in time (slow start, crash on startup, or the process died while loading). | Check the log lines just above for the backend's stderr. A crash here often means out of memory, a missing shared library, or an incompatible CPU (see `SIGILL`). Increase available RAM/VRAM or pick a smaller quantization. |
| `failed to load model: ...` | Returned by the load endpoints and several feature paths (voice, realtime, audio transform) when the model config could not be resolved or the backend load failed. | Confirm the model name exists (`local-ai models list`) and its YAML is valid. The trailing text carries the specific reason. |
| HTTP `503` with a `Retry-After` header, after a load failed | Model-load failure cooldown. After a model fails to load, LocalAI refuses new load attempts for that model for a short window so a client that keeps polling a broken model does not respawn a crashing backend on every request. The window starts at `--model-load-failure-cooldown` (default `10s`) and doubles per consecutive failure up to 5m; it resets on the first success. | Fix the underlying load failure (see the rows above), then wait out the `Retry-After` seconds before retrying, or restart LocalAI to clear the cooldown. Set `--model-load-failure-cooldown 0` (or `LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN=0`) to disable the cooldown entirely. See {{% relref "reference/cli-reference" %}}. |
| HTTP `503` with a `Retry-After` header, under load | Per-model concurrency limit reached. When a model config sets a `MaxConcurrent` limit, extra requests are rejected with `503` and a `Retry-After` (whole seconds, floor 1) instead of queueing. | Retry after the advised delay, raise the model's concurrency limit, or run more replicas. |
| `invalid pitch` (with CUDA) | The prompt exceeded the model's context size. | Reduce the prompt length, or raise the model's context size (`context_size:` in the model YAML). |
| `SIGILL` (illegal instruction) on startup | The prebuilt backend binary uses CPU instructions your CPU does not have (for example AVX512, AVX2, F16C, FMA). | Rebuild the backend for your CPU. In a container, set `REBUILD=true` and disable the unsupported instructions, for example `CMAKE_ARGS="-DGGML_F16C=OFF -DGGML_AVX512=OFF -DGGML_AVX2=OFF -DGGML_FMA=OFF" make build`. |
| CUDA / VRAM out of memory (backend log shows `out of memory`, `CUDA error: out of memory`, or the process is killed loading) | The model plus its KV cache does not fit in GPU memory. | Use a smaller quantization, reduce `context_size:`, offload fewer layers to the GPU (lower `gpu_layers:`), or free VRAM held by other processes. On multi-GPU hosts, confirm the model is not trying to load entirely onto one device. |
| Backend was terminated by the watchdog | The idle or busy watchdog stopped a backend that ran longer than its threshold. This is expected behavior when the watchdog is enabled, not a crash. | If the backend was killed prematurely, raise `--watchdog-idle-timeout` / `--watchdog-busy-timeout`, or disable the relevant watchdog (`LOCALAI_WATCHDOG_IDLE=false`, `LOCALAI_WATCHDOG_BUSY=false`). The next request reloads the model. |
| Context size exceeded / truncated output | The combined prompt and generation exceeded the model's context window. | Shorten the prompt or raise `context_size:` in the model YAML (bounded by what the model and your memory allow). |
{{% notice note %}}
The exact wording of a backend's own error (the `...` part above) is produced by the backend engine (llama.cpp, vLLM, whisper.cpp, and so on), not by LocalAI, and can change between backend versions. Match on the LocalAI-side prefix (`could not load model`, `grpc service not ready`) and treat the trailing text as the backend's diagnosis.
{{% /notice %}}
## How to read the real error
Most user-facing failures surface as an HTTP `500` whose body is a short, generic message. The real cause is in the LocalAI server log, where the backend gRPC error is logged in full:
1. **Look at the server log, not just the HTTP response.** A `500` from a chat or completion request usually wraps a backend gRPC error. The log line that matters is the one printed by LocalAI when the backend replied (or failed to start).
2. **Turn on debug output.** Run with `DEBUG=true` in the environment, or pass `--log-level=debug` (equivalently `--debug`) on the command line. This prints the backend's stdout/stderr, the load parameters, and per-token timing.
```bash
DEBUG=true local-ai run
# or
local-ai run --log-level=debug
```
3. **Read the backend's own output.** Backend engines write their diagnostics to stderr, which LocalAI captures into its log at debug level. A load failure (`could not load model`, `grpc service not ready`) almost always has the concrete reason (out of memory, missing library, unsupported quantization, illegal instruction) in the backend lines immediately above the LocalAI error.
## Performance: everything is slow
Slow inference is usually a configuration or hardware issue rather than an error:
- **Do not store models on an HDD.** Prefer an SSD. If you are stuck with an HDD, disable `mmap` in the model config so the model loads fully into memory.
- **Do not overbook the CPU.** Ideally `--threads` matches the number of physical cores. For a 4-core CPU, allocate `<= 4` threads per model.
- **Run with `DEBUG=true`** to see token-inference stats and confirm where time is going.
- **Confirm you are getting output at all:** send a simple request with `"stream": true` and watch how fast tokens arrive.
- **Check GPU offload.** If you expect GPU acceleration, confirm the model actually loaded onto the GPU (the backend log reports offloaded layers). See {{% relref "features/GPU-acceleration" %}}.
## See also
- {{% relref "reference/api-errors" %}} for API-envelope error formats and status codes.
- {{% relref "getting-started/first-agent" %}} for the "if your agent will not run" checklist (agent-specific failures).
- {{% relref "reference/cli-reference" %}} for the full flag list, including the model-load failure cooldown and watchdog options.

464
docs/content/whats-new.md
View File

@@ -1,7 +1,7 @@
+++
disableToc = false
title = "News"
weight = 7
weight = 10
url = '/basics/news/'
icon = "newspaper"
+++
@@ -12,464 +12,20 @@ You can see the release notes [here](https://github.com/mudler/LocalAI/releases)
## 2026 Highlights
- **July 2026**: [LongCat video and avatar generation](/features/longcat-video/) — dedicated CUDA backend for `LongCat-Video` text/image-to-video and `LongCat-Video-Avatar-1.5` speech-driven avatars. Includes multi-segment continuation, portrait and recorded-audio inputs in Studio, and an SDPA CUDA 13 ARM64 build for DGX Spark.
- **April 2026**: [Audio Transform](/features/audio-transform/) — generic audio-in / audio-out endpoint with optional reference signal. First implementation: [LocalVQE](https://github.com/localai-org/LocalVQE) C++ backend (joint AEC + noise suppression + dereverberation, DeepVQE-style). Both batch (`POST /audio/transformations`) and bidirectional WebSocket streaming (`/audio/transformations/stream`). Studio "Transform" tab with synchronized waveform players for input / reference / output.
- **April 2026**: [Face recognition backend](/features/face-recognition/) — `insightface`-powered 1:1 verification, 1:N identification, face embedding, face detection, and demographic analysis. Ships both a non-commercial `buffalo_l` model and an Apache 2.0 OpenCV Zoo alternative.
- **May 2026**: [Speaker diarization](/features/audio-diarization/) — new `/v1/audio/diarization` endpoint returning "who spoke when" segments. Backed by `sherpa-onnx` (pyannote-3.0 + speaker embeddings + clustering) for pure diarization, and `vibevoice-cpp` for diarization bundled with long-form ASR. Supports `json` / `verbose_json` / `rttm` response formats.
- **June 2026**: [Sound classification](/features/audio-classification/) — new `/v1/audio/classification` endpoint for audio tagging / sound-event classification, returning scored [AudioSet](https://research.google.com/audioset/) labels (baby cry, glass breaking, alarms, ...). Backed by [ced.cpp](https://github.com/mudler/ced.cpp), a 527-class AudioSet tagger ported to ggml.
- **June 2026**: [PII analyze / redact API](/features/middleware/#analyze--redact-api) — the PII detection pipeline (NER + restricted-regex pattern tiers) is now a standalone service: `POST /api/pii/analyze` returns detected entity spans and `POST /api/pii/redact` returns the sanitised text (or `400 pii_blocked`), without routing a chat request through the middleware. Events gain an `origin` (`middleware` / `proxy` / `pii_analyze` / `pii_redact`) so `/api/pii/events` can be filtered by source.
- **July 2026**: [Model capabilities endpoint](/features/api-discovery/#model-capabilities) — `GET /v1/models/capabilities`, an additive superset of `/v1/models` that reports each model's `capabilities` plus its `input_modalities` / `output_modalities` (`text` / `image` / `audio` / `video`). Lets clients route attachments using inferred or explicitly declared model modalities instead of backend-name checks.
- **June 2026**: Concurrent scoring and PII NER on llama.cpp — the `Score` (router classifier) and `TokenClassify` (PII NER) primitives now ride llama.cpp's server task queue instead of locking the context, so they run concurrently with chat/completion/embedding traffic and with each other. The `known_usecases` restriction that forced dedicated scorer/NER model configs on llama-cpp is lifted, repeated scoring calls reuse the prompt KV cache across candidates, and scoring inputs are no longer capped by the physical batch size.
- **July 2026**: [LongCat video and avatar generation](/features/video-generation/) - dedicated CUDA backend for `LongCat-Video` text/image-to-video and `LongCat-Video-Avatar-1.5` speech-driven avatars. Includes multi-segment continuation, portrait and recorded-audio inputs in Studio, and an SDPA CUDA 13 ARM64 build for DGX Spark.
- **April 2026**: [Audio Transform](/features/audio-transform/) - generic audio-in / audio-out endpoint with optional reference signal. First implementation: [LocalVQE](https://github.com/localai-org/LocalVQE) C++ backend (joint AEC + noise suppression + dereverberation, DeepVQE-style). Both batch (`POST /audio/transformations`) and bidirectional WebSocket streaming (`/audio/transformations/stream`). Studio "Transform" tab with synchronized waveform players for input / reference / output.
- **April 2026**: [Face recognition backend](/features/face-recognition/) - `insightface`-powered 1:1 verification, 1:N identification, face embedding, face detection, and demographic analysis. Ships both a non-commercial `buffalo_l` model and an Apache 2.0 OpenCV Zoo alternative.
- **May 2026**: [Speaker diarization](/features/audio-diarization/) - new `/v1/audio/diarization` endpoint returning "who spoke when" segments. Backed by `sherpa-onnx` (pyannote-3.0 + speaker embeddings + clustering) for pure diarization, and `vibevoice-cpp` for diarization bundled with long-form ASR. Supports `json` / `verbose_json` / `rttm` response formats.
- **June 2026**: [Sound classification](/features/audio-classification/) - new `/v1/audio/classification` endpoint for audio tagging / sound-event classification, returning scored [AudioSet](https://research.google.com/audioset/) labels (baby cry, glass breaking, alarms, ...). Backed by [ced.cpp](https://github.com/mudler/ced.cpp), a 527-class AudioSet tagger ported to ggml.
- **June 2026**: [PII analyze / redact API](/features/middleware/#analyze--redact-api) - the PII detection pipeline (NER + restricted-regex pattern tiers) is now a standalone service: `POST /api/pii/analyze` returns detected entity spans and `POST /api/pii/redact` returns the sanitised text (or `400 pii_blocked`), without routing a chat request through the middleware. Events gain an `origin` (`middleware` / `proxy` / `pii_analyze` / `pii_redact`) so `/api/pii/events` can be filtered by source.
- **July 2026**: [Model capabilities endpoint](/features/api-discovery/#model-capabilities) - `GET /v1/models/capabilities`, an additive superset of `/v1/models` that reports each model's `capabilities` plus its `input_modalities` / `output_modalities` (`text` / `image` / `audio` / `video`). Lets clients route attachments using inferred or explicitly declared model modalities instead of backend-name checks.
- **June 2026**: Concurrent scoring and PII NER on llama.cpp - the `Score` (router classifier) and `TokenClassify` (PII NER) primitives now ride llama.cpp's server task queue instead of locking the context, so they run concurrently with chat/completion/embedding traffic and with each other. The `known_usecases` restriction that forced dedicated scorer/NER model configs on llama-cpp is lifted, repeated scoring calls reuse the prompt KV cache across candidates, and scoring inputs are no longer capped by the physical batch size.
## 2024 Highlights
- **April 2024**: [Reranker API](https://github.com/mudler/LocalAI/pull/2121)
- **May 2024**: [Distributed inferencing](https://github.com/mudler/LocalAI/pull/2324), [Decentralized P2P llama.cpp](https://github.com/mudler/LocalAI/pull/2343) — [Docs](https://localai.io/features/distribute/)
- **May 2024**: [Distributed inferencing](https://github.com/mudler/LocalAI/pull/2324), [Decentralized P2P llama.cpp](https://github.com/mudler/LocalAI/pull/2343) - [Docs](https://localai.io/features/distribute/)
- **July/August 2024**: [P2P Dashboard, Federated mode and AI Swarms](https://github.com/mudler/LocalAI/pull/2723), [P2P Global community pools](https://github.com/mudler/LocalAI/issues/3113), FLUX-1 support, [P2P Explorer](https://explorer.localai.io)
- **October 2024**: Examples moved to [LocalAI-examples](https://github.com/mudler/LocalAI-examples)
- **November 2024**: [Voice Activity Detection (VAD)](https://github.com/mudler/LocalAI/pull/4204), [Bark.cpp backend](https://github.com/mudler/LocalAI/pull/4287)
- **December 2024**: [stablediffusion.cpp backend (ggml)](https://github.com/mudler/LocalAI/pull/4289)
---
## 04-12-2023: __v2.0.0__
This release brings a major overhaul in some backends.
Breaking/important changes:
- Backend rename: `llama-stable` renamed to `llama-ggml` {{< pr "1287" >}}
- Prompt template changes: {{< pr "1254" >}} (extra space in roles)
- Apple metal bugfixes: {{< pr "1365" >}}
New:
- Added support for LLaVa and OpenAI Vision API support ({{< pr "1254" >}})
- Python based backends are now using conda to track env dependencies ( {{< pr "1144" >}} )
- Support for parallel requests ( {{< pr "1290" >}} )
- Support for transformers-embeddings ( {{< pr "1308" >}})
- Watchdog for backends ( {{< pr "1341" >}}). As https://github.com/ggerganov/llama.cpp/issues/3969 is hitting LocalAI's llama-cpp implementation, we have now a watchdog that can be used to make sure backends are not stalling. This is a generic mechanism that can be enabled for all the backends now.
- Whisper.cpp updates ( {{< pr "1302" >}} )
- Petals backend ( {{< pr "1350" >}} )
- Full LLM fine-tuning example to use with LocalAI: https://localai.io/advanced/fine-tuning/
Due to the python dependencies size of images grew in size.
If you still want to use smaller images without python dependencies, you can use the corresponding images tags ending with `-core`.
Full changelog: https://github.com/mudler/LocalAI/releases/tag/v2.0.0
## 30-10-2023: __v1.40.0__
This release is a preparation before v2 - the efforts now will be to refactor, polish and add new backends. Follow up on: https://github.com/mudler/LocalAI/issues/1126
## Hot topics
This release now brings the `llama-cpp` backend which is a c++ backend tied to llama.cpp. It follows more closely and tracks recent versions of llama.cpp. It is not feature compatible with the current `llama` backend but plans are to sunset the current `llama` backend in favor of this one. This one will be probably be the latest release containing the older `llama` backend written in go and c++. The major improvement with this change is that there are less layers that could be expose to potential bugs - and as well it ease out maintenance as well.
### Support for ROCm/HIPBLAS
This release bring support for AMD thanks to @65a . See more details in {{< pr "1100" >}}
### More CLI commands
Thanks to @jespino now the local-ai binary has more subcommands allowing to manage the gallery or try out directly inferencing, check it out!
[Release notes](https://github.com/mudler/LocalAI/releases/tag/v1.40.0)
## 25-09-2023: __v1.30.0__
This is an exciting LocalAI release! Besides bug-fixes and enhancements this release brings the new backend to a whole new level by extending support to vllm and vall-e-x for audio generation!
Check out the documentation for vllm [here](https://localai.io/model-compatibility/vllm/) and Vall-E-X [here](https://localai.io/model-compatibility/vall-e-x/)
[Release notes](https://github.com/mudler/LocalAI/releases/tag/v1.30.0)
## 26-08-2023: __v1.25.0__
Hey everyone, [Ettore](https://github.com/mudler/) here, I'm so happy to share this release out - while this summer is hot apparently doesn't stop LocalAI development :)
This release brings a lot of new features, bugfixes and updates! Also a big shout out to the community, this was a great release!
### Attention 🚨
From this release the `llama` backend supports only `gguf` files (see {{< pr "943" >}}). LocalAI however still supports `ggml` files. We ship a version of llama.cpp before that change in a separate backend, named `llama-stable` to allow still loading `ggml` files. If you were specifying the `llama` backend manually to load `ggml` files from this release you should use `llama-stable` instead, or do not specify a backend at all (LocalAI will automatically handle this).
### Image generation enhancements
The [Diffusers]({{%relref "features/image-generation" %}}) backend got now various enhancements, including support to generate images from images, longer prompts, and support for more kernels schedulers. See the [Diffusers]({{%relref "features/image-generation" %}}) documentation for more information.
### Lora adapters
Now it's possible to load lora adapters for llama.cpp. See {{< pr "955" >}} for more information.
### Device management
It is now possible for single-devices with one GPU to specify `--single-active-backend` to allow only one backend active at the time {{< pr "925" >}}.
### Community spotlight
#### Resources management
Thanks to the continuous community efforts (another cool contribution from {{< github "dave-gray101" >}} ) now it's possible to shutdown a backend programmatically via the API.
There is an ongoing effort in the community to better handling of resources. See also the [🔥Roadmap](https://localai.io/#-hot-topics--roadmap).
#### New how-to section
Thanks to the community efforts now we have a new [how-to website](https://io.midori-ai.xyz/howtos/) with various examples on how to use LocalAI. This is a great starting point for new users! We are currently working on improving it, a huge shout out to {{< github "lunamidori5" >}} from the community for the impressive efforts on this!
#### 💡 More examples!
- Open source autopilot? See the new addition by {{< github "gruberdev" >}} in our [examples](https://github.com/mudler/LocalAI-examples/tree/main/continue) on how to use Continue with LocalAI!
- Want to try LocalAI with Insomnia? Check out the new [Insomnia example](https://github.com/mudler/LocalAI-examples/tree/main/insomnia) by {{< github "dave-gray101" >}}!
#### LocalAGI in discord!
Did you know that we have now few cool bots in our Discord? come check them out! We also have an instance of [LocalAGI](https://github.com/mudler/LocalAGI) ready to help you out!
### Changelog summary
#### Breaking Changes 🛠
* feat: bump llama.cpp, add gguf support by {{< github "mudler" >}} in {{< pr "943" >}}
#### Exciting New Features 🎉
* feat(Makefile): allow to restrict backend builds by {{< github "mudler" >}} in {{< pr "890" >}}
* feat(diffusers): various enhancements by {{< github "mudler" >}} in {{< pr "895" >}}
* feat: make initializer accept gRPC delay times by {{< github "mudler" >}} in {{< pr "900" >}}
* feat(diffusers): add DPMSolverMultistepScheduler++, DPMSolverMultistepSchedulerSDE++, guidance_scale by {{< github "mudler" >}} in {{< pr "903" >}}
* feat(diffusers): overcome prompt limit by {{< github "mudler" >}} in {{< pr "904" >}}
* feat(diffusers): add img2img and clip_skip, support more kernels schedulers by {{< github "mudler" >}} in {{< pr "906" >}}
* Usage Features by {{< github "dave-gray101" >}} in {{< pr "863" >}}
* feat(diffusers): be consistent with pipelines, support also depthimg2img by {{< github "mudler" >}} in {{< pr "926" >}}
* feat: add --single-active-backend to allow only one backend active at the time by {{< github "mudler" >}} in {{< pr "925" >}}
* feat: add llama-stable backend by {{< github "mudler" >}} in {{< pr "932" >}}
* feat: allow to customize rwkv tokenizer by {{< github "dave-gray101" >}} in {{< pr "937" >}}
* feat: backend monitor shutdown endpoint, process based by {{< github "dave-gray101" >}} in {{< pr "938" >}}
* feat: Allow to load lora adapters for llama.cpp by {{< github "mudler" >}} in {{< pr "955" >}}
Join our Discord community! our vibrant community is growing fast, and we are always happy to help! https://discord.gg/uJAeKSAGDy
The full changelog is available [here](https://github.com/go-skynet/LocalAI/releases/tag/v.1.25.0).
---
## 🔥🔥🔥🔥 12-08-2023: __v1.24.0__ 🔥🔥🔥🔥
This is release brings four(!) new additional backends to LocalAI: [🐶 Bark]({{%relref "features/text-to-audio#bark" %}}), 🦙 [AutoGPTQ]({{%relref "features/text-generation#autogptq" %}}), [🧨 Diffusers]({{%relref "features/image-generation" %}}), 🦙 [exllama]({{%relref "features/text-generation#exllama" %}}) and a lot of improvements!
### Major improvements:
* feat: add bark and AutoGPTQ by {{< github "mudler" >}} in {{< pr "871" >}}
* feat: Add Diffusers by {{< github "mudler" >}} in {{< pr "874" >}}
* feat: add API_KEY list support by {{< github "neboman11" >}} and {{< github "bnusunny" >}} in {{< pr "877" >}}
* feat: Add exllama by {{< github "mudler" >}} in {{< pr "881" >}}
* feat: pre-configure LocalAI galleries by {{< github "mudler" >}} in {{< pr "886" >}}
### 🐶 Bark
[Bark]({{%relref "features/text-to-audio#bark" %}}) is a text-prompted generative audio model - it combines GPT techniques to generate Audio from text. It is a great addition to LocalAI, and it's available in the container images by default.
It can also generate music, see the example: [lion.webm](https://user-images.githubusercontent.com/5068315/230684766-97f5ea23-ad99-473c-924b-66b6fab24289.webm)
### 🦙 AutoGPTQ
[AutoGPTQ]({{%relref "features/text-generation#autogptq" %}}) is an easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
It is targeted mainly for GPU usage only. Check out the [ documentation]({{%relref "features/text-generation" %}}) for usage.
### 🦙 Exllama
[Exllama]({{%relref "features/text-generation#exllama" %}}) is a "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights". It is a faster alternative to run LLaMA models on GPU.Check out the [Exllama documentation]({{%relref "features/text-generation#exllama" %}}) for usage.
### 🧨 Diffusers
[Diffusers]({{%relref "features/image-generation#diffusers" %}}) is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Currently it is experimental, and supports generation only of images so you might encounter some issues on models which weren't tested yet. Check out the [Diffusers documentation]({{%relref "features/image-generation" %}}) for usage.
### 🔑 API Keys
Thanks to the community contributions now it's possible to specify a list of API keys that can be used to gate API requests.
API Keys can be specified with the `API_KEY` environment variable as a comma-separated list of keys.
### 🖼️ Galleries
Now by default the model-gallery repositories are configured in the container images
### 💡 New project
[LocalAGI](https://github.com/mudler/LocalAGI) is a simple agent that uses LocalAI functions to have a full locally runnable assistant (with no API keys needed).
See it [here in action](https://github.com/mudler/LocalAGI/assets/2420543/9ba43b82-dec5-432a-bdb9-8318e7db59a4) planning a trip for San Francisco!
The full changelog is available [here](https://github.com/go-skynet/LocalAI/releases/tag/v.1.24.0).
---
## 🔥🔥 29-07-2023: __v1.23.0__ 🚀
This release focuses mostly on bugfixing and updates, with just a couple of new features:
* feat: add rope settings and negative prompt, drop grammar backend by {{< github "mudler" >}} in {{< pr "797" >}}
* Added CPU information to entrypoint.sh by @finger42 in {{< pr "794" >}}
* feat: cancel stream generation if client disappears by @tmm1 in {{< pr "792" >}}
Most notably, this release brings important fixes for CUDA (and not only):
* fix: add rope settings during model load, fix CUDA by {{< github "mudler" >}} in {{< pr "821" >}}
* fix: select function calls if 'name' is set in the request by {{< github "mudler" >}} in {{< pr "827" >}}
* fix: symlink libphonemize in the container by {{< github "mudler" >}} in {{< pr "831" >}}
{{% notice note %}}
From this release [OpenAI functions]({{%relref "features/openai-functions" %}}) are available in the `llama` backend. The `llama-grammar` has been deprecated. See also [OpenAI functions]({{%relref "features/openai-functions" %}}).
{{% /notice %}}
The full [changelog is available here](https://github.com/go-skynet/LocalAI/releases/tag/v1.23.0)
---
## 🔥🔥🔥 23-07-2023: __v1.22.0__ 🚀
* feat: add llama-master backend by {{< github "mudler" >}} in {{< pr "752" >}}
* [build] pass build type to cmake on libtransformers.a build by @TonDar0n in {{< pr "741" >}}
* feat: resolve JSONSchema refs (planners) by {{< github "mudler" >}} in {{< pr "774" >}}
* feat: backends improvements by {{< github "mudler" >}} in {{< pr "778" >}}
* feat(llama2): add template for chat messages by {{< github "dave-gray101" >}} in {{< pr "782" >}}
{{% notice note %}}
From this release to use the OpenAI functions you need to use the `llama-grammar` backend. It has been added a `llama` backend for tracking `llama.cpp` master and `llama-grammar` for the grammar functionalities that have not been merged yet upstream. See also [OpenAI functions]({{%relref "features/openai-functions" %}}). Until the feature is merged we will have two llama backends.
{{% /notice %}}
## Huggingface embeddings
In this release is now possible to specify to LocalAI external `gRPC` backends that can be used for inferencing {{< pr "778" >}}. It is now possible to write internal backends in any language, and a `huggingface-embeddings` backend is now available in the container image to be used with https://github.com/UKPLab/sentence-transformers. See also [Embeddings]({{%relref "features/embeddings" %}}).
## LLaMa 2 has been released!
Thanks to the community effort now LocalAI supports templating for LLaMa2! more at: {{< pr "782" >}} until we update the model gallery with LLaMa2 models!
## Official langchain integration
Progress has been made to support LocalAI with `langchain`. See: https://github.com/langchain-ai/langchain/pull/8134
---
## 🔥🔥🔥 17-07-2023: __v1.21.0__ 🚀
* [whisper] Partial support for verbose_json format in transcribe endpoint by `@ldotlopez` in {{< pr "721" >}}
* LocalAI functions by `@mudler` in {{< pr "726" >}}
* `gRPC`-based backends by `@mudler` in {{< pr "743" >}}
* falcon support (7b and 40b) with `ggllm.cpp` by `@mudler` in {{< pr "743" >}}
### LocalAI functions
This allows to run OpenAI functions as described in the OpenAI blog post and documentation: https://openai.com/blog/function-calling-and-other-api-updates.
This is a video of running the same example, locally with `LocalAI`:
![localai-functions-1](https://github.com/ggerganov/llama.cpp/assets/2420543/5bd15da2-78c1-4625-be90-1e938e6823f1)
And here when it actually picks to reply to the user instead of using functions!
![functions-2](https://github.com/ggerganov/llama.cpp/assets/2420543/e3f89d15-1d2c-45ab-974f-6c9eb8eae41d)
Note: functions are supported only with `llama.cpp`-compatible models.
A full example is available here: https://github.com/mudler/LocalAI-examples/tree/main/functions
### gRPC backends
This is an internal refactor which is not user-facing, however, it allows to ease out maintenance and addition of new backends to LocalAI!
### `falcon` support
Now Falcon 7b and 40b models compatible with https://github.com/cmp-nct/ggllm.cpp are supported as well.
The former, ggml-based backend has been renamed to `falcon-ggml`.
### Default pre-compiled binaries
From this release the default behavior of images has changed. Compilation is not triggered on start automatically, to recompile `local-ai` from scratch on start and switch back to the old behavior, you can set `REBUILD=true` in the environment variables. Rebuilding can be necessary if your CPU and/or architecture is old and the pre-compiled binaries are not compatible with your platform. See the [build section]({{%relref "installation/build" %}}) for more information.
[Full release changelog](https://github.com/go-skynet/LocalAI/releases/tag/v1.21.0)
---
## 🔥🔥🔥 28-06-2023: __v1.20.0__ 🚀
### Exciting New Features 🎉
* Add Text-to-Audio generation with `go-piper` by {{< github "mudler" >}} in {{< pr "649" >}} See [API endpoints]({{%relref "features/text-to-audio" %}}) in our documentation.
* Add gallery repository by {{< github "mudler" >}} in {{< pr "663" >}}. See [models]({{%relref "features/model-gallery" %}}) for documentation.
### Container images
- Standard (GPT + `stablediffusion`): `quay.io/go-skynet/local-ai:v1.20.0`
- FFmpeg: `quay.io/go-skynet/local-ai:v1.20.0-ffmpeg`
- CUDA 11+FFmpeg: `quay.io/go-skynet/local-ai:v1.20.0-gpu-nvidia-cuda11-ffmpeg`
- CUDA 12+FFmpeg: `quay.io/go-skynet/local-ai:v1.20.0-gpu-nvidia-cuda12-ffmpeg`
### Updates
Updates to `llama.cpp`, `go-transformers`, `gpt4all.cpp` and `rwkv.cpp`.
The NUMA option was enabled by {{< github "mudler" >}} in {{< pr "684" >}}, along with many new parameters (`mmap`,`mmlock`, ..). See [advanced]({{%relref "advanced" %}}) for the full list of parameters.
### Gallery repositories
In this release there is support for gallery repositories. These are repositories that contain models, and can be used to install models. The default gallery which contains only freely licensed models is in Github: https://github.com/go-skynet/model-gallery, but you can use your own gallery by setting the `GALLERIES` environment variable. An automatic index of huggingface models is available as well.
For example, now you can start `LocalAI` with the following environment variable to use both galleries:
```bash
GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}, {"url": "github:ci-robbot/localai-huggingface-zoo/index.yaml","name":"huggingface"}]
```
And in runtime you can install a model from huggingface now with:
```bash
curl http://localhost:8000/models/apply -H "Content-Type: application/json" -d '{ "id": "huggingface@thebloke__open-llama-7b-open-instruct-ggml__open-llama-7b-open-instruct.ggmlv3.q4_0.bin" }'
```
or a `tts` voice with:
```bash
curl http://localhost:8080/models/apply -H "Content-Type: application/json" -d '{ "id": "model-gallery@voice-en-us-kathleen-low" }'
```
See also [models]({{%relref "features/model-gallery" %}}) for a complete documentation.
### Text to Audio
Now `LocalAI` uses [piper](https://github.com/rhasspy/piper) and [go-piper](https://github.com/mudler/go-piper) to generate audio from text. This is an experimental feature, and it requires `GO_TAGS=tts` to be set during build. It is enabled by default in the pre-built container images.
To setup audio models, you can use the new galleries, or setup the models manually as described in [the API section of the documentation]({{%relref "features/text-to-audio" %}}).
You can check the full changelog in [Github](https://github.com/go-skynet/LocalAI/releases/tag/v1.20.0)
---
## 🔥🔥🔥 19-06-2023: __v1.19.0__ 🚀
- Full CUDA GPU offload support ( [PR](https://github.com/go-skynet/go-llama.cpp/pull/105) by [mudler](https://github.com/mudler). Thanks to [chnyda](https://github.com/chnyda) for handing over the GPU access, and [lu-zero](https://github.com/lu-zero) to help in debugging )
- Full GPU Metal Support is now fully functional. Thanks to [Soleblaze](https://github.com/Soleblaze) to iron out the Metal Apple silicon support!
Container images:
- Standard (GPT + `stablediffusion`): `quay.io/go-skynet/local-ai:v1.19.2`
- FFmpeg: `quay.io/go-skynet/local-ai:v1.19.2-ffmpeg`
- CUDA 11+FFmpeg: `quay.io/go-skynet/local-ai:v1.19.2-gpu-nvidia-cuda11-ffmpeg`
- CUDA 12+FFmpeg: `quay.io/go-skynet/local-ai:v1.19.2-gpu-nvidia-cuda12-ffmpeg`
---
## 🔥🔥🔥 06-06-2023: __v1.18.0__ 🚀
This LocalAI release is plenty of new features, bugfixes and updates! Thanks to the community for the help, this was a great community release!
We now support a vast variety of models, while being backward compatible with prior quantization formats, this new release allows still to load older formats and new [k-quants](https://github.com/ggerganov/llama.cpp/pull/1684)!
### New features
- ✨ Added support for `falcon`-based model families (7b) ( [mudler](https://github.com/mudler) )
- ✨ Experimental support for Metal Apple Silicon GPU - ( [mudler](https://github.com/mudler) and thanks to [Soleblaze](https://github.com/Soleblaze) for testing! ). See the [build section]({{%relref "installation/build#Acceleration" %}}).
- ✨ Support for token stream in the `/v1/completions` endpoint ( [samm81](https://github.com/samm81) )
- ✨ Added huggingface backend ( [Evilfreelancer](https://github.com/EvilFreelancer) )
- 📷 Stablediffusion now can output `2048x2048` images size with `esrgan`! ( [mudler](https://github.com/mudler) )
### Container images
- 🐋 CUDA container images (arm64, x86_64) ( [sebastien-prudhomme](https://github.com/sebastien-prudhomme) )
- 🐋 FFmpeg container images (arm64, x86_64) ( [mudler](https://github.com/mudler) )
### Dependencies updates
- 🆙 Bloomz has been updated to the latest ggml changes, including new quantization format ( [mudler](https://github.com/mudler) )
- 🆙 RWKV has been updated to the new quantization format( [mudler](https://github.com/mudler) )
- 🆙 [k-quants](https://github.com/ggerganov/llama.cpp/pull/1684) format support for the `llama` models ( [mudler](https://github.com/mudler) )
- 🆙 gpt4all has been updated, incorporating upstream changes allowing to load older models, and with different CPU instruction set (AVX only, AVX2) from the same binary! ( [mudler](https://github.com/mudler) )
### Generic
- 🐧 Fully Linux static binary releases ( [mudler](https://github.com/mudler) )
- 📷 Stablediffusion has been enabled on container images by default ( [mudler](https://github.com/mudler) )
Note: You can disable container image rebuilds with `REBUILD=false`
### Examples
- 💡 [AutoGPT](https://github.com/mudler/LocalAI-examples/tree/main/autoGPT) example ( [mudler](https://github.com/mudler) )
- 💡 [PrivateGPT](https://github.com/mudler/LocalAI-examples/tree/main/privateGPT) example ( [mudler](https://github.com/mudler) )
- 💡 [Flowise](https://github.com/mudler/LocalAI-examples/tree/main/flowise) example ( [mudler](https://github.com/mudler) )
Two new projects offer now direct integration with LocalAI!
- [Flowise](https://github.com/FlowiseAI/Flowise/pull/123)
- [Mods](https://github.com/charmbracelet/mods)
[Full release changelog](https://github.com/go-skynet/LocalAI/releases/tag/v1.18.0)
---
## 29-05-2023: __v1.17.0__
Support for OpenCL has been added while building from sources.
You can now build LocalAI from source with `BUILD_TYPE=clblas` to have an OpenCL build. See also the [build section]({{%relref "getting-started/build#Acceleration" %}}).
For instructions on how to install OpenCL/CLBlast see [here](https://github.com/ggerganov/llama.cpp#blas-build).
rwkv.cpp has been updated to the new ggml format [commit](https://github.com/saharNooby/rwkv.cpp/commit/dea929f8cad90b7cf2f820c5a3d6653cfdd58c4e).
---
## 27-05-2023: __v1.16.0__
Now it's possible to automatically download pre-configured models before starting the API.
Start local-ai with the `PRELOAD_MODELS` containing a list of models from the gallery, for instance to install `gpt4all-j` as `gpt-3.5-turbo`:
```bash
PRELOAD_MODELS=[{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo"}]
```
`llama.cpp` models now can also automatically save the prompt cache state as well by specifying in the model YAML configuration file:
```yaml
prompt_cache_path: "alpaca-cache"
prompt_cache_all: true
```
See also the [advanced section]({{%relref "advanced" %}}).
## Media, Blogs, Social
- [Create a slackbot for teams and OSS projects that answer to documentation](https://mudler.pm/posts/smart-slackbot-for-teams/)
- [LocalAI meets k8sgpt](https://www.youtube.com/watch?v=PKrDNuJ_dfE) - CNCF Webinar showcasing LocalAI and k8sgpt.
- [Question Answering on Documents locally with LangChain, LocalAI, Chroma, and GPT4All](https://mudler.pm/posts/localai-question-answering/) by Ettore Di Giacinto
- [Tutorial to use k8sgpt with LocalAI](https://medium.com/@tyler_97636/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65) - excellent usecase for localAI, using AI to analyse Kubernetes clusters. by Tyller Gillson
## Previous
- 23-05-2023: __v1.15.0__ released. `go-gpt2.cpp` backend got renamed to `go-ggml-transformers.cpp` updated including https://github.com/ggerganov/llama.cpp/pull/1508 which breaks compatibility with older models. This impacts RedPajama, GptNeoX, MPT(not `gpt4all-mpt`), Dolly, GPT2 and Starcoder based models. [Binary releases available](https://github.com/go-skynet/LocalAI/releases), various fixes, including {{< pr "341" >}} .
- 21-05-2023: __v1.14.0__ released. Minor updates to the `/models/apply` endpoint, `llama.cpp` backend updated including https://github.com/ggerganov/llama.cpp/pull/1508 which breaks compatibility with older models. `gpt4all` is still compatible with the old format.
- 19-05-2023: __v1.13.0__ released! 🔥🔥 updates to the `gpt4all` and `llama` backend, consolidated CUDA support ( {{< pr "310" >}} thanks to @bubthegreat and @Thireus ), preliminar support for [installing models via API]({{%relref "advanced#" %}}).
- 17-05-2023: __v1.12.0__ released! 🔥🔥 Minor fixes, plus CUDA ({{< pr "258" >}}) support for `llama.cpp`-compatible models and image generation ({{< pr "272" >}}).
- 16-05-2023: 🔥🔥🔥 Experimental support for CUDA ({{< pr "258" >}}) in the `llama.cpp` backend and Stable diffusion CPU image generation ({{< pr "272" >}}) in `master`.
Now LocalAI can generate images too:
| mode=0 | mode=1 (winograd/sgemm) |
|------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
| ![b6441997879](https://github.com/go-skynet/LocalAI/assets/2420543/d50af51c-51b7-4f39-b6c2-bf04c403894c) | ![winograd2](https://github.com/go-skynet/LocalAI/assets/2420543/1935a69a-ecce-4afc-a099-1ac28cb649b3) |
- 14-05-2023: __v1.11.1__ released! `rwkv` backend patch release
- 13-05-2023: __v1.11.0__ released! 🔥 Updated `llama.cpp` bindings: This update includes a breaking change in the model files ( https://github.com/ggerganov/llama.cpp/pull/1405 ) - old models should still work with the `gpt4all-llama` backend.
- 12-05-2023: __v1.10.0__ released! 🔥🔥 Updated `gpt4all` bindings. Added support for GPTNeox (experimental), RedPajama (experimental), Starcoder (experimental), Replit (experimental), MosaicML MPT. Also now `embeddings` endpoint supports tokens arrays. See the [langchain-chroma](https://github.com/mudler/LocalAI-examples/tree/main/langchain-chroma) example! Note - this update does NOT include https://github.com/ggerganov/llama.cpp/pull/1405 which makes models incompatible.
- 11-05-2023: __v1.9.0__ released! 🔥 Important whisper updates ( {{< pr "233" >}} {{< pr "229" >}} ) and extended gpt4all model families support ( {{< pr "232" >}} ). Redpajama/dolly experimental ( {{< pr "214" >}} )
- 10-05-2023: __v1.8.0__ released! 🔥 Added support for fast and accurate embeddings with `bert.cpp` ( {{< pr "222" >}} )
- 09-05-2023: Added experimental support for transcriptions endpoint ( {{< pr "211" >}} )
- 08-05-2023: Support for embeddings with models using the `llama.cpp` backend ( {{< pr "207" >}} )
- 02-05-2023: Support for `rwkv.cpp` models ( {{< pr "158" >}} ) and for `/edits` endpoint
- 01-05-2023: Support for SSE stream of tokens in `llama.cpp` backends ( {{< pr "152" >}} )