{{ define "main" }}

Open source · MIT · v4.8.0

Make AI run onevery machine.

Text, voice, vision, images, video, 3D and agents, from one open runtime. It works on the laptop you already own, and scales to a room full of GPUs when you have one. When the engine we need is too heavy, too closed, or does not exist, we write it.

Install LocalAI → Read the docs ★ Star on GitHub
0GitHub stars
0Backends
0Engines we wrote
0Models, one click
localai · chat CPU only, no GPU

APEX quantization

The model you could not fit, on the card you already own.

{{/* APEX was being used as a known term on first appearance, in a section that opened straight onto a benchmark table. The first two sentences say what it is and why it follows the engines. */}}

The engine decides how fast a model runs. The weights decide whether it runs at all, so we build those too. APEX assigns a different precision to every tensor and every layer: a 35B mixture-of-experts model goes from 64.6 GB, out of reach of any consumer GPU, to 12.2 GB at 74 tokens a second. That is more than twice the speed of the original, and quality barely moves. The file is an ordinary GGUF, so stock llama.cpp opens it with no patches, and 201 of them are already sitting in the LocalAI gallery.

F16 · 64.6 GB30.4 t/s
Q8_0 · 34.4 GB52.5 t/s
APEX Quality · 21.3 GB62.3 t/s
APEX Mini · 12.2 GB74.4 t/s
BuildSizePerplexityHellaSwagMMLUtg128 t/s
F1664.6 GB6.53782.5%41.5%30.4
Q8_034.4 GB6.53383.0%41.2%52.5
Unsloth UD-Q8_K_XL45.3 GB6.53682.5%41.3%36.4
APEX Quality21.3 GB6.52783.0%41.2%62.3
APEX I-Quality21.3 GB6.55283.5%41.4%63.1
APEX Compact16.1 GB6.78382.5%40.9%69.8
APEX Mini12.2 GB7.08881.0%41.3%74.4

Qwen3.5-35B-A3B on an NVIDIA DGX Spark (GB10). Perplexity on wikitext-2-raw at context 2048. Full methodology in the technical report.

APEX models on Hugging Face ↗ Technical report ↗

the agent

The agent is already in the binary.

Run local-ai chat and you are talking to an agent that already knows where your models are. It runs shell commands behind an approval gate you control, delegates to sub-agents, and loads MCP servers, plugins and skills. Claude Code plugins load as they are.

It is also nib, a single ~20 MB Go binary with no runtime and no daemon, so you can drop the same agent on any box you SSH into and press Ctrl+Space.

local-ai chatMCPpluginsskillssub-agentszero dependencies
Read the docs → nib on GitHub ↗
local-ai chat the agent, in the binary

From the team

We publish the numbers, including the ones that cost us.

APEX takes a 35B model from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second. Perplexity goes from 6.537 to 7.088. That trade is in the post too, with the command that produced it.

{{/* Pulled from the posts themselves. The cards used to be hand-written, which is how one of them ended up advertising a post that did not exist, and how all three linked to the index instead of the article. */}}
{{- range first 3 (where .Site.RegularPages "Section" "blog") }}

{{ .Date.Format "2 January 2006" }}

{{ .Title }}

{{ .Params.summary | truncate 155 }}

Read the post → {{- end }}
{{ end }}