{{ define "main" }}

Open source · MIT · v4.8.0

Make AI run onevery machine.

Text, voice, vision, images, video, 3D and agents, from one open runtime. It works on the laptop you already own, and scales to a room full of GPUs when you have one. When the engine we need is too heavy, too closed, or does not exist, we write it.

Install LocalAI → Read the docs ★ Star on GitHub
0GitHub stars
0Backends
0Engines we wrote
0Models, one click
localai · chat CPU only, no GPU

APEX quantization

The model you could not fit, on the card you already own.

{{/* APEX was being used as a known term on first appearance, in a section that opened straight onto a benchmark table. The first two sentences say what it is and why it follows the engines. */}}

The engine decides how fast a model runs. The weights decide whether it runs at all, so we build those too. APEX assigns a different precision to every tensor and every layer: a 35B mixture-of-experts model goes from 64.6 GB, out of reach of any consumer GPU, to 12.2 GB at 74 tokens a second. That is more than twice the speed of the original, and quality barely moves. The file is an ordinary GGUF, so stock llama.cpp opens it with no patches, and 201 of them are already sitting in the LocalAI gallery.

F16 · 64.6 GB30.4 t/s
Q8_0 · 34.4 GB52.5 t/s
APEX Quality · 21.3 GB62.3 t/s
APEX Mini · 12.2 GB74.4 t/s
BuildSizePerplexityHellaSwagMMLUtg128 t/s
F1664.6 GB6.53782.5%41.5%30.4
Q8_034.4 GB6.53383.0%41.2%52.5
Unsloth UD-Q8_K_XL45.3 GB6.53682.5%41.3%36.4
APEX Quality21.3 GB6.52783.0%41.2%62.3
APEX I-Quality21.3 GB6.55283.5%41.4%63.1
APEX Compact16.1 GB6.78382.5%40.9%69.8
APEX Mini12.2 GB7.08881.0%41.3%74.4

Qwen3.5-35B-A3B on an NVIDIA DGX Spark (GB10). Perplexity on wikitext-2-raw at context 2048. Full methodology in the technical report.

APEX models on Hugging Face ↗ Technical report ↗

nib

An agent you can drop on any box you SSH into.

One Go binary, about 20 MB, no runtime and no daemon. Press Ctrl+Space anywhere and it opens. Point it at any OpenAI-compatible endpoint, including a model running on your own laptop, and it works. Tool calls go through an approval gate you control, and Claude Code plugins load as they are.

Gozero dependenciesMCPpluginsskillssub-agents

nib on GitHub ↗

nib ~20 MB, one binary

From the team

We publish the numbers, including the ones that cost us.

APEX takes a 35B model from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second. Perplexity goes from 6.537 to 7.088. That trade is in the post too, with the command that produced it.

{{/* Pulled from the posts themselves. The cards used to be hand-written, which is how one of them ended up advertising a post that did not exist, and how all three linked to the index instead of the article. */}}
{{- range first 3 (where .Site.RegularPages "Section" "blog") }}

{{ .Date.Format "2 January 2006" }}

{{ .Title }}

{{ .Params.summary | truncate 155 }}

Read the post → {{- end }}
{{ end }}