mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-08-04 12:22:22 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
60bee92a2cb2ee9d02b619e9d5852fc79bef6faa
LocalAI/website/layouts/index.html
mudler's LocalAI [bot] d0119bf62c feat(chat): local-ai chat is now a terminal agent (#11291) 
* chore(deps): bump cogito to v0.11 ahead of the nib harness

nib is the agent harness that becomes 'local-ai chat'. It requires cogito
v0.11, so pull that bump forward on its own: minimal version selection would
apply it to LocalAI anyway, and both repos use cogito and cogito/clients.
Landing it separately keeps the harness change reviewable.

nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod
tidy' runs as a goreleaser before-hook in CI, so an unimported require line
does not survive. It lands with its first importer.

No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers
guard the argument above zero, so v0.11's new clamp is unreachable, and
LocalAI's Multimedia values implement only URL(), so v0.11's new
TypedMultimedia routing treats them as images exactly as v0.10 did.

Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664).
A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over
the pre-change baseline).

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(chat): resolve and seed the agent state directory

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): write the agent config atomically and tighten its modes

Replacing config.yaml in place truncated it first, so an interrupted write
would have destroyed the api_key nib keeps in the same file. Stage through a
sibling temp file and rename over the target instead, and match nib's 0700
directory mode.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(chat): probe the endpoint and classify failures

Probe lists what a LocalAI endpoint advertises and separates the two
failures that need different advice: nothing listening, and rejected
credentials.

go-openai reports a rejected key as one of two concrete types depending
on the error body, and both occur against a real LocalAI. The normal
error handler sends an OpenAI error envelope, which arrives as
*openai.APIError; the opaque-errors handler replies with a bare status
and no body, which arrives as *openai.RequestError. Classifying on only
one of them misses half the cases, so the status is read from either.

A cancelled probe is not reported as an unreachable server, because it
learned nothing about the endpoint, and neither is a reply that could
not be parsed, because something did answer. Both would otherwise send
the user off to start a server that may already be running.

The model list is returned verbatim and in server order. LocalAI lists
whatever it finds in the models directory, including stray archives and
dotfiles, and deciding which advertised ids are real belongs to whoever
presents them.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(chat): resolve the model from flag, config, or the server

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* test(chat): pin that model resolution sorts a copy of the caller's slice

The sort spec asserted only on what the chooser was offered, so replacing the
defensive copy with an in-place sort of req.Available still passed all 37
specs. Assert the input slice's order after the call, so the guarantee cannot
be dropped silently by a later refactor.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(chat): offer to start a server when none is reachable

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): bound the server wait and pin readiness and stop semantics

Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe
cannot block cmd.Wait forever, which would leave exited unclosed, burn the
whole shutdown grace on a clean exit, and leak the waiter goroutine.

Two test gaps closed alongside it: the readiness spec now counts polls, so
treating 503 as ready is observable, and Stop's single-interrupt contract is
pinned by giving StartedServer interrupt/kill hooks that a spec can count.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(chat): drive Stop through one process interface, hide exec plumbing

Two independent interrupt/kill func fields plus a nil check admitted wirings
no test could distinguish: the pair swapped, so a SIGKILL would strand the
backends SIGINT exists to let local-ai run clean up, or kill left nil, so a
wedged server never escalates. One two-method interface that *os.Process
already satisfies leaves nothing to swap and nothing to nil.

Also translate exec.ErrWaitDelay, whose text names an os/exec struct field,
into what the user can act on. os/exec only substitutes that sentinel when the
process exited without an error of its own, so no exit status is swallowed.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* feat(chat): replace the REPL with the built-in agent

local-ai chat is now the nib agent harness compiled into the binary: tool
use behind an approval gate, sub-agents, MCP, plugins, and skills, all
auto-configured against the local server.

The REPL goes with it. Its model listing and its 401 classifier were
duplicates of the ones Probe now owns, and the classifier was the version
that misreads a bare 401 with no OpenAI error envelope, so keeping either
would leave the package with two divergent answers to the same question.

github.com/mudler/nib lands in go.mod in this commit rather than earlier:
go mod tidy runs as a goreleaser before-hook on every PR, so a require
line with no importer is stripped before it reaches CI.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* refactor(chat): split the pre-agent phase out of Run and pin it

Everything before the handoff is testable and nothing after it is: once
app.Run owns the terminal there is no seam left. prepare draws that line,
takes interactivity as a parameter so the prompts can be driven over a
pipe, and hands Run the state dir, the model, and any server it started.

The questions move onto one prompter that owns its buffered reader. A
fresh bufio.Reader per question reads ahead and discards what it buffered,
so the model choice typed behind an answer to "start a server?" was lost
and the next question saw EOF.

choose answers with a list index and refuses an empty offer, so a value
that was never on the list cannot reach ResolveModel, which persists it
and starts every later run against it.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): bound each server check with a deadline

Nothing bounded the model listing, so pointing chat at an address that
accepts the connection and then never replies left the user with no
output and no offer to start a server.

The budget is context.WithTimeout rather than a cancel plus a timer.
Probe deliberately refuses to call an endpoint unreachable on a
context.Canceled, since a caller who gave up learned nothing about the
server, and only honours a deadline. A cancel-based budget therefore
expires as the one error that suppresses ErrUnreachable, exactly for the
hung servers the offer exists to rescue.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): tell the user when their model choice cannot be saved

The choice is meant to be asked for once. When saving it fails the user
is silently asked again on the next run, and the only trace was an
xlog.Warn: the agent runs at log level error, and a --log-level=error run
swallows it entirely.

ModelRequest gains Notify for exactly this class of problem, one that is
worth telling the user about but not worth failing over, and the chat
wiring points it at the same writer the question was asked on.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): stop a session's server when the process is signalled

A server started for the session is stopped by a deferred call, and a
signal skips deferred calls: a SIGTERM between the spawn and the exit
left 'local-ai run' reparented to init with nothing left that knew to
shut it down. Ctrl+C was already safe, but only incidentally, because the
child shares this process' foreground process group.

A signal handler rather than Pdeathsig on the child. Pdeathsig is
Linux-only and, in Go, is delivered when the OS thread that forked exits
rather than when the process does, so it can fire on a healthy parent.
Setpgid would break the Ctrl+C that works today by taking the child out
of the foreground group.

SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is
gone has nobody left to talk to. The same context is what cancels the
agent, which nib leaves to its embedder.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): only skip the server checks for work that stays local

Two argument shapes were classified wrongly. Every 'mcp ...' invocation
counted as management, so 'local-ai chat mcp --stdio', which serves the
agent over MCP and needs a model like any other session, was handed an
empty one. And --init, whose shell snippet a user pastes into an rc file
long before any server exists, went the other way: it demanded a running
server to print a static string.

The mcp split is asked of nib's own IsMCPManageSubcommand rather than
restated here, so a verb added upstream cannot drift out of this list.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): exit with the agent's status instead of reporting it twice

nib writes what went wrong to stderr and returns nothing but an exit
code, so returning that error unchanged had main log "Error running the
application error=exit status 1" underneath the message the user had just
read. The refusal to render the full-screen interface into a pipe is the
one they meet in practice: it names --cli, and burying that hides the fix.

ExitCodeError says "already reported, exit with this status". main
honours it and prints nothing more, so a piped or redirected chat still
fails a script the way it should.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* style(chat): route interactive chatter through one writer helper

The prompts and notices all write to a terminal, where a failed write is
not worth failing the session over and the read that follows the question
reports the real problem. say says that once instead of five discarded
error returns.

The command's one-line help comes along: chat is no longer "an
interactive chat session".

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(chat): record why the agent gets this process' streams

Injecting them is what makes nib refuse to draw its full-screen interface
into a pipe and name --cli, instead of rendering onto a terminal the
caller may not own. The tradeoff is worth stating where the wiring is.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): stop the session's server on cancellation, not on the way out

The deferred Stop is reached only if the agent returns, and cancelling
the context does not make it: nib hands the TUI to bubbletea without the
context, so what actually unwinds a running session today is bubbletea's
own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and
registering for it removed the default disposition that used to end the
process outright, so kill -HUP left a live TUI with a cancelled context
and the started server still running.

runSession watches the context alongside the agent and stops the server
the moment it is cancelled, so the guarantee no longer depends on what
the agent does with cancellation. Stop is idempotent, so the deferred
call stays correct and free.

The doc comment on shutdownContext described the mechanism it was
supposed to work by rather than the one that does. Corrected, bubbletea's
handler included.

ResolveModel now checks the chooser's answer against what it offered.
The shipped chooser answers by list index and cannot be wrong, but
ModelChooser is exported, the answer is persisted, and every later run
starts against it, so the invariant belongs at the consumer.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* chore(chat): bump nib to v0.5.1

v0.5.1 carries four fixes that matter to 'local-ai chat':

- --init now names the embedder's command, so the emitted widget invokes
  'local-ai chat' rather than a bare 'nib' the user does not have.
- A piped CLI session that succeeds exits 0 instead of failing with EOF.
- EOF at a tool-approval prompt denies the call rather than approving it,
  and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell
  "answered" from "refused to act" without reading stdout. Read-only tools
  are unaffected and still run. ExitStatus already unwraps app.ExitError,
  so the code propagates with no change here.
- RunTUI passes the context to bubbletea and gives up bubbletea's own signal
  handler, which makes shutdownContext the single owner of the signal and
  stops a SIGHUP leaving a wedged TUI behind.

Verified against a live server on 127.0.0.1:8080: the three --init shells,
a piped prompt exiting 0, a denied 'touch' that left no file and exited 3,
a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a
TUI running under a pty.

Two comment blocks in run.go described the old TUI behavior and are now
wrong, so they are corrected in the same change. No behavior change: both
shutdownContext and runSession are untouched, and stopping the server on
cancellation is still worth keeping independent of how promptly nib unwinds.

One known gap, not addressed here. The widget --init now emits runs
'output=$(local-ai chat --height 50%)', and runAgent injects Stdout
unconditionally, so under $(...) nib refuses the TUI for a non-terminal
stream. This is the cost the runAgent comment already anticipated, now that
the snippets no longer hardcode standalone nib. Ctrl+Space should not be
documented until that is decided.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): let nib own stdout, so the Ctrl+Space widget works

The widget 'local-ai chat --init' emits runs
'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by
construction. runAgent injected os.Stdout unconditionally, and nib refuses
every mode but --cli when a stream it was handed is not a terminal, so
Ctrl+Space printed "Re-run with --cli to use the injected streams" and
inserted nothing. Verified against a pty before and after.

nib reads a nil stream as "not injected" and falls back to the process
stream, which is how an embedder asks for nib's own behavior. That is what
stdout needs: the interface renders on /dev/tty but writes the chosen
command to stdout even when stdout is a pipe, and that write is the whole
of the shell-capture idiom.

Stdin is deliberately left injected. A piped or redirected stdin really is
ignored by the interface, so the refusal is the honest answer there, and it
is the one users meet: 'echo q | local-ai chat' still says to re-run with
--cli, once, exit 1. Nilling stdin the way stdout is nilled would delete
that silently. Stderr is not gated by nib at all and is unchanged.

One case does change and cannot be kept: 'local-ai chat > out.txt' from a
terminal no longer refuses, because it is indistinguishable from the
widget. It renders on /dev/tty and writes the capture line to the file,
which is what standalone nib does.

The app.Options literal moves into agentOptions so the decision is
reachable from a spec rather than being a detail of a function that takes
the terminal. Both sides of the asymmetry are pinned: reinstating
'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and
nilling stdin fails "hands the process stdin over".

Also rewrites the last comments describing the pre-v0.5.1 behavior.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs(chat): say what the stream refusal actually keys on

Two comments still called it the refusal to render the interface "into a
pipe". That was true when both stdin and stdout were injected, but a pipe on
stdout no longer refuses, so the wording now points at precisely the case
that was un-refused to make Ctrl+Space work. Only a stdin that cannot be
read triggers it, and both comments now say so and name the command a user
meets it with, 'echo q | local-ai chat'.

The agentOptions doc also said a "file a caller chose" stays injected and
refused, which reads as though 'local-ai chat > out.txt' still refuses. It
does not: a shell redirect arrives as os.Stdout and is nil-ed like the
widget's pipe, because the two differ only in being a regular file rather
than a FIFO and nib's gate does not look at that. What stays injected is a
writer an in-process caller chose for itself. Says that now, in the doc and
in the spec comment that had the same ambiguity.

Comments only. No behavior change.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* docs: local-ai chat is now the built-in terminal agent

`local-ai chat` was a plain chat prompt and is now an agent that runs
shell commands behind an approval gate, so the pages that described a
REPL were wrong rather than merely thin.

Adds a Terminal agent feature page at /features/terminal-agent covering
the approval gate, piped runs and their exit codes, Ctrl+Space, model
resolution, state directory, and the pass-through management commands
(including the `--yes` caveat that leaves a plugin installed but
disabled in a script).

The three-way "looking for something else" notice becomes four-way and
moves into an agentic-routing shortcode. Four hand-kept copies of the
same paragraph is what produced the drift the new page would otherwise
have added to; the shortcode takes `current=` so each page still marks
itself, and errors the build on a name that is not one of the four.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* website: the agent is in the binary, not a second install

The nib section sold a separate tool you also install, with a GitHub
link as the only way in, which is now the wrong order: the agent ships
compiled into local-ai, and the standalone binary is the second reason
to care rather than the first.

Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and
adds a docs CTA pointing at the new Terminal agent page. id="nib" is
left alone because localai.io/#nib is linked from outside.

The two credits on the demo clip named nib as the thing that drove the
machine; they now credit the agent in LocalAI, which is the same agent.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* website: fix the exit keys, the plugin warning, and the redirect gap

Three claims on the chat-agent pages that the code does not back.

try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the
full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit
in --cli, where it arrives as ordinary tty EOF. That sentence had
replaced the removed /exit and /quit text, so the page was left with no
working way to leave a session. Document both modes, since they differ.

The plugin warning said nothing tells you the install stopped short. It
does: the command prints that the plugin was left disabled. What it does
not do is say so in its exit code, which is 0 either way. That is the
part a script cannot work around, and it is the reason to pass --yes.
Overstating it in the paragraph that gives the advice only makes the
advice easier to dismiss.

Redirecting stdout no longer refuses; the interface goes to /dev/tty and
only the yanked command reaches the file. It is what lets the Ctrl+Space
widget capture a command at all, since a redirect and out=$(...) are the
same thing to the stream gate. It was documented nowhere. A non-terminal
stdin is still refused, and the new text says which of the two it is.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): make the CLI flags outrank the agent config file

local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo
through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the
config file, so the file silently undoes them. That made the flags accepted and
inert, and not in an edge case, since EnsureStateDir writes base_url on the
first run and the interactive picker writes model, so from the second run on
the file carried a value for both.

Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the
config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit
8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0
in the config, --model lfm2.5-8b-a1b was ignored on the wire.

nib v0.6.0 adds app.Options.Overrides, applied above the config file and above
the bare environment block. Move the whole block there: all five values are
decisions this invocation already made on the user's behalf, and a flag the
config file can undo is not a flag. Nothing is left in Defaults, because
LocalAI's one genuine seed, the initial base_url, is written into the config
file by EnsureStateDir rather than handed to nib.

Two limits come with the channel and are documented on agentOptions rather than
worked around. An override can only raise a field, since nib cannot tell "set
to the zero value" from "not set", so --yolo can turn approval off but nothing
on the command line turns it back on over an approval_mode: auto in the file.
And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and
still outrank these, deliberately, upstream.

The existing spec pinned that the right values reach app.Options, which they
always did, which is exactly why it could not see nib discarding them. The new
specs resolve the config the way app.Run resolves it, against a real config
file that disagrees with every flag, and one asserts Defaults stays empty.

docs/content/features/terminal-agent.md already documented --model as winning
over the saved model; that was false before this change and is true now, so no
docs edit was needed.

Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(chat): document intentional config file read

Assisted-by: Codex:gpt-5 [gosec]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
2026-08-02 09:23:26 +02:00

586 lines
41 KiB
HTML
Raw Blame History

{{ define "main" }}
<main id="top">
<!-- HERO -->
<section class="hero navy">
<div class="shell">
<div class="hero__grid">
<div>
<p class="kicker fd" style="margin-top:0">Open source · MIT · v4.8.0</p>
<h1><u><b>Make AI run on</b></u><u><b><s>every machine.</s></b></u></h1>
<div class="bars" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="lede fd mt2">Text, voice, vision, images, video, 3D and agents, from one open runtime. It works on the laptop you already own, and scales to a room full of GPUs when you have one. When the engine we need is too heavy, too closed, or does not exist, we write it.</p>
<div class="acts fd">
<a class="btn" href="#start">Install LocalAI <span>→</span></a>
<a class="btn btn--o" href="/docs/">Read the docs</a>
<a class="btn btn--o" href="{{ .Site.Params.github }}">&#9733; Star on GitHub</a>
</div>
<div class="figures fd">
<div><b class="tnum" data-count="{{ .Site.Data.stats.stars }}">0</b><span>GitHub stars</span></div>
<div><b class="tnum" data-count="73">0</b><span>Backends</span></div>
<div><b class="tnum" data-count="{{ len .Site.Data.engines.engines }}">0</b><span>Engines we wrote</span></div>
<div><b class="tnum" data-count="1585">0</b><span>Models, one click</span></div>
</div>
</div>
<div class="fd">
<figure class="screen" style="margin:0">
<figcaption class="screen__bar"><i></i> localai · chat <b>CPU only, no GPU</b></figcaption>
<video src="/media/hero-ui.mp4" poster="/img/hero-poster.jpg" autoplay muted loop playsinline preload="metadata" aria-label="The LocalAI interface running a chat completion on CPU"></video>
</figure>
</div>
</div>
</div>
</section>
<!-- LOCALAI -->
<section class="navy" id="localai">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">The runtime</p>
<h2 class="rv mt1" style="max-width:21ch">LocalAI is the engine everything else plugs into.</h2>
<p class="lede rv mt2">One binary with an OpenAI-compatible API in front of it. Point an existing client at it and the calls keep working, except now the model is on your machine. It also speaks the Anthropic, Ollama and ElevenLabs APIs, so most tools need a URL change and nothing else.</p>
<p class="lede rv mt2">Underneath, a small core pulls each engine in as a separate backend, only when a model asks for it. That is why one install covers this much ground without becoming a 9 GB download.</p>
<div class="apis rv">
<span>OpenAI API</span><span>Anthropic API</span><span>Ollama API</span><span>ElevenLabs API</span><span>Realtime over WebRTC</span>
</div>
<div class="duo mt3">
<div>
<div class="lanes rv">
<a class="lane" href="/docs/features/text-generation/"><span class="lane__k">Reason</span><span class="lane__d">Language models, tool calling, structured output</span><span class="lane__t">llama.cpp · vLLM · MLX</span></a>
<a class="lane" href="/docs/features/openai-realtime/"><span class="lane__k">Listen</span><span class="lane__d">Realtime voice, transcription, diarization</span><span class="lane__t">parakeet · whisper</span></a>
<a class="lane" href="/docs/features/text-to-audio/"><span class="lane__k">Speak</span><span class="lane__d">Speech synthesis and voice cloning</span><span class="lane__t">moss-tts · piper</span></a>
<a class="lane" href="/docs/features/object-detection/"><span class="lane__k">See</span><span class="lane__d">Vision, detection, recognition, depth, 3D</span><span class="lane__t">rf-detr · depth-anything</span></a>
<a class="lane" href="/docs/features/image-generation/"><span class="lane__k">Create</span><span class="lane__d">Images, video, music and sound</span><span class="lane__t">diffusers · ace-step</span></a>
<a class="lane" href="/docs/features/agents/"><span class="lane__k">Act</span><span class="lane__d">Agents, MCP, skills, RAG, interactive tools</span><span class="lane__t">agents · MCP apps</span></a>
</div>
<p class="rv mt2"><a class="btn btn--o" href="/docs/">Read the documentation →</a></p>
</div>
<div class="duo__m rv">
<figure class="screen" style="margin:0">
<figcaption class="screen__bar"><i></i> localai · model gallery <b>1,585 models</b></figcaption>
<video src="/media/gallery.mp4" muted loop playsinline preload="none" data-lazy aria-label="Installing a model from the LocalAI gallery"></video>
</figure>
</div>
</div>
</div>
</section>
<!-- MISSION -->
<section class="navy" id="mission">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Why the project exists</p>
<h2 class="rv mt1" style="max-width:19ch">Local AI should not need a datacenter.</h2>
<div class="mission">
<div class="mi rv">
<p class="mi__n">01 / HARDWARE</p>
<h3>Every feature ships a CPU path first.</h3>
<p>Not a degraded mode that technically runs. The real one, tested in CI, on the hardware most people already have. GPUs make it faster, they are not the price of entry.</p>
<p class="mi__meta">x86_64 · ARM64 · CUDA · ROCm · SYCL · Metal · Vulkan</p>
</div>
<div class="mi rv">
<p class="mi__n">02 / REALTIME</p>
<h3>You can talk to it, and it answers.</h3>
<p>Speech in, tool calls in the middle, speech out over WebRTC, fast enough to feel like a conversation. Transcription, diarization and speech synthesis all run without a GPU.</p>
<p class="mi__meta">Realtime API · WebRTC · streaming ASR · TTS · VAD</p>
</div>
<div class="mi rv">
<p class="mi__n">03 / DISTRIBUTED</p>
<h3>Plug in a second machine and stop there.</h3>
<p>Routing, VRAM-aware placement, prefix-cache affinity and failover are the runtime's problem. You add hardware, the cluster works out what to do with it.</p>
<p class="mi__meta">Smart routing · autoscaling · P2P · NATS · federation</p>
</div>
</div>
</div>
</section>
<!-- SENSES -->
<section class="navy" id="senses">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Senses</p>
<h2 class="rv mt1" style="max-width:20ch">Give the model eyes and ears.</h2>
<p class="lede rv mt2">A chat model can only work with what somebody types at it. Our engines change what it has access to: what is happening in the room, who walked into it, where things are in space, and how to answer out loud. All of it runs locally, most of it on a CPU.</p>
<div class="senses">
<div class="sn rv"><p class="sn__v">Hear <u>words</u></p>
<p>Live transcription with speaker labels and timestamps, fast enough to keep up with a meeting while it is still happening.</p>
<div class="sn__e"><span>parakeet.cpp</span><span>moss-transcribe.cpp</span></div></div>
<div class="sn rv"><p class="sn__v">Hear <u>the room</u></p>
<p>527 kinds of sound event: a door, a dog, breaking glass, a smoke alarm. The model notices things nobody thought to type.</p>
<div class="sn__e"><span>ced.cpp</span></div></div>
<div class="sn rv"><p class="sn__v">Know <u>who</u></p>
<p>Recognise a voice, recognise a face, and tell a live person from a photo held up to the camera.</p>
<div class="sn__e"><span>voice-detect.cpp</span><span>face-detect.cpp</span></div></div>
<div class="sn rv"><p class="sn__v">See <u>things</u></p>
<p>Ask for "the red mug on the left" in plain language and get back coordinates, not a caption.</p>
<div class="sn__e"><span>locate-anything.cpp</span><span>rf-detr.cpp</span></div></div>
<div class="sn rv"><p class="sn__v">See <u>space</u></p>
<p>Distance in metres from one ordinary photo, and a full 3D reconstruction from a handful of them. No rig, no camera poses, no GPU.</p>
<div class="sn__e"><span>depth-anything.cpp</span><span>free-splatter.cpp</span><span>trellis2.cpp</span></div></div>
<div class="sn rv"><p class="sn__v">Speak</p>
<p>Long-form speech in a cloned voice, across dozens of languages, up to 48 kHz.</p>
<div class="sn__e"><span>moss-tts.cpp</span><span>magpie-tts.cpp</span><span>vibevoice.cpp</span><span>voxtral-tts.c</span></div></div>
<div class="sn rv"><p class="sn__v">Hear <u>clearly</u></p>
<p>Echo cancellation, noise suppression and dereverberation, so a voice loop survives a real room with a real speaker in it.</p>
<div class="sn__e"><span>LocalVQE</span></div></div>
<div class="sn rv"><p class="sn__v">Forget <u>on purpose</u></p>
<p>Names, addresses and card numbers get caught and redacted on the machine, before anything is sent anywhere.</p>
<div class="sn__e"><span>privacy-filter.cpp</span></div></div>
</div>
<p class="lede rv mt3" style="max-width:74ch;color:var(--ink)">One session can do all of it at once: hear the room, work out who is talking, read what is on the desk, call a tool, and answer out loud. One API, one machine, nothing leaves the building.</p>
</div>
</section>
<!-- MADE WITH -->
<section class="navy" id="made">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Made with LocalAI</p>
<h2 class="rv mt1" style="max-width:23ch">Nobody was at the keyboard.</h2>
<div class="made">
<div class="rv">
<div class="player" id="player">
<span class="player__tag">Sound on</span>
<video id="pv" src="/media/presenter.mp4" poster="/img/presenter-poster.jpg" playsinline preload="none" aria-label="A generated presenter explaining voice cloning, produced entirely with LocalAI"></video>
<div class="player__ui"><span class="player__btn"><i></i> Play with sound</span></div>
</div>
</div>
<div>
<p class="lede rv">Every part of this clip came out of LocalAI, and the agent in the binary drove the machine that made it. It opened the app, ran the demo and captured the screen, while a local model wrote the script, a cloned voice read it, and the video endpoint generated and lip-synced the presenter. No human touched the keyboard, and nothing left the building.</p>
<div class="recipe rv">
<div><b>Direction</b><span>The agent in LocalAI drove the machine end to end</span></div>
<div><b>Script</b><span>Written by a local language model</span></div>
<div><b>Voice</b><span>Cloned from a few seconds of reference audio</span></div>
<div><b>Presenter</b><span>Generated and lip-synced through the video endpoint</span></div>
</div>
<p class="rv mt2"><a class="btn btn--o" href="/docs/features/text-to-audio/">How the voice pipeline works &#8594;</a></p>
</div>
</div>
</div>
</section>
<!-- ENGINES -->
<section class="navy" id="engines">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Engines we build</p>
<h2 class="rv mt1" style="max-width:22ch">{{ len .Site.Data.engines.engines }} engines, written from scratch.</h2>
{{/* Names the link back to the runtime section explicitly. Readers were
arriving here and taking these for unrelated side projects, because
nothing on the page said they are the backends the core loads. */}}
<p class="lede rv mt2">Those backends the core pulls in on demand are mostly ours. Most projects wrap somebody else's engine. We wrote ours, because the thing we needed came as a 9 GB Python install, or was closed, or did not exist. Each one is a binary and a GGUF file, checked against the reference implementation in CI.</p>
<div class="spot rv">
<div>
<h3>parakeet.cpp</h3>
<p class="spot__h">Twenty-seven times faster than whisper.cpp, on a CPU.</p>
<p>NVIDIA NeMo Parakeet, ported to C++ and ggml. Ten checkpoints, all of them verified at WER 0 against NeMo, which means the transcript comes out byte for byte identical while finishing first. Cache-aware streaming with end-of-utterance detection handles live audio, and the multilingual streaming model covers 40 or more locales.</p>
<div class="facts">
<div><b>27x</b><span>vs whisper.cpp, CPU</span></div>
<div><b>1.40x</b><span>vs NeMo, CPU median</span></div>
<div><b>WER 0</b><span>Parity with NeMo</span></div>
<div><b>37%</b><span>Size at q8_0</span></div>
</div>
<p class="mt2"><a class="btn btn--o" href="https://github.com/mudler/parakeet.cpp">parakeet.cpp on GitHub ↗</a></p>
</div>
<div class="spot__m">
<figure><video src="/media/parakeet-duel.mp4" muted loop playsinline preload="none" data-lazy aria-label="parakeet.cpp finishing ahead of NeMo on the same audio"></video>
<figcaption>Same audio, same words, ours finishes first</figcaption></figure>
<figure><img src="/img/parakeet-speedup.jpg" alt="Bar chart of parakeet.cpp CPU speedup over NeMo per dtype" loading="lazy">
<figcaption>CPU speedup over NeMo, by dtype</figcaption></figure>
</div>
</div>
<div class="spot spot--r rv">
<div>
<h3>depth-anything.cpp</h3>
<p class="spot__h">Beats PyTorch on CPU, in half the memory.</p>
<p>Depth Anything 3 as a 99 MB file. It gives you metric depth, per-pixel confidence, camera intrinsics and extrinsics, and a back-projected point cloud you can export to glb or COLMAP. Output correlates 1.0 with the reference implementation, component by component, and there is no Python or CUDA toolkit anywhere at inference.</p>
<div class="facts">
<div><b>1.31x</b><span>vs PyTorch, CPU</span></div>
<div><b>363 MB</b><span>Peak RAM, q8_0</span></div>
<div><b>6.7x</b><span>Faster to load</span></div>
<div><b>99 MB</b><span>Smallest build</span></div>
</div>
<p class="mt2"><a class="btn btn--o" href="https://github.com/mudler/depth-anything.cpp">depth-anything.cpp on GitHub ↗</a></p>
</div>
<div class="spot__m">
<figure><video src="/media/depth-race.mp4" muted loop playsinline preload="none" data-lazy aria-label="depth-anything.cpp finishing ahead of PyTorch on CPU"></video>
<figcaption>One photo in, distance in metres out, ahead of PyTorch on the same CPU</figcaption></figure>
</div>
</div>
<div class="wallhead rv">
<h3>And the rest of them, running.</h3>
<p>Every clip comes out of the benchmark suite that runs in CI on that engine. Where you see two panes, ours is racing the reference implementation on the same machine, on the same input.</p>
</div>
<div class="wall">
<a class="wi rv" href="https://github.com/mudler/parakeet.cpp"><video src="/media/parakeet-long.mp4" muted loop playsinline preload="none" data-lazy aria-label="parakeet.cpp racing NeMo on long audio"></video>
<div class="wi__c"><em>parakeet.cpp</em> long-form transcription <b>vs NeMo</b></div></a>
<a class="wi rv" href="https://github.com/localai-org/ced.cpp"><video src="/media/ced.mp4" muted loop playsinline preload="none" data-lazy aria-label="ced.cpp tagging sound events live"></video>
<div class="wi__c"><em>ced.cpp</em> sound events, live <b>527 classes</b></div></a>
<a class="wi rv" href="https://github.com/mudler/face-detect.cpp"><video src="/media/face.mp4" muted loop playsinline preload="none" data-lazy aria-label="face-detect.cpp detecting, landmarking and recognising faces"></video>
<div class="wi__c"><em>face-detect.cpp</em> detect, landmark, recognise <b>no Python</b></div></a>
<a class="wi rv" href="https://github.com/mudler/face-detect.cpp"><video src="/media/face-id.mp4" muted loop playsinline preload="none" data-lazy aria-label="face-detect.cpp finding the same person in a lineup"></video>
<div class="wi__c"><em>face-detect.cpp</em> same person, different photo <b>1 to N</b></div></a>
<a class="wi rv" href="https://github.com/localai-org/voice-detect.cpp"><video src="/media/voice.mp4" muted loop playsinline preload="none" data-lazy aria-label="voice-detect.cpp speaker recognition benchmark"></video>
<div class="wi__c"><em>voice-detect.cpp</em> who is speaking <b>vs reference</b></div></a>
<a class="wi rv" href="https://github.com/mudler/depth-anything.cpp"><video src="/media/depth.mp4" muted loop playsinline preload="none" data-lazy aria-label="depth-anything.cpp racing PyTorch on CPU"></video>
<div class="wi__c"><em>depth-anything.cpp</em> metric depth <b>vs PyTorch, CPU</b></div></a>
<a class="wi rv" href="https://github.com/mudler/locate-anything.cpp"><video src="/media/locate.mp4" muted loop playsinline preload="none" data-lazy aria-label="locate-anything.cpp open vocabulary detection race"></video>
<div class="wi__c"><em>locate-anything.cpp</em> say it, find it <b>open vocabulary</b></div></a>
<a class="wi rv" href="https://github.com/mudler/moss-tts.cpp"><video src="/media/moss.mp4" muted loop playsinline preload="none" data-lazy aria-label="moss-tts.cpp synthesis benchmark"></video>
<div class="wi__c"><em>moss-tts.cpp</em> 48 kHz voice cloning <b>vs reference</b></div></a>
<a class="wi rv" href="https://github.com/mudler/magpie-tts.cpp"><video src="/media/magpie.mp4" muted loop playsinline preload="none" data-lazy aria-label="magpie-tts.cpp synthesis benchmark"></video>
<div class="wi__c"><em>magpie-tts.cpp</em> 9 languages, 5 voices <b>vs reference</b></div></a>
</div>
<div class="reel rv" aria-hidden="true">
<div class="reel__t">
<span>vllm.cpp</span><span>parakeet.cpp</span><span>moss-transcribe.cpp</span><span>moss-tts.cpp</span><span>magpie-tts.cpp</span><span>ced.cpp</span><span>voice-detect.cpp</span><span>voxtral-tts.c</span><span>vibevoice.cpp</span><span>rf-detr.cpp</span><span>locate-anything.cpp</span><span>depth-anything.cpp</span><span>face-detect.cpp</span><span>free-splatter.cpp</span><span>trellis2.cpp</span><span>privacy-filter.cpp</span><span>LocalVQE</span><span>local-store</span><span>apex-quant</span>
<span>vllm.cpp</span><span>parakeet.cpp</span><span>moss-transcribe.cpp</span><span>moss-tts.cpp</span><span>magpie-tts.cpp</span><span>ced.cpp</span><span>voice-detect.cpp</span><span>voxtral-tts.c</span><span>vibevoice.cpp</span><span>rf-detr.cpp</span><span>locate-anything.cpp</span><span>depth-anything.cpp</span><span>face-detect.cpp</span><span>free-splatter.cpp</span><span>trellis2.cpp</span><span>privacy-filter.cpp</span><span>LocalVQE</span><span>local-store</span><span>apex-quant</span>
</div>
</div>
<div class="quiet rv">
<div>
<p class="quiet__tag">In development, not announced yet</p>
<h4>vllm.cpp</h4>
<p>vLLM ported to C++20, with paged attention, continuous batching and prefix caching, on CPU, CUDA, Metal and Vulkan. It installs as 66 MB instead of a 9.1 GB virtualenv, and it stays ahead of vLLM at every concurrency level we have measured so far. Still being finished, so treat the numbers as provisional.</p>
</div>
<video src="/media/vllm-race.mp4" muted loop playsinline preload="none" data-lazy aria-label="vllm.cpp ahead of vLLM at every concurrency level"></video>
</div>
<p class="rv mt2"><a class="btn btn--o" href="/engines/">All {{ len .Site.Data.engines.engines }} engines →</a></p>
</div>
</section>
<!-- APEX -->
<section class="paper" id="apex">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">APEX quantization</p>
<h2 class="rv mt1" style="max-width:19ch">The model you could not fit, on the card you already own.</h2>
{{/* APEX was being used as a known term on first appearance, in a section
that opened straight onto a benchmark table. The first two sentences
say what it is and why it follows the engines. */}}
<p class="lede rv mt2">The engine decides how fast a model runs. The weights decide whether it runs at all, so we build those too. APEX assigns a different precision to every tensor and every layer: a 35B mixture-of-experts model goes from 64.6 GB, out of reach of any consumer GPU, to 12.2 GB at 74 tokens a second. That is more than twice the speed of the original, and quality barely moves. The file is an ordinary GGUF, so stock llama.cpp opens it with no patches, and 201 of them are already sitting in the LocalAI gallery.</p>
<div class="sizes rv">
<div class="sz"><span>F16 · 64.6 GB</span><i style="width:100%"></i><u>30.4 t/s</u></div>
<div class="sz"><span>Q8_0 · 34.4 GB</span><i style="width:53%"></i><u>52.5 t/s</u></div>
<div class="sz"><span>APEX Quality · 21.3 GB</span><i style="width:33%"></i><u>62.3 t/s</u></div>
<div class="sz"><span>APEX Mini · 12.2 GB</span><i style="width:19%"></i><u>74.4 t/s</u></div>
</div>
<div class="tw rv">
<table>
<thead><tr><th>Build</th><th>Size</th><th>Perplexity</th><th>HellaSwag</th><th>MMLU</th><th>tg128 t/s</th></tr></thead>
<tbody>
<tr><td>F16</td><td>64.6 GB</td><td>6.537</td><td>82.5%</td><td>41.5%</td><td>30.4</td></tr>
<tr><td>Q8_0</td><td>34.4 GB</td><td>6.533</td><td>83.0%</td><td>41.2%</td><td>52.5</td></tr>
<tr><td>Unsloth UD-Q8_K_XL</td><td>45.3 GB</td><td>6.536</td><td>82.5%</td><td>41.3%</td><td>36.4</td></tr>
<tr data-a><td>APEX Quality</td><td>21.3 GB</td><td>6.527</td><td>83.0%</td><td>41.2%</td><td>62.3</td></tr>
<tr data-a><td>APEX I-Quality</td><td>21.3 GB</td><td>6.552</td><td>83.5%</td><td>41.4%</td><td>63.1</td></tr>
<tr data-a><td>APEX Compact</td><td>16.1 GB</td><td>6.783</td><td>82.5%</td><td>40.9%</td><td>69.8</td></tr>
<tr data-a><td>APEX Mini</td><td>12.2 GB</td><td>7.088</td><td>81.0%</td><td>41.3%</td><td>74.4</td></tr>
</tbody>
</table>
</div>
<p class="mono rv" style="margin-top:.9rem;font-size:.66rem;color:var(--p-dim)">Qwen3.5-35B-A3B on an NVIDIA DGX Spark (GB10). Perplexity on wikitext-2-raw at context 2048. Full methodology in the technical report.</p>
<div class="acts rv">
<a class="btn" href="https://huggingface.co/collections/mudler/apex-quants-gguf">APEX models on Hugging Face ↗</a>
<a class="btn btn--o" href="https://github.com/localai-org/apex-quant">Technical report ↗</a>
</div>
</div>
</section>
<!-- CLUSTER -->
<section class="navy" id="cluster">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Distributed</p>
<h2 class="rv mt1" style="max-width:17ch">Two machines behave like one.</h2>
<div class="scene">
<div>
<div class="st rv"><p class="st__k">01</p><h3>Start a worker</h3>
<p>One command on any box. It reports what hardware it has and which backends it can run, then joins the pool.</p></div>
<div class="st rv"><p class="st__k">02</p><h3>Work goes where it is cheapest</h3>
<p>Requests land on the replica that already holds the model and the matching prefix cache, sized against real free VRAM rather than a guess.</p></div>
<div class="st rv"><p class="st__k">03</p><h3>Losing a node is boring</h3>
<p>In-flight work reschedules, the model loads somewhere else, and the client never finds out.</p></div>
</div>
<div class="pin rv">
<div class="rig">
<div class="nodes" id="nodes">
<div class="nd"><b>node-01</b>24 GB · CUDA</div>
<div class="nd"><b>node-02</b>16 GB · ROCm</div>
<div class="nd"><b>node-03</b>CPU · 64 GB</div>
</div>
<div class="link"></div>
<div class="nodes"><div class="nd" style="grid-column:1/4"><b>router</b>prefix affinity · VRAM aware · autoscaling</div></div>
</div>
</div>
</div>
</div>
</section>
<!-- MODELS -->
<section class="navy" id="models">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">The gallery</p>
<h2 class="rv mt1" style="max-width:20ch">1,585 models. No notebook, no conversion script.</h2>
<div class="cards">
<a class="cd rv" href="/docs/getting-started/models/"><p class="cd__k">Quantizations</p><h3>201 APEX builds</h3>
<p>Every tier of every model we quantize, ranked against the hardware you actually have and installed with one click.</p><span class="cd__go">Browse the gallery →</span></a>
<a class="cd rv" href="#"><p class="cd__k">Speech</p><h3>italian-asr</h3>
<p>Italian speech recognition trained and published by the team, streaming on a CPU through parakeet.cpp.</p><span class="cd__go">Model card →</span></a>
<a class="cd rv" href="/docs/features/text-to-audio/"><p class="cd__k">Voices</p><h3>60 Piper voices</h3>
<p>Forty-two languages of text to speech, small enough to run on a Raspberry Pi, installed from the web interface.</p><span class="cd__go">Voice catalogue →</span></a>
</div>
</div>
</section>
<!-- NIB -->
<section class="deepbg" id="nib">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">the agent</p>
<div class="duo">
<div>
<h2 class="rv mt1" style="max-width:18ch">The agent is already in the binary.</h2>
<p class="lede rv mt2">Run <span class="mono" style="color:var(--cyan-hi)">local-ai chat</span> and you are talking to an agent that already knows where your models are. It runs shell commands behind an approval gate you control, delegates to sub-agents, and loads MCP servers, plugins and skills. Claude Code plugins load as they are.</p>
<p class="lede rv mt2">It is also nib, a single ~20 MB Go binary with no runtime and no daemon, so you can drop the same agent on any box you SSH into and press <span class="mono" style="color:var(--cyan-hi)">Ctrl+Space</span>.</p>
<div class="chips rv"><span>local-ai chat</span><span>MCP</span><span>plugins</span><span>skills</span><span>sub-agents</span><span>zero dependencies</span></div>
<div class="acts rv">
<a class="btn" href="/docs/features/terminal-agent/">Read the docs <span>→</span></a>
<a class="btn btn--o" href="https://github.com/mudler/nib">nib on GitHub ↗</a>
</div>
</div>
<div class="duo__m rv">
<figure class="screen" style="margin:0">
<figcaption class="screen__bar"><i></i> local-ai chat <b>the agent, in the binary</b></figcaption>
<video src="/media/nib.mp4" muted loop playsinline preload="none" data-lazy aria-label="The terminal agent built into LocalAI running a task"></video>
</figure>
</div>
</div>
</div>
</section>
<!-- PROOF -->
<section class="navy" id="proof">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Since March 2023</p>
<h2 class="rv mt1" style="max-width:22ch">Forty-eight thousand stars, and still shipping every week.</h2>
<div class="trend rv">
<img src="/img/trendshift.svg" alt="LocalAI on Trendshift">
<p>LocalAI has been <b>trending on GitHub</b> repeatedly since it launched, and it is one of the most starred self-hosted AI projects there is. <b>{{ lang.FormatNumberCustom 0 .Site.Data.stats.contributors }} people</b> have contributed code, <b>{{ lang.FormatNumberCustom 0 .Site.Data.stats.discord }}</b> are in the Discord, and the README is kept translated into <b>eight languages</b> because the users are everywhere.</p>
</div>
<div class="big rv">
<div><b class="tnum" data-count="{{ .Site.Data.stats.stars }}">0</b><span>Stars</span></div>
<div><b class="tnum" data-count="{{ .Site.Data.stats.forks }}">0</b><span>Forks</span></div>
<div><b class="tnum" data-count="{{ .Site.Data.stats.contributors }}">0</b><span>Contributors</span></div>
<div><b class="tnum" data-count="{{ .Site.Data.stats.releases }}">0</b><span>Releases</span></div>
<div><b class="tnum" data-count="{{ .Site.Data.stats.discord }}">0</b><span>In Discord</span></div>
<div><b class="tnum" data-count="0" data-text="3 yrs">0</b><span>Shipping since</span></div>
</div>
<div class="tl rv">
<div class="tl__t">
<div class="tl__i"><p class="tl__d">MAR 2023</p><h4>First commit</h4>
<p>An OpenAI-compatible API in front of llama.cpp, so a laptop could answer the same calls as the cloud.</p></div>
<div class="tl__i"><p class="tl__d">JUL 2025</p><h4>The core gets small</h4>
<p>Every backend moves out of the binary. You install only the engines your models need.</p></div>
<div class="tl__i"><p class="tl__d">MAR 2026</p><h4>Agents and a new interface</h4>
<p>Native agentic orchestration, a full React rewrite with canvas mode, WebRTC realtime audio.</p></div>
<div class="tl__i"><p class="tl__d">APR 2026</p><h4>It becomes a cluster</h4>
<p>Distributed mode with VRAM-aware routing, autoscaling, multi-user auth and per-user quotas.</p></div>
<div class="tl__i"><p class="tl__d">MAY 2026</p><h4>It sees and hears</h4>
<p>Voice recognition, face recognition with liveness, diarization, video generation, drop-in Ollama API.</p></div>
<div class="tl__i"><p class="tl__d">JUL 2026</p><h4>Nineteen engines of our own</h4>
<p>The native C and C++ ports take over the heavy Python backends, one modality at a time.</p></div>
</div>
</div>
<p class="kicker rv mt3">Where it runs</p>
<div class="marks rv">
<span>NVIDIA CUDA</span><span>AMD ROCm</span><span>Intel SYCL</span><span>Apple Metal</span><span>Vulkan</span>
<span>Jetson</span><span>Raspberry Pi</span><span>x86_64</span><span>ARM64</span><span>Kubernetes</span><span>Docker</span>
</div>
<div class="langs rv">
<span>Deutsch</span><span>Español</span><span>français</span><span>日本語</span>
<span>한국어</span><span>Português</span><span>Русский</span><span>中文</span>
</div>
{{/* The strongest sentence in this section is somebody else's, so it opens
the section rather than closing it. Everything below is supporting
evidence for what these three people already said. */}}
<p class="kicker rv mt3">What other people say</p>
<div class="headline">
<a class="hq rv" href="https://x.com/ggerganov/status/2065447087311917459">
<blockquote>&ldquo;Some cool ggml-based work by @mudler_it recently, make sure to check it out.&rdquo;</blockquote>
<p class="hq__w">@ggerganov<b>Georgi Gerganov, author of llama.cpp and ggml</b></p>
<span class="hq__go">On X, 2026 &#8599;</span></a>
<a class="hq rv" href="https://x.com/badlogicgames/status/2061201400059531729">
<blockquote>&ldquo;What a wonderful project: parakeet.cpp. A ggml based parakeet inference pipeline that is 2x faster than my ONNX parakeet pipeline on Apple Silicon.&rdquo;</blockquote>
<p class="hq__w">@badlogicgames<b>Mario Zechner, author of pi.agent</b></p>
<span class="hq__go">On X, 2026 &#8599;</span></a>
<a class="hq rv" href="https://www.linkedin.com/posts/adimargolink_spinoza-saw-all-things-straining-to-become-activity-7468135820076634113-u3Hc">
<blockquote>&ldquo;Build something good enough that the community chooses to carry it beyond your reach. This week, Ettore Di Giacinto brought NVIDIA Parakeet to the CPU.&rdquo;</blockquote>
<p class="hq__w">Adi Margolin<b>On LinkedIn</b></p>
<span class="hq__go">Read the post &#8599;</span></a>
</div>
{{/* Names run as a sentence rather than a grid of pills. A pill wall of
employers reads as a customer logo wall, which is a claim we are not
making; a sentence keeps it about the people, which is the true one. */}}
<p class="kicker rv mt3">Who shows up</p>
<h3 class="eco__h rv">{{ lang.FormatNumberCustom 0 .Site.Data.stats.contributors }} people have put code in this repository.</h3>
{{- $co := slice }}
{{- range .Site.Data.ecosystem.contributors.companies }}{{ $co = $co | append (printf "<b>%s</b>" .name) }}{{ end }}
{{- $ac := slice }}
{{- range .Site.Data.ecosystem.contributors.academia }}{{ $ac = $ac | append (printf "<b>%s</b>" .) }}{{ end }}
<p class="eco__names rv">Some of them do it from a desk at {{ delimit $co ", " " and " | safeHTML }}.
Others from labs at {{ delimit $ac ", " " and " | safeHTML }}.</p>
<p class="eco__note rv">That is where they work, not who sent them.
<a href="{{ .Site.Params.github }}/graphs/contributors">Check out the full contributor list, 200+ and growing &#8599;</a></p>
{{/* The list runs as a marquee because the count is the argument: any one
of these is a weak signal, and the whole moving line is the strong one.
It pauses on hover so the links stay usable. */}}
<h3 class="eco__h rv mt3">And many projects integrate it</h3>
<p class="eco__note rv">Click any name to see where they say so.</p>
<div class="reel reel--int rv">
<div class="reel__t">
{{- range .Site.Data.ecosystem.integrations.projects }}
<a href="{{ .url }}">{{ .name }}</a>
{{- end }}
{{- range .Site.Data.ecosystem.integrations.projects }}
<a href="{{ .url }}" tabindex="-1" aria-hidden="true">{{ .name }}</a>
{{- end }}
</div>
</div>
<p class="eco__aside rv">Using LocalAI at work? Add your organisation to
<a href="{{ .Site.Params.github }}/blob/master/ADOPTERS.md">ADOPTERS.md</a>.</p>
<h3 class="eco__h rv mt3">Written about elsewhere</h3>
<div class="eco eco--press rv">
{{- range .Site.Data.ecosystem.press.articles }}
<a href="{{ .url }}"><b>{{ .outlet }}</b><span>{{ .title }}</span><em>{{ .what }}</em></a>
{{- end }}
</div>
<p class="kicker rv mt3">Built on, integrated with, written about</p>
<div class="posts">
<a class="post rv" href="https://x.com/sozercan/status/1769769695081546236">
<p class="post__h">@sozercan <em>builds on</em></p>
<p>AIKit now offers an extensible solution for finetuning LLMs! Thanks to @UnslothAI, you can finetune fast and efficiently. Then, deploy seamlessly with AIKit using @LocalAI_API for an end-to-end solution!</p>
<span class="post__go">On X, 2024 &#8599;</span></a>
<a class="post rv" href="https://x.com/ivanfioravanti/status/2062526685484851440">
<p class="post__h">@ivanfioravanti <em>benchmarks</em></p>
<p>An M5 Max with 40 GPU cores just beat an M3 Ultra with 80 on parakeet.cpp. Every model. ~1.7x faster on average, up to ~2x. Half the cores.</p>
<span class="post__go">On X, 2026 &#8599;</span></a>
<a class="post rv" href="https://x.com/PulumiCorp/status/1794038185061663083">
<p class="post__h">@PulumiCorp <em>integration</em></p>
<p>Explore how to build and deploy a LLM app using @FlowiseAI and @LocalAI_API with AWS EKS, Pulumi, and TypeScript! Run your models locally or on-prem.</p>
<span class="post__go">On X, 2024 &#8599;</span></a>
<a class="post rv" href="https://x.com/enricoros/status/1912401037794898354">
<p class="post__h">@enricoros <em>ecosystem</em></p>
<p>Congrats to @LocalAI_API for launching LocalAGI (Agents), and LocalRecall (Memory). The Local stack is well designed and expanding.</p>
<span class="post__go">On X, 2025 &#8599;</span></a>
<a class="post rv" href="https://x.com/UniverseAdam/status/1779854715519459432">
<p class="post__h">@UniverseAdam <em>in print</em></p>
<p>My hardcopies of the official @Raspberry_Pi magazine @TheMagPi have arrived! And they have my Automatic Speech Recognition project based around @NordVPN&#x27;s Meshnet and a self-hosted @LocalAI_API language model inside.</p>
<span class="post__go">On X, 2024 &#8599;</span></a>
<a class="post rv" href="https://x.com/ivanfioravanti/status/2038141571678212580">
<p class="post__h">@ivanfioravanti <em>community</em></p>
<p>LocalAI is becoming stronger and better release, after release! Keep pushing @mudler_it and @LocalAI_API</p>
<span class="post__go">On X, 2026 &#8599;</span></a>
<a class="post rv" href="https://x.com/alepiad/status/1654502947697442816">
<p class="post__h">@alepiad <em>early days</em></p>
<p>I&#x27;m really excited about the prospect of open-source LLMs. In that respect, take a look at @LocalAI_API, a drop-in replacement for OpenAI&#x27;s API but serving GGML models right on your own infrastructure.</p>
<span class="post__go">On X, 2023 &#8599;</span></a>
<a class="post rv" href="https://x.com/mattapperson/status/1727390465543041423">
<p class="post__h">@mattapperson <em>builds on</em></p>
<p>Oh, high there @LocalAI_API, nice to see a terminal based UI for ya! (it&#x27;s a WIP, but just wanted something cleaner then CURL calls)</p>
<span class="post__go">On X, 2023 &#8599;</span></a>
</div>
</div>
</section>
<!-- BLOG -->
<section class="deepbg" id="blog">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">From the team</p>
<h2 class="rv mt1" style="max-width:22ch">We publish the numbers, including the ones that cost us.</h2>
<p class="lede rv mt2">APEX takes a 35B model from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second. Perplexity goes from 6.537 to 7.088. That trade is in the post too, with the command that produced it.</p>
{{/* Pulled from the posts themselves. The cards used to be hand-written,
which is how one of them ended up advertising a post that did not
exist, and how all three linked to the index instead of the article. */}}
<div class="cards">
{{- range first 3 (where .Site.RegularPages "Section" "blog") }}
<a class="cd rv" href="{{ .RelPermalink }}">
<p class="cd__k">{{ .Date.Format "2 January 2006" }}</p><h3>{{ .Title }}</h3>
<p>{{ .Params.summary | truncate 155 }}</p><span class="cd__go">Read the post →</span></a>
{{- end }}
</div>
</div>
</section>
<!-- START -->
<section class="navy" id="start">
<div class="shell">
<div class="bars rv" aria-hidden="true"><i></i><i></i><i></i><i></i></div>
<p class="kicker rv">Get started</p>
<div class="duo">
<div>
<h2 class="rv mt1" style="max-width:14ch">Running in about a minute.</h2>
<p class="lede rv mt2">A container on any platform, a DMG on macOS, a binary on Linux, or a chart on Kubernetes. Backends download themselves the first time a model asks for one, so the base install stays small.</p>
<div class="acts rv">
<a class="btn" href="/docs/getting-started/">Installation guide →</a>
<a class="btn btn--o" href="https://discord.gg/uJAeKSAGDy">Join the Discord</a>
</div>
</div>
<div class="rv">
<div class="tabs" role="tablist" aria-label="Installation method">
<button class="tab" type="button" role="tab" aria-selected="true" data-pane="p-script">Script</button>
<button class="tab" type="button" role="tab" aria-selected="false" data-pane="p-docker">Docker</button>
<button class="tab" type="button" role="tab" aria-selected="false" data-pane="p-k8s">Kubernetes</button>
</div>
<div class="term">
<div class="term__b"><span id="term-label">bash</span> <button class="cpy" type="button" id="cpy" data-copy="curl -sSL https://localai.io/install.sh | sh">copy</button></div>
<div class="pane" id="p-script">
<pre><span class="p">$</span> <span class="c">curl -sSL https://localai.io/install.sh | sh</span>
<span class="o">detecting platform</span> <span class="h">linux/amd64</span>
<span class="o">fetching latest release binary</span>
<span class="o">installed to</span> <span class="h">/usr/local/bin/local-ai</span>
<span class="p">$</span> <span class="c">local-ai run qwen3.5-35b-a3b-apex</span>
<span class="o">pulling backend llama-cpp</span>
<span class="o">API ready on</span> <span class="h">http://localhost:8080</span>
<span class="o">model ready ·</span> <span class="h">74.4 tok/s</span></pre>
</div>
<div class="pane" id="p-docker" hidden>
<pre><span class="p">$</span> <span class="c">docker run -p 8080:8080 --name local-ai -ti localai/localai:latest</span>
<span class="o">starting local-ai</span>
<span class="o">detected: CPU (AVX-512), 32 GB RAM</span>
<span class="o">API ready on</span> <span class="h">http://localhost:8080</span>
<span class="o"># Podman works the same way</span>
<span class="p">$</span> <span class="c">podman run -p 8080:8080 --name local-ai -ti localai/localai:latest</span></pre>
</div>
<div class="pane" id="p-k8s" hidden>
<pre><span class="p">$</span> <span class="c">kubectl apply -f https://localai.io/install/kubernetes.yaml</span>
<span class="o">deployment.apps/local-ai created</span>
<span class="o">service/local-ai created</span>
<span class="o"># or with Helm</span>
<span class="p">$</span> <span class="c">helm install local-ai go-skynet/local-ai</span></pre>
</div>
</div>
<div class="alts"><span>macOS: DMG</span><span>Linux: binary</span><span>Podman</span><span>Helm chart</span><span>Build from source</span></div>
</div>
</div>
</div>
</section>
</main>
{{ end }}
Reference in New Issue View Git Blame Copy Permalink