mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
8b01ac2d4e8880099b17013eaed88a4773d20981
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler's LocalAI [bot]andEttore Di Giacinto
|
1271b97a46 |
docs(blog): cover the terminal agent in the 4.8 post (#11372)
docs(blog): cover the terminal agent, and fix the counts in the intro The 4.8 post never mentions that `local-ai chat` stopped being a REPL and became an agent (#11291): the nib harness compiled into the binary, with tool use behind an approval gate, sub-agents, MCP servers, plugins and skills, auto-configured against the local instance. It also ships a shell integration script for zsh, bash and fish that binds Ctrl+Space. That is one of the larger user-facing changes in the release and it was missing from both the post and the release-notes highlights. Added a section after 3D generation, including the breaking changes for anyone who had habits around the old REPL: `/clear` is gone in favour of `/compact`, and a model switch now keeps the conversation. While in the intro, corrected the counts. The post said 374 pull requests in twenty-one days, which was accurate when it was drafted on the 4th but not once v4.8.0 was tagged on the 5th. The published release notes say 386 in twenty-two days, and the intro now matches them rather than contradicting them. For the record, neither figure is exactly right: `git log --format=%s v4.7.1..v4.8.0 | grep -cE '\(#[0-9]+\)$'` counts 388 squash-merged pull requests, and 389 from v4.7.0. The notes were cut before the last few landed. Matching the published notes was the priority here, since that is the artifact everyone else quotes, and 386 is the number already in circulation. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
c61b6f2286 |
docs(blog): new DeepSeek and Laguna numbers, visuals, humanizer pass (#11369)
* docs(blog): new DeepSeek and Laguna numbers, visuals, humanizer pass vllm.cpp master moved 26 commits past what the post was written against, and two results changed enough to matter. Both came from the same lever: staging weights device-resident at load instead of reading them from the GGUF mmap over unified memory, which the GB10 reads about 20% slower per GEMV than device memory. - DeepSeek-V4-Flash against DwarfStar: 0.997x parity becomes 1.144x ahead, 18.69 vs 16.33 tok/s decode, same generated tokens. - Laguna-XS-2.1 against vLLM: 87% becomes 1.03x, 44.46 vs 43.10 tok/s. New row in the scoreboard. Adds three visuals. A chart of throughput against every reference engine, which is worth having now that the spread is 0.976 to 1.144 rather than a flat line at parity. The Activity page with four installs running, and the model detail pane with all four pocket-35b variants. Both screenshots were recaptured on 2026-08-04 because #11288, #11305, #11307 and #11222 had all changed those pages since the earlier set. llama.cpp is deliberately absent from the chart: its 1.18x is a prefill ratio, and putting it on the same axis as throughput ratios would be comparing two different measurements. Also carries the media the release notes embed, since a GitHub release body needs URLs that survive publishing and drag-and-drop has no CLI. Supersedes #11364. Humanizer pass on the prose. The post had collected five exactness idioms in one section (token-for-token, byte-exact twice, byte-identical, token-identical). One is precision, five is a tic, so the 27B row keeps its "token-for-token identical" where identical output is the actual claim and the rest say what they mean. That also fixed a hyphen in predicate position ("is token-identical"). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): redraw the benchmark chart as a branded card The Flint bar chart was generic: default palette, no brand, and drawn from zero, which made five ratios between 0.976 and 1.144 look like five bars of roughly equal length. Redrawn in the style of recorder-for-agents' render-card.sh cards, the same shape as the vllm.cpp README GIF. Palette taken from the two logos rather than invented (LocalAI navy #0E2632 and teal #469AAF, vllm.cpp teal #3AB4CA), SVG generated by a small JS loop so the geometry is exact at any scale, headless Chrome to PNG at 2x. The substantive change is that bars now run from the 1.00 parity line instead of from zero. Deviation is what the data is about, so DeepSeek's +14.4% and MLX-LM's -2.4% are both legible, and the one row that is behind is the one row in amber. Each bar carries its ratio and the raw measurement under it. Keeps the .html source next to the .png so the chart is editable later: change a number, re-run render-card.sh. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
4fec33966a |
docs(blog): final figures for the 4.8 post, and the MLX provider (#11362)
* docs(blog): final figures for the 4.8 post, and the MLX provider The cycle closed at 374 PRs over twenty-one days, not the 321 over eighteen the post was written against. Corrects the summary, the opening line, the contributor count and the gallery total, and moves the date to the day the release is cut. Adds the MLX GEMM provider (#11137), which merged after the post was written and is the one number an Apple Silicon reader wants: 1.54x to 2.19x on an M4 with time to first token roughly halving, both arms toggled on one binary. The +/-10% caveat travels with the table rather than being left in the PR. Two lines edited against the no-ai-slop skill while I was in the file, the same pass #11324 ran over the engines post: - The opener balanced two clauses across a colon and closed on "without lying to you", which is the built-to-be-quoted shape readers picked out of the HN thread. It is a flat statement now. - "This is a new modality rather than a new backend under an existing one" is a binary contrast that says nothing the next clause does not. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): call vllm.cpp alpha, and finish the no-ai-slop pass vllm.cpp is not a released backend and the post read like it was. The old wording buried the caveat in a block quote at the end of the section and still said "first release of a young engine". It now says plainly, before the caveat can be skipped, that these are alpha development builds, that shipping them in 4.8 is about letting people try the thing rather than recommending it, and that llama-cpp stays the default. Also completes the no-ai-slop pass I had only half run. Counting the lines built to be quoted, headings and section endings included, the post is in reasonable shape: long flat informational stretches, tables followed by a plain finding, headings that are labels rather than epigram-verdicts. Three patterns survived, each one an item in eval.md: - "and inverts that:" set the usual shape against ours across a colon. The sentence works without the frame. - "Two things were conflated there: a signal, which needs one line, and the detail, which needs somewhere to put it" is a role-assignment pair. Says what happens instead. - "The maturity statement from the release notes is worth repeating in full" is throat-clearing in front of a quote, and the quote is gone. Left the rest alone. Minimum effective edit, not a rewrite. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): present vllm.cpp as a community project, with its own numbers The post described vllm.cpp as "a from-scratch port of vLLM, written and maintained by the LocalAI team". Two things wrong with that. It is a community project, and it has stopped being only a port: it loads GGUF, runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload, and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because those are the engines it competes with on that hardware. vLLM's role is now stated for what it is, the reference implementation. Correctness is checked against it and the scoreboard is kept against it. Also flags that the name will probably change, since it is drifting far enough that vllm.cpp will eventually mislead. Adds real numbers from the project's own docs/BENCHMARKS.md rather than adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the 35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution travels with them: it treats c2 through c32 as ties because its noise band is 0.5% and those margins are 0.7% to 1.7%. Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md rather than restated from memory. The heading is marked alpha to match the section body. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): say who maintains vllm.cpp, and add the DeepSeek Flash result Two corrections to the previous commit. "A community project" says nothing and was not quite true either. The LocalAI team maintains vllm.cpp. Community-first is the intent, not a description, so it now says that and says what backs it: its own repository, its own docs, benchmark record and issue tracker, and it runs without LocalAI anywhere in the picture. Adds the DeepSeek-V4-Flash result, which makes the divergence point better than any of the prose around it. That model does not run on vLLM on a single GB10: every vLLM-loadable checkpoint is 156 GB or more against a 119 GiB unified pool, and the only quant that fits is an extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP speculative decoding, token-identical to vLLM's and about 4% faster at concurrency 1. Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): lead the DeepSeek result with what we run, not with what vLLM cannot The previous version opened on "that model does not run on vLLM on a single GB10 at all". Wrong emphasis twice over: it makes a strong negative claim about another project the headline, and it buries the actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at 16.28 tok/s against DwarfStar's 16.33. The size constraint is still there, stated as the reason the quant is what it is rather than as a point about vLLM: at 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits the Spark's 119 GiB unified pool. The table row now names the quant and the box (IQ2_XXS, one DGX Spark) instead of just "GGUF, GB10", since that is the part a reader with a Spark wants. Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] * docs(blog): say the new name is undecided "The name will probably change at some point" invited the obvious question. It now says the rename is expected and the name is still to be decided, which is the actual state. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
2e14511fe2 |
docs(blog): add release write-ups for 3.10 through 4.3 (#11330)
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.
Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
|
6e7c0a4df8 |
blog, website: edit out the AI writing tells readers called out on HN (#11324)
* blog: rewrite the engines post without the AI tells
The HN thread on this post (item 49125065) spent most of its comments on the
writing rather than the engines. Readers quoted specific lines back as tells.
This is the same post with the same numbers, edited against the updated
no-ai-slop skill.
Every figure, table and link is unchanged, except that "27% of the memory"
is now the underlying 363 MB against 1328 MB from the table.
Two substantive framing fixes, both from the reply draft in
hn-reply-engines-post.md:
- vllm.cpp is no longer implied to be a speed win. The table is a tie, the
result is the install size, and the post now says so before a reader has to
work it out and post about it.
- Added one line on the language mix. Readers took the C++/Python/Go tree as
incoherence rather than as a Go core with per-ecosystem backends.
Cut throughout: the ledger metaphor ("what those ports buy", "not paid for in
throughput"), unearned framing ("the honest reading is", "has nothing to do
with"), the shape summary ("that is the general shape of these wins"),
confident deference ("people who are better at those models than we are"),
self-grading numbers ("a good result for a 66 MiB binary"), verbless
comparisons, three of the four exactness idioms, and the aphoristic headings
and verdicts. The double-tricolon summary is one plain clause now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* blog, website: same anti-slop sweep over the rest of the site
One-by-one pass over the other four posts and the site templates, with the
same rules used on the engines post. All figures, tables, links and PR
numbers are unchanged everywhere; the edits are to prose only.
apex-moe-quantization: ledger metaphors were the main issue, eight uses of
buy/cost/pay/spend for things that are not money. Also "the honest reading
is", "that is the comparison that matters", and two section-ending aphorisms
("Size is a speed knob as much as a memory knob", "Q6_K is the ceiling worth
paying for").
localai-since-march-2023: light touch, this one already reads like a person.
Removed "the curve is not the point", a "not the feature list, but the four
decisions" contrast, and two "X is what made / is the piece that" forms.
parakeet-cpp-asr-on-cpu: six exactness idioms across one post, "byte for
byte" twice, "character for character" twice, "byte-identical" twice and
"bit-identical" once, including in the title. Down to one, kept where the
precision is load-bearing. Also the "what end-of-utterance detection buys
you" heading and the "we say so rather than averaging it away" flex.
what-landed-in-localai-4-8: no changes. It is dense, flat and ends every
section on a PR number or a plain fact, which is the shape the other posts
should look like.
Site templates: "Most backends wrap somebody else's engine. These do not."
was the same contrast the engines post opened with. Also "Not a degraded mode
that technically runs", "A port only ships once it matches the original",
"Speed is the part we then go and win ... not a marketing run", and the last
"byte for byte" on the landing page.
Hugo builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* website: it is eighteen engines, not nineteen
Three places said nineteen: the /engines/ page description, the JUL 2026
timeline entry on the landing page, and the header comment in
data/engines.yaml.
Eighteen is right, confirmed two ways. The "Backends built by us" table in
the README has exactly 18 rows, and data/engines.yaml has 19 entries of which
one is apex-quant, which is a quantization recipe rather than an engine. The
two lists otherwise match name for name.
The yaml comment is the likely origin: it read "the nineteen native engines
the LocalAI team wrote, and the one quantization recipe that feeds them",
which counts apex-quant twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
359bd4850d |
docs(blog): bring the 4.8 release post up to the final changelog (#11287)
The post was written against the first draft of the release notes, when the cycle stood at 214 PRs over thirteen days. It closed at 321 PRs over eighteen days, and three of the larger user-facing changes landed after it was written. - Correct the counts throughout: 321 PRs, eighteen days, 24 contributors (11 first-time), gallery 1,221 to 1,505. - Add sections for the three new capabilities: 3D generation as a modality (Generate3D, FLAG_3D, /v1/3d/generations, trellis2cpp), audio.cpp serving six audio endpoints from one process, and the operations bar becoming the Activity page. - Cover the two further hardening fixes (tar hardlink escape, cyclic $ref stack overflow) alongside the TRL one. - Note the Valkey store, systemd socket activation, persistent trace history, in-place chat edits, the self-contained SYCL backend and the site split. - Group the new-engine sections together rather than splitting them across the operational ones. Embeds the existing vllm-race and magpie clips, and adds a 3D generation clip cut from the demo recording to the conventions in .agents/preparing-a-release.md (no audio track, 14s, named for the feature). blog.css styled figure img but not figure video, so a clip in a post rendered outside the card; both selectors now share the rule. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
94d5affcea |
feat(website): split the site, move docs to /docs, add a landing page (#11243)
* feat(website): split the site, move docs to /docs, add a landing page The Hugo docs site has always been localai.io itself, which left nowhere to explain what LocalAI is or show what the team builds. This adds a separate marketing site at the root and moves the documentation under /docs/. Docs: The existing site keeps its content tree and its Relearn theme, and now builds with baseURL <root>/docs/. Its _index.md, which held a hand written landing page, becomes a real documentation home. Every previously published URL keeps working. GitHub Pages has no server side rewrites, so .github/ci/gen-redirects.sh walks the built docs output and leaves a meta refresh plus a canonical link at each old root path. It covers bare .html files too, which is what keeps /gallery.html alive, and it never overwrites a path the marketing site already owns. Website: A second Hugo site under website/ with its own layouts and no external theme, so the marketing side does not have to fight Relearn's home rooted menu and asset pipeline. CI builds both and merges them into one Pages artifact. The design is derived from the project logo rather than invented: the navy of the triangle, the cyan of the llama, the purple of the speed bars. Those offset bars became the motion signature. The background renders a real depth-anything.cpp depth map as contour lines and switches to a locate-anything.cpp style detection overlay over the engines section. Also included: an /engines/ index driven entirely by data/engines.yaml, a /blog/ section with five posts written from the release notes and the engine benchmark suites, install.sh and a Kubernetes manifest since the site advertises both, and a rule in .agents/ that release preparation now includes a blog post and demo clips. Every figure on the site is derived from the repository or the GitHub API, not from memory. Correcting them against their sources found one error in README.md: voxtral-tts.c is text to speech, not speech to text. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent] * feat(website): add a star history chart, rewrite the history post in first person The history post read like a changelog written by a committee. It is now in Ettore's voice, first person, with the admissions left in. The numbers paragraph in particular read like a directory listing. It now says what the figures mean rather than which file they came from. Adds an interactive star history chart, built from the GitHub stargazers API rather than embedded from a third party, so the page makes no external request and cannot break when someone else's service is down. The four releases the post is organised around are marked on the curve, and the labels stack into rows because three of them land within two months of each other. The API stops paginating at 40,000 items, so the curve is measured up to December 2025 and the segment from there to today's total is drawn dashed, labelled as an estimate in the caption and in the tooltip. It is a straight line between two known points, and the chart says so rather than implying it is data. Also drops "marketing site" from the README heading and everywhere else it appeared, and calls it the main site instead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |