diff --git a/website/content/blog/what-landed-in-localai-4-8.md b/website/content/blog/what-landed-in-localai-4-8.md index 9a2a5fe35..41d98818f 100644 --- a/website/content/blog/what-landed-in-localai-4-8.md +++ b/website/content/blog/what-landed-in-localai-4-8.md @@ -36,6 +36,11 @@ The third one was `/api/traces` returning a 21 MB unpaginated blob that the UI p ## One gallery entry, several builds +
+The model detail pane listing every variant +
One entry, four builds. LocalAI picks the largest that fits and marks it auto-selected.
+
+ Installing a model no longer means reading a list of quantizations and guessing which one your card will hold. A gallery entry can now declare `variants:`, a list of references to other entries that are alternative builds of the same weights: ```yaml @@ -63,6 +68,11 @@ It has grown features vLLM does not have, which is most of the reason the port e Tool calling is at llama.cpp parity by construction, because chat deliberately reuses the same autoparser path: full minja chat templates, `tool_choice: auto` lowered to a lazy structural-tag decode constraint, 30 tool dialects, 7 reasoning parsers, and streamed `ChatDelta` and `ToolCallDelta`. +
+Throughput of vllm.cpp relative to each reference engine, drawn as deviation from parity +
llama.cpp is left out because its 1.18x is a prefill ratio, and putting that on the same axis as throughput would compare two different measurements.
+
+ Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/blob/master/docs/BENCHMARKS.md), which calls ties ties and losses losses. Above 1.0 means vllm.cpp is ahead:
@@ -73,16 +83,19 @@ Numbers from the project's own [scoreboard](https://github.com/mudler/vllm.cpp/b vLLMQwen3.6-35B-A3B NVFP4, GB101.010x at c16 and 1.013x at c32, behind from c1 to c8 (0.817x at c1) llama.cppQwen3.5-2B GGUF, CPU aarch64prefill 1.18x, decode a tie, memory parity MLX-LMQwen3-0.6B, Apple M497.6% of warm total, prefill ahead -DwarfStar (ds4)DeepSeek-V4-Flash IQ2_XXS, one DGX Spark16.28 vs 16.33 tok/s decode, 0.997x, a parity result +DwarfStar (ds4)DeepSeek-V4-Flash IQ2_XXS, one DGX Spark18.69 vs 16.33 tok/s decode, 1.144x, same output +vLLMLaguna-XS-2.1 NVFP4, GB1044.46 vs 43.10 tok/s, 1.03x, same output
The upstream page is careful about its own noise: on the 27B grid the run-to-run spread is 0.5% and c2 through c32 land between 0.7% and 1.7%, so it calls those five ties rather than wins. The concurrency-1 result is the one it stands behind. -The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 16.28 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible. +The DeepSeek-V4-Flash row is the one that shows how far this has moved from being a vLLM port. It runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark, decoding at 18.69 tok/s against DwarfStar's 16.33. At 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits inside the Spark's 119 GiB unified pool, and reading GGUF is what makes that possible. -Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 is token-identical to vLLM's MTP and about 4% faster at concurrency 1. +That number moved twice in a week, and the second move came from one lever. The dense Q8_0 projection tower was being read from the GGUF mmap over unified memory, which the GB10 reads about 20% slower per-GEMV than device memory. Staging that 6 GiB tower device-resident once at load, same bytes and same kernels, took decode from 16.23 to 18.69, generating the same tokens and using no more peak memory. The same change took Laguna-XS-2.1 from 87% of vLLM to 1.03x ahead of it. + +Speculative decoding is in similar shape: MTP on Qwen3.6-27B NVFP4 generates the same tokens as vLLM's MTP and runs about 4% faster at concurrency 1. Configuration is a normal backend install: @@ -168,6 +181,11 @@ The `bonsai` backend serves the 1-bit (Q1_0) and ternary (Q2_0) Bonsai quantizat ## The operations bar became a page +
+The Activity page with four installs running +
Four backend installs in flight, and the record of what already finished.
+
+ The old operations bar rendered one row per in-flight operation above every page. Queue four model installs and a backend and it took most of the viewport, on every route, until the last one finished. It was doing two jobs at once. A global "something is happening" signal only needs one line, and the detail of what is happening needs a page of its own. The strip is now one line, permanently, showing a failure first and otherwise the least-advanced running operation, with a `+N more` pill. Its `✕` hides the strip and no longer cancels anything. That is a deliberate behavior change worth knowing about before you click it out of habit: the same glyph used to cancel a 17 GB download in one row and dismiss a message in the next. Cancelling moved to the new page, behind a button that says so. diff --git a/website/static/media/3d-generation.gif b/website/static/media/3d-generation.gif new file mode 100644 index 000000000..5f7105d24 Binary files /dev/null and b/website/static/media/3d-generation.gif differ diff --git a/website/static/media/v4-8-0-ui-activity.png b/website/static/media/v4-8-0-ui-activity.png new file mode 100644 index 000000000..3e25a0a37 Binary files /dev/null and b/website/static/media/v4-8-0-ui-activity.png differ diff --git a/website/static/media/v4-8-0-ui-home.png b/website/static/media/v4-8-0-ui-home.png new file mode 100644 index 000000000..0e1d6f687 Binary files /dev/null and b/website/static/media/v4-8-0-ui-home.png differ diff --git a/website/static/media/v4-8-0-ui-model-variants.png b/website/static/media/v4-8-0-ui-model-variants.png new file mode 100644 index 000000000..bde6d4151 Binary files /dev/null and b/website/static/media/v4-8-0-ui-model-variants.png differ diff --git a/website/static/media/v4-8-0-vllm-cpp-scoreboard.html b/website/static/media/v4-8-0-vllm-cpp-scoreboard.html new file mode 100644 index 000000000..bed1ee912 --- /dev/null +++ b/website/static/media/v4-8-0-vllm-cpp-scoreboard.html @@ -0,0 +1,100 @@ + + + + + + + +
vllm.cpp · throughput vs the reference engine
+

Measured against what each workload actually runs on

+
Throughput relative to the reference. 1.00 is parity, bars run from it. Higher is faster.
+ + + +
+ github.com/mudler/vllm.cpp + GB10 unless noted · greedy, reference in its own production config · docs/BENCHMARKS.md +
+ + + + diff --git a/website/static/media/v4-8-0-vllm-cpp-scoreboard.png b/website/static/media/v4-8-0-vllm-cpp-scoreboard.png new file mode 100644 index 000000000..0d24077d4 Binary files /dev/null and b/website/static/media/v4-8-0-vllm-cpp-scoreboard.png differ