mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-13 06:45:26 -04:00
ea60f62d66d4e548bf2b2d2d83f4e2569e6f2db7
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler's LocalAI [bot]andmudler
|
2ba3265c67 |
chore: ⬆️ Update mudler/vllm.cpp to 6e99e4540b8d0811542a9d7690dc9aa0bda5812f (#11589)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
|
6d4c64c6c3 |
chore: ⬆️ Update mudler/vllm.cpp to 76f2a6d84e41cc9a593146a9a958f63645542731 (#11555)
* ⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(vllm-cpp): track ABI v21 The updated engine exposes ABI v21 after extending its speech API. The Go backend does not bind that API, so its existing mirrors remain valid. The qwen3.5 warning fix is now present upstream, so the old patch no longer applies. Assisted-by: Codex:gpt-5 --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
|
869b30d18f |
chore: ⬆️ Update mudler/vllm.cpp to 4880c5715f36445a30bd39d3349a06dc96085a11 (#11515)
* ⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(vllm-cpp): mirror ABI v20 layouts The dependency bump advances the engine ABI from v17 to v20. The old binding stops every backend build at the ABI guard and undersizes structures used at runtime. Mirror the appended model and video fields so every platform uses the pinned engine layout. Assisted-by: Codex:gpt-5 * fix(vllm-cpp): satisfy Apple Clang The new engine pin captures a namespace-scope help string in a lambda. Apple Clang rejects the redundant capture because upstream enables -Werror. Carry the one-line source patch until the engine pin includes the fix. Assisted-by: Codex:gpt-5 * fix(vllm-cpp): align the carry patch The Apple Clang patch used context from another source revision. Source preparation rejected it before every backend build. Align the patch with the pinned engine revision. Assisted-by: Codex:gpt-5 [monitoring-prs] --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0c9d4bf9cc |
fix(vllm-cpp): build every CUDA architecture the platform can host (#11512)
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64. vllm.cpp's own release archive builds ten architectures, so LocalAI shipped one or two of them. The failure mode is the problem. An unlisted card is not slower, it dies at the first request with "no kernel image is available for execution on the device", long after `backends install` reported success. That covers A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor, and it is how a Jetson Thor node was found serving nothing at all. amd64 now builds 80;86;89;90a;100a;103a;120a;121a and arm64 builds 87;90a;100a;110;121a, split by where the silicon exists: Jetson is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of GH200/GB200. Triton-AOT stays ON for both, which the old comment said was impossible. It is not, at the version we pin: only maintainer REGEN needs a single arch, while the BUILDER path embeds every vendored cubin tree and selects by exact SM, so 87/103a/110/120a take the portable CUDA kernels and can never load a neighbouring cubin. Upstream ships its ten-SM archive that way. The CUDA 13 guard now covers both branches rather than amd64 alone. arm64 needs compute_121a just as much, and CI already builds it with 13. Cost is smaller than the arch count suggests, because gencode is per-source: fp4-mma still resolves to 120a;121a, and the CUTLASS scaled-mm kernels to one arch each, so the added architectures do not multiply the expensive translation units. Verified: flag generation checked for both branches, CUDA 12 still refused, CPU build untouched; both arch lists expanded through vllm.cpp's own vt_cuda_gencode_options and per-feature arch gating, and all six vendored Triton trees confirmed intact, at the exact pinned commit. A real compile is CI-only: there is no CUDA toolchain on the dev box. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andmudler
|
3aa097af44 |
chore: ⬆️ Update mudler/vllm.cpp to 9fd9e8f34408d5dd21d7f9385e96fc755708950b (#11472)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andmudler
|
f92e4e1303 |
chore: ⬆️ Update mudler/vllm.cpp to 24306364ab8beaed9197604a042a56aaccfde493 (#11385)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0f50b2af2 |
feat(vllm-cpp): serve MiniMax-H3 video+audio generation (#11424)
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
ea438cdeaf |
feat(vllm-cpp): wire the full engine config surface through engine_args (#11159)
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
102d91414e |
fix(vllm-cpp): mirror the engine's ABI v10 so the backend loads again (#11386)
The Go bindings mirror vllm.h by hand and refuse a library whose vllm_abi_version differs from what they were written against. Two automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines declaring ABI v10 while govllmcpp.go still mirrored v5, so every vllm-cpp image built since then panics at startup on every platform: panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5 Grow both PODs to the v10 layout: vllm_model_params gains speculative_config, enable_prefix_caching, max_num_batched_tokens, scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes), vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The offsets in the specs come from offsetof() against the pinned header. All of the new fields are inert when zeroed, so the engine behaves exactly as it did under v5; the backend sets none of them. Nothing cross-checked the two files, which is why a blind pin bump could ship a backend that cannot load. The library build now runs abi-check first: it compares VLLM_ABI_VERSION in the fetched header against abiVersion in govllmcpp.go and fails the build naming both, instead of leaving the mismatch for a user's runtime. Fixes #11379 Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andmudler
|
a8d310573e |
chore: ⬆️ Update mudler/vllm.cpp to 0757cac231ecd571a83c4fd2f50805c9251fc225 (#11352)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
|
3f0db2a9c2 |
feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal (#11137)
* feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* fix(vllm-cpp): default the MLX GEMM provider OFF on darwin
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.
Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s
MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s
Shipping the previous default would have halved Apple Silicon throughput.
MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.
The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.
Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.
Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM
MLX ungated (older pins) TTFT 537 ms 12.7 tok/s
MLX off TTFT 602 ms 23.9 tok/s
This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.
The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.
One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.
Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.
Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX gate from mainline
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): restore backend build portability
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): demote MLX header VLA warning
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): suppress MLX header VLA warning
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin source-scoped MLX warning fix
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin effective MLX warning suppression
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin diagnostic pragma fix
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin remaining Darwin build fixes
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin MLX system dependency boundary
Pin the companion vllm.cpp change that models MLX as an imported system dependency, keeping third-party header diagnostics out of the project's warnings-as-errors policy while retaining fatal warnings for project sources.
Assisted-by: Codex:gpt-5 [Codex]
* fix(vllm-cpp): pin scoped MLX warning guard
Advance vllm.cpp to the companion fix that keeps MLX headers on a SYSTEM dependency and scopes AppleClang folding-constant suppression to the external includes.
Assisted-by: Codex:gpt-5 [systematic-debugging] [test-driven-development]
* fix(vllm-cpp): use available MLX wheel
MLX 0.29.3 is no longer available to the Darwin runner, so the backend build stopped before CMake. Pin the first available compatible wheel and keep the documented default in sync.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
|
||
|
mudler's LocalAI [bot]andmudler
|
211aa0a536 |
chore: ⬆️ Update mudler/vllm.cpp to a42b8187caff02c570c28e19e4dc2b1d7f55ed14 (#11174)
⬆️ Update mudler/vllm.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
4baa36ddd8 |
feat(backend): vllm-cpp - text-generation backend for vllm.cpp with llama.cpp-parity tool calling (#11100)
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp) Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20 port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, no Python at inference) - as a Go gRPC backend over its stable C ABI (ABI v2) via purego. Backend (backend/go/vllm-cpp): - Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir (anything else is refused, satisfying the greedy-probe rule); context_size maps to max_model_len, options block_size/num_blocks/max_num_seqs size the KV cache and scheduler admission. - Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream with the per-delta C callback bridged into the gRPC stream. The backend embeds base.Base (not SingleThread): concurrent requests batch continuously in the engine's shared AsyncLLM scheduler. - PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving grammar-constrained tool calling at parity with llama-cpp; the ABI also exposes JSON-schema/regex/choice constraints. - Hand-mirrored POD structs with layout locked by unit tests (unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate. - One portable library per platform (vllm.cpp uses per-file SIMD tiers with runtime dispatch), so no avx/avx2/avx512 variant builds. Wiring: - backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13 amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark - the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal. - backend/index.yaml meta + 12 image entries (latest/development x cpu, cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit specs (pure Go, no engine build). - Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors) advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp without tokenizer templating (the C ABI takes the FINAL prompt; templating and tool parsing stay LocalAI-side). No auto-detect importer. - Docs: backends list, top-level README maintained-engines table, compatibility table. Verified: 20/20 Ginkgo specs against the real pinned engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy determinism, stop words, GBNF-constrained generation, and 4 concurrent streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary. Upstream ABI v2 + production structured-output wiring landed as mudler/vllm.cpp@86013f3. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3) The backend now implements AIModelRich (PredictRich / PredictStreamRich) over vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME code path as the llama.cpp autoparser: the ENGINE renders the model's chat template, decides when a tool call engages, and parses it - LocalAI receives pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp. - With use_tokenizer_template + structured Messages, PredictOptions lowers to ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling, stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice auto lowers engine-side to a LAZY structural-tag decode constraint - free text until the model emits the tool trigger, then the call is grammar-constrained; required/named force a call. Tool output is parsed by the engine's streaming Hermes-style parser; each chat.completion.chunk maps onto ChatDeltas (content / reasoning_content / tool_calls) which the host already prefers over Go-side tag extraction. Without structured messages the plain path (LocalAI templating + optional GBNF grammar) applies unchanged. - The engine resolves the chat template from the GGUF tokenizer.chat_template metadata (or tokenizer_config.json); templates beyond its minja subset - e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with a stderr witness, so structural-tag engagement keeps working. - Importers now emit the same config shape as llama-cpp for vllm-cpp (use_tokenizer_template: true, no-grammar autoparser flow); only the llama-cpp-specific use_jinja option and the vllm-python parser options are dropped. - Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution). Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full suite green - blocking chat, streaming deltas concatenating byte-equal to the blocking answer, a REQUIRED tool call returning schema-valid arguments JSON, and an AUTO run where the engine itself engages get_weather and streams parsed tool deltas; plus unit specs for the request lowering, chunk->ChatDelta mapping, and the C struct mirrors (ABI gate now v3). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects (every pure-text parser in the pinned vLLM registry, each ported 1:1 with its upstream tests), 7 reasoning parsers, google/minja as the template renderer (the full Qwen3.5 template now renders engine-side), per-family structural tags (tool_choice required/named compiles the model's NATIVE syntax where expressible), and template auto-detection for both parser axes. Backend changes: - cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields, layout-locked by the offset tests; ABI gate now v5). - New model options tool_parser:<name> / reasoning_parser:<name> pass through to the engine; unset means template auto-detection (18-row tool marker table; [THINK]->mistral, <think>->think_auto for reasoning); "none" disables the reasoning split; unknown names fail the first chat call. - Chat chunks parse the `reasoning` field (the pin renamed reasoning_content), flowing into ChatDelta.ReasoningContent which the host already prefers. Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the real chat template renders (no more fallback), reasoning auto-detection picks think_auto so markerless answers stay pure content (the live run caught the deepseek_r1 content-swallow upstream and drove the think_auto fix), required tool_choice returns schema-valid arguments, auto tool_choice engages engine-side and streams parsed deltas, and blocking/streaming stay byte-identical. Turn latency also dropped (proper template EOS behavior). Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5, minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry, think_auto). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(vllm-cpp): bump the engine pin to the ENG-wave close-out mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families (qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss) text-reimplemented from their wire formats and held to the upstream test suites - 39 registered dialects; the pinned vLLM registry is now covered except the three Rust/Harmony-backed families, descoped by decision. kimi_k2 also gains a full native structural-tag builder; four new template auto-detection rows land with test-pinned ordering. Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): add the vllm-cpp-development gallery meta The gallery grew the twelve latest/development image entries but was missing the separate vllm-cpp-development meta (own capabilities map targeting the -development image names), which every backend ships so the development gallery resolves per-platform. Validated: all capability targets in both metas resolve to existing entries, and every image URI's tag suffix matches a backend-matrix build. Also full-stack verified in this change's context (single-node local-ai from this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF): /v1/chat/completions non-stream (clean content + usage), streaming (SSE deltas), tool_choice auto engaging get_weather engine-side with schema-valid arguments and finish_reason=tool_calls, and streamed tool-call deltas in the standard name-first cadence. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton Two distinct failures took down all five vllm-cpp backend builds on the PR: 1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the engine build with -Werror=maybe-uninitialized in InputBatch::condense - a false positive through a staging std::optional's raw storage. Fixed upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly; verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two diagnostics exactly, fixed compiles clean) with the engine's behavior suites green. Pin bumped to that sha. 2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a multi-arch (120a;121a) fat build unless pinned to one tree, which would be unsound for the other arch. Triton is now enabled only on the single-arch arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the engine's non-AOT GDN path. Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8) image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13), killing the amd64 cuda-12 build at nvcc. Gate the architecture list on CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t (cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so the amd64 cuda-12 image never served it - no capability change. Verified by Makefile dry-run variable dumps for all three combinations (cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off). Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain Third CI round, third layer: with the arch list already narrowed to 120a, the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4 path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels would be a crippled build of an engine whose whole GPU story is fp4, so the variant is dropped instead: - backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64, cpu, vulkan, metal remain). - gallery: cuda12 image entries removed; the nvidia capability now resolves to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so older-driver hosts fall back to the CPU image instead of an unrunnable one. - backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails fast with a clear message; cuda-13 keeps the 120a;121a fat binary and arm64/l4t keeps 121a with the Triton cubins. Verified: Makefile branch dumps for all four combinations (cuda12 loud error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter tests green, gallery capability targets all resolve. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool" replies and Message.ReasoningContent on assistant history, so a second turn after tool execution reached the engine's chat template without the fields that bind a tool result to the call it answers. Forward all three (present-only, matching the OpenAI wire shape) and pin vllm.cpp to 6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id, name and reasoning and the minja adapter exposes them to the template context. Adds the round-trip request-lowering spec (user -> assistant tool_call -> tool reply -> lowered request) and re-ran the gated e2e suite against the new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming parity, required-tool and auto-tool cases all green. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix The darwin-metal CI job was the first build to compile the engine's arm CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> / <sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite re-run green against the new pin with a real Qwen3.5 GGUF. Assisted-by: Claude Code:claude-fable-5 [Bash] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent The macOS runners have no nproc, so JOBS evaluated empty and `cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a 3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log shows "nproc: Command not found" and 7+ concurrent clang processes being reaped at the cutoff). Use the same portable fallback chain as the other darwin backends: nproc, then sysctl hw.ncpu, then 4. Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |