5945b0eb11
The vllm-cpp backend could configure four of the engine's knobs - block size,
KV block count, max sequence length and max concurrent sequences - out of a
config surface that is considerably larger. Speculative decoding, prefix
caching, the chunked-prefill token budget, the scheduling policy and the
external KV connector were all reachable from vllm.cpp's own HTTP server and
from nothing LocalAI could write in a model config.
Part of that gap was the C ABI itself, which carried strictly less than
EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin
to it). The rest was here: the backend parsed a flat `options:` list with five
recognised keys and had no way to express a nested JSON document at all.
Configuration now goes through `engine_args:`, the same map the vLLM and SGLang
backends already take, with keys spelled as vLLM's own CLI flags - so a
`speculative_config` or `kv_transfer_config` block written for vLLM works
verbatim:
engine_args:
max_num_batched_tokens: 8192
enable_prefix_caching: true
scheduling_policy: lpm
speculative_config:
method: dflash
model: z-lab/Qwen3.6-27B-DFlash
num_speculative_tokens: 4
kv_transfer_config:
kv_connector: LMCacheConnector
kv_role: kv_both
kv_connector_extra_config: {host: 127.0.0.1, port: 65432}
The `options:` list keeps working, and now reads every key too, so no existing
config breaks; engine_args wins where both set the same key.
Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's
force-OFF state (2), not the 0 that means "let the model capability decide" -
collapsing them would silently turn the cache ON for the dense architectures
that default it on. And cSamplingParams grows the ABI v8 logits-processor tail:
LocalAI installs no processor, but the C side reads those fields off the pointer
we hand it, so a Go struct that stopped short would have had the engine read 16
bytes past our allocation and call whatever sat there.
The importer gets the safetensors counterpart of the llama-cpp MTP hook: a
`vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint
that declares an MTP head, writes speculative_config {method: mtp} into the
generated engine_args. DFlash draft repositories are detected and refused with a
warning rather than configured as standalone models, since a drafter cannot
serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp`
options when the chosen backend is vllm-cpp: those are llama.cpp option keys
vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway
because the `mtp.*` draft tensors only exist in the safetensors checkpoint.
docs/content/features/text-generation.md gains a vllm.cpp section - the backend
had no documentation page at all - covering the engine_args table, all three
speculative methods, LMCache, and the legacy options list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
LocalAI website
LocalAI documentation website
Requirement
In this project, the Docsy theme component is pulled in as a Hugo module, together with other module dependencies:
$ hugo mod graph
hugo: collected modules in 566 ms
hugo: collected modules in 578 ms
github.com/google/docsy-example github.com/google/docsy@v0.5.1-0.20221017155306-99eacb09ffb0
github.com/google/docsy-example github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/twbs/bootstrap@v4.6.2+incompatible
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/FortAwesome/Font-Awesome@v0.0.0-20220831210243-d3a7818c253f
If you want to do SCSS edits and want to publish these, you need to install PostCSS
npm install
Running the website locally
Building and running the site locally requires a recent extended version of Hugo.
You can find out more about how to install Hugo for your environment in our
Getting started guide.
Once you've made your working copy of the site repo, from the repo root folder, run:
hugo server
Running a container locally
You can run docsy-example inside a Docker
container, the container runs with a volume bound to the docsy-example
folder. This approach doesn't require you to install any dependencies other
than Docker Desktop on
Windows and Mac, and Docker Compose
on Linux.
-
Build the docker image
docker-compose build -
Run the built image
docker-compose upNOTE: You can run both commands at once with
docker-compose up --build. -
Verify that the service is working.
Open your web browser and type
http://localhost:1313in your navigation bar, This opens a local instance of the docsy-example homepage. You can now make changes to the docsy example and those changes will immediately show up in your browser after you save.
Cleanup
To stop Docker Compose, on your terminal window, press Ctrl + C.
To remove the produced images run:
docker-compose rm
For more information see the Docker Compose documentation.
Troubleshooting
As you run the website locally, you may run into the following error:
➜ hugo server
INFO 2021/01/21 21:07:55 Using config file:
Building sites … INFO 2021/01/21 21:07:55 syncing static files to /
Built in 288 ms
Error: Error building site: TOCSS: failed to transform "scss/main.scss" (text/x-scss): resource "scss/scss/main.scss_9fadf33d895a46083cdd64396b57ef68" not found in file cache
This error occurs if you have not installed the extended version of Hugo. See this section of the user guide for instructions on how to install Hugo.
Or you may encounter the following error:
➜ hugo server
Error: failed to download modules: binary with name "go" not found
This error occurs if you have not installed the go programming language on your system.
See this section of the user guide for instructions on how to install go.