mudler-agentandEttore Di Giacinto
b540c3e1fd
* chore(vllm-cpp): bump vllm.cpp to 967883486 (ABI v30) Moves the pin from c3bebc357 to 967883486. On top of the Nimble decision adapter and the Qwen3.5 vision-loader fix, this brings Tev1 on /v1/systemone and vllm_decide (opt-in through a "Tev1Model" architecture in config.json), a tokenizer/ subdirectory fallback so the Laya HF snapshot loads as downloaded, a stop-token fix, a logprobs fix under async scheduling and a pinned parakeet.cpp fetch for the diarization build. ABI v30 only adds the diarization and speaker-attributed ASR entry points; no existing struct or signature changed, so the purego mirrors keep their layout and only abiVersion moves to 30. Between 4479dc99f and 967883486 vllm.h changed only in a comment. v30 turns VLLM_CPP_WITH_DIARIZATION on by default. The fetch is pinned now, but ON still downloads parakeet.cpp at configure time and links a second ggml into libvllm for calls this backend never makes, so build with the option off: the symbols stay present as refusing stubs. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-sonnet-5-5 * feat(vllm-cpp): add the hf_overrides engine arg vLLM parity: engine_args.hf_overrides is a JSON object of top-level config.json keys merged over the model directory's own config.json. The main use is opting a published checkpoint into an engine adapter its config does not name, such as {"architectures": ["Tev1Model"]} on the Tev1 snapshots, which declare Qwen3_5ForConditionalGeneration. The C ABI has no override input and the engine reads config.json from the directory it is given, so Load builds a private overlay directory: the merged config.json plus a symlink to every other entry of the model directory, and passes that to the engine. The download is never written. Free, a failed load and the next Load remove the overlay. validModelPath and the DFlash draft resolution still see the real directory. A value that is not a JSON object, a .gguf model or a directory without config.json fails the load instead of being skipped like an unknown engine_args key, because loading the unmodified config would serve a different architecture than the one configured. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-sonnet-5-5 * fix(gallery): nest vllm-cpp artifacts under overrides artifacts: is a model-config key, and the installer reads model-config keys only from overrides:. Five vllm-cpp entries (laya, gliner25-decide, qwen3-vl-4b, cua-s1-forms and gliner2.5) declared it at the entry top level, where it is silently dropped: the install reports success, writes a config whose model is the bare HF repo id and downloads nothing, and vllm-cpp (which does not infer artifacts) then fails the first load with "model path not found". Move each block under overrides:, and add a guard test that refuses a top-level artifacts: key in the index. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-sonnet-5-5 * feat(gallery): add Tev1 4B and 0.8B on vllm-cpp Two decisions entries for Together AI's Tev1 checkpoints, pinned to the current HF revisions. Tev1 is autoregressive: vllm.cpp answers /v1/systemone by scoring the option letters, and the same engine still serves chat completions. The published config.json names Qwen3_5ForConditionalGeneration, so each entry sets hf_overrides: {architectures: [Tev1Model]} to enable the decision route without editing the download. known_usecases is [decisions] only, since a declared decisions list is authoritative for reservation. The descriptions state what was checked: agreement with transformers on CPU over seven questions (4B 7/7, max probability difference 0.0004; 0.8B 6/7 with one near tie), CPU-only for the decision route, and a fine-tune license the model card says is still being finalized, so no license key is set. The Decisions API page lists both entries, drops the note that Tev1 does not serve /v1/systemone and documents the 24-option limit (Ollama allows 26). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-sonnet-5-5 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
LocalAI website
LocalAI documentation website
Requirement
In this project, the Docsy theme component is pulled in as a Hugo module, together with other module dependencies:
$ hugo mod graph
hugo: collected modules in 566 ms
hugo: collected modules in 578 ms
github.com/google/docsy-example github.com/google/docsy@v0.5.1-0.20221017155306-99eacb09ffb0
github.com/google/docsy-example github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/twbs/bootstrap@v4.6.2+incompatible
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/FortAwesome/Font-Awesome@v0.0.0-20220831210243-d3a7818c253f
If you want to do SCSS edits and want to publish these, you need to install PostCSS
npm install
Running the website locally
Building and running the site locally requires a recent extended version of Hugo.
You can find out more about how to install Hugo for your environment in our
Getting started guide.
From the LocalAI repository root, run:
make docs
The Hugo configuration lives in the docs directory. To invoke Hugo
directly instead, run:
cd docs
hugo server
Running a container locally
You can run docsy-example inside a Docker
container, the container runs with a volume bound to the docsy-example
folder. This approach doesn't require you to install any dependencies other
than Docker Desktop on
Windows and Mac, and Docker Compose
on Linux.
-
Build the docker image
docker-compose build -
Run the built image
docker-compose upNOTE: You can run both commands at once with
docker-compose up --build. -
Verify that the service is working.
Open your web browser and type
http://localhost:1313in your navigation bar, This opens a local instance of the docsy-example homepage. You can now make changes to the docsy example and those changes will immediately show up in your browser after you save.
Cleanup
To stop Docker Compose, on your terminal window, press Ctrl + C.
To remove the produced images run:
docker-compose rm
For more information see the Docker Compose documentation.
Troubleshooting
As you run the website locally, you may run into the following error:
➜ hugo server
INFO 2021/01/21 21:07:55 Using config file:
Building sites … INFO 2021/01/21 21:07:55 syncing static files to /
Built in 288 ms
Error: Error building site: TOCSS: failed to transform "scss/main.scss" (text/x-scss): resource "scss/scss/main.scss_9fadf33d895a46083cdd64396b57ef68" not found in file cache
This error occurs if you have not installed the extended version of Hugo. See this section of the user guide for instructions on how to install Hugo.
Or you may encounter the following error:
➜ hugo server
Error: failed to download modules: binary with name "go" not found
This error occurs if you have not installed the go programming language on your system.
See this section of the user guide for instructions on how to install go.