mudler's LocalAI [bot]andEttore Di Giacinto
0c9d4bf9cc
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64. vllm.cpp's own release archive builds ten architectures, so LocalAI shipped one or two of them. The failure mode is the problem. An unlisted card is not slower, it dies at the first request with "no kernel image is available for execution on the device", long after `backends install` reported success. That covers A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor, and it is how a Jetson Thor node was found serving nothing at all. amd64 now builds 80;86;89;90a;100a;103a;120a;121a and arm64 builds 87;90a;100a;110;121a, split by where the silicon exists: Jetson is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of GH200/GB200. Triton-AOT stays ON for both, which the old comment said was impossible. It is not, at the version we pin: only maintainer REGEN needs a single arch, while the BUILDER path embeds every vendored cubin tree and selects by exact SM, so 87/103a/110/120a take the portable CUDA kernels and can never load a neighbouring cubin. Upstream ships its ten-SM archive that way. The CUDA 13 guard now covers both branches rather than amd64 alone. arm64 needs compute_121a just as much, and CI already builds it with 13. Cost is smaller than the arch count suggests, because gencode is per-source: fp4-mma still resolves to 120a;121a, and the CUTLASS scaled-mm kernels to one arch each, so the added architectures do not multiply the expensive translation units. Verified: flag generation checked for both branches, CUDA 12 still refused, CPU build untouched; both arch lists expanded through vllm.cpp's own vt_cuda_gencode_options and per-feature arch gating, and all six vendored Triton trees confirmed intact, at the exact pinned commit. A real compile is CI-only: there is no CUDA toolchain on the dev box. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
LocalAI website
LocalAI documentation website
Requirement
In this project, the Docsy theme component is pulled in as a Hugo module, together with other module dependencies:
$ hugo mod graph
hugo: collected modules in 566 ms
hugo: collected modules in 578 ms
github.com/google/docsy-example github.com/google/docsy@v0.5.1-0.20221017155306-99eacb09ffb0
github.com/google/docsy-example github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/twbs/bootstrap@v4.6.2+incompatible
github.com/google/docsy/dependencies@v0.5.1-0.20221014161617-be5da07ecff1 github.com/FortAwesome/Font-Awesome@v0.0.0-20220831210243-d3a7818c253f
If you want to do SCSS edits and want to publish these, you need to install PostCSS
npm install
Running the website locally
Building and running the site locally requires a recent extended version of Hugo.
You can find out more about how to install Hugo for your environment in our
Getting started guide.
From the LocalAI repository root, run:
make docs
The Hugo configuration lives in the docs directory. To invoke Hugo
directly instead, run:
cd docs
hugo server
Running a container locally
You can run docsy-example inside a Docker
container, the container runs with a volume bound to the docsy-example
folder. This approach doesn't require you to install any dependencies other
than Docker Desktop on
Windows and Mac, and Docker Compose
on Linux.
-
Build the docker image
docker-compose build -
Run the built image
docker-compose upNOTE: You can run both commands at once with
docker-compose up --build. -
Verify that the service is working.
Open your web browser and type
http://localhost:1313in your navigation bar, This opens a local instance of the docsy-example homepage. You can now make changes to the docsy example and those changes will immediately show up in your browser after you save.
Cleanup
To stop Docker Compose, on your terminal window, press Ctrl + C.
To remove the produced images run:
docker-compose rm
For more information see the Docker Compose documentation.
Troubleshooting
As you run the website locally, you may run into the following error:
➜ hugo server
INFO 2021/01/21 21:07:55 Using config file:
Building sites … INFO 2021/01/21 21:07:55 syncing static files to /
Built in 288 ms
Error: Error building site: TOCSS: failed to transform "scss/main.scss" (text/x-scss): resource "scss/scss/main.scss_9fadf33d895a46083cdd64396b57ef68" not found in file cache
This error occurs if you have not installed the extended version of Hugo. See this section of the user guide for instructions on how to install Hugo.
Or you may encounter the following error:
➜ hugo server
Error: failed to download modules: binary with name "go" not found
This error occurs if you have not installed the go programming language on your system.
See this section of the user guide for instructions on how to install go.