feat(gallery): add kev-0.8b-vllm-cpp decision model

Add the kev 0.8B decision model, converted for vllm.cpp and pinned to
revision c17e7366 of mudler/kev-0.8b-vllm-cpp. It is a redistribution of
jaredpalmer/kev-0.8b with the LoRA merged and the PointerHead stored as
head.safetensors, so only the vllm-cpp backend can load it.

The artifact sits under overrides, where the installer reads it. The
entry sets a 2048-token context and an explicit KV pool: with the
default 4096-token context the CPU KV pool holds only 4064 tokens and the
load fails.

List the entry in the decisions gallery table and drop kev from the
list of decision models that are not gallery entries yet.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-sonnet-5-5
This commit is contained in:
Ettore Di Giacinto committed 2026-09-30 22:28:13 +00:00
1 parent 4b34c54e75
commit 975ff5ca42
2 files changed
+58 -2

No files matched your search

+4 -2
View File
@@ -107,10 +107,12 @@ Install one from the gallery and filter on the `decisions` tag:
| `gliner25-decide-vllm-cpp` | GLiNER2.5-Decide | DeBERTa-v3-large with a classification head, about 2 GB |
| `tev1-4b-vllm-cpp` | Tev1 4B | Autoregressive Qwen3.5-4B fine-tune that answers with an option letter, about 9.3 GB |
| `tev1-0.8b-vllm-cpp` | Tev1 0.8B | Autoregressive Qwen3.5-0.8B fine-tune that answers with an option letter, about 1.8 GB |
| `kev-0.8b-vllm-cpp` | kev 0.8B | Qwen3.5-0.8B-Base with a merged LoRA and a PointerHead readout, converted for vllm.cpp only, about 1.53 GB |
The engine, [vllm.cpp]({{% relref "features/vllm-cpp" %}}), also supports the
kev, CLM and xor decision models. Those checkpoints need a conversion step, so
they are not gallery entries yet.
CLM and xor decision models. Those checkpoints need a conversion step, so
they are not gallery entries yet. The kev entry installs a checkpoint that was
already converted with the vllm.cpp `convert-kev.py` script.
Tev1 is an autoregressive decision model. The engine answers each question by
scoring the option letters, so its `confidence` is the entropy measure Ollama