mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-03 03:24:34 -04:00
feat(gallery): add kev-0.8b-vllm-cpp decision model
Add the kev 0.8B decision model, converted for vllm.cpp and pinned to revision c17e7366 of mudler/kev-0.8b-vllm-cpp. It is a redistribution of jaredpalmer/kev-0.8b with the LoRA merged and the PointerHead stored as head.safetensors, so only the vllm-cpp backend can load it. The artifact sits under overrides, where the installer reads it. The entry sets a 2048-token context and an explicit KV pool: with the default 4096-token context the CPU KV pool holds only 4064 tokens and the load fails. List the entry in the decisions gallery table and drop kev from the list of decision models that are not gallery entries yet. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-sonnet-5-5
This commit is contained in:
1 parent
4b34c54e75
commit
975ff5ca42
2 files changed
+58
-2
No files matched your search
@@ -107,10 +107,12 @@ Install one from the gallery and filter on the `decisions` tag:
|
||||
| `gliner25-decide-vllm-cpp` | GLiNER2.5-Decide | DeBERTa-v3-large with a classification head, about 2 GB |
|
||||
| `tev1-4b-vllm-cpp` | Tev1 4B | Autoregressive Qwen3.5-4B fine-tune that answers with an option letter, about 9.3 GB |
|
||||
| `tev1-0.8b-vllm-cpp` | Tev1 0.8B | Autoregressive Qwen3.5-0.8B fine-tune that answers with an option letter, about 1.8 GB |
|
||||
| `kev-0.8b-vllm-cpp` | kev 0.8B | Qwen3.5-0.8B-Base with a merged LoRA and a PointerHead readout, converted for vllm.cpp only, about 1.53 GB |
|
||||
|
||||
The engine, [vllm.cpp]({{% relref "features/vllm-cpp" %}}), also supports the
|
||||
kev, CLM and xor decision models. Those checkpoints need a conversion step, so
|
||||
they are not gallery entries yet.
|
||||
CLM and xor decision models. Those checkpoints need a conversion step, so
|
||||
they are not gallery entries yet. The kev entry installs a checkpoint that was
|
||||
already converted with the vllm.cpp `convert-kev.py` script.
|
||||
|
||||
Tev1 is an autoregressive decision model. The engine answers each question by
|
||||
scoring the option letters, so its `confidence` is the entropy measure Ollama
|
||||
|
||||
Reference in new issue
Block a user