mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-25 07:34:58 -04:00
feat(gallery): add Qwen3.8 9B Distill Q5 and Q6
Fill the quantization gap between the existing Q4 and Q8 builds. Pin the new downloads to a Hugging Face revision and retain the existing llama.cpp chat configuration. Document explicit selection. Assisted-by: Codex:GPT-6
This commit is contained in:
1 parent
3dfb34376a
commit
fbed25f1f7
2 files changed
+80
-1
No files matched your search
@@ -90,6 +90,21 @@ temperature 1, top_k 64, and top_p 0.95. They are separate from the
|
||||
existing QAT builds. See the [source model](https://huggingface.co/google/gemma-4-12B-it)
|
||||
and [GGUF files](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF).
|
||||
|
||||
### Qwen3.8 9B Distill quantizations
|
||||
|
||||
The text-only Qwen3.8 9B Distill entry offers Q4_K_M, Q5_K_M, Q6_K, and
|
||||
Q8_0 GGUF builds for llama.cpp. Select Q5 or Q6 explicitly with:
|
||||
|
||||
```bash
|
||||
local-ai models install qwen3.8-9b-distill-q4 --variant qwen3.8-9b-distill-q5
|
||||
local-ai models install qwen3.8-9b-distill-q4 --variant qwen3.8-9b-distill-q6
|
||||
```
|
||||
|
||||
All four builds use the embedded chat template, a 32,768-token context,
|
||||
and the same sampling defaults. See the
|
||||
[GGUF repository](https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF)
|
||||
for the published quantizations.
|
||||
|
||||
## Add other galleries
|
||||
|
||||
You can add other galleries by:
|
||||
|
||||
+65
-1
@@ -5902,6 +5902,8 @@
|
||||
- &qwen3-8-9b-distill
|
||||
name: "qwen3.8-9b-distill-q4"
|
||||
variants:
|
||||
- model: qwen3.8-9b-distill-q5
|
||||
- model: qwen3.8-9b-distill-q6
|
||||
- model: qwen3.8-9b-distill-q8
|
||||
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
|
||||
urls:
|
||||
@@ -5912,7 +5914,7 @@
|
||||
distilled from Qwen3.8 2.4T A95B reasoning traces. It targets mathematics,
|
||||
coding, instruction following, and function calling with a 262K native
|
||||
context window. This entry uses the balanced Q4_K_M GGUF quantization; the
|
||||
Q8_0 variant offers higher fidelity.
|
||||
Q5_K_M, Q6_K, and Q8_0 variants offer higher fidelity.
|
||||
license: apache-2.0
|
||||
icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/logo_qwen.jpg
|
||||
tags:
|
||||
@@ -5978,6 +5980,68 @@
|
||||
- filename: llama-cpp/models/qwen3.8-9b-distill/Qwen3.8-9B-Q8_0.gguf
|
||||
uri: huggingface://empero-ai/Qwen3.8-9B-Distill-GGUF/Qwen3.8-9B-Q8_0.gguf
|
||||
sha256: 79ca5d342a07922f2bbf38c8d892a79a3c8620c65feaf4b1c66b7830ae724db8
|
||||
- !!merge <<: *qwen3-8-9b-distill
|
||||
name: "qwen3.8-9b-distill-q5"
|
||||
variants: []
|
||||
last_checked: "2026-09-14"
|
||||
description: |
|
||||
Qwen3.8 9B Distill in the higher-fidelity Q5_K_M GGUF format. This text-only
|
||||
Qwen3.5 9B fine-tune targets reasoning, coding, instruction following, and
|
||||
function calling with a 262K native context window.
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
context_size: 32768
|
||||
function:
|
||||
automatic_tool_parsing_fallback: true
|
||||
grammar:
|
||||
disable: true
|
||||
known_usecases:
|
||||
- chat
|
||||
options:
|
||||
- use_jinja:true
|
||||
- reasoning_format:deepseek
|
||||
parameters:
|
||||
model: llama-cpp/models/qwen3.8-9b-distill/Qwen3.8-9B-Q5_K_M.gguf
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
template:
|
||||
use_tokenizer_template: true
|
||||
files:
|
||||
- filename: llama-cpp/models/qwen3.8-9b-distill/Qwen3.8-9B-Q5_K_M.gguf
|
||||
uri: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF/resolve/760121cd70bb4c36b2b5ec58eb765e0df5987efe/Qwen3.8-9B-Q5_K_M.gguf
|
||||
sha256: c6667345d4e45d8cddca3c8e997a483a4f9ae04ee9402273c24e959dee7173dc
|
||||
- !!merge <<: *qwen3-8-9b-distill
|
||||
name: "qwen3.8-9b-distill-q6"
|
||||
variants: []
|
||||
last_checked: "2026-09-14"
|
||||
description: |
|
||||
Qwen3.8 9B Distill in the higher-fidelity Q6_K GGUF format. This text-only
|
||||
Qwen3.5 9B fine-tune targets reasoning, coding, instruction following, and
|
||||
function calling with a 262K native context window.
|
||||
overrides:
|
||||
backend: llama-cpp
|
||||
context_size: 32768
|
||||
function:
|
||||
automatic_tool_parsing_fallback: true
|
||||
grammar:
|
||||
disable: true
|
||||
known_usecases:
|
||||
- chat
|
||||
options:
|
||||
- use_jinja:true
|
||||
- reasoning_format:deepseek
|
||||
parameters:
|
||||
model: llama-cpp/models/qwen3.8-9b-distill/Qwen3.8-9B-Q6_K.gguf
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
template:
|
||||
use_tokenizer_template: true
|
||||
files:
|
||||
- filename: llama-cpp/models/qwen3.8-9b-distill/Qwen3.8-9B-Q6_K.gguf
|
||||
uri: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF/resolve/760121cd70bb4c36b2b5ec58eb765e0df5987efe/Qwen3.8-9B-Q6_K.gguf
|
||||
sha256: 0f1271373f899912bfe4ea76299af7dd83722d98ea421b0827501c3a2c6da22b
|
||||
- name: "btl-4-compact"
|
||||
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
|
||||
urls:
|
||||
|
||||
Reference in new issue
Block a user