mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-11 18:27:32 -04:00
feat(dllm): default gallery entry on Q4_K_M; add Q8_0 variant
Q4_K_M (~17 GB, GB10-validated: cosine 0.9862, coherent generation) is the friendlier default download than the 50 GB BF16; Q8_0 (~27 GB) is the higher-fidelity middle ground. Both descriptions carry the measured caveat that BF16 is ~5x faster per denoise step on BF16-native hardware, with a pointer to fetch it manually when it fits. sha256 values are the HF LFS oids. Assisted-by: Claude Code (Fable 5) Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
@@ -13,8 +13,11 @@
|
||||
Honest expectations:
|
||||
* Experimental: both the model family and the dllm backend are young -
|
||||
expect rough edges.
|
||||
* BF16 weights (~50 GB): CUDA-13-class hardware (DGX Spark / large-VRAM or
|
||||
unified-memory machines) is recommended; CPU works but is slow.
|
||||
* Q4_K_M weights (~17 GB): the memory-friendly default. Validated on
|
||||
GB10 (quality holds: golden cosine 0.9862, coherent generation), but
|
||||
note that on hardware with native BF16 tensor cores (GB10-class) the
|
||||
BF16 file is ~5x FASTER per denoise step than K-quants - if ~50 GB
|
||||
fits, fetch diffusiongemma-26B-A4B-it-BF16.gguf manually instead.
|
||||
* Throughput: every denoise step currently recomputes the full
|
||||
prompt+canvas - the prefix-KV cache that removes this lands with the
|
||||
dllm backend's P3 work - so long prompts cost proportionally more per
|
||||
@@ -31,11 +34,36 @@
|
||||
- dllm
|
||||
overrides:
|
||||
parameters:
|
||||
model: dllm/diffusiongemma-26B-A4B-it-BF16.gguf
|
||||
model: dllm/diffusiongemma-26B-A4B-it-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: dllm/diffusiongemma-26B-A4B-it-BF16.gguf
|
||||
sha256: b0ef5dbf246608953ee9945fb03c6056af9e2459799fb179651a20a8bbaa2921
|
||||
uri: https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF/resolve/main/diffusiongemma-26B-A4B-it-BF16.gguf
|
||||
- filename: dllm/diffusiongemma-26B-A4B-it-Q4_K_M.gguf
|
||||
sha256: d2ca2c032ebfb23cf2d1794a3465e615c7545634d46b3c30652a26d8b07c4ad3
|
||||
uri: https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF/resolve/main/diffusiongemma-26B-A4B-it-Q4_K_M.gguf
|
||||
- name: "diffusiongemma-26b-a4b-it-q8_0"
|
||||
url: "github:mudler/LocalAI/gallery/diffusiongemma.yaml@master"
|
||||
urls:
|
||||
- https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF
|
||||
- https://github.com/mudler/dllm.cpp
|
||||
description: |
|
||||
DiffusionGemma 26B A4B (instruction-tuned), Q8_0 quantization (~27 GB):
|
||||
the higher-fidelity middle ground between Q4_K_M (~17 GB) and BF16
|
||||
(~50 GB). Served by LocalAI's dllm backend (dllm.cpp); see the Q4_K_M
|
||||
entry for the full notes on experimental status, throughput, and the
|
||||
K-quant-vs-BF16 speed trade-off on BF16-native hardware.
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- llm
|
||||
- gguf
|
||||
- gemma
|
||||
- diffusion
|
||||
- dllm
|
||||
overrides:
|
||||
parameters:
|
||||
model: dllm/diffusiongemma-26B-A4B-it-Q8_0.gguf
|
||||
files:
|
||||
- filename: dllm/diffusiongemma-26B-A4B-it-Q8_0.gguf
|
||||
sha256: fa5180660b80d52aae94ed814a6183af303841d8bb425a27f13ea27400a7b430
|
||||
uri: https://huggingface.co/unsloth/diffusiongemma-26B-A4B-it-GGUF/resolve/main/diffusiongemma-26B-A4B-it-Q8_0.gguf
|
||||
- name: "gemma-4-26b-a4b-it-qat"
|
||||
url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
|
||||
urls:
|
||||
|
||||
Reference in New Issue
Block a user