feat(stablediffusion-ggml): Qwen-Image 2.1 support + gallery GGUF (#12190)

* fix(stablediffusion-ggml): bump sd.cpp for Qwen-Image 2.1, fix RPC build Bump stable-diffusion.cpp to c678dfe70, which adds Qwen-Image 2.1 support (leejet/stable-diffusion.cpp#1994). The same range pulls a ggml update that adds GGML_OP_SAGE_ATTN but does not update the GGML_OP_COUNT static_assert in ggml-rpc.h. We build with SD_RPC=ON and upstream CI does not, so every backend image failed to compile (see #12170). Add a sync-rpc-op-count step after checkout. It sets the ggml-rpc.h assert to the count that ggml.c asserts. The new op is appended before GGML_OP_COUNT, so existing op ids on the wire do not change, and the RPC handshake compares only major and minor versions. When upstream fixes the header, the step does nothing, so future automated bumps stay green. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Qwen-Image 2.1 GGUF for stablediffusion-ggml Add qwen-image-2.1-q4_k-ggml, with qwen-image-2.1-q8_0-ggml as a variant, from leejet/Qwen-Image-2.1-GGUF. The config follows the upstream stable-diffusion.cpp recipe: Qwen3-VL-8B-Instruct text encoder, the Qwen-Image 2.1 VAE, cfg scale 6, euler sampler. The bundle also pulls the Qwen3-VL mmproj as llm_vision_path, so that image editing with reference images works. The text encoder and mmproj use the same filenames and checksums as the qwen3-vl-8b-instruct entry, so both entries share the files on disk. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-09-22 14:14:54 -04:00 · 2026-09-22 08:20:59 +02:00
1 parent 66e3a06ce6
commit 32457137a3
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)

 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=1330cebae8f2ba99249df846cc0c9444fcbd4308
+STABLEDIFFUSION_GGML_VERSION?=c678dfe704a2230342376b46add9c8ca736a653d

 CMAKE_ARGS+=-DGGML_MAX_NAME=128

@@ -70,6 +70,20 @@ sources/stablediffusion-ggml.cpp:
 	cd sources/stablediffusion-ggml.cpp && \
 	git checkout $(STABLEDIFFUSION_GGML_VERSION) && \
 	git submodule update --init --recursive --depth 1 --single-branch
+	$(MAKE) sync-rpc-op-count
+
+# leejet's ggml fork adds ops (e.g. GGML_OP_SAGE_ATTN) without updating the
+# GGML_OP_COUNT static_assert in ggml-rpc.h, which breaks the build because we
+# enable SD_RPC while upstream CI does not. The new ops are appended before
+# GGML_OP_COUNT, so existing op ids on the wire are unchanged and the RPC
+# handshake only compares major/minor versions. Align the assert with the
+# count that ggml.c asserts; this is a no-op once upstream fixes it.
+sync-rpc-op-count:
+	@count=$$(grep -oE 'GGML_OP_COUNT == [0-9]+' sources/stablediffusion-ggml.cpp/ggml/src/ggml.c | head -n1 | grep -oE '[0-9]+'); \
+	if [ -n "$$count" ]; then \
+		sed -i.bak -E "s/GGML_OP_COUNT == [0-9]+/GGML_OP_COUNT == $$count/" sources/stablediffusion-ggml.cpp/ggml/include/ggml-rpc.h && \
+		rm -f sources/stablediffusion-ggml.cpp/ggml/include/ggml-rpc.h.bak; \
+	fi

 UNAME_S := $(shell uname -s)

@@ -93,7 +93,7 @@ All backends listed here can be installed on demand from the [Backend Gallery]({

 | Backend | Description | Acceleration |
 |---------|-------------|-------------|
-| [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) | Stable Diffusion, Flux, PhotoMaker, Ideogram in C/C++ | CPU, CUDA 12/13, Intel SYCL, Vulkan, Metal, Jetson L4T |
+| [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp) | Stable Diffusion, Flux, PhotoMaker, Ideogram, Qwen-Image in C/C++ | CPU, CUDA 12/13, Intel SYCL, Vulkan, Metal, Jetson L4T |
 | [diffusers](https://github.com/huggingface/diffusers) | HuggingFace diffusion models (image and video generation) | CPU, CUDA 12/13, ROCm, Intel SYCL, Metal, Jetson L4T |
 | [mlx-video](https://github.com/Blaizzy/mlx-video) | LTX-2 and converted Wan2.1/Wan2.2 video generation | Metal (Apple Silicon) |
 | [vLLM Omni](https://github.com/vllm-project/vllm-omni) | Multimodal generation including text-to-image and text-to-video | CUDA 12/13, ROCm, Jetson L4T |
@@ -45553,6 +45553,110 @@
    - filename: flux2-vae.safetensors
      sha256: 868fe7b343cc8f3a19dbcfcafbc3d5f888802be3f89bd81b65b3621a066ce8f3
      uri: https://huggingface.co/Comfy-Org/Ideogram-4/resolve/main/vae/flux2-vae.safetensors
+- name: qwen-image-2.1-q4_k-ggml
+  variants:
+    - model: qwen-image-2.1-q8_0-ggml
+  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+  urls:
+    - https://huggingface.co/Qwen/Qwen-Image-2.1
+    - https://huggingface.co/leejet/Qwen-Image-2.1-GGUF
+  description: |
+    Qwen-Image 2.1 is the Qwen image generation foundation model, with strong prompt adherence and accurate text rendering (English and Chinese). It uses Qwen3-VL-8B as the text encoder and its own VAE, and supports both text-to-image and image editing: pass reference images to edit them.
+
+    This is the Q4_K (4-bit) quantization (~4.2GB diffusion model) by leejet for stable-diffusion.cpp. The bundle also pulls the Qwen3-VL-8B-Instruct text encoder with its vision projector (used for image editing) and the Qwen-Image 2.1 VAE. Use image dimensions divisible by 32.
+  license: qwen-research
+  icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/qwen_image_logo.png
+  tags:
+    - qwen
+    - qwen-image
+    - text-to-image
+    - image-to-image
+    - image-editing
+    - text-rendering
+    - multilingual
+    - gguf
+    - quantized
+    - diffusion
+  last_checked: "2026-09-21"
+  overrides:
+    backend: stablediffusion-ggml
+    step: 25
+    # Qwen-Image 2.1 is not guidance-distilled: 6 matches the upstream
+    # stable-diffusion.cpp example.
+    cfg_scale: 6
+    options:
+      - diffusion_model
+      - llm_path:Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+      - llm_vision_path:mmproj/mmproj-Qwen3-VL-8B-Instruct-F16.gguf
+      - vae_path:qwen_image_2.1_vae_bf16.safetensors
+      - sampler:euler
+      - offload_params_to_cpu:true
+    parameters:
+      model: qwen_image_2.1-Q4_K.gguf
+  files:
+    - filename: qwen_image_2.1-Q4_K.gguf
+      sha256: 29f9c83c249ff0292fb2943fceddfa2319b446601866c82a4f8be062abea72c2
+      uri: huggingface://leejet/Qwen-Image-2.1-GGUF/qwen_image_2.1-Q4_K.gguf
+    - filename: Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+      sha256: 108e7ff92b78eefd3db4741885104acba514255c11b617d3c7b197a5f46efe89
+      uri: huggingface://unsloth/Qwen3-VL-8B-Instruct-GGUF/Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+    - filename: mmproj/mmproj-Qwen3-VL-8B-Instruct-F16.gguf
+      sha256: d406d03ebabefdef86a2c86bf0c1b65f9e046f7a81c218f25de4931b46a07fc4
+      uri: huggingface://unsloth/Qwen3-VL-8B-Instruct-GGUF/mmproj-F16.gguf
+    - filename: qwen_image_2.1_vae_bf16.safetensors
+      sha256: bb21f7473051e1ac368515dd3f2e15cd44d7a11748ee8823e1ddca3e4876b7c9
+      uri: https://huggingface.co/Comfy-Org/Qwen-Image-2.1/resolve/main/vae/qwen_image_2.1_vae_bf16.safetensors
+- name: qwen-image-2.1-q8_0-ggml
+  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
+  urls:
+    - https://huggingface.co/Qwen/Qwen-Image-2.1
+    - https://huggingface.co/leejet/Qwen-Image-2.1-GGUF
+  description: |
+    Qwen-Image 2.1 is the Qwen image generation foundation model, with strong prompt adherence and accurate text rendering (English and Chinese). It uses Qwen3-VL-8B as the text encoder and its own VAE, and supports both text-to-image and image editing: pass reference images to edit them.
+
+    This is the Q8_0 (8-bit) quantization (~7.7GB diffusion model) by leejet for stable-diffusion.cpp. The bundle also pulls the Qwen3-VL-8B-Instruct text encoder with its vision projector (used for image editing) and the Qwen-Image 2.1 VAE. Use image dimensions divisible by 32.
+  license: qwen-research
+  icon: https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-Image/qwen_image_logo.png
+  tags:
+    - qwen
+    - qwen-image
+    - text-to-image
+    - image-to-image
+    - image-editing
+    - text-rendering
+    - multilingual
+    - gguf
+    - quantized
+    - diffusion
+  last_checked: "2026-09-21"
+  overrides:
+    backend: stablediffusion-ggml
+    step: 25
+    # Qwen-Image 2.1 is not guidance-distilled: 6 matches the upstream
+    # stable-diffusion.cpp example.
+    cfg_scale: 6
+    options:
+      - diffusion_model
+      - llm_path:Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+      - llm_vision_path:mmproj/mmproj-Qwen3-VL-8B-Instruct-F16.gguf
+      - vae_path:qwen_image_2.1_vae_bf16.safetensors
+      - sampler:euler
+      - offload_params_to_cpu:true
+    parameters:
+      model: qwen_image_2.1-Q8_0.gguf
+  files:
+    - filename: qwen_image_2.1-Q8_0.gguf
+      sha256: f8b244b00937f0e444a40dbf7866460871b89b30142594973b6012d1b471dc0a
+      uri: huggingface://leejet/Qwen-Image-2.1-GGUF/qwen_image_2.1-Q8_0.gguf
+    - filename: Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+      sha256: 108e7ff92b78eefd3db4741885104acba514255c11b617d3c7b197a5f46efe89
+      uri: huggingface://unsloth/Qwen3-VL-8B-Instruct-GGUF/Qwen3-VL-8B-Instruct-Q4_K_M.gguf
+    - filename: mmproj/mmproj-Qwen3-VL-8B-Instruct-F16.gguf
+      sha256: d406d03ebabefdef86a2c86bf0c1b65f9e046f7a81c218f25de4931b46a07fc4
+      uri: huggingface://unsloth/Qwen3-VL-8B-Instruct-GGUF/mmproj-F16.gguf
+    - filename: qwen_image_2.1_vae_bf16.safetensors
+      sha256: bb21f7473051e1ac368515dd3f2e15cd44d7a11748ee8823e1ddca3e4876b7c9
+      uri: https://huggingface.co/Comfy-Org/Qwen-Image-2.1/resolve/main/vae/qwen_image_2.1_vae_bf16.safetensors
 - name: whisper-1
  url: github:mudler/LocalAI/gallery/whisper-base.yaml@master
  urls: