stablediffusion-ggml: run MiniMax-H3 video with references and add its gallery entries (#12668)

* feat(stablediffusion-ggml): run MiniMax-H3 video with reference conditioning

Bump stable-diffusion.cpp to 2988060 and follow its tiling rename
(tile_size_x/y -> tile_size_w/h, rel_size_x/y -> rel_size_w/h).

GenerateVideo now passes params["ref_image"] and the request audio to
the MiniMax-H3 Ref2VA inputs (ref_images, ref_audios). The audio must
be a WAV file. params["ref_video"] is refused, since the proto cannot
carry a frame sequence yet.

stable-diffusion.cpp cannot tell the FL2VA and REF2VA DiTs apart, and a
reference input on FL2VA renders a wrong video without an error. The
backend now reads the video_partition option, or the model file name,
and fails early on that mix. Keyframes plus references are refused too.

Add gallery entries for the FL2VA and REF2VA Q4_K_M sets. They do not
enable any bf16 dequantisation.

Assisted-by: Claude:claude-sonnet-5-5 go vet gofmt

* feat(stablediffusion-ggml): take several reference images, videos and audio clips

Phase 1 passed one reference image and one reference audio clip to
stable-diffusion.cpp and refused ref_video. Add repeated reference fields
to GenerateVideoRequest (ref_images, ref_videos, ref_audios) so the
MiniMax-H3 Ref2VA path can use as many references as upstream accepts. A
reference video is a directory of frames with an fps and an optional
paired audio clip.

The /video endpoint stages the new inputs through the same path as
start_image, end_image and audio. Frames of one reference video are
written to a private directory under zero-padded names so the backend can
read them in order. The input count and frame count per request are
bounded.

The backend merges the new fields with params.ref_image, params.ref_video
and audio, which keep working. The FL2VA partition refuses any reference
input, REF2VA accepts all of them, and keyframes stay exclusive with
references. Reference audio is still WAV only.

Assisted-by: Claude:claude-sonnet-5-5 [Bash]

* feat(gallery): offer MiniMax-H3 by partition, DiT layout and format

Name each MiniMax-H3 entry after its partition (fl2va or ref2va), DiT
layout (unpruned or pruned) and format. Add the vllm.cpp builds that its
model page lists: pruned Q8_0 (renders), pruned Q6_K, Q5_0, Q4_K, Q3_K and
Q2_K (load only), and the pruned FL2VA NVFP4 file (loads only). The
earlier names minimax-h3-fl2va-q4 and minimax-h3-ref2va-q4 stay as
aliases. The two stable-diffusion.cpp entries become
minimax-h3-fl2va-unpruned-q4-ggml and minimax-h3-ref2va-pruned-q4-ggml,
and the latter now lists video input.

Fix two bad sources. The stable-diffusion.cpp entries took the VAEs from
a path that does not exist in the GGUF repository; they now use the
Comfy-Org copies. The vllm.cpp entries paired the fp16 video VAE weights
with a config from a different directory; they now use the official
weights and config from FL2VA/video_vae/source/.

Drop video_dequant_bf16 from every vllm.cpp entry. Expanding a quantised
DiT to bf16 is the suspected cause of a host out-of-memory error, so the
quantised weights stay resident. The model page records its renders with
the expansion on, and each description says so.

Assisted-by: Claude:claude-sonnet-5-5 [Bash]

* feat(gallery): add pruned FL2VA Q4 MiniMax-H3 for sd.cpp

Add minimax-h3-fl2va-pruned-q4-ggml, the 11.4 GB pruned FL2VA
counterpart of the existing pruned REF2VA entry.

Assisted-by: Claude Code:claude-sonnet-5-5 [rc]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

backend/backend.proto

@@ -728,6 +728,19 @@ message GenerateVideoRequest {
   // PredictOptions.ModelIdentity for the full rationale. Empty means "no
   // identity supplied" and backends MUST skip the check.
   string ModelIdentity = 15;
+  // Reference conditioning (MiniMax-H3 Ref2VA). Additive: `audio` and
+  // params["ref_image"] keep working and count as one more reference each.
+  // Keyframes (start_image/end_image) and references are mutually exclusive.
+  repeated string ref_images = 16;          // Staged paths of reference images
+  repeated VideoReference ref_videos = 17;  // Reference videos given as frame directories
+  repeated string ref_audios = 18;          // Staged paths of reference audio clips (WAV)
+}
+
+// VideoReference is one reference video for GenerateVideoRequest.
+message VideoReference {
+  string frames_dir = 1;  // Staged directory of frame images, read in lexicographic file-name order
+  int32 fps = 2;          // Frame rate of the source frames; 0 means the backend default
+  string audio = 3;       // Optional staged WAV path paired with this video
 }
 
 // A named conditioning input. Media references are staged local paths by the

backend/go/stablediffusion-ggml/cpp/gosd.cpp

@@ -1148,6 +1148,87 @@ static uint8_t* load_and_resize_image(const char* path, int target_width, int ta
     return buf;
 }
 
+// Read a RIFF/WAVE file (PCM 16/24/32-bit or IEEE float 32-bit) into an
+// sd_audio_t holding planar float samples (all of channel 0, then channel 1,
+// ...), the layout sd.cpp expects. The caller owns out->data (free()).
+// Returns false and prints the reason on any unsupported or malformed input.
+static bool load_wav_audio(const char* path, sd_audio_t* out) {
+    *out = {0, 0, 0, nullptr};
+    FILE* f = fopen(path, "rb");
+    if (!f) { fprintf(stderr, "Failed to open reference audio '%s'\n", path); return false; }
+
+    auto rd16 = [](const uint8_t* b) { return (uint32_t)b[0] | ((uint32_t)b[1] << 8); };
+    auto rd32 = [](const uint8_t* b) { return (uint32_t)b[0] | ((uint32_t)b[1] << 8) | ((uint32_t)b[2] << 16) | ((uint32_t)b[3] << 24); };
+
+    uint8_t hdr[12];
+    if (fread(hdr, 1, 12, f) != 12 || memcmp(hdr, "RIFF", 4) != 0 || memcmp(hdr + 8, "WAVE", 4) != 0) {
+        fprintf(stderr, "Reference audio '%s' is not a WAV file (convert it to WAV first)\n", path);
+        fclose(f);
+        return false;
+    }
+
+    uint32_t format = 0, channels = 0, rate = 0, bits = 0;
+    std::vector<uint8_t> data;
+    bool have_data = false;
+    uint8_t ch[8];
+    while (fread(ch, 1, 8, f) == 8) {
+        uint32_t size = rd32(ch + 4);
+        if (memcmp(ch, "fmt ", 4) == 0 && size >= 16) {
+            std::vector<uint8_t> fmt(size);
+            if (fread(fmt.data(), 1, size, f) != size) break;
+            format = rd16(fmt.data());
+            channels = rd16(fmt.data() + 2);
+            rate = rd32(fmt.data() + 4);
+            bits = rd16(fmt.data() + 14);
+            // WAVE_FORMAT_EXTENSIBLE: the real format tag is the first two bytes of the sub-format GUID.
+            if (format == 0xFFFE && size >= 26) format = rd16(fmt.data() + 24);
+        } else if (memcmp(ch, "data", 4) == 0) {
+            data.resize(size);
+            size_t got = fread(data.data(), 1, size, f);
+            data.resize(got);
+            have_data = true;
+            break;
+        } else if (fseek(f, (long)size, SEEK_CUR) != 0) {
+            break;
+        }
+        if (size & 1) fseek(f, 1, SEEK_CUR);
+    }
+    fclose(f);
+
+    bool is_float = format == 3 && bits == 32;
+    bool is_pcm = format == 1 && (bits == 16 || bits == 24 || bits == 32);
+    if (!have_data || channels == 0 || rate == 0 || (!is_float && !is_pcm)) {
+        fprintf(stderr, "Unsupported or malformed WAV '%s' (format=%u bits=%u channels=%u rate=%u)\n",
+                path, format, bits, channels, rate);
+        return false;
+    }
+
+    size_t bytes_per_sample = bits / 8;
+    uint64_t frames = data.size() / (bytes_per_sample * channels);
+    if (frames == 0) { fprintf(stderr, "Reference audio '%s' has no samples\n", path); return false; }
+    float* planar = (float*)malloc((size_t)frames * channels * sizeof(float));
+    if (!planar) return false;
+    for (uint64_t i = 0; i < frames; i++) {
+        for (uint32_t c = 0; c < channels; c++) {
+            const uint8_t* b = data.data() + (i * channels + c) * bytes_per_sample;
+            float v;
+            if (is_float) {
+                memcpy(&v, b, 4);
+            } else if (bits == 16) {
+                v = (int16_t)rd16(b) / 32768.0f;
+            } else if (bits == 24) {
+                int32_t x = (int32_t)((uint32_t)b[0] << 8 | (uint32_t)b[1] << 16 | (uint32_t)b[2] << 24) >> 8;
+                v = x / 8388608.0f;
+            } else {
+                v = (int32_t)rd32(b) / 2147483648.0f;
+            }
+            planar[(uint64_t)c * frames + i] = v;
+        }
+    }
+    *out = {rate, channels, frames, planar};
+    return true;
+}
+
 // Write sd.cpp's audio buffer to a temp WAV file (IEEE float, interleaved).
 // sd_audio_t.data is planar (all channel 0 samples, then channel 1, etc.) — we
 // interleave on the fly so ffmpeg's standard wav demuxer can read it directly.
@@ -1365,7 +1446,7 @@ static int ffmpeg_mux_raw_to_mp4(sd_image_t* frames, int num_frames, int fps,
     return 0;
 }
 
-int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int fps, char *init_image, char *end_image) {
+int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int fps, char *init_image, char *end_image, char *ref_images, char *ref_videos, char *ref_audios) {
     if (!p) return 1;
     if (!dst || strlen(dst) == 0) {
         fprintf(stderr, "gen_video: dst is empty\n");
@@ -1415,6 +1496,96 @@ int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int
     p->init_image = init_img;
     p->end_image  = end_img;
 
+    // MiniMax-H3 Ref2VA references. Keyframes and references are mutually
+    // exclusive upstream; the Go side rejects the mix before getting here.
+    // The lists arrive flattened because purego cannot pass arrays of strings
+    // here: ref_images and ref_audios are newline-separated paths, ref_videos
+    // is one "frames_dir\x1ffps\x1faudio" record per line.
+    std::vector<uint8_t*> ref_bufs;                  // owned pixel buffers
+    std::vector<sd_image_t> ref_imgs;
+    std::vector<std::vector<sd_image_t>> ref_vid_frames;
+    std::vector<sd_ref_video_t> ref_vids;
+    std::vector<sd_audio_t> ref_auds;                // standalone clips, then video audio via ref_vids
+    std::vector<float*> ref_audio_bufs;              // owned audio buffers
+    auto free_inputs = [&]() {
+        if (init_buf) free(init_buf);
+        if (end_buf) free(end_buf);
+        for (auto* b : ref_bufs) free(b);
+        for (auto* b : ref_audio_bufs) free(b);
+    };
+    auto fail = [&]() { free_inputs(); std::free(p); return 1; };
+    auto split = [](const char* list, char sep) {
+        std::vector<std::string> out;
+        if (!list) return out;
+        std::string cur;
+        for (const char* c = list; *c; c++) {
+            if (*c == sep) { if (!cur.empty()) out.push_back(cur); cur.clear(); }
+            else cur += *c;
+        }
+        if (!cur.empty()) out.push_back(cur);
+        return out;
+    };
+    auto load_ref_image = [&](const std::string& path, sd_image_t* out) {
+        // Keep the native size: upstream keeps the aspect ratio of a reference
+        // image and only downscales it when it exceeds the output canvas.
+        int c = 0, w = 0, h = 0;
+        uint8_t* buf = stbi_load(path.c_str(), &w, &h, &c, 3);
+        if (!buf) { fprintf(stderr, "Failed to load reference image from '%s'\n", path.c_str()); return false; }
+        ref_bufs.push_back(buf);
+        *out = {(uint32_t)w, (uint32_t)h, 3, buf};
+        return true;
+    };
+    auto load_ref_audio = [&](const std::string& path, sd_audio_t* out) {
+        if (!load_wav_audio(path.c_str(), out)) return false;
+        ref_audio_bufs.push_back(out->data);
+        return true;
+    };
+
+    for (const auto& path : split(ref_images, '\n')) {
+        sd_image_t img;
+        if (!load_ref_image(path, &img)) return fail();
+        ref_imgs.push_back(img);
+    }
+    for (const auto& path : split(ref_audios, '\n')) {
+        sd_audio_t aud;
+        if (!load_ref_audio(path, &aud)) return fail();
+        ref_auds.push_back(aud);
+    }
+    auto video_records = split(ref_videos, '\n');
+    ref_vid_frames.resize(video_records.size());  // fixed size: frame pointers must stay valid
+    for (size_t v = 0; v < video_records.size(); v++) {
+        const std::string& rec = video_records[v];
+        size_t sep1 = rec.find('\x1f');
+        size_t sep2 = sep1 == std::string::npos ? std::string::npos : rec.find('\x1f', sep1 + 1);
+        if (sep2 == std::string::npos) { fprintf(stderr, "Malformed reference video record\n"); return fail(); }
+        std::string dir = rec.substr(0, sep1);
+        int vid_fps = atoi(rec.substr(sep1 + 1, sep2 - sep1 - 1).c_str());
+        std::string vid_audio = rec.substr(sep2 + 1);
+
+        std::vector<std::string> files;
+        std::error_code ec;
+        for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) {
+            if (entry.is_regular_file()) files.push_back(entry.path().string());
+        }
+        if (ec || files.empty()) {
+            fprintf(stderr, "Reference video directory '%s' is missing or has no frames\n", dir.c_str());
+            return fail();
+        }
+        std::sort(files.begin(), files.end());
+        ref_vid_frames[v].reserve(files.size());
+        for (const auto& file : files) {
+            sd_image_t frame;
+            if (!load_ref_image(file, &frame)) return fail();
+            ref_vid_frames[v].push_back(frame);
+        }
+        sd_audio_t vid_aud = {0, 0, 0, nullptr};
+        if (!vid_audio.empty() && !load_ref_audio(vid_audio, &vid_aud)) return fail();
+        ref_vids.push_back({ref_vid_frames[v].data(), (int)ref_vid_frames[v].size(), vid_fps, vid_aud});
+    }
+    if (!ref_imgs.empty()) { p->ref_images = ref_imgs.data(); p->ref_images_count = (int)ref_imgs.size(); }
+    if (!ref_vids.empty()) { p->ref_videos = ref_vids.data(); p->ref_videos_count = (int)ref_vids.size(); }
+    if (!ref_auds.empty()) { p->ref_audios = ref_auds.data(); p->ref_audios_count = (int)ref_auds.size(); }
+
     // Generate
     int num_frames_out = 0;
     int effective_fps = fps;
@@ -1426,8 +1597,7 @@ int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int
     if (!ok || !frames || num_frames_out == 0) {
         fprintf(stderr, "generate_video produced no frames\n");
         if (audio) free_sd_audio(audio);
-        if (init_buf) free(init_buf);
-        if (end_buf) free(end_buf);
+        free_inputs();
         return 1;
     }
 
@@ -1440,8 +1610,7 @@ int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int
     }
     free(frames);
     if (audio) free_sd_audio(audio);
-    if (init_buf) free(init_buf);
-    if (end_buf) free(end_buf);
+    free_inputs();
 
     if (rc == 0) {
         fprintf(stderr, "gen_video done: %s\n", dst);

backend/go/stablediffusion-ggml/cpp/gosd.h

@@ -32,7 +32,7 @@ void sd_vid_gen_params_set_prompts(sd_vid_gen_params_t *params, const char *prom
 void sd_vid_gen_params_set_dimensions(sd_vid_gen_params_t *params, int width, int height);
 void sd_vid_gen_params_set_seed(sd_vid_gen_params_t *params, int64_t seed);
 void sd_vid_gen_params_set_video_frames(sd_vid_gen_params_t *params, int n);
-int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int fps, char *init_image, char *end_image);
+int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int fps, char *init_image, char *end_image, char *ref_images, char *ref_videos, char *ref_audios);
 #ifdef __cplusplus
 }
 #endif

backend/go/stablediffusion-ggml/gosd.go

@@ -26,6 +26,8 @@ type SDGGML struct {
 	cfgScale     float32
 	customSigmas []float32
 	vaeTiling    vaeTiling
+	// videoPartition is "fl2va", "ref2va" or "" (unknown) for MiniMax-H3.
+	videoPartition string
 }
 
 var (
@@ -35,7 +37,7 @@ var (
 	UpscalerDestroy func(uintptr)
 	LoadModel       func(model, model_apth string, options []uintptr, threads int32, diff int) int
 	GenImage        func(params uintptr, steps int, dst string, cfgScale float32, srcImage string, strength float32, maskImage string, refImages []uintptr, refImagesCount int) int
-	GenVideo        func(params uintptr, steps int, dst string, cfgScale float32, fps int, initImage string, endImage string) int
+	GenVideo        func(params uintptr, steps int, dst string, cfgScale float32, fps int, initImage string, endImage string, refImages string, refVideos string, refAudios string) int
 
 	TilingParamsSetEnabled       func(params uintptr, enabled bool)
 	TilingParamsSetTileSizes     func(params uintptr, tileSizeX int, tileSizeY int)
@@ -146,6 +148,167 @@ func parseTileSize(value string) (int, int, bool) {
 	return x, y, true
 }
 
+// Video partitions of MiniMax-H3. The FL2VA DiT serves t2va and keyframe
+// conditioning; the REF2VA DiT serves reference conditioning.
+const (
+	partitionFL2VA  = "fl2va"
+	partitionRef2VA = "ref2va"
+)
+
+// detectVideoPartition tells the two MiniMax-H3 DiTs apart. stable-diffusion.cpp
+// has no key in the weights for it, so the loaded model cannot be asked.
+// Choice made here: the explicit backend option "video_partition:fl2va|ref2va"
+// wins; otherwise the model file name is matched for "fl2va" or "ref2va"
+// (the upstream file names are minimax_h3_fl2va-*.gguf and minimax_h3_ref2va-*.gguf).
+// An unrecognised name returns "" and the guard is then not applied.
+func detectVideoPartition(modelFile string, options []string) string {
+	for _, op := range options {
+		if name, value, ok := strings.Cut(op, ":"); ok && name == "video_partition" {
+			switch v := strings.ToLower(strings.TrimSpace(value)); v {
+			case partitionFL2VA, partitionRef2VA:
+				return v
+			}
+		}
+	}
+	// The DiT may also be given as an option path (diffusion_model_path:...).
+	candidates := []string{modelFile}
+	for _, op := range options {
+		if name, value, ok := strings.Cut(op, ":"); ok && name == "diffusion_model_path" {
+			candidates = append(candidates, value)
+		}
+	}
+	for _, c := range candidates {
+		base := strings.ToLower(filepath.Base(c))
+		switch {
+		case strings.Contains(base, partitionRef2VA):
+			return partitionRef2VA
+		case strings.Contains(base, partitionFL2VA):
+			return partitionFL2VA
+		}
+	}
+	return ""
+}
+
+// videoExtraParams holds the per-request knobs that have no proto field.
+type videoExtraParams struct {
+	refImage string
+	// refVideo is a directory of frames; it is read at the backend default fps.
+	refVideo string
+}
+
+// videoReferences is every reference input of a request, from the proto fields
+// and from the legacy params/audio spellings, in the order they are passed to
+// stable-diffusion.cpp.
+type videoReferences struct {
+	images []string
+	videos []*pb.VideoReference
+	audios []string
+}
+
+func (r videoReferences) any() bool {
+	return len(r.images) > 0 || len(r.videos) > 0 || len(r.audios) > 0
+}
+
+// parseVideoRequestParams reads GenerateVideoRequest.Params. Unknown keys are
+// ignored, as before.
+func parseVideoRequestParams(params map[string]string) (videoExtraParams, error) {
+	var extra videoExtraParams
+	extra.refImage = strings.TrimSpace(params["ref_image"])
+	extra.refVideo = strings.TrimSpace(params["ref_video"])
+	return extra, nil
+}
+
+// gatherVideoReferences merges the repeated proto fields with params.ref_image,
+// params.ref_video and the single `audio` field, which stay valid. It rejects
+// entries the flattened hand-off to C++ cannot carry.
+func gatherVideoReferences(opts *pb.GenerateVideoRequest, extra videoExtraParams) (videoReferences, error) {
+	var refs videoReferences
+	clean := func(what, v string) (string, error) {
+		v = strings.TrimSpace(v)
+		if v == "" {
+			return "", fmt.Errorf("stablediffusion-ggml: %s is empty", what)
+		}
+		if strings.ContainsAny(v, "\n\x1f") {
+			return "", fmt.Errorf("stablediffusion-ggml: %s contains a control character", what)
+		}
+		return v, nil
+	}
+	if extra.refImage != "" {
+		refs.images = append(refs.images, extra.refImage)
+	}
+	for i, v := range opts.GetRefImages() {
+		path, err := clean(fmt.Sprintf("ref_images[%d]", i), v)
+		if err != nil {
+			return refs, err
+		}
+		refs.images = append(refs.images, path)
+	}
+	if extra.refVideo != "" {
+		refs.videos = append(refs.videos, &pb.VideoReference{FramesDir: extra.refVideo})
+	}
+	for i, v := range opts.GetRefVideos() {
+		what := fmt.Sprintf("ref_videos[%d]", i)
+		dir, err := clean(what+".frames_dir", v.GetFramesDir())
+		if err != nil {
+			return refs, err
+		}
+		if v.GetFps() < 0 {
+			return refs, fmt.Errorf("stablediffusion-ggml: %s.fps must not be negative", what)
+		}
+		entry := &pb.VideoReference{FramesDir: dir, Fps: v.GetFps()}
+		if strings.TrimSpace(v.GetAudio()) != "" {
+			if entry.Audio, err = clean(what+".audio", v.GetAudio()); err != nil {
+				return refs, err
+			}
+		}
+		refs.videos = append(refs.videos, entry)
+	}
+	if opts.GetAudio() != "" {
+		path, err := clean("audio", opts.GetAudio())
+		if err != nil {
+			return refs, err
+		}
+		refs.audios = append(refs.audios, path)
+	}
+	for i, v := range opts.GetRefAudios() {
+		path, err := clean(fmt.Sprintf("ref_audios[%d]", i), v)
+		if err != nil {
+			return refs, err
+		}
+		refs.audios = append(refs.audios, path)
+	}
+	return refs, nil
+}
+
+// encodeVideoReferences flattens the references for the C++ side: paths are
+// newline-separated and each video is "frames_dir\x1ffps\x1faudio".
+func encodeVideoReferences(refs videoReferences) (images, videos, audios string) {
+	vids := make([]string, 0, len(refs.videos))
+	for _, v := range refs.videos {
+		vids = append(vids, fmt.Sprintf("%s\x1f%d\x1f%s", v.GetFramesDir(), v.GetFps(), v.GetAudio()))
+	}
+	return strings.Join(refs.images, "\n"), strings.Join(vids, "\n"), strings.Join(refs.audios, "\n")
+}
+
+// checkPartitionConditioning refuses conditioning the loaded MiniMax-H3 DiT
+// cannot serve. A reference input on the FL2VA checkpoint does not fail in
+// stable-diffusion.cpp; it renders a wrong video, so fail before generating.
+func checkPartitionConditioning(partition string, opts *pb.GenerateVideoRequest, refs videoReferences) error {
+	hasKeyframe := opts.GetStartImage() != "" || opts.GetEndImage() != ""
+	hasReference := refs.any()
+
+	if hasKeyframe && hasReference {
+		return fmt.Errorf("stablediffusion-ggml: keyframes (start_image/end_image) and reference " +
+			"conditioning (ref_images/ref_videos/ref_audios/audio/params.ref_image/params.ref_video) cannot be used together")
+	}
+	if partition == partitionFL2VA && hasReference {
+		return fmt.Errorf("stablediffusion-ggml: the FL2VA checkpoint serves t2va and keyframes only; " +
+			"reference conditioning (ref_images/ref_videos/ref_audios/audio) needs the REF2VA checkpoint. " +
+			"Use start_image for first-frame conditioning instead")
+	}
+	return nil
+}
+
 // Copied from Purego internal/strings
 // TODO: We should upstream sending []string
 func hasSuffix(s, suffix string) bool {
@@ -227,6 +390,7 @@ func (sd *SDGGML) Load(opts *pb.ModelOptions) error {
 	// Read from the unfiltered list: none of the tiling options name a path, so
 	// the resolution pass above neither rewrites nor drops them.
 	sd.vaeTiling = parseVAETiling(opts.Options)
+	sd.videoPartition = detectVideoPartition(modelFile, opts.Options)
 
 	ret := LoadModel(modelFile, modelPathC, options, opts.Threads, diffusionModel)
 	runtime.KeepAlive(keepAlive)
@@ -344,6 +508,19 @@ func (sd *SDGGML) GenerateVideo(opts *pb.GenerateVideoRequest) error {
 		cfg = 5.0
 	}
 
+	extra, err := parseVideoRequestParams(opts.Params)
+	if err != nil {
+		return err
+	}
+	refs, err := gatherVideoReferences(opts, extra)
+	if err != nil {
+		return err
+	}
+	if err := checkPartitionConditioning(sd.videoPartition, opts, refs); err != nil {
+		return err
+	}
+	refImages, refVideos, refAudios := encodeVideoReferences(refs)
+
 	// sd_vid_gen_params_new allocates; gen_video frees it after the generation call.
 	p := VidGenParamsNew()
 	VidGenParamsSetPrompts(p, opts.Prompt, opts.NegativePrompt)
@@ -354,7 +531,7 @@ func (sd *SDGGML) GenerateVideo(opts *pb.GenerateVideoRequest) error {
 	fmt.Fprintf(os.Stderr, "GenerateVideo: dst=%s size=%dx%d frames=%d fps=%d steps=%d cfg=%.2f\n",
 		dst, width, height, numFrames, fps, steps, cfg)
 
-	ret := GenVideo(p, steps, dst, cfg, fps, opts.StartImage, opts.EndImage)
+	ret := GenVideo(p, steps, dst, cfg, fps, opts.StartImage, opts.EndImage, refImages, refVideos, refAudios)
 	if ret != 0 {
 		return fmt.Errorf("video inference failed (code %d)", ret)
 	}

backend/go/stablediffusion-ggml/gosd_test.go

@@ -501,3 +501,112 @@ func upscaleTestRoot() string {
 	Expect(os.WriteFile(filepath.Join(root, "model.gguf"), []byte("model"), 0600)).To(Succeed())
 	return root
 }
+
+var _ = DescribeTable("detectVideoPartition",
+	func(modelFile string, options []string, want string) {
+		Expect(detectVideoPartition(modelFile, options)).To(Equal(want))
+	},
+	Entry("fl2va from the file name", "minimax_h3_fl2va-Q4_K_M.gguf", nil, partitionFL2VA),
+	Entry("ref2va from the file name", "minimax_h3_ref2va-Q4_K_M.gguf", nil, partitionRef2VA),
+	Entry("name match ignores case and directories", "/m/MiniMax_H3_REF2VA-Q4.gguf", nil, partitionRef2VA),
+	Entry("diffusion_model_path option", "x.gguf", []string{"diffusion_model_path:minimax_h3_fl2va-Q4_K_M.gguf"}, partitionFL2VA),
+	Entry("explicit option beats the name", "minimax_h3_fl2va-Q4_K_M.gguf", []string{"video_partition:ref2va"}, partitionRef2VA),
+	Entry("bad option value falls back to the name", "minimax_h3_ref2va.gguf", []string{"video_partition:nope"}, partitionRef2VA),
+	Entry("unknown model", "wan2.2.gguf", nil, ""),
+)
+
+var _ = Describe("MiniMax-H3 video conditioning", func() {
+	gather := func(opts *pb.GenerateVideoRequest, params map[string]string) videoReferences {
+		extra, err := parseVideoRequestParams(params)
+		Expect(err).ToNot(HaveOccurred())
+		refs, err := gatherVideoReferences(opts, extra)
+		Expect(err).ToNot(HaveOccurred())
+		return refs
+	}
+
+	It("maps ref_image and ref_video params and ignores unknown keys", func() {
+		extra, err := parseVideoRequestParams(map[string]string{"ref_image": " /tmp/a.png ", "ref_video": "/tmp/frames", "other": "1"})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(extra.refImage).To(Equal("/tmp/a.png"))
+		Expect(extra.refVideo).To(Equal("/tmp/frames"))
+	})
+
+	It("keeps the phase 1 spellings: params.ref_image and audio", func() {
+		refs := gather(&pb.GenerateVideoRequest{Audio: "a.wav"}, map[string]string{"ref_image": "a.png"})
+		Expect(refs.images).To(Equal([]string{"a.png"}))
+		Expect(refs.audios).To(Equal([]string{"a.wav"}))
+		Expect(refs.videos).To(BeEmpty())
+	})
+
+	It("appends the repeated proto fields after the legacy ones", func() {
+		refs := gather(&pb.GenerateVideoRequest{
+			Audio:     "a.wav",
+			RefImages: []string{"b.png", "c.png"},
+			RefAudios: []string{"d.wav"},
+			RefVideos: []*pb.VideoReference{{FramesDir: "/f1", Fps: 12, Audio: "v.wav"}, {FramesDir: "/f2"}},
+		}, map[string]string{"ref_image": "a.png"})
+		Expect(refs.images).To(Equal([]string{"a.png", "b.png", "c.png"}))
+		Expect(refs.audios).To(Equal([]string{"a.wav", "d.wav"}))
+		Expect(refs.videos).To(HaveLen(2))
+		Expect(refs.videos[0].GetFps()).To(Equal(int32(12)))
+	})
+
+	It("encodes the references for the C++ side", func() {
+		images, videos, audios := encodeVideoReferences(videoReferences{
+			images: []string{"a.png", "b.png"},
+			videos: []*pb.VideoReference{{FramesDir: "/f1", Fps: 12, Audio: "v.wav"}, {FramesDir: "/f2"}},
+			audios: []string{"a.wav"},
+		})
+		Expect(images).To(Equal("a.png\nb.png"))
+		Expect(videos).To(Equal("/f1\x1f12\x1fv.wav\n/f2\x1f0\x1f"))
+		Expect(audios).To(Equal("a.wav"))
+	})
+
+	DescribeTable("rejects references the hand-off cannot carry",
+		func(opts *pb.GenerateVideoRequest) {
+			_, err := gatherVideoReferences(opts, videoExtraParams{})
+			Expect(err).To(HaveOccurred())
+		},
+		Entry("empty image", &pb.GenerateVideoRequest{RefImages: []string{" "}}),
+		Entry("newline in an image path", &pb.GenerateVideoRequest{RefImages: []string{"a\nb.png"}}),
+		Entry("empty frames dir", &pb.GenerateVideoRequest{RefVideos: []*pb.VideoReference{{}}}),
+		Entry("negative fps", &pb.GenerateVideoRequest{RefVideos: []*pb.VideoReference{{FramesDir: "/f", Fps: -1}}}),
+		Entry("separator in the paired audio", &pb.GenerateVideoRequest{RefVideos: []*pb.VideoReference{{FramesDir: "/f", Audio: "a\x1fb"}}}),
+		Entry("empty reference audio", &pb.GenerateVideoRequest{RefAudios: []string{""}}),
+	)
+
+	It("accepts t2va and keyframes on FL2VA", func() {
+		Expect(checkPartitionConditioning(partitionFL2VA, &pb.GenerateVideoRequest{}, videoReferences{})).To(Succeed())
+		Expect(checkPartitionConditioning(partitionFL2VA, &pb.GenerateVideoRequest{StartImage: "a.png"}, videoReferences{})).To(Succeed())
+	})
+
+	DescribeTable("refuses every kind of reference on FL2VA",
+		func(refs videoReferences) {
+			err := checkPartitionConditioning(partitionFL2VA, &pb.GenerateVideoRequest{}, refs)
+			Expect(err).To(MatchError(ContainSubstring("REF2VA")))
+		},
+		Entry("image", videoReferences{images: []string{"a.png"}}),
+		Entry("video", videoReferences{videos: []*pb.VideoReference{{FramesDir: "/f"}}}),
+		Entry("audio", videoReferences{audios: []string{"a.wav"}}),
+	)
+
+	It("accepts every kind of reference on REF2VA and on an unknown partition", func() {
+		refs := videoReferences{
+			images: []string{"a.png", "b.png"},
+			videos: []*pb.VideoReference{{FramesDir: "/f", Fps: 24}},
+			audios: []string{"a.wav", "b.wav"},
+		}
+		Expect(checkPartitionConditioning(partitionRef2VA, &pb.GenerateVideoRequest{}, refs)).To(Succeed())
+		Expect(checkPartitionConditioning("", &pb.GenerateVideoRequest{}, refs)).To(Succeed())
+	})
+
+	DescribeTable("refuses keyframes mixed with references",
+		func(opts *pb.GenerateVideoRequest, refs videoReferences) {
+			err := checkPartitionConditioning(partitionRef2VA, opts, refs)
+			Expect(err).To(MatchError(ContainSubstring("cannot be used together")))
+		},
+		Entry("end image and a reference image", &pb.GenerateVideoRequest{EndImage: "e.png"}, videoReferences{images: []string{"a.png"}}),
+		Entry("start image and a reference video", &pb.GenerateVideoRequest{StartImage: "s.png"}, videoReferences{videos: []*pb.VideoReference{{FramesDir: "/f"}}}),
+		Entry("start image and reference audio", &pb.GenerateVideoRequest{StartImage: "s.png"}, videoReferences{audios: []string{"a.wav"}}),
+	)
+})

core/backend/video.go

@@ -2,6 +2,7 @@ package backend
 
 import (
 	"maps"
+	"slices"
 	"time"
 
 	"github.com/mudler/LocalAI/core/config"
@@ -12,6 +13,14 @@ import (
 
 // VideoGenerationOptions is the backend-neutral request passed to video generators.
 // Media fields contain staged local paths by the time they reach this layer.
+// VideoReferenceOptions is a reference video whose frames were staged into one
+// directory, named so that lexicographic order is frame order.
+type VideoReferenceOptions struct {
+	FramesDir string
+	FPS       int32
+	Audio     string
+}
+
 type VideoGenerationOptions struct {
 	Height         int32
 	Width          int32
@@ -20,6 +29,9 @@ type VideoGenerationOptions struct {
 	StartImage     string
 	EndImage       string
 	Audio          string
+	RefImages      []string
+	RefVideos      []VideoReferenceOptions
+	RefAudios      []string
 	Destination    string
 	NumFrames      int32
 	FPS            int32
@@ -29,6 +41,14 @@ type VideoGenerationOptions struct {
 	Params         map[string]string
 }
 
+func refVideosToProto(in []VideoReferenceOptions) []*proto.VideoReference {
+	out := make([]*proto.VideoReference, 0, len(in))
+	for _, r := range in {
+		out = append(out, &proto.VideoReference{FramesDir: r.FramesDir, Fps: r.FPS, Audio: r.Audio})
+	}
+	return out
+}
+
 func VideoGeneration(options VideoGenerationOptions, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (func() error, error) {
 	opts := ModelOptions(modelConfig, appConfig)
 	inferenceModel, err := loader.Load(opts...)
@@ -51,6 +71,9 @@ func VideoGeneration(options VideoGenerationOptions, loader *model.ModelLoader,
 				StartImage:     options.StartImage,
 				EndImage:       options.EndImage,
 				Audio:          options.Audio,
+				RefImages:      slices.Clone(options.RefImages),
+				RefVideos:      refVideosToProto(options.RefVideos),
+				RefAudios:      slices.Clone(options.RefAudios),
 				NumFrames:      options.NumFrames,
 				Fps:            options.FPS,
 				Seed:           options.Seed,
@@ -79,6 +102,9 @@ func VideoGeneration(options VideoGenerationOptions, loader *model.ModelLoader,
 			"has_start_image": options.StartImage != "",
 			"has_end_image":   options.EndImage != "",
 			"has_audio":       options.Audio != "",
+			"ref_images":      len(options.RefImages),
+			"ref_videos":      len(options.RefVideos),
+			"ref_audios":      len(options.RefAudios),
 		}
 
 		originalFn := fn

core/http/endpoints/localai/video.go

@@ -31,6 +31,16 @@ import (
 
 const maxVideoInputBytes = 128 << 20
 
+// Reference inputs are bounded so one request cannot stage an unbounded number
+// of files. A reference frame is a single image, so its limit is lower than the
+// one for audio or keyframes.
+const (
+	maxVideoRefsPerKind     = 16
+	maxVideoRefFrames       = 512
+	maxVideoRefFrameBytes   = 32 << 20
+	videoRefFrameNameFormat = "%06d"
+)
+
 func newVideoDownloadClient() *http.Client {
 	client := httpclient.NewWithTimeout(30*time.Second, httpclient.WithFollowRedirects())
 	checkRedirect := client.CheckRedirect
@@ -130,6 +140,107 @@ func stageVideoMediaWithLimit(ctx context.Context, directory, ref string, maxByt
 	return outputPath, nil
 }
 
+// stagedVideoRefs holds the staged reference inputs of one request.
+type stagedVideoRefs struct {
+	images []string
+	videos []backend.VideoReferenceOptions
+	audios []string
+	// paths are the files and directories to remove once the request is done.
+	paths []string
+}
+
+func (s *stagedVideoRefs) cleanup() {
+	for _, p := range s.paths {
+		_ = os.RemoveAll(p)
+	}
+}
+
+// stageVideoReferences stages ref_images, ref_videos and ref_audios with the
+// same path as start_image/end_image/audio (stageVideoMedia). A reference video
+// becomes a private directory of frames renamed to a zero-padded index, so the
+// backend can read them in order. On error everything staged so far is removed.
+func stageVideoReferences(ctx context.Context, directory string, input *schema.VideoRequest) (refs *stagedVideoRefs, err error) {
+	refs = &stagedVideoRefs{}
+	defer func() {
+		if err != nil {
+			refs.cleanup()
+			refs = nil
+		}
+	}()
+
+	if len(input.RefImages) > maxVideoRefsPerKind || len(input.RefVideos) > maxVideoRefsPerKind || len(input.RefAudios) > maxVideoRefsPerKind {
+		return refs, echo.NewHTTPError(http.StatusBadRequest,
+			fmt.Sprintf("too many reference inputs: at most %d of each kind", maxVideoRefsPerKind))
+	}
+	invalid := func(name string, cause error) error {
+		return echo.NewHTTPError(http.StatusBadRequest, fmt.Sprintf("invalid %s: %v", name, cause))
+	}
+
+	for i, ref := range input.RefImages {
+		if ref == "" {
+			return refs, invalid(fmt.Sprintf("ref_images[%d]", i), fmt.Errorf("empty reference"))
+		}
+		path, serr := stageVideoMedia(ctx, directory, ref)
+		if serr != nil {
+			return refs, invalid(fmt.Sprintf("ref_images[%d]", i), serr)
+		}
+		refs.paths = append(refs.paths, path)
+		refs.images = append(refs.images, path)
+	}
+	for i, ref := range input.RefAudios {
+		if ref == "" {
+			return refs, invalid(fmt.Sprintf("ref_audios[%d]", i), fmt.Errorf("empty reference"))
+		}
+		path, serr := stageVideoMedia(ctx, directory, ref)
+		if serr != nil {
+			return refs, invalid(fmt.Sprintf("ref_audios[%d]", i), serr)
+		}
+		refs.paths = append(refs.paths, path)
+		refs.audios = append(refs.audios, path)
+	}
+	for i, video := range input.RefVideos {
+		name := fmt.Sprintf("ref_videos[%d]", i)
+		if len(video.Frames) == 0 {
+			return refs, invalid(name, fmt.Errorf("frames is empty"))
+		}
+		if len(video.Frames) > maxVideoRefFrames {
+			return refs, invalid(name, fmt.Errorf("at most %d frames are accepted", maxVideoRefFrames))
+		}
+		if video.FPS < 0 {
+			return refs, invalid(name, fmt.Errorf("fps must not be negative"))
+		}
+		dir, merr := os.MkdirTemp(directory, "video-ref-frames-*")
+		if merr != nil {
+			return refs, fmt.Errorf("creating reference frame directory: %w", merr)
+		}
+		refs.paths = append(refs.paths, dir)
+		for j, frame := range video.Frames {
+			if frame == "" {
+				return refs, invalid(fmt.Sprintf("%s.frames[%d]", name, j), fmt.Errorf("empty reference"))
+			}
+			path, serr := stageVideoMediaWithLimit(ctx, dir, frame, maxVideoRefFrameBytes)
+			if serr != nil {
+				return refs, invalid(fmt.Sprintf("%s.frames[%d]", name, j), serr)
+			}
+			if rerr := os.Rename(path, filepath.Join(dir, fmt.Sprintf(videoRefFrameNameFormat, j))); rerr != nil {
+				_ = os.Remove(path)
+				return refs, fmt.Errorf("naming reference frame: %w", rerr)
+			}
+		}
+		entry := backend.VideoReferenceOptions{FramesDir: dir, FPS: video.FPS}
+		if video.Audio != "" {
+			path, serr := stageVideoMedia(ctx, directory, video.Audio)
+			if serr != nil {
+				return refs, invalid(name+".audio", serr)
+			}
+			refs.paths = append(refs.paths, path)
+			entry.Audio = path
+		}
+		refs.videos = append(refs.videos, entry)
+	}
+	return refs, nil
+}
+
 //
 
 /*
@@ -200,6 +311,12 @@ func VideoEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfi
 			defer func() { _ = os.Remove(audioSrc) }()
 		}
 
+		refs, err := stageVideoReferences(c.Request().Context(), appConfig.GeneratedContentDir, input)
+		if err != nil {
+			return err
+		}
+		defer refs.cleanup()
+
 		xlog.Debug("Parameter Config", "config", config)
 
 		switch config.Backend {
@@ -274,6 +391,9 @@ func VideoEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfi
 				StartImage:     src,
 				EndImage:       endSrc,
 				Audio:          audioSrc,
+				RefImages:      refs.images,
+				RefVideos:      refs.videos,
+				RefAudios:      refs.audios,
 				Destination:    output,
 				NumFrames:      input.NumFrames,
 				FPS:            input.FPS,

core/http/endpoints/localai/video_internal_test.go

@@ -3,9 +3,14 @@ package localai
 import (
 	"context"
 	"encoding/base64"
+	"errors"
+	"fmt"
+	"net/http"
 	"os"
 	"path/filepath"
 
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/schema"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 )
@@ -66,3 +71,74 @@ var _ = Describe("video media staging", func() {
 		Expect(err).To(MatchError(ContainSubstring("base64 payload")))
 	})
 })
+
+var _ = Describe("video reference staging", func() {
+	enc := func(s string) string { return base64.StdEncoding.EncodeToString([]byte(s)) }
+
+	It("stages images, audio and a frame directory in order", func() {
+		directory := GinkgoT().TempDir()
+		input := &schema.VideoRequest{
+			RefImages: []string{enc("img0"), enc("img1")},
+			RefAudios: []string{enc("wav0")},
+			RefVideos: []schema.VideoReference{{
+				Frames: []string{enc("f0"), enc("f1"), enc("f2")},
+				FPS:    12,
+				Audio:  enc("vwav"),
+			}},
+		}
+
+		refs, err := stageVideoReferences(context.Background(), directory, input)
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(refs.images).To(HaveLen(2))
+		Expect(os.ReadFile(refs.images[1])).To(Equal([]byte("img1")))
+		Expect(refs.audios).To(HaveLen(1))
+		Expect(refs.videos).To(HaveLen(1))
+		Expect(refs.videos[0].FPS).To(Equal(int32(12)))
+		Expect(os.ReadFile(refs.videos[0].Audio)).To(Equal([]byte("vwav")))
+		entries, err := os.ReadDir(refs.videos[0].FramesDir)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(entries).To(HaveLen(3))
+		for i, want := range []string{"f0", "f1", "f2"} {
+			Expect(entries[i].Name()).To(Equal(fmt.Sprintf("%06d", i)))
+			Expect(os.ReadFile(filepath.Join(refs.videos[0].FramesDir, entries[i].Name()))).To(Equal([]byte(want)))
+		}
+
+		refs.cleanup()
+		left, err := os.ReadDir(directory)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(left).To(BeEmpty())
+	})
+
+	It("stages nothing when no reference is given", func() {
+		refs, err := stageVideoReferences(context.Background(), GinkgoT().TempDir(), &schema.VideoRequest{})
+		Expect(err).NotTo(HaveOccurred())
+		Expect(refs.images).To(BeEmpty())
+		Expect(refs.videos).To(BeEmpty())
+		Expect(refs.audios).To(BeEmpty())
+	})
+
+	DescribeTable("rejects bad input and leaves nothing behind",
+		func(input *schema.VideoRequest, want string) {
+			directory := GinkgoT().TempDir()
+
+			refs, err := stageVideoReferences(context.Background(), directory, input)
+
+			Expect(refs).To(BeNil())
+			var httpErr *echo.HTTPError
+			Expect(errors.As(err, &httpErr)).To(BeTrue())
+			Expect(httpErr.Code).To(Equal(http.StatusBadRequest))
+			Expect(fmt.Sprint(httpErr.Message)).To(ContainSubstring(want))
+			left, readErr := os.ReadDir(directory)
+			Expect(readErr).NotTo(HaveOccurred())
+			Expect(left).To(BeEmpty())
+		},
+		Entry("empty image", &schema.VideoRequest{RefImages: []string{enc("ok"), ""}}, "ref_images[1]"),
+		Entry("bad base64 audio", &schema.VideoRequest{RefAudios: []string{"not%%%base64"}}, "ref_audios[0]"),
+		Entry("video without frames", &schema.VideoRequest{RefVideos: []schema.VideoReference{{}}}, "ref_videos[0]"),
+		Entry("negative fps", &schema.VideoRequest{RefVideos: []schema.VideoReference{{Frames: []string{enc("f")}, FPS: -1}}}, "fps"),
+		Entry("bad frame after good ones", &schema.VideoRequest{RefVideos: []schema.VideoReference{{Frames: []string{enc("f"), "not%%%base64"}}}}, "frames[1]"),
+		Entry("bad paired audio", &schema.VideoRequest{RefVideos: []schema.VideoReference{{Frames: []string{enc("f")}, Audio: "not%%%base64"}}}, "ref_videos[0].audio"),
+		Entry("too many images", &schema.VideoRequest{RefImages: make([]string, maxVideoRefsPerKind+1)}, "too many"),
+	)
+})

core/schema/localai.go

@@ -49,6 +49,13 @@ type GalleryResponse struct {
 	EstimatedSizeDisplay string `json:"estimated_size_display,omitempty"`
 }
 
+// VideoReference is one reference video, given as an ordered list of frames.
+type VideoReference struct {
+	Frames []string `json:"frames" yaml:"frames"`                   // ordered frame images: URL, base64 or data URI
+	FPS    int32    `json:"fps,omitempty" yaml:"fps,omitempty"`     // frame rate of the source frames; 0 means the backend default
+	Audio  string   `json:"audio,omitempty" yaml:"audio,omitempty"` // optional audio paired with this video
+}
+
 type VideoRequest struct {
 	BasicModelRequest
 	Prompt         string            `json:"prompt" yaml:"prompt"`                                       // text description of the video to generate
@@ -56,6 +63,9 @@ type VideoRequest struct {
 	StartImage     string            `json:"start_image" yaml:"start_image"`                             // URL or base64 of the first frame
 	EndImage       string            `json:"end_image" yaml:"end_image"`                                 // URL or base64 of the last frame
 	Audio          string            `json:"audio,omitempty" yaml:"audio,omitempty"`                     // URL or base64 audio for audio-conditioned generation
+	RefImages      []string          `json:"ref_images,omitempty" yaml:"ref_images,omitempty"`           // reference images (URL, base64 or data URI); exclusive with start_image/end_image
+	RefVideos      []VideoReference  `json:"ref_videos,omitempty" yaml:"ref_videos,omitempty"`           // reference videos as frame lists; exclusive with start_image/end_image
+	RefAudios      []string          `json:"ref_audios,omitempty" yaml:"ref_audios,omitempty"`           // reference audio clips (WAV); exclusive with start_image/end_image
 	Width          int32             `json:"width" yaml:"width"`                                         // output width in pixels
 	Height         int32             `json:"height" yaml:"height"`                                       // output height in pixels
 	NumFrames      int32             `json:"num_frames" yaml:"num_frames"`                               // total number of frames to generate
@@ -74,15 +84,15 @@ type VideoRequest struct {
 // (TRELLIS.2 has no text-prompt path); the response is a binary glTF (.glb).
 type Model3DRequest struct {
 	BasicModelRequest
-	Image          string            `json:"image" yaml:"image"`                                           // conditioning image: URL, base64, or data URI (required)
-	Seed           int32             `json:"seed,omitempty" yaml:"seed,omitempty"`                         // random seed; <=0 picks a random seed
-	Step           int32             `json:"step,omitempty" yaml:"step,omitempty"`                         // flow sampling steps (backend default 12)
-	CFGScale       float32           `json:"cfg_scale,omitempty" yaml:"cfg_scale,omitempty"`               // classifier-free guidance scale (backend default 7.5)
-	TextureSteps   int32             `json:"texture_steps,omitempty" yaml:"texture_steps,omitempty"`       // texture flow sampling steps (backend default 12)
-	Quality        string            `json:"quality,omitempty" yaml:"quality,omitempty"`                   // mesh pipeline: auto|coarse|512|1024
-	Background     string            `json:"background,omitempty" yaml:"background,omitempty"`             // background handling: auto|keep|black|white
-	ResponseFormat string            `json:"response_format,omitempty" yaml:"response_format,omitempty"`   // output format (url or b64_json)
-	Params         map[string]string `json:"params,omitempty" yaml:"params,omitempty"`                     // backend-specific generation parameters
+	Image          string            `json:"image" yaml:"image"`                                         // conditioning image: URL, base64, or data URI (required)
+	Seed           int32             `json:"seed,omitempty" yaml:"seed,omitempty"`                       // random seed; <=0 picks a random seed
+	Step           int32             `json:"step,omitempty" yaml:"step,omitempty"`                       // flow sampling steps (backend default 12)
+	CFGScale       float32           `json:"cfg_scale,omitempty" yaml:"cfg_scale,omitempty"`             // classifier-free guidance scale (backend default 7.5)
+	TextureSteps   int32             `json:"texture_steps,omitempty" yaml:"texture_steps,omitempty"`     // texture flow sampling steps (backend default 12)
+	Quality        string            `json:"quality,omitempty" yaml:"quality,omitempty"`                 // mesh pipeline: auto|coarse|512|1024
+	Background     string            `json:"background,omitempty" yaml:"background,omitempty"`           // background handling: auto|keep|black|white
+	ResponseFormat string            `json:"response_format,omitempty" yaml:"response_format,omitempty"` // output format (url or b64_json)
+	Params         map[string]string `json:"params,omitempty" yaml:"params,omitempty"`                   // backend-specific generation parameters
 }
 
 // @Description Print-remesh an existing trellis2.cpp GLB. The multipart mesh

docs/content/features/video-generation.md

@@ -50,6 +50,9 @@ The request body is JSON with the following fields:
 | `start_image`     | `string` | No       |         | Starting image as base64 string or URL                   |
 | `end_image`       | `string` | No       |         | Ending image for guided generation                       |
 | `audio`           | `string` | No       |         | Audio conditioning as base64, a data URI, or URL          |
+| `ref_images`      | `string[]` | No     |         | Reference images as base64, data URIs or URLs. Cannot be combined with `start_image` or `end_image` |
+| `ref_videos`      | `object[]` | No     |         | Reference videos: `{"frames": [...], "fps": 24, "audio": "..."}`. `frames` is an ordered list of images, `fps` and `audio` are optional |
+| `ref_audios`      | `string[]` | No     |         | Reference audio clips. The `stablediffusion` backend reads WAV only |
 | `width`           | `int`    | No       | backend | Video width in pixels; omit it to get the model's own default canvas |
 | `height`          | `int`    | No       | backend | Video height in pixels; omit it to get the model's own default canvas |
 | `num_frames`      | `int`    | No       |         | Number of frames                                         |
@@ -448,6 +451,10 @@ The `/video` request's `params` object accepts string values. Unknown keys are r
 
 `negative_prompt`, `cfg_scale` and `fps` have no MiniMax-H3 equivalent: H3 has no negative prompt or CFG scale, and it renders at a fixed frame rate that the audio track is synchronised to. Setting them is logged and ignored rather than silently honoured.
 
+### Reference inputs with the `stablediffusion` backend
+
+A MiniMax-H3 Ref2VA checkpoint on the `stablediffusion` backend takes several reference images, reference videos (each with an fps and an optional paired audio clip) and reference audio clips through `ref_images`, `ref_videos` and `ref_audios`. The single `audio` field counts as one more reference audio. Reference inputs need the Ref2VA checkpoint and cannot be mixed with `start_image` or `end_image`. A request accepts at most 16 entries of each kind and at most 512 frames per reference video. Reference audio must be WAV.
+
 ### MiniMax-H3 troubleshooting
 
 - **`ffmpeg not found`**: install ffmpeg on the host or set `options: [ffmpeg:<path>]`. The frames and WAV are already rendered; only the mux failed.