mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-17 08:50:57 -04:00
feat(vllm-cpp): serve MiniMax-H3 video+audio generation (#11424)
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
mudler's LocalAI [bot]andEttore Di Giacinto authored and GitHub committed
1 parent
f31c3bbf1b
commit
a0f50b2af2
14 files changed
+1579
-27
No files matched your search
@@ -231,7 +231,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
|
||||
|
||||
| Backend | What it does |
|
||||
|---------|-------------|
|
||||
| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
|
||||
| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan. Also serves MiniMax-H3 joint video+audio generation |
|
||||
| [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
|
||||
| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
|
||||
| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
|
||||
|
||||
@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || e
|
||||
|
||||
# vllm.cpp version
|
||||
VLLM_CPP_REPO?=https://github.com/mudler/vllm.cpp
|
||||
VLLM_CPP_VERSION?=0757cac231ecd571a83c4fd2f50805c9251fc225
|
||||
VLLM_CPP_VERSION?=2b08dd246e04b3f0a4bf1f276170fd28004ced01
|
||||
|
||||
# MLX GEMM provider (darwin/metal only; see the metal branch below for why).
|
||||
# Consumed as the prebuilt pip wheel: building MLX from source needs `xcrun
|
||||
@@ -106,13 +106,24 @@ else
|
||||
LIB=libvllm.so
|
||||
endif
|
||||
|
||||
sources/vllm.cpp:
|
||||
# patches/ carries fixes the pinned engine SHA does not have yet. `git apply`
|
||||
# is deliberately unguarded: a patch that no longer applies must FAIL the clone
|
||||
# loudly, because the alternative is a pin that silently ships without a fix it
|
||||
# is documented to carry. Each patch header says which pin retires it.
|
||||
VLLM_CPP_PATCHES=$(wildcard patches/*.patch)
|
||||
|
||||
sources/vllm.cpp: $(VLLM_CPP_PATCHES)
|
||||
rm -rf sources/vllm.cpp
|
||||
mkdir -p sources/vllm.cpp
|
||||
cd sources/vllm.cpp && \
|
||||
git init && \
|
||||
git remote add origin $(VLLM_CPP_REPO) && \
|
||||
git fetch --depth 1 origin $(VLLM_CPP_VERSION) && \
|
||||
git checkout FETCH_HEAD
|
||||
git checkout FETCH_HEAD && \
|
||||
for p in $(VLLM_CPP_PATCHES); do \
|
||||
echo "==> applying $$p"; \
|
||||
git apply ../../$$p || exit 1; \
|
||||
done
|
||||
|
||||
ifeq ($(MLX_ENABLED),1)
|
||||
# A stamp FILE, not a phony target: a phony prerequisite is always "newer" than
|
||||
@@ -165,7 +176,7 @@ $(LIB): sources/vllm.cpp $(MLX_STAMP)
|
||||
cmake --build . --config Release -j$(JOBS) --target vllm_shared
|
||||
cp -fL build/$(LIB) ./$(LIB)
|
||||
|
||||
vllm-cpp: main.go govllmcpp.go backend.go options.go $(LIB)
|
||||
vllm-cpp: main.go govllmcpp.go backend.go chat.go options.go video.go $(LIB)
|
||||
CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o vllm-cpp ./
|
||||
|
||||
package: vllm-cpp
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
# vllm-cpp backend
|
||||
|
||||
LocalAI text-generation backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
|
||||
LocalAI backend for [vllm.cpp](https://github.com/mudler/vllm.cpp),
|
||||
the LocalAI-team C++20 port of vLLM (paged KV cache, continuous batching,
|
||||
safetensors + GGUF loading, CUDA / CPU / Metal / Vulkan) with no Python at
|
||||
inference time.
|
||||
|
||||
It serves two things: text generation, and MiniMax-H3 joint video+audio
|
||||
generation.
|
||||
|
||||
The backend dlopens the engine's stable C ABI (`libvllm`, `include/vllm.h`,
|
||||
ABI v10) through purego:
|
||||
ABI v16) through purego:
|
||||
|
||||
- `Load` -> `vllm_engine_load`: accepts a `.gguf` file or a HF-style model
|
||||
directory (`config.json` + safetensors). `context_size` maps to
|
||||
@@ -29,6 +32,12 @@ ABI v10) through purego:
|
||||
LocalAI's Go-side grammar-constrained tool calling; JSON-schema / regex /
|
||||
choice constraints are also exposed by the ABI.
|
||||
|
||||
`patches/` carries fixes the pinned engine SHA does not have yet, applied to
|
||||
the clone the same way `longcat-video` patches its upstream. `git apply` is
|
||||
unguarded on purpose: a patch that stops applying must fail the clone loudly
|
||||
rather than leave a pin silently missing a fix it is documented to carry. Each
|
||||
patch header says what retires it.
|
||||
|
||||
The struct mirrors in `govllmcpp.go` are hand-written against one ABI version,
|
||||
and the engine refuses to load against any other. Moving `VLLM_CPP_VERSION` in
|
||||
the Makefile therefore means updating `abiVersion` plus the mirrors (and their
|
||||
@@ -47,6 +56,68 @@ options:
|
||||
- max_num_seqs:16
|
||||
```
|
||||
|
||||
## MiniMax-H3 video+audio generation
|
||||
|
||||
`GenerateVideo` -> `vllm_video_generate` (ABI v12). H3 renders picture and sound
|
||||
together, so the output MP4 carries a real AAC track.
|
||||
|
||||
The video engine is a SECOND handle (`vllm_video_engine`), not a mode of the
|
||||
text one, because H3 is a checkpoint SET rather than a model directory: the DiT,
|
||||
the text encoder and two VAEs are separate artifacts, and vllm.cpp has the two
|
||||
loaders refuse each other's checkpoints. `Load` takes the video branch when the
|
||||
model config carries any of the video options below; `parameters.model` is the
|
||||
DiT and everything else is named in `options:`.
|
||||
|
||||
```yaml
|
||||
name: minimax-h3-fl2va-q4
|
||||
backend: vllm-cpp
|
||||
cuda: true
|
||||
known_usecases: [video]
|
||||
parameters:
|
||||
model: minimax-h3/MiniMax-H3-FL2VA-Q4_K_M.gguf
|
||||
options:
|
||||
- video_encoder:minimax-h3/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
|
||||
- video_tokenizer:minimax-h3/tokenizer.json
|
||||
- video_vae:minimax-h3/video_vae.safetensors
|
||||
- video_vae_config:minimax-h3/video_vae_config.json
|
||||
- audio_vae:minimax-h3/audio_vae.safetensors
|
||||
- audio_vae_config:minimax-h3/audio_vae_config.json
|
||||
- video_partition:fl2va
|
||||
- video_device:cuda
|
||||
- video_dequant_bf16:true
|
||||
- video_width:1344
|
||||
- video_height:768
|
||||
- video_num_frames:124
|
||||
```
|
||||
|
||||
Three things are worth knowing before touching this path.
|
||||
|
||||
**The partition is declared, not detected, and a mismatch does not fail
|
||||
cleanly.** The FL2VA DiT serves `t2va` and `fl2va`; `ref2va` is a different
|
||||
checkpoint. The community GGUF/NVFP4 quantisations strip the release metadata
|
||||
and the two DiTs are byte-structurally identical, so the engine refuses every
|
||||
generate until `video_partition` says which one it has. Handing reference
|
||||
conditioning to an FL2VA DiT renders for hours and returns a coloured lattice
|
||||
over the frame, so `checkPartitionConditioning` refuses that combination here,
|
||||
before the engine is called.
|
||||
|
||||
**ffmpeg comes from the host.** libvllm writes the frames and the WAV and
|
||||
COMPOSES the mux argv, then spawns nothing — that process boundary is upstream's
|
||||
decision. `muxVideo` takes the composed argv, substitutes `argv[0]` with the
|
||||
resolved binary and execs it; the backend image is `FROM scratch` and carries no
|
||||
ffmpeg, the same arrangement `vibevoice-cpp` uses for transcoding. ffmpeg also
|
||||
converts a `start_image`/`end_image` upload into the binary PPM at the exact
|
||||
output canvas the engine requires, since libvllm vendors neither an image codec
|
||||
nor a resampler.
|
||||
|
||||
**It is slow.** Roughly 176 s per denoise step at 1344x768 on a 20-SM device, so
|
||||
the 50-step default is hours. Nothing here imposes a deadline.
|
||||
|
||||
Geometry mirrors the engine so the two agree: the canvas is truncated onto a
|
||||
32-pixel grid, the frame count sits on the 17n+5 grid, and an unspecified canvas
|
||||
with a keyframe is derived from that image's aspect on a 768-pixel short edge
|
||||
(`MiniMaxH3ResolveShape`, `minimax_h3_planner.cpp`).
|
||||
|
||||
## Apple Silicon: the MLX GEMM provider (ON by default, gated to prefill)
|
||||
|
||||
`BUILD_TYPE=metal` builds vllm.cpp's MLX provider for the dense GEMM
|
||||
|
||||
@@ -28,7 +28,12 @@ type VllmCpp struct {
|
||||
base.Base
|
||||
|
||||
engine uintptr
|
||||
opts loadOptions
|
||||
// videoEngine is the MiniMax-H3 handle (ABI v12). It is deliberately a
|
||||
// SECOND handle, not a mode of the first: H3 is a checkpoint set rather
|
||||
// than a model directory, and vllm.cpp has the two loaders refuse each
|
||||
// other's checkpoints. Exactly one of the two is ever non-zero.
|
||||
videoEngine uintptr
|
||||
opts loadOptions
|
||||
}
|
||||
|
||||
// Stream registry: the per-request bridge between the C token callback and
|
||||
@@ -109,6 +114,14 @@ func (v *VllmCpp) Load(opts *pb.ModelOptions) error {
|
||||
|
||||
v.opts = parseOptions(opts)
|
||||
|
||||
// MiniMax-H3 is a checkpoint SET behind its own engine handle, so the
|
||||
// branch is taken before any text-engine knob is resolved. The two loaders
|
||||
// refuse each other's checkpoints, which is why this is decided from the
|
||||
// config rather than probed.
|
||||
if v.opts.video.engaged() {
|
||||
return v.loadVideo(opts, model)
|
||||
}
|
||||
|
||||
// A DFlash draft is a second checkpoint the engine opens by path, and the
|
||||
// engine never downloads one. Resolve it against LocalAI's models directory
|
||||
// now so a repo-id spelling works, and so a missing draft fails here with an
|
||||
@@ -194,6 +207,10 @@ func (v *VllmCpp) Free() error {
|
||||
vllmEngineFree(v.engine)
|
||||
v.engine = 0
|
||||
}
|
||||
if v.videoEngine != 0 {
|
||||
vllmVideoEngineFree(v.videoEngine)
|
||||
v.videoEngine = 0
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
package main
|
||||
|
||||
// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v10).
|
||||
// purego bindings for the vllm.cpp stable C ABI (include/vllm.h, ABI v16).
|
||||
//
|
||||
// The structs below are hand-mirrored PODs of the C declarations, with
|
||||
// explicit padding so the Go layout matches the C layout on linux/darwin
|
||||
@@ -21,7 +21,7 @@ import (
|
||||
// the header of the VLLM_CPP_VERSION pinned in the Makefile: the build checks
|
||||
// the two against each other, because a mismatch is only caught at runtime by
|
||||
// registerLib, where it takes the backend down on every load (issue #11379).
|
||||
const abiVersion = 10
|
||||
const abiVersion = 17
|
||||
|
||||
// The ABI's tri-state toggles (enable_prefix_caching ABI v7,
|
||||
// enable_jump_forward ABI v10) share one encoding: 0 is NOT "off", it is
|
||||
@@ -70,7 +70,15 @@ type cModelParams struct {
|
||||
SchedulingPolicy uintptr // const char*; NULL = "fcfs" (ABI v9)
|
||||
KVTransferConfig uintptr // const char* JSON; NULL = no connector (ABI v9)
|
||||
EnableJumpForward int32 // tri-state 0/1/2 (ABI v10)
|
||||
_ [4]byte // trailing pad to the struct's 8-byte alignment
|
||||
// v14/v16 tail. LocalAI sets none of these (0 is "auto" for the device and
|
||||
// "unset" for both sizing knobs, i.e. the pre-v14 engine byte for byte), but
|
||||
// the fields MUST be mirrored: the C side reads sizeof(vllm_model_params)
|
||||
// bytes off the pointer we hand it, so a Go struct that stopped at
|
||||
// EnableJumpForward would have vllm_engine_load read 24 bytes past our
|
||||
// allocation and size the KV pool from whatever sat there.
|
||||
Device int32 // 0 auto, 1 cpu, 2 cuda (ABI v14)
|
||||
GPUMemoryUtil float64 // 0 => 0.92 (ABI v16)
|
||||
KVCacheMemoryBytes int64 // 0 => unset (ABI v16)
|
||||
}
|
||||
|
||||
// cSamplingParams mirrors vllm_sampling_params (structured fields included).
|
||||
@@ -117,6 +125,79 @@ type cCompletion struct {
|
||||
CompletionTokens int32
|
||||
}
|
||||
|
||||
// ── Video+audio generation (ABI v12, MiniMax-H3) ────────────────────────────
|
||||
//
|
||||
// A video engine is a SEPARATE handle from vllm_engine: H3 is a checkpoint SET
|
||||
// (DiT + text encoder + two VAEs), not one model directory, and the two loaders
|
||||
// refuse each other's checkpoints on purpose. Offsets are asserted in
|
||||
// video_test.go the same way the text PODs are in vllmcpp_test.go.
|
||||
|
||||
// cVideoModelParams mirrors vllm_video_model_params. Nine pointers then three
|
||||
// int32s, so only the trailing pad is implicit.
|
||||
type cVideoModelParams struct {
|
||||
DitPath uintptr // const char*
|
||||
EncoderPath uintptr // const char*
|
||||
TokenizerPath uintptr // const char*
|
||||
VideoVaePath uintptr // const char*
|
||||
VideoVaeConfigPath uintptr // const char*
|
||||
AudioVaePath uintptr // const char*
|
||||
AudioVaeConfigPath uintptr // const char*
|
||||
PromptEmbedsPath uintptr // const char*
|
||||
Partition uintptr // const char*; "fl2va" | "ref2va", REQUIRED
|
||||
Device int32 // 0 cpu, 1 cuda
|
||||
DequantBf16 int32 // 0 keep-quant, 1 dequant/stream bf16
|
||||
Fp4Resident int32 // NVFP4+cuda: keep FP4 packed, Marlin W4A16
|
||||
_ [4]byte // trailing pad to the struct's 8-byte alignment
|
||||
}
|
||||
|
||||
// cVideoParams mirrors vllm_video_params. `width`/`height` and `num_frames`/
|
||||
// `steps` pair up into 8-byte slots; the uint64 seed forces the alignment after
|
||||
// them, and the float noise_aug leaves a pad before output_dir.
|
||||
type cVideoParams struct {
|
||||
Prompt uintptr // const char*
|
||||
Width int32
|
||||
Height int32
|
||||
NumFrames int32 // <= 1 => per-task default (124 for t2va/fl2va)
|
||||
Steps int32 // <= 0 => the H3 default (50)
|
||||
Seed uint64
|
||||
HasSeed int32
|
||||
_ [4]byte
|
||||
FirstFrame uintptr // const char*; fl2va keyframe, binary PPM (P6)
|
||||
LastFrame uintptr // const char*
|
||||
RefImage uintptr // const char*; ref2va only
|
||||
RefVideo uintptr // const char*; ref2va only, a frame_%06d.ppm DIRECTORY
|
||||
RefAudio uintptr // const char*; ref2va only, 16-bit PCM WAV
|
||||
NoiseAug float32 // <= 0 => 1.0
|
||||
_ [4]byte
|
||||
OutputDir uintptr // const char*; REQUIRED
|
||||
}
|
||||
|
||||
// cVideoResult mirrors vllm_video_result. Every member is library-allocated and
|
||||
// released together by vllm_video_result_free.
|
||||
type cVideoResult struct {
|
||||
FrameDir uintptr // char*, holds frame_%06d.ppm
|
||||
AudioPath uintptr // char*, 16-bit PCM WAV
|
||||
FrameCount int32
|
||||
Width int32
|
||||
Height int32
|
||||
Fps int32
|
||||
SampleRate int32
|
||||
_ [4]byte
|
||||
MuxArgv uintptr // char**, NULL-terminated at MuxArgc
|
||||
MuxArgc int32
|
||||
_ [4]byte
|
||||
}
|
||||
|
||||
// cVideoMuxParams mirrors vllm_video_mux_params. The library composes the argv;
|
||||
// spawning it is the CALLER's job, which is why no ffmpeg lives in libvllm.
|
||||
type cVideoMuxParams struct {
|
||||
Frames uintptr // const char*; printf pattern, dir/frame_%06d.ppm
|
||||
AudioPath uintptr // const char*; NULL/empty => a silent clip
|
||||
OutputPath uintptr // const char*; the .mp4 to write
|
||||
Fps int32 // <= 0 => the H3 default (24)
|
||||
Crf int32 // <= 0 => the library default (18)
|
||||
}
|
||||
|
||||
// defaultSamplingParams mirrors vllm_sampling_params_default().
|
||||
func defaultSamplingParams() cSamplingParams {
|
||||
return cSamplingParams{
|
||||
@@ -148,6 +229,14 @@ var (
|
||||
vllmLastError func() string
|
||||
vllmVersion func() string
|
||||
vllmABIVersion func() int32
|
||||
|
||||
// Video+audio generation (ABI v12).
|
||||
vllmVideoEngineLoad func(params, out unsafe.Pointer) int32
|
||||
vllmVideoEngineFree func(engine uintptr)
|
||||
vllmVideoGenerate func(engine uintptr, params, out unsafe.Pointer) int32
|
||||
vllmVideoResultFree func(out unsafe.Pointer)
|
||||
vllmVideoMuxArgv func(params, outArgv, outArgc unsafe.Pointer) int32
|
||||
vllmVideoMuxArgvFre func(argv uintptr, argc int32)
|
||||
)
|
||||
|
||||
type libFunc struct {
|
||||
@@ -175,6 +264,12 @@ func registerLib(libName string) error {
|
||||
{&vllmLastError, "vllm_last_error"},
|
||||
{&vllmVersion, "vllm_version"},
|
||||
{&vllmABIVersion, "vllm_abi_version"},
|
||||
{&vllmVideoEngineLoad, "vllm_video_engine_load"},
|
||||
{&vllmVideoEngineFree, "vllm_video_engine_free"},
|
||||
{&vllmVideoGenerate, "vllm_video_generate"},
|
||||
{&vllmVideoResultFree, "vllm_video_result_free"},
|
||||
{&vllmVideoMuxArgv, "vllm_video_mux_argv"},
|
||||
{&vllmVideoMuxArgvFre, "vllm_video_mux_argv_free"},
|
||||
} {
|
||||
purego.RegisterLibFunc(lf.ptr, lib, lf.name)
|
||||
}
|
||||
@@ -222,3 +317,19 @@ func goString(p uintptr) string {
|
||||
}
|
||||
return string(unsafe.Slice((*byte)(base), n))
|
||||
}
|
||||
|
||||
// goStringSlice copies a C `char*` array of n entries. Used for the ffmpeg argv
|
||||
// the library composes: it is copied out immediately so the caller can free the
|
||||
// C allocation before ever spawning the process.
|
||||
func goStringSlice(p uintptr, n int32) []string {
|
||||
if p == 0 || n <= 0 {
|
||||
return nil
|
||||
}
|
||||
//nolint:govet // C-owned pointer handed over by purego, valid for this call
|
||||
entries := unsafe.Slice((**byte)(unsafe.Pointer(p)), int(n)) // #nosec G103 -- C-owned, copied out immediately
|
||||
out := make([]string, 0, n)
|
||||
for _, e := range entries {
|
||||
out = append(out, goString(uintptr(unsafe.Pointer(e)))) // #nosec G103 -- ditto
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -62,6 +62,66 @@ type loadOptions struct {
|
||||
// Override for the tokenizer_config.json the chat template is read from
|
||||
// (ABI v9). Empty = <model_dir>/tokenizer_config.json.
|
||||
tokenizerConfigPath string
|
||||
// MiniMax-H3 video+audio generation (ABI v12). Present only when the config
|
||||
// carries at least one of its keys; see videoOptions.engaged.
|
||||
video videoOptions
|
||||
}
|
||||
|
||||
// videoOptions is the MiniMax-H3 checkpoint SET plus its generation defaults.
|
||||
//
|
||||
// H3 is not one model directory: the DiT, the text encoder and the two VAEs are
|
||||
// separate artifacts, which is why vllm.cpp gives video its own engine handle
|
||||
// (vllm_video_engine, ABI v12) rather than another vllm_engine. The DiT is the
|
||||
// model config's `parameters.model`; everything else arrives through these
|
||||
// options, so one gallery entry can name five files.
|
||||
//
|
||||
// The geometry/frame defaults exist because H3's trained canvas is nothing like
|
||||
// the generic /video defaults: 1344x768 at 124 frames is a ~5.2 s clip, and the
|
||||
// frame count must sit on the 17n+5 grid. A request that leaves a field unset
|
||||
// gets the model's own default from here instead of a canvas the checkpoint was
|
||||
// never trained at.
|
||||
type videoOptions struct {
|
||||
encoderPath string // H3-Encoder GGUF or bf16 shard dir
|
||||
tokenizerPath string // tokenizer.json, needed with an encoder
|
||||
videoVaePath string
|
||||
videoVaeConfig string
|
||||
audioVaePath string
|
||||
audioVaeConfig string
|
||||
promptEmbedsPath string // fallback conditioning when there is no encoder
|
||||
// The served checkpoint PARTITION. Community GGUF/NVFP4 files strip the
|
||||
// release metadata and the FL2VA/Ref2VA DiTs are byte-structurally
|
||||
// identical, so the engine refuses every generate until it is DECLARED.
|
||||
// "fl2va" serves t2va + fl2va; "ref2va" serves reference conditioning.
|
||||
partition string
|
||||
device int32 // 0 cpu, 1 cuda (the ABI's own encoding, no auto slot)
|
||||
deviceSet bool
|
||||
dequantBf16 int32
|
||||
fp4Resident int32
|
||||
// Per-model generation defaults, applied when the request leaves the field
|
||||
// at 0.
|
||||
width int32
|
||||
height int32
|
||||
numFrames int32
|
||||
steps int32
|
||||
// Where frames + WAV are written. Empty = a temporary directory beside the
|
||||
// requested output, removed once the mux succeeds. Set it to keep the
|
||||
// frame_%06d.ppm runs around (they are what ref2va's ref_video consumes).
|
||||
workdir string
|
||||
// The ffmpeg binary the composed mux argv is exec'd with. Empty = "ffmpeg"
|
||||
// from PATH. libvllm composes the argv and spawns nothing, by design.
|
||||
ffmpeg string
|
||||
crf int32
|
||||
}
|
||||
|
||||
// engaged reports whether this config describes an H3 video engine. Load uses
|
||||
// it to choose which of the two mutually exclusive engine handles to open: the
|
||||
// checkpoints refuse each other, so guessing is not an option, and every key
|
||||
// below is meaningless to the text engine.
|
||||
func (v videoOptions) engaged() bool {
|
||||
return v.encoderPath != "" || v.tokenizerPath != "" ||
|
||||
v.videoVaePath != "" || v.videoVaeConfig != "" ||
|
||||
v.audioVaePath != "" || v.audioVaeConfig != "" ||
|
||||
v.promptEmbedsPath != "" || v.partition != ""
|
||||
}
|
||||
|
||||
func parseOptions(opts *pb.ModelOptions) loadOptions {
|
||||
@@ -110,10 +170,94 @@ func applyOptionsList(lo *loadOptions, options []string) {
|
||||
if b, err := strconv.ParseBool(strings.TrimSpace(v)); err == nil {
|
||||
lo.enableJumpForward = boolTriState(b)
|
||||
}
|
||||
default:
|
||||
applyVideoOption(&lo.video, strings.TrimSpace(k), v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// applyVideoOption reads one MiniMax-H3 key. Split out of applyOptionsList so
|
||||
// the video surface stays legible next to the videoOptions it fills, and so
|
||||
// video_test.go can exercise it directly.
|
||||
func applyVideoOption(vo *videoOptions, key, value string) bool {
|
||||
v := strings.TrimSpace(value)
|
||||
switch key {
|
||||
case "video_encoder":
|
||||
vo.encoderPath = v
|
||||
case "video_tokenizer":
|
||||
vo.tokenizerPath = v
|
||||
case "video_vae":
|
||||
vo.videoVaePath = v
|
||||
case "video_vae_config":
|
||||
vo.videoVaeConfig = v
|
||||
case "audio_vae":
|
||||
vo.audioVaePath = v
|
||||
case "audio_vae_config":
|
||||
vo.audioVaeConfig = v
|
||||
case "video_prompt_embeds":
|
||||
vo.promptEmbedsPath = v
|
||||
case "video_partition":
|
||||
vo.partition = strings.ToLower(v)
|
||||
case "video_device":
|
||||
switch strings.ToLower(v) {
|
||||
case "cpu":
|
||||
vo.device, vo.deviceSet = videoDeviceCPU, true
|
||||
case "cuda", "gpu":
|
||||
vo.device, vo.deviceSet = videoDeviceCUDA, true
|
||||
default:
|
||||
xlog.Warn("[vllm-cpp] ignoring unknown video_device", "value", v)
|
||||
}
|
||||
case "video_dequant_bf16":
|
||||
if b, err := strconv.ParseBool(v); err == nil {
|
||||
vo.dequantBf16 = boolInt32(b)
|
||||
}
|
||||
case "video_fp4_resident":
|
||||
if b, err := strconv.ParseBool(v); err == nil {
|
||||
vo.fp4Resident = boolInt32(b)
|
||||
}
|
||||
case "video_width":
|
||||
vo.width = parseInt32(v, vo.width)
|
||||
case "video_height":
|
||||
vo.height = parseInt32(v, vo.height)
|
||||
case "video_num_frames":
|
||||
vo.numFrames = parseInt32(v, vo.numFrames)
|
||||
case "video_steps":
|
||||
vo.steps = parseInt32(v, vo.steps)
|
||||
case "video_workdir":
|
||||
vo.workdir = v
|
||||
case "video_crf":
|
||||
vo.crf = parseInt32(v, vo.crf)
|
||||
case "ffmpeg", "ffmpeg_path":
|
||||
vo.ffmpeg = v
|
||||
default:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// videoScalarString renders an engine_args scalar so the video keys can share
|
||||
// one parser with the "key:value" list. Objects and arrays have no video
|
||||
// meaning and are left to the caller's unknown-key path.
|
||||
func videoScalarString(v any) (string, bool) {
|
||||
switch t := v.(type) {
|
||||
case string:
|
||||
return t, true
|
||||
case bool:
|
||||
return strconv.FormatBool(t), true
|
||||
case float64:
|
||||
return strconv.FormatFloat(t, 'f', -1, 64), true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
func boolInt32(b bool) int32 {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// applyEngineArgs overlays the `engine_args:` JSON object. A document that does
|
||||
// not parse is logged and skipped: engine_args is shared with the other engines
|
||||
// (the vLLM and SGLang backends read the same field), so a stray key must not
|
||||
@@ -160,6 +304,9 @@ func applyEngineArgs(lo *loadOptions, engineArgs string) {
|
||||
lo.enableJumpForward = boolTriState(b)
|
||||
}
|
||||
default:
|
||||
if s, ok := videoScalarString(v); ok && applyVideoOption(&lo.video, k, s) {
|
||||
continue
|
||||
}
|
||||
xlog.Debug("[vllm-cpp] ignoring unknown engine_args key", "key", k)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,634 @@
|
||||
package main
|
||||
|
||||
// MiniMax-H3 video+audio generation over the vllm.cpp C ABI (v12).
|
||||
//
|
||||
// Two things make this different from the text path, and both come from the
|
||||
// engine's own shape rather than from LocalAI:
|
||||
//
|
||||
// 1. A video engine is loaded from a checkpoint SET - the DiT, the text
|
||||
// encoder and two VAEs are separate artifacts - so it is its own handle
|
||||
// (vllm_video_engine) and its own Load branch. The two loaders refuse each
|
||||
// other's checkpoints on purpose.
|
||||
// 2. libvllm writes frames + a WAV and COMPOSES the ffmpeg argv, but spawns
|
||||
// nothing. That process boundary is deliberate upstream, so the mux lives
|
||||
// here: we take the composed argv, substitute argv[0], and exec it. ffmpeg
|
||||
// comes from PATH the same way the vibevoice-cpp backend takes it.
|
||||
//
|
||||
// Generation is SLOW - roughly 176 s per denoise step at 1344x768 on a 20-SM
|
||||
// device, so a default 50-step render is hours, not seconds. Nothing here
|
||||
// imposes a deadline: GenerateVideo blocks for as long as the engine needs and
|
||||
// the gRPC call carries LocalAI's application context.
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"image"
|
||||
"math"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unsafe"
|
||||
|
||||
// Registered for image.DecodeConfig only: a staged keyframe arrives as
|
||||
// whatever the caller uploaded, and we need its geometry to size the canvas.
|
||||
_ "image/gif"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
|
||||
pb "github.com/mudler/LocalAI/pkg/grpc/proto"
|
||||
"github.com/mudler/xlog"
|
||||
)
|
||||
|
||||
// vllm_video_model_params.device (vllm.h): no auto slot, unlike the text
|
||||
// engine's v14 device field.
|
||||
const (
|
||||
videoDeviceCPU int32 = 0
|
||||
videoDeviceCUDA int32 = 1
|
||||
)
|
||||
|
||||
// H3's shipped geometry. The canvas is truncated onto a 32-pixel grid and the
|
||||
// frame count onto the 17n+5 grid by the engine itself
|
||||
// (MiniMaxH3ResolveShape / MiniMaxH3AlignFrameCount in
|
||||
// src/vllm/model_executor/models/minimax_h3_planner.cpp); mirrored here only so
|
||||
// a keyframe can be resampled to the exact canvas the engine will render at.
|
||||
const (
|
||||
h3CanvasMultiple int32 = 32
|
||||
h3FrameGrid int32 = 17
|
||||
h3FrameOffset int32 = 5
|
||||
h3ShortEdge int32 = 768
|
||||
)
|
||||
|
||||
// videoPartitions are the two DECLARED partitions of the H3 release. The FL2VA
|
||||
// checkpoint serves t2va and fl2va; ref2va is a different checkpoint. Passing
|
||||
// reference conditioning against an fl2va DiT is a partition mismatch that
|
||||
// renders a coloured lattice over the frame rather than failing cleanly, which
|
||||
// is why it is refused here before the engine is ever called.
|
||||
const (
|
||||
partitionFL2VA = "fl2va"
|
||||
partitionRef2VA = "ref2va"
|
||||
)
|
||||
|
||||
// videoRequestParams are the per-request `params` keys this backend accepts.
|
||||
// Unknown keys are an error rather than a silent drop: a misspelled reference
|
||||
// path would otherwise produce a perfectly successful render of the wrong
|
||||
// thing, hours later.
|
||||
var videoRequestParams = []string{"noise_aug", "ref_image", "ref_video", "crf"}
|
||||
|
||||
// loadVideo opens the H3 checkpoint set. `dit` is the model config's
|
||||
// parameters.model; every other artifact comes from the options.
|
||||
func (v *VllmCpp) loadVideo(opts *pb.ModelOptions, dit string) error {
|
||||
vo := &v.opts.video
|
||||
|
||||
// Relative option paths resolve against LocalAI's models directory, which
|
||||
// is where the gallery lands the five H3 files.
|
||||
resolve := func(p string) string {
|
||||
if p == "" || filepath.IsAbs(p) || opts.ModelPath == "" {
|
||||
return p
|
||||
}
|
||||
return filepath.Join(opts.ModelPath, p)
|
||||
}
|
||||
vo.encoderPath = resolve(vo.encoderPath)
|
||||
vo.tokenizerPath = resolve(vo.tokenizerPath)
|
||||
vo.videoVaePath = resolve(vo.videoVaePath)
|
||||
vo.videoVaeConfig = resolve(vo.videoVaeConfig)
|
||||
vo.audioVaePath = resolve(vo.audioVaePath)
|
||||
vo.audioVaeConfig = resolve(vo.audioVaeConfig)
|
||||
vo.promptEmbedsPath = resolve(vo.promptEmbedsPath)
|
||||
vo.workdir = resolve(vo.workdir)
|
||||
|
||||
// A VAE config carries the per-channel latents_mean/latents_std and the
|
||||
// temporal clip_length/token_drop; decode is wrong without it. The release
|
||||
// ships it beside the weights, so default to that rather than making every
|
||||
// config repeat it.
|
||||
if vo.videoVaeConfig == "" && vo.videoVaePath != "" {
|
||||
vo.videoVaeConfig = siblingConfigJSON(vo.videoVaePath)
|
||||
}
|
||||
if vo.audioVaeConfig == "" && vo.audioVaePath != "" {
|
||||
vo.audioVaeConfig = siblingConfigJSON(vo.audioVaePath)
|
||||
}
|
||||
|
||||
if vo.partition == "" {
|
||||
// The community GGUF/NVFP4 quantisations strip the release metadata and
|
||||
// the two DiTs are byte-structurally identical, so the engine cannot
|
||||
// infer this and refuses every generate until it is declared. The
|
||||
// shipped FL2VA checkpoint is the one the gallery entry installs.
|
||||
vo.partition = partitionFL2VA
|
||||
xlog.Warn("[vllm-cpp] video partition not declared, assuming the FL2VA checkpoint",
|
||||
"hint", "set options: [video_partition:fl2va] or [video_partition:ref2va] to match the DiT you installed")
|
||||
}
|
||||
if vo.partition != partitionFL2VA && vo.partition != partitionRef2VA {
|
||||
return fmt.Errorf("vllm-cpp: video_partition must be %q or %q, got %q",
|
||||
partitionFL2VA, partitionRef2VA, vo.partition)
|
||||
}
|
||||
if vo.videoVaePath == "" || vo.audioVaePath == "" {
|
||||
return fmt.Errorf("vllm-cpp: MiniMax-H3 needs both VAEs: set options: " +
|
||||
"[video_vae:<video vae .safetensors>, audio_vae:<audio vae .safetensors>]")
|
||||
}
|
||||
if vo.encoderPath == "" && vo.promptEmbedsPath == "" {
|
||||
return fmt.Errorf("vllm-cpp: MiniMax-H3 needs text conditioning: set options: " +
|
||||
"[video_encoder:<encoder .gguf>, video_tokenizer:<tokenizer.json>] " +
|
||||
"or [video_prompt_embeds:<f32 embeddings>]")
|
||||
}
|
||||
if !vo.deviceSet && opts.GetCUDA() {
|
||||
vo.device = videoDeviceCUDA
|
||||
}
|
||||
|
||||
mp := cVideoModelParams{
|
||||
Device: vo.device,
|
||||
DequantBf16: vo.dequantBf16,
|
||||
Fp4Resident: vo.fp4Resident,
|
||||
}
|
||||
var keep [][]byte
|
||||
setStr := func(dst *uintptr, s string) {
|
||||
if s == "" {
|
||||
return
|
||||
}
|
||||
b := cString(s)
|
||||
keep = append(keep, b)
|
||||
*dst = uintptr(unsafe.Pointer(&b[0])) // #nosec G103 -- borrowed by C for the load call only
|
||||
}
|
||||
setStr(&mp.DitPath, dit)
|
||||
setStr(&mp.EncoderPath, vo.encoderPath)
|
||||
setStr(&mp.TokenizerPath, vo.tokenizerPath)
|
||||
setStr(&mp.VideoVaePath, vo.videoVaePath)
|
||||
setStr(&mp.VideoVaeConfigPath, vo.videoVaeConfig)
|
||||
setStr(&mp.AudioVaePath, vo.audioVaePath)
|
||||
setStr(&mp.AudioVaeConfigPath, vo.audioVaeConfig)
|
||||
setStr(&mp.PromptEmbedsPath, vo.promptEmbedsPath)
|
||||
setStr(&mp.Partition, vo.partition)
|
||||
|
||||
xlog.Info("[vllm-cpp] Load (MiniMax-H3 video)", "dit", dit, "engine", vllmVersion(),
|
||||
"encoder", vo.encoderPath, "tokenizer", vo.tokenizerPath,
|
||||
"videoVae", vo.videoVaePath, "audioVae", vo.audioVaePath,
|
||||
"partition", vo.partition, "device", videoDeviceName(vo.device),
|
||||
"dequantBf16", vo.dequantBf16 == 1, "fp4Resident", vo.fp4Resident == 1)
|
||||
|
||||
var engine uintptr
|
||||
rc := vllmVideoEngineLoad(unsafe.Pointer(&mp), unsafe.Pointer(&engine)) // #nosec G103 -- POD out-params
|
||||
runtime.KeepAlive(keep)
|
||||
if rc != vllmOK {
|
||||
return fmt.Errorf("vllm-cpp: video engine load failed: %s", vllmLastError())
|
||||
}
|
||||
v.videoEngine = engine
|
||||
return nil
|
||||
}
|
||||
|
||||
// GenerateVideo renders one clip and muxes it to opts.Dst as an MP4 carrying
|
||||
// H3's jointly generated AAC audio track. It blocks for the whole render.
|
||||
func (v *VllmCpp) GenerateVideo(opts *pb.GenerateVideoRequest) error {
|
||||
if v.videoEngine == 0 {
|
||||
return fmt.Errorf("vllm-cpp: this model is not a MiniMax-H3 video engine " +
|
||||
"(load it with the video_vae / audio_vae / video_encoder options)")
|
||||
}
|
||||
if strings.TrimSpace(opts.GetPrompt()) == "" {
|
||||
return fmt.Errorf("vllm-cpp: video generation needs a prompt")
|
||||
}
|
||||
dst := opts.GetDst()
|
||||
if dst == "" {
|
||||
return fmt.Errorf("vllm-cpp: video generation needs an output path")
|
||||
}
|
||||
vo := v.opts.video
|
||||
|
||||
extra, err := parseVideoRequestParams(opts.GetParams())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := checkPartitionConditioning(vo.partition, opts, extra); err != nil {
|
||||
return err
|
||||
}
|
||||
if opts.GetNegativePrompt() != "" {
|
||||
xlog.Warn("[vllm-cpp] MiniMax-H3 has no negative prompt; ignoring it")
|
||||
}
|
||||
if opts.GetCfgScale() != 0 {
|
||||
xlog.Warn("[vllm-cpp] MiniMax-H3 has no classifier-free guidance scale; ignoring cfg_scale")
|
||||
}
|
||||
|
||||
workdir, cleanup, err := v.videoWorkdir(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
width, height := firstPositive(opts.GetWidth(), vo.width), firstPositive(opts.GetHeight(), vo.height)
|
||||
frames := firstPositive(opts.GetNumFrames(), vo.numFrames)
|
||||
steps := firstPositive(opts.GetStep(), vo.steps)
|
||||
|
||||
vp := cVideoParams{
|
||||
NumFrames: frames,
|
||||
Steps: steps,
|
||||
NoiseAug: extra.noiseAug,
|
||||
}
|
||||
if opts.GetSeed() > 0 {
|
||||
vp.Seed = uint64(opts.GetSeed())
|
||||
vp.HasSeed = 1
|
||||
}
|
||||
if aligned := alignFrameCount(frames); aligned != frames {
|
||||
xlog.Warn("[vllm-cpp] frame count is not on H3's 17n+5 grid; the engine rounds up",
|
||||
"requested", frames, "rendered", aligned)
|
||||
}
|
||||
|
||||
// Keyframes must be binary PPM (P6) at the exact output canvas: no image
|
||||
// codec and no resampler is vendored in libvllm. Resolve the canvas first,
|
||||
// then stage the frames through ffmpeg into it.
|
||||
//
|
||||
// The REQUEST's geometry is what is honoured here, not the model-level
|
||||
// default: that default is a t2va canvas, and applying it to a keyframe
|
||||
// would stretch a portrait photo into a 1344x768 letterbox. With no
|
||||
// requested geometry the canvas comes from the keyframe's own aspect, which
|
||||
// is the rule the engine itself applies (MiniMaxH3ResolveShape).
|
||||
first, last := opts.GetStartImage(), opts.GetEndImage()
|
||||
if first != "" || last != "" {
|
||||
width, height, err = resolveCanvas(opts.GetWidth(), opts.GetHeight(), first, last)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if first, err = stageKeyframe(vo.ffmpeg, first, width, height, workdir, "first"); err != nil {
|
||||
return err
|
||||
}
|
||||
if last, err = stageKeyframe(vo.ffmpeg, last, width, height, workdir, "last"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
vp.Width, vp.Height = truncateToGrid(width), truncateToGrid(height)
|
||||
|
||||
var keep [][]byte
|
||||
setStr := func(dst *uintptr, s string) {
|
||||
if s == "" {
|
||||
return
|
||||
}
|
||||
b := cString(s)
|
||||
keep = append(keep, b)
|
||||
*dst = uintptr(unsafe.Pointer(&b[0])) // #nosec G103 -- borrowed by C for the call only
|
||||
}
|
||||
setStr(&vp.Prompt, opts.GetPrompt())
|
||||
setStr(&vp.OutputDir, workdir)
|
||||
setStr(&vp.FirstFrame, first)
|
||||
setStr(&vp.LastFrame, last)
|
||||
setStr(&vp.RefImage, extra.refImage)
|
||||
setStr(&vp.RefVideo, extra.refVideo)
|
||||
setStr(&vp.RefAudio, opts.GetAudio())
|
||||
|
||||
xlog.Info("[vllm-cpp] GenerateVideo", "dst", dst, "workdir", workdir,
|
||||
"width", vp.Width, "height", vp.Height, "frames", vp.NumFrames,
|
||||
"steps", vp.Steps, "seeded", vp.HasSeed == 1, "partition", vo.partition)
|
||||
|
||||
var out cVideoResult
|
||||
rc := vllmVideoGenerate(v.videoEngine, unsafe.Pointer(&vp), unsafe.Pointer(&out)) // #nosec G103 -- POD in/out params
|
||||
runtime.KeepAlive(keep)
|
||||
if rc != vllmOK {
|
||||
return fmt.Errorf("vllm-cpp: video generation failed: %s", vllmLastError())
|
||||
}
|
||||
defer vllmVideoResultFree(unsafe.Pointer(&out)) // #nosec G103 -- frees the library-owned members
|
||||
|
||||
frameDir, audioPath := goString(out.FrameDir), goString(out.AudioPath)
|
||||
xlog.Info("[vllm-cpp] rendered", "frames", out.FrameCount,
|
||||
"width", out.Width, "height", out.Height, "fps", out.Fps,
|
||||
"audio", audioPath, "sampleRate", out.SampleRate)
|
||||
if opts.GetFps() > 0 && opts.GetFps() != out.Fps {
|
||||
// Muxing at any other rate desynchronises the jointly generated audio.
|
||||
xlog.Warn("[vllm-cpp] MiniMax-H3 renders at a fixed frame rate; ignoring the requested fps",
|
||||
"requested", opts.GetFps(), "rendered", out.Fps)
|
||||
}
|
||||
|
||||
return v.muxVideo(frameDir, audioPath, dst, out.Fps, extra.crf)
|
||||
}
|
||||
|
||||
// muxVideo execs the argv libvllm composed. The encoding contract (h264 /
|
||||
// yuv420p + AAC, -shortest, +faststart) belongs to the library; only the spawn
|
||||
// is ours.
|
||||
func (v *VllmCpp) muxVideo(frameDir, audioPath, dst string, fps, crf int32) error {
|
||||
mx := cVideoMuxParams{Fps: fps, Crf: crf}
|
||||
var keep [][]byte
|
||||
setStr := func(dst *uintptr, s string) {
|
||||
if s == "" {
|
||||
return
|
||||
}
|
||||
b := cString(s)
|
||||
keep = append(keep, b)
|
||||
*dst = uintptr(unsafe.Pointer(&b[0])) // #nosec G103 -- borrowed by C for the call only
|
||||
}
|
||||
setStr(&mx.Frames, filepath.Join(frameDir, "frame_%06d.ppm"))
|
||||
setStr(&mx.AudioPath, audioPath)
|
||||
setStr(&mx.OutputPath, dst)
|
||||
|
||||
var argvPtr uintptr
|
||||
var argc int32
|
||||
rc := vllmVideoMuxArgv(unsafe.Pointer(&mx), unsafe.Pointer(&argvPtr), unsafe.Pointer(&argc)) // #nosec G103 -- POD out-params
|
||||
runtime.KeepAlive(keep)
|
||||
if rc != vllmOK {
|
||||
return fmt.Errorf("vllm-cpp: composing the mux command failed: %s", vllmLastError())
|
||||
}
|
||||
argv := goStringSlice(argvPtr, argc)
|
||||
vllmVideoMuxArgvFre(argvPtr, argc)
|
||||
if len(argv) == 0 {
|
||||
return fmt.Errorf("vllm-cpp: the library composed an empty mux command")
|
||||
}
|
||||
|
||||
ffmpegBin, err := resolveFfmpeg(v.opts.video.ffmpeg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
argv[0] = ffmpegBin
|
||||
|
||||
xlog.Debug("[vllm-cpp] muxing", "argv", argv)
|
||||
output, err := exec.Command(argv[0], argv[1:]...).CombinedOutput() // #nosec G204 -- argv is composed by libvllm, argv[0] is a resolved binary
|
||||
if err != nil {
|
||||
return fmt.Errorf("vllm-cpp: ffmpeg mux failed: %w (output: %s)", err, strings.TrimSpace(string(output)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveFfmpeg locates the mux binary. The backend image is FROM scratch and
|
||||
// carries no ffmpeg, exactly like vibevoice-cpp's transcode path: the host must
|
||||
// provide one, and saying so plainly beats a bare "exec: not found" after an
|
||||
// hours-long render.
|
||||
func resolveFfmpeg(configured string) (string, error) {
|
||||
name := configured
|
||||
if name == "" {
|
||||
name = "ffmpeg"
|
||||
}
|
||||
path, err := exec.LookPath(name)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("vllm-cpp: %q not found: MiniMax-H3 output is muxed with ffmpeg, "+
|
||||
"install it on the host or point options: [ffmpeg:<path>] at a binary: %w", name, err)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
// videoWorkdir returns the directory the engine writes frame_%06d.ppm and
|
||||
// audio.wav into, plus its cleanup.
|
||||
//
|
||||
// It is ALWAYS a fresh directory. Reusing one would leave a longer previous
|
||||
// run's trailing frames in place for the mux to pick up, silently splicing two
|
||||
// renders together. With video_workdir set the run is kept (its frames are what
|
||||
// ref2va's ref_video consumes); otherwise it is removed once the mux succeeds.
|
||||
func (v *VllmCpp) videoWorkdir(dst string) (string, func(), error) {
|
||||
parent := v.opts.video.workdir
|
||||
keep := parent != ""
|
||||
if parent == "" {
|
||||
parent = filepath.Dir(dst)
|
||||
}
|
||||
if err := os.MkdirAll(parent, 0o750); err != nil {
|
||||
return "", nil, fmt.Errorf("vllm-cpp: creating the video work directory: %w", err)
|
||||
}
|
||||
dir, err := os.MkdirTemp(parent, "vllm-cpp-h3-")
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("vllm-cpp: creating the video work directory: %w", err)
|
||||
}
|
||||
if keep {
|
||||
return dir, func() {}, nil
|
||||
}
|
||||
return dir, func() {
|
||||
if err := os.RemoveAll(dir); err != nil {
|
||||
xlog.Warn("[vllm-cpp] could not remove the video work directory", "dir", dir, "error", err)
|
||||
}
|
||||
}, nil
|
||||
}
|
||||
|
||||
// videoExtraParams holds the per-request knobs that have no proto field.
|
||||
type videoExtraParams struct {
|
||||
noiseAug float32
|
||||
refImage string
|
||||
refVideo string
|
||||
crf int32
|
||||
}
|
||||
|
||||
func parseVideoRequestParams(params map[string]string) (videoExtraParams, error) {
|
||||
var extra videoExtraParams
|
||||
for k, raw := range params {
|
||||
v := strings.TrimSpace(raw)
|
||||
switch k {
|
||||
case "noise_aug":
|
||||
f, err := strconv.ParseFloat(v, 32)
|
||||
if err != nil {
|
||||
return extra, fmt.Errorf("vllm-cpp: params.noise_aug must be a number, got %q", raw)
|
||||
}
|
||||
extra.noiseAug = float32(f)
|
||||
case "ref_image":
|
||||
extra.refImage = v
|
||||
case "ref_video":
|
||||
extra.refVideo = v
|
||||
case "crf":
|
||||
n, err := strconv.ParseInt(v, 10, 32)
|
||||
if err != nil {
|
||||
return extra, fmt.Errorf("vllm-cpp: params.crf must be an integer, got %q", raw)
|
||||
}
|
||||
extra.crf = int32(n)
|
||||
default:
|
||||
return extra, fmt.Errorf("vllm-cpp: unknown params key %q (accepted: %s)",
|
||||
k, strings.Join(videoRequestParams, ", "))
|
||||
}
|
||||
}
|
||||
return extra, nil
|
||||
}
|
||||
|
||||
// checkPartitionConditioning refuses conditioning the loaded checkpoint cannot
|
||||
// serve.
|
||||
//
|
||||
// This is the failure this backend most needs to catch early. The FL2VA
|
||||
// partition serves t2va and fl2va; handing it a reference image or audio is a
|
||||
// partition mismatch, and H3 does not fail cleanly on one - it renders, for
|
||||
// hours, and returns a coloured lattice over the frame. The engine's own #77
|
||||
// guard covers a missing declaration; this covers a declaration that does not
|
||||
// match the request.
|
||||
func checkPartitionConditioning(partition string, opts *pb.GenerateVideoRequest, extra videoExtraParams) error {
|
||||
hasKeyframe := opts.GetStartImage() != "" || opts.GetEndImage() != ""
|
||||
hasReference := extra.refImage != "" || extra.refVideo != "" || opts.GetAudio() != ""
|
||||
|
||||
if hasKeyframe && hasReference {
|
||||
return fmt.Errorf("vllm-cpp: fl2va keyframes (start_image/end_image) and ref2va reference " +
|
||||
"conditioning (params.ref_image/params.ref_video/audio) are exclusive in the H3 pipeline")
|
||||
}
|
||||
switch partition {
|
||||
case partitionFL2VA:
|
||||
if hasReference {
|
||||
return fmt.Errorf("vllm-cpp: the FL2VA checkpoint serves t2va and fl2va only - " +
|
||||
"reference conditioning (params.ref_image/params.ref_video/audio) needs a ref2va DiT. " +
|
||||
"Use start_image for first-frame conditioning instead")
|
||||
}
|
||||
case partitionRef2VA:
|
||||
if hasKeyframe {
|
||||
return fmt.Errorf("vllm-cpp: the Ref2VA checkpoint does not serve fl2va keyframes - " +
|
||||
"pass the image as params.ref_image, or install the FL2VA checkpoint")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveCanvas settles the output geometry BEFORE a keyframe is resampled,
|
||||
// because the two have to agree exactly: the engine refuses a keyframe that is
|
||||
// not already at the output resolution, and when no geometry is requested it
|
||||
// derives one from the keyframe's own aspect. Mirrors _resolve_shape
|
||||
// (src/vllm/model_executor/models/minimax_h3_planner.cpp:264-308).
|
||||
func resolveCanvas(width, height int32, keyframes ...string) (int32, int32, error) {
|
||||
if width > 0 && height > 0 {
|
||||
return width, height, nil
|
||||
}
|
||||
for _, k := range keyframes {
|
||||
if k == "" {
|
||||
continue
|
||||
}
|
||||
w, h, err := imageDimensions(k)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
if w <= 0 || h <= 0 {
|
||||
continue
|
||||
}
|
||||
// A 768 short edge, the long edge snapped onto the 32 grid.
|
||||
if w >= h {
|
||||
return alignMultiple(float64(h3ShortEdge)*float64(w)/float64(h), h3CanvasMultiple), h3ShortEdge, nil
|
||||
}
|
||||
return h3ShortEdge, alignMultiple(float64(h3ShortEdge)*float64(h)/float64(w), h3CanvasMultiple), nil
|
||||
}
|
||||
// The shipped canvas.
|
||||
return 1344, h3ShortEdge, nil
|
||||
}
|
||||
|
||||
// stageKeyframe converts a staged upload into the binary PPM (P6) at exactly
|
||||
// width x height that the engine requires. libvllm vendors no image codec and
|
||||
// no resampler, so ffmpeg does both; a P6 already at the canvas passes through
|
||||
// untouched.
|
||||
func stageKeyframe(ffmpegPath, src string, width, height int32, workdir, name string) (string, error) {
|
||||
if src == "" {
|
||||
return "", nil
|
||||
}
|
||||
if w, h, err := ppmDimensions(src); err == nil && w == width && h == height {
|
||||
return src, nil
|
||||
}
|
||||
ffmpegBin, err := resolveFfmpeg(ffmpegPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("converting the %s keyframe to PPM: %w", name, err)
|
||||
}
|
||||
out := filepath.Join(workdir, name+"_frame.ppm")
|
||||
// -frames:v 1 because an animated upload (GIF) would otherwise write a
|
||||
// sequence; -pix_fmt rgb24 is what the image2/ppm muxer needs for P6.
|
||||
cmd := exec.Command(ffmpegBin, "-y", "-loglevel", "error", "-i", src, // #nosec G204 -- the binary is resolved, the rest are literals and staged paths
|
||||
"-frames:v", "1",
|
||||
"-vf", fmt.Sprintf("scale=%d:%d", width, height),
|
||||
"-pix_fmt", "rgb24", "-f", "image2", out)
|
||||
if output, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("vllm-cpp: converting the %s keyframe to PPM failed: %w (output: %s)",
|
||||
name, err, strings.TrimSpace(string(output)))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// imageDimensions reads geometry from a staged upload, PPM included (the Go
|
||||
// standard library has no netpbm decoder).
|
||||
func imageDimensions(path string) (int32, int32, error) {
|
||||
if w, h, err := ppmDimensions(path); err == nil {
|
||||
return w, h, nil
|
||||
}
|
||||
f, err := os.Open(path) // #nosec G304 -- a path staged by LocalAI for this request
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("vllm-cpp: reading the keyframe %q: %w", path, err)
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
cfg, _, err := image.DecodeConfig(f)
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("vllm-cpp: the keyframe %q is not a PNG, JPEG, GIF or binary PPM: %w", path, err)
|
||||
}
|
||||
return int32(cfg.Width), int32(cfg.Height), nil
|
||||
}
|
||||
|
||||
// ppmDimensions parses a binary PPM (P6) header: magic, then width, height and
|
||||
// maxval as ASCII decimals separated by whitespace, with # comments allowed.
|
||||
func ppmDimensions(path string) (int32, int32, error) {
|
||||
f, err := os.Open(path) // #nosec G304 -- a path staged by LocalAI for this request
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
|
||||
// A P6 header is a handful of bytes; 512 covers any sane comment run.
|
||||
buf := make([]byte, 512)
|
||||
n, err := f.Read(buf)
|
||||
if n < 2 || (err != nil && n == 0) {
|
||||
return 0, 0, fmt.Errorf("not a PPM")
|
||||
}
|
||||
if buf[0] != 'P' || buf[1] != '6' {
|
||||
return 0, 0, fmt.Errorf("not a binary PPM (P6)")
|
||||
}
|
||||
fields := make([]int32, 0, 2)
|
||||
for i := 2; i < n && len(fields) < 2; {
|
||||
switch {
|
||||
case buf[i] == '#':
|
||||
for i < n && buf[i] != '\n' {
|
||||
i++
|
||||
}
|
||||
case buf[i] >= '0' && buf[i] <= '9':
|
||||
value := int32(0)
|
||||
for i < n && buf[i] >= '0' && buf[i] <= '9' {
|
||||
value = value*10 + int32(buf[i]-'0')
|
||||
i++
|
||||
}
|
||||
fields = append(fields, value)
|
||||
default:
|
||||
i++
|
||||
}
|
||||
}
|
||||
if len(fields) < 2 {
|
||||
return 0, 0, fmt.Errorf("truncated PPM header")
|
||||
}
|
||||
return fields[0], fields[1], nil
|
||||
}
|
||||
|
||||
// alignMultiple mirrors MiniMaxH3AlignMultiple: round-half-to-even onto the
|
||||
// multiple, floored at one multiple. Half-to-even, not half-away-from-zero,
|
||||
// because the reference pipeline uses Python's round().
|
||||
func alignMultiple(value float64, multiple int32) int32 {
|
||||
snapped := int32(math.RoundToEven(value/float64(multiple))) * multiple
|
||||
if snapped < multiple {
|
||||
return multiple
|
||||
}
|
||||
return snapped
|
||||
}
|
||||
|
||||
// truncateToGrid mirrors the engine's canvas snap: truncation, not rounding.
|
||||
func truncateToGrid(v int32) int32 {
|
||||
if v <= 0 {
|
||||
return 0
|
||||
}
|
||||
return v / h3CanvasMultiple * h3CanvasMultiple
|
||||
}
|
||||
|
||||
// alignFrameCount mirrors MiniMaxH3AlignFrameCount: the next value on the
|
||||
// 17n+5 grid. Used only to warn - the engine does the real alignment.
|
||||
func alignFrameCount(frames int32) int32 {
|
||||
if frames <= 0 {
|
||||
return frames
|
||||
}
|
||||
for frames%h3FrameGrid != h3FrameOffset {
|
||||
frames++
|
||||
}
|
||||
return frames
|
||||
}
|
||||
|
||||
func firstPositive(values ...int32) int32 {
|
||||
for _, v := range values {
|
||||
if v > 0 {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func videoDeviceName(device int32) string {
|
||||
if device == videoDeviceCUDA {
|
||||
return "cuda"
|
||||
}
|
||||
return "cpu"
|
||||
}
|
||||
|
||||
// siblingConfigJSON is the release layout: each VAE ships its config.json in
|
||||
// the directory holding its weights.
|
||||
func siblingConfigJSON(weights string) string {
|
||||
candidate := filepath.Join(filepath.Dir(weights), "config.json")
|
||||
if _, err := os.Stat(candidate); err != nil {
|
||||
return ""
|
||||
}
|
||||
return candidate
|
||||
}
|
||||
@@ -0,0 +1,298 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"unsafe"
|
||||
|
||||
pb "github.com/mudler/LocalAI/pkg/grpc/proto"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
)
|
||||
|
||||
// The video PODs carry the same contract as the text ones in vllmcpp_test.go:
|
||||
// these are the C offsets of vllm.h on LP64, and a drift here is silent memory
|
||||
// corruption rather than a compile error.
|
||||
var _ = Describe("C ABI video struct mirrors", func() {
|
||||
It("cVideoModelParams matches vllm_video_model_params", func() {
|
||||
var p cVideoModelParams
|
||||
Expect(unsafe.Offsetof(p.DitPath)).To(Equal(uintptr(0)))
|
||||
Expect(unsafe.Offsetof(p.EncoderPath)).To(Equal(uintptr(8)))
|
||||
Expect(unsafe.Offsetof(p.TokenizerPath)).To(Equal(uintptr(16)))
|
||||
Expect(unsafe.Offsetof(p.VideoVaePath)).To(Equal(uintptr(24)))
|
||||
Expect(unsafe.Offsetof(p.VideoVaeConfigPath)).To(Equal(uintptr(32)))
|
||||
Expect(unsafe.Offsetof(p.AudioVaePath)).To(Equal(uintptr(40)))
|
||||
Expect(unsafe.Offsetof(p.AudioVaeConfigPath)).To(Equal(uintptr(48)))
|
||||
Expect(unsafe.Offsetof(p.PromptEmbedsPath)).To(Equal(uintptr(56)))
|
||||
Expect(unsafe.Offsetof(p.Partition)).To(Equal(uintptr(64)))
|
||||
Expect(unsafe.Offsetof(p.Device)).To(Equal(uintptr(72)))
|
||||
Expect(unsafe.Offsetof(p.DequantBf16)).To(Equal(uintptr(76)))
|
||||
Expect(unsafe.Offsetof(p.Fp4Resident)).To(Equal(uintptr(80)))
|
||||
Expect(unsafe.Sizeof(p)).To(Equal(uintptr(88)))
|
||||
})
|
||||
|
||||
It("cVideoParams matches vllm_video_params", func() {
|
||||
var p cVideoParams
|
||||
Expect(unsafe.Offsetof(p.Prompt)).To(Equal(uintptr(0)))
|
||||
Expect(unsafe.Offsetof(p.Width)).To(Equal(uintptr(8)))
|
||||
Expect(unsafe.Offsetof(p.Height)).To(Equal(uintptr(12)))
|
||||
Expect(unsafe.Offsetof(p.NumFrames)).To(Equal(uintptr(16)))
|
||||
Expect(unsafe.Offsetof(p.Steps)).To(Equal(uintptr(20)))
|
||||
Expect(unsafe.Offsetof(p.Seed)).To(Equal(uintptr(24)))
|
||||
Expect(unsafe.Offsetof(p.HasSeed)).To(Equal(uintptr(32)))
|
||||
Expect(unsafe.Offsetof(p.FirstFrame)).To(Equal(uintptr(40)))
|
||||
Expect(unsafe.Offsetof(p.LastFrame)).To(Equal(uintptr(48)))
|
||||
Expect(unsafe.Offsetof(p.RefImage)).To(Equal(uintptr(56)))
|
||||
Expect(unsafe.Offsetof(p.RefVideo)).To(Equal(uintptr(64)))
|
||||
Expect(unsafe.Offsetof(p.RefAudio)).To(Equal(uintptr(72)))
|
||||
Expect(unsafe.Offsetof(p.NoiseAug)).To(Equal(uintptr(80)))
|
||||
Expect(unsafe.Offsetof(p.OutputDir)).To(Equal(uintptr(88)))
|
||||
Expect(unsafe.Sizeof(p)).To(Equal(uintptr(96)))
|
||||
})
|
||||
|
||||
It("cVideoResult matches vllm_video_result", func() {
|
||||
var r cVideoResult
|
||||
Expect(unsafe.Offsetof(r.FrameDir)).To(Equal(uintptr(0)))
|
||||
Expect(unsafe.Offsetof(r.AudioPath)).To(Equal(uintptr(8)))
|
||||
Expect(unsafe.Offsetof(r.FrameCount)).To(Equal(uintptr(16)))
|
||||
Expect(unsafe.Offsetof(r.Width)).To(Equal(uintptr(20)))
|
||||
Expect(unsafe.Offsetof(r.Height)).To(Equal(uintptr(24)))
|
||||
Expect(unsafe.Offsetof(r.Fps)).To(Equal(uintptr(28)))
|
||||
Expect(unsafe.Offsetof(r.SampleRate)).To(Equal(uintptr(32)))
|
||||
Expect(unsafe.Offsetof(r.MuxArgv)).To(Equal(uintptr(40)))
|
||||
Expect(unsafe.Offsetof(r.MuxArgc)).To(Equal(uintptr(48)))
|
||||
Expect(unsafe.Sizeof(r)).To(Equal(uintptr(56)))
|
||||
})
|
||||
|
||||
It("cVideoMuxParams matches vllm_video_mux_params", func() {
|
||||
var p cVideoMuxParams
|
||||
Expect(unsafe.Offsetof(p.Frames)).To(Equal(uintptr(0)))
|
||||
Expect(unsafe.Offsetof(p.AudioPath)).To(Equal(uintptr(8)))
|
||||
Expect(unsafe.Offsetof(p.OutputPath)).To(Equal(uintptr(16)))
|
||||
Expect(unsafe.Offsetof(p.Fps)).To(Equal(uintptr(24)))
|
||||
Expect(unsafe.Offsetof(p.Crf)).To(Equal(uintptr(28)))
|
||||
Expect(unsafe.Sizeof(p)).To(Equal(uintptr(32)))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("video load options", func() {
|
||||
It("stays disengaged for a plain text config", func() {
|
||||
lo := parseOptions(&pb.ModelOptions{Options: []string{"max_num_seqs:16"}})
|
||||
Expect(lo.video.engaged()).To(BeFalse())
|
||||
})
|
||||
|
||||
It("reads the H3 checkpoint set from the options list", func() {
|
||||
lo := parseOptions(&pb.ModelOptions{Options: []string{
|
||||
"video_encoder:qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf",
|
||||
"video_tokenizer:tokenizer.json",
|
||||
"video_vae:vae/diffusion_pytorch_model.safetensors",
|
||||
"audio_vae:audio_vae/model.safetensors",
|
||||
"video_partition:fl2va",
|
||||
"video_device:cuda",
|
||||
"video_dequant_bf16:true",
|
||||
"video_width:1344",
|
||||
"video_height:768",
|
||||
"video_num_frames:124",
|
||||
"video_steps:50",
|
||||
}})
|
||||
Expect(lo.video.engaged()).To(BeTrue())
|
||||
Expect(lo.video.encoderPath).To(Equal("qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf"))
|
||||
Expect(lo.video.tokenizerPath).To(Equal("tokenizer.json"))
|
||||
Expect(lo.video.videoVaePath).To(Equal("vae/diffusion_pytorch_model.safetensors"))
|
||||
Expect(lo.video.audioVaePath).To(Equal("audio_vae/model.safetensors"))
|
||||
Expect(lo.video.partition).To(Equal(partitionFL2VA))
|
||||
Expect(lo.video.device).To(Equal(videoDeviceCUDA))
|
||||
Expect(lo.video.deviceSet).To(BeTrue())
|
||||
Expect(lo.video.dequantBf16).To(Equal(int32(1)))
|
||||
Expect(lo.video.width).To(Equal(int32(1344)))
|
||||
Expect(lo.video.height).To(Equal(int32(768)))
|
||||
Expect(lo.video.numFrames).To(Equal(int32(124)))
|
||||
Expect(lo.video.steps).To(Equal(int32(50)))
|
||||
})
|
||||
|
||||
It("reads the same keys from engine_args", func() {
|
||||
lo := parseOptions(&pb.ModelOptions{
|
||||
EngineArgs: `{"video_vae":"vae/v.safetensors","audio_vae":"a.safetensors","video_num_frames":124,"video_dequant_bf16":true}`,
|
||||
})
|
||||
Expect(lo.video.engaged()).To(BeTrue())
|
||||
Expect(lo.video.videoVaePath).To(Equal("vae/v.safetensors"))
|
||||
Expect(lo.video.audioVaePath).To(Equal("a.safetensors"))
|
||||
Expect(lo.video.numFrames).To(Equal(int32(124)))
|
||||
Expect(lo.video.dequantBf16).To(Equal(int32(1)))
|
||||
})
|
||||
|
||||
It("ignores an unknown video_device rather than guessing", func() {
|
||||
lo := parseOptions(&pb.ModelOptions{Options: []string{"video_vae:v", "video_device:tpu"}})
|
||||
Expect(lo.video.deviceSet).To(BeFalse())
|
||||
Expect(lo.video.device).To(Equal(videoDeviceCPU))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("per-request params", func() {
|
||||
It("maps the accepted keys", func() {
|
||||
extra, err := parseVideoRequestParams(map[string]string{
|
||||
"noise_aug": "0.5", "ref_image": "/tmp/ref.ppm", "crf": "20",
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(extra.noiseAug).To(BeNumerically("~", 0.5, 1e-6))
|
||||
Expect(extra.refImage).To(Equal("/tmp/ref.ppm"))
|
||||
Expect(extra.crf).To(Equal(int32(20)))
|
||||
})
|
||||
|
||||
It("refuses an unknown key instead of dropping it", func() {
|
||||
_, err := parseVideoRequestParams(map[string]string{"resolution": "480p"})
|
||||
Expect(err).To(MatchError(ContainSubstring("unknown params key")))
|
||||
})
|
||||
|
||||
It("refuses a non-numeric noise_aug", func() {
|
||||
_, err := parseVideoRequestParams(map[string]string{"noise_aug": "high"})
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
// The partition guard is the correctness rule this backend exists to enforce:
|
||||
// the FL2VA DiT serves t2va and fl2va, and handing it reference conditioning
|
||||
// renders a broken lattice over the frame after a multi-hour generation rather
|
||||
// than failing.
|
||||
var _ = Describe("partition conditioning guard", func() {
|
||||
It("accepts a plain t2va request on fl2va", func() {
|
||||
Expect(checkPartitionConditioning(partitionFL2VA,
|
||||
&pb.GenerateVideoRequest{Prompt: "a llama"}, videoExtraParams{})).To(Succeed())
|
||||
})
|
||||
|
||||
It("accepts fl2va keyframes on fl2va", func() {
|
||||
Expect(checkPartitionConditioning(partitionFL2VA,
|
||||
&pb.GenerateVideoRequest{StartImage: "/tmp/a.png"}, videoExtraParams{})).To(Succeed())
|
||||
})
|
||||
|
||||
It("refuses a reference image on fl2va", func() {
|
||||
err := checkPartitionConditioning(partitionFL2VA,
|
||||
&pb.GenerateVideoRequest{}, videoExtraParams{refImage: "/tmp/ref.ppm"})
|
||||
Expect(err).To(MatchError(ContainSubstring("ref2va")))
|
||||
})
|
||||
|
||||
It("refuses reference audio on fl2va", func() {
|
||||
err := checkPartitionConditioning(partitionFL2VA,
|
||||
&pb.GenerateVideoRequest{Audio: "/tmp/voice.wav"}, videoExtraParams{})
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("refuses fl2va keyframes on ref2va", func() {
|
||||
err := checkPartitionConditioning(partitionRef2VA,
|
||||
&pb.GenerateVideoRequest{StartImage: "/tmp/a.png"}, videoExtraParams{})
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("refuses keyframes and references together on either partition", func() {
|
||||
err := checkPartitionConditioning(partitionRef2VA,
|
||||
&pb.GenerateVideoRequest{StartImage: "/tmp/a.png"}, videoExtraParams{refVideo: "/tmp/clip"})
|
||||
Expect(err).To(MatchError(ContainSubstring("exclusive")))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("H3 geometry", func() {
|
||||
It("keeps an explicitly requested canvas", func() {
|
||||
w, h, err := resolveCanvas(1280, 720)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(w).To(Equal(int32(1280)))
|
||||
Expect(h).To(Equal(int32(720)))
|
||||
})
|
||||
|
||||
It("falls back to the shipped 1344x768 canvas", func() {
|
||||
w, h, err := resolveCanvas(0, 0)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(w).To(Equal(int32(1344)))
|
||||
Expect(h).To(Equal(int32(768)))
|
||||
})
|
||||
|
||||
It("derives a landscape canvas from a keyframe's aspect", func() {
|
||||
path := writePPM(1920, 1080)
|
||||
w, h, err := resolveCanvas(0, 0, path)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(h).To(Equal(int32(768)))
|
||||
// 768 * 16/9 = 1365.33; /32 = 42.67, round-half-to-even to 43, x32.
|
||||
Expect(w).To(Equal(int32(1376)))
|
||||
})
|
||||
|
||||
It("derives a portrait canvas from a keyframe's aspect", func() {
|
||||
path := writePPM(1080, 1920)
|
||||
w, h, err := resolveCanvas(0, 0, path)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(w).To(Equal(int32(768)))
|
||||
Expect(h).To(Equal(int32(1376)))
|
||||
})
|
||||
|
||||
It("truncates onto the 32 grid the way the engine does", func() {
|
||||
Expect(truncateToGrid(1000)).To(Equal(int32(992)))
|
||||
Expect(truncateToGrid(768)).To(Equal(int32(768)))
|
||||
})
|
||||
|
||||
It("reports the 17n+5 frame grid", func() {
|
||||
Expect(alignFrameCount(124)).To(Equal(int32(124)))
|
||||
Expect(alignFrameCount(120)).To(Equal(int32(124)))
|
||||
Expect(alignFrameCount(100)).To(Equal(int32(107)))
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("keyframe staging", func() {
|
||||
It("parses a binary PPM header, comments included", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
path := filepath.Join(dir, "commented.ppm")
|
||||
Expect(os.WriteFile(path, []byte("P6\n# made by a test\n64 32\n255\n"), 0o600)).To(Succeed())
|
||||
w, h, err := ppmDimensions(path)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(w).To(Equal(int32(64)))
|
||||
Expect(h).To(Equal(int32(32)))
|
||||
})
|
||||
|
||||
It("refuses an ASCII PPM (P3): the engine reads P6 only", func() {
|
||||
dir := GinkgoT().TempDir()
|
||||
path := filepath.Join(dir, "ascii.ppm")
|
||||
Expect(os.WriteFile(path, []byte("P3\n64 32\n255\n"), 0o600)).To(Succeed())
|
||||
_, _, err := ppmDimensions(path)
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("passes a P6 already at the canvas straight through, without ffmpeg", func() {
|
||||
path := writePPM(64, 32)
|
||||
out, err := stageKeyframe("", path, 64, 32, GinkgoT().TempDir(), "first")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out).To(Equal(path))
|
||||
})
|
||||
|
||||
It("is a no-op for an absent keyframe", func() {
|
||||
out, err := stageKeyframe("", "", 64, 32, GinkgoT().TempDir(), "first")
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out).To(BeEmpty())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("GenerateVideo preconditions", func() {
|
||||
It("refuses when the model is not a video engine", func() {
|
||||
v := &VllmCpp{}
|
||||
Expect(v.GenerateVideo(&pb.GenerateVideoRequest{Prompt: "x", Dst: "/tmp/o.mp4"})).
|
||||
To(MatchError(ContainSubstring("not a MiniMax-H3 video engine")))
|
||||
})
|
||||
})
|
||||
|
||||
// writePPM writes a valid P6 header of the given geometry. Only the header is
|
||||
// read by anything under test, so the pixel payload is left off.
|
||||
func writePPM(width, height int) string {
|
||||
dir := GinkgoT().TempDir()
|
||||
path := filepath.Join(dir, "frame.ppm")
|
||||
header := []byte("P6\n" + itoa(width) + " " + itoa(height) + "\n255\n")
|
||||
Expect(os.WriteFile(path, header, 0o600)).To(Succeed())
|
||||
return path
|
||||
}
|
||||
|
||||
func itoa(v int) string {
|
||||
if v == 0 {
|
||||
return "0"
|
||||
}
|
||||
digits := ""
|
||||
for v > 0 {
|
||||
digits = string(rune('0'+v%10)) + digits
|
||||
v /= 10
|
||||
}
|
||||
return digits
|
||||
}
|
||||
@@ -16,7 +16,7 @@ func TestVllmCpp(t *testing.T) {
|
||||
RunSpecs(t, "vllm-cpp suite")
|
||||
}
|
||||
|
||||
// The Go POD mirrors must match the C struct layout of vllm.h (ABI v10)
|
||||
// The Go POD mirrors must match the C struct layout of vllm.h (ABI v16)
|
||||
// byte-for-byte: these offsets are the C offsets on LP64 (linux/darwin
|
||||
// amd64+arm64). A failure here means govllmcpp.go drifted from vllm.h.
|
||||
var _ = Describe("C ABI struct mirrors", func() {
|
||||
@@ -24,7 +24,7 @@ var _ = Describe("C ABI struct mirrors", func() {
|
||||
// VLLM_ABI_VERSION in the vllm.h of VLLM_CPP_VERSION (Makefile).
|
||||
// Moving the pin past this without growing the mirrors below ships a
|
||||
// backend that refuses every load at startup (issue #11379).
|
||||
Expect(abiVersion).To(Equal(10))
|
||||
Expect(abiVersion).To(Equal(16))
|
||||
})
|
||||
|
||||
It("cModelParams matches vllm_model_params", func() {
|
||||
@@ -43,9 +43,12 @@ var _ = Describe("C ABI struct mirrors", func() {
|
||||
Expect(unsafe.Offsetof(p.SchedulingPolicy)).To(Equal(uintptr(64)))
|
||||
Expect(unsafe.Offsetof(p.KVTransferConfig)).To(Equal(uintptr(72)))
|
||||
Expect(unsafe.Offsetof(p.EnableJumpForward)).To(Equal(uintptr(80)))
|
||||
// 88, not 84: the struct is 8-aligned (it holds pointers), so the
|
||||
// trailing int32 is padded out. Go pads identically.
|
||||
Expect(unsafe.Sizeof(p)).To(Equal(uintptr(88)))
|
||||
Expect(unsafe.Offsetof(p.Device)).To(Equal(uintptr(84)))
|
||||
// 88, not 92: gpu_memory_utilization is a double, so it takes the next
|
||||
// 8-aligned slot after the int32 pair. Go pads identically.
|
||||
Expect(unsafe.Offsetof(p.GPUMemoryUtil)).To(Equal(uintptr(88)))
|
||||
Expect(unsafe.Offsetof(p.KVCacheMemoryBytes)).To(Equal(uintptr(96)))
|
||||
Expect(unsafe.Sizeof(p)).To(Equal(uintptr(104)))
|
||||
})
|
||||
|
||||
It("cSamplingParams matches vllm_sampling_params (ABI v8)", func() {
|
||||
|
||||
@@ -304,6 +304,20 @@ var BackendCapabilities = map[string]BackendCapability{
|
||||
AcceptsImages: true,
|
||||
Description: "SGLang — fast LLM inference with structured generation and optional vision",
|
||||
},
|
||||
// vllm-cpp serves two mutually exclusive engine handles from one backend:
|
||||
// a text engine, and MiniMax-H3's video+audio engine when the model config
|
||||
// declares the H3 checkpoint set. Both usecases are possible, and chat is
|
||||
// the default because a config that says nothing is a text model.
|
||||
//
|
||||
// AcceptsImages is the fl2va keyframe (start_image/end_image), the same
|
||||
// reason longcat-video declares it; the text path takes no image input.
|
||||
"vllm-cpp": {
|
||||
GRPCMethods: []GRPCMethod{MethodPredict, MethodPredictStream, MethodGenerateVideo},
|
||||
PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseVideo},
|
||||
DefaultUsecases: []string{UsecaseChat},
|
||||
AcceptsImages: true,
|
||||
Description: "vllm.cpp — the LocalAI team's C++20 port of vLLM; text generation plus MiniMax-H3 video+audio generation",
|
||||
},
|
||||
"vllm-omni": {
|
||||
GRPCMethods: []GRPCMethod{MethodPredict, MethodPredictStream, MethodGenerateImage, MethodGenerateVideo, MethodTTS},
|
||||
PossibleUsecases: []string{UsecaseChat, UsecaseCompletion, UsecaseImage, UsecaseVideo, UsecaseTTS, UsecaseVision},
|
||||
|
||||
@@ -209,16 +209,15 @@ func VideoEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfi
|
||||
config.Backend = model.StableDiffusionGGMLBackend
|
||||
}
|
||||
|
||||
// Unset geometry is passed through as 0 so the BACKEND supplies its own
|
||||
// default canvas. Every video backend already does: stablediffusion-ggml
|
||||
// falls back to 512x512, diffusers to 1280x720, longcat-video to 832x480,
|
||||
// vllm-cpp to MiniMax-H3's trained 1344x768. Forcing 512x512 here made a
|
||||
// request that asked for nothing render at a size three of the four were
|
||||
// never trained at, and there is no size that is right for all of them.
|
||||
width := input.Width
|
||||
height := input.Height
|
||||
|
||||
if width == 0 {
|
||||
width = 512
|
||||
}
|
||||
if height == 0 {
|
||||
height = 512
|
||||
}
|
||||
|
||||
b64JSON := input.ResponseFormat == "b64_json"
|
||||
|
||||
tempDir := ""
|
||||
|
||||
@@ -172,7 +172,7 @@ LocalAI supports various types of backends:
|
||||
- **Text-to-Speech Backends**: For speech synthesis (e.g., piper, Kokoro, VibeVoice, Qwen3-TTS, [NeMo-Speech.cpp]({{%relref "features/nemo-speech-cpp" %}}), [audio.cpp]({{%relref "features/audio-cpp" %}}))
|
||||
- **Sound Generation Backends**: For music and audio generation (e.g., ACE-Step, [audio.cpp]({{%relref "features/audio-cpp" %}}))
|
||||
- **Sound Classification Backends**: For sound-event classification / audio tagging - identifying everyday sounds like baby cry, glass breaking, alarms (e.g., ced.cpp)
|
||||
- **Image & Video Generation Backends**: For diffusion and audio-conditioned avatar models (e.g., stable-diffusion.cpp, diffusers, vLLM-Omni, [LongCat-Video]({{%relref "features/video-generation" %}}))
|
||||
- **Image & Video Generation Backends**: For diffusion and audio-conditioned avatar models (e.g., stable-diffusion.cpp, diffusers, vLLM-Omni, [LongCat-Video]({{%relref "features/video-generation" %}}), [vllm.cpp / MiniMax-H3]({{%relref "features/video-generation" %}}))
|
||||
- **3D Generation Backends**: For image-to-3D mesh generation ([trellis2.cpp]({{%relref "features/3d-generation" %}}) — Microsoft TRELLIS.2, producing GLB assets with PBR textures)
|
||||
- **Vision & Detection Backends**: For object detection, segmentation, depth, and face/voice recognition (e.g., rf-detr.cpp, locate-anything.cpp, sam3.cpp, insightface)
|
||||
- **Audio Processing Backends**: For voice activity detection and audio enhancement (e.g., Silero VAD, LocalVQE, [audio.cpp]({{%relref "features/audio-cpp" %}}))
|
||||
|
||||
@@ -6,7 +6,7 @@ url = "/features/video-generation/"
|
||||
aliases = ["/features/longcat-video/"]
|
||||
+++
|
||||
|
||||
LocalAI can generate videos from text prompts and optional image or audio conditioning via the `/video` endpoint. Supported backends include `diffusers`, `stablediffusion`, `vllm-omni`, and the dedicated `longcat-video` backend.
|
||||
LocalAI can generate videos from text prompts and optional image or audio conditioning via the `/video` endpoint. Supported backends include `diffusers`, `stablediffusion`, `vllm-omni`, `vllm-cpp` (MiniMax-H3, which generates video **and** audio together), and the dedicated `longcat-video` backend.
|
||||
|
||||
## API
|
||||
|
||||
@@ -25,8 +25,8 @@ The request body is JSON with the following fields:
|
||||
| `start_image` | `string` | No | | Starting image as base64 string or URL |
|
||||
| `end_image` | `string` | No | | Ending image for guided generation |
|
||||
| `audio` | `string` | No | | Audio conditioning as base64, a data URI, or URL |
|
||||
| `width` | `int` | No | 512 | Video width in pixels |
|
||||
| `height` | `int` | No | 512 | Video height in pixels |
|
||||
| `width` | `int` | No | backend | Video width in pixels; omit it to get the model's own default canvas |
|
||||
| `height` | `int` | No | backend | Video height in pixels; omit it to get the model's own default canvas |
|
||||
| `num_frames` | `int` | No | | Number of frames |
|
||||
| `fps` | `int` | No | | Frames per second |
|
||||
| `seconds` | `string` | No | | Duration in seconds |
|
||||
@@ -279,6 +279,158 @@ With distillation enabled, Avatar uses eight inference steps and fixed text/audi
|
||||
- **Out of memory while loading**: use BF16 on unified-memory hardware, close other GPU workloads, or reduce model concurrency. INT8 is not guaranteed to reduce peak load memory.
|
||||
- **Slow first request**: the backend and checkpoints are downloaded and loaded on demand; subsequent requests reuse the loaded pipeline.
|
||||
|
||||
## MiniMax-H3 (vllm.cpp)
|
||||
|
||||
The `vllm-cpp` backend — LocalAI's own C++ port of vLLM — also serves MiniMax-H3, which generates **video and audio jointly**. The clip comes back as an MP4 with a real AAC track rather than a silent render.
|
||||
|
||||
| Gallery model | Upstream checkpoint | Inputs | Output |
|
||||
|---------------|---------------------|--------|--------|
|
||||
| `minimax-h3-fl2va-q4` | `MiniMaxAI/MiniMax-H3`, Q4_K_M FL2VA partition | text, optional start/end frame | video with generated audio |
|
||||
|
||||
```bash
|
||||
local-ai models install minimax-h3-fl2va-q4
|
||||
```
|
||||
|
||||
{{% notice warning %}}
|
||||
This is a large, slow model. The five weight files total roughly 40 GB, and generation was measured at about 176 seconds per denoise step at the default 1344x768 canvas on a 20-SM device — so the 50-step default is a **multi-hour** request, not a multi-second one. Nothing in the path imposes a deadline, but plan for a long-running HTTP call, and use a CUDA host.
|
||||
{{% /notice %}}
|
||||
|
||||
### Ask for the sound
|
||||
|
||||
The model generates picture and sound from the same prompt, so a prompt that only describes what is *seen* produces room tone and ambience. To get speech, say that the character talks and put the words in the prompt:
|
||||
|
||||
```text
|
||||
It is TALKING to the camera: its mouth moves clearly in sync with its speech,
|
||||
in a dry, deadpan tone.
|
||||
|
||||
It says, clearly and audibly: "Michael scheduled another all-hands.
|
||||
It is about the printer. Again."
|
||||
|
||||
Audio: a single clear voice, close-miked, with quiet room tone underneath.
|
||||
```
|
||||
|
||||
### Geometry and clip length
|
||||
|
||||
The trained canvas is **1344x768 at 124 frames and 24 fps**, about 5.2 seconds, and that is what the gallery entry defaults to. Two rules the engine enforces:
|
||||
|
||||
- The canvas is truncated onto a 32-pixel grid.
|
||||
- The frame count sits on a **17n+5** grid (…, 90, 107, 124, 141, …). A count off the grid is rounded up, and LocalAI logs the value it actually rendered.
|
||||
|
||||
The trained clip range is roughly 124 to 362 frames (about 5 to 15 seconds).
|
||||
|
||||
### Text-to-video with sound
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "minimax-h3-fl2va-q4",
|
||||
"prompt": "A cyan llama mascot in a grey office chair, talking to the camera. It says, clearly and audibly: \"the printer is down again\". Audio: one clear close-miked voice.",
|
||||
"num_frames": 124,
|
||||
"step": 50,
|
||||
"seed": 42
|
||||
}'
|
||||
```
|
||||
|
||||
### First-frame conditioning
|
||||
|
||||
`start_image` pins the supplied image as frame 0 (H3's `fl2va` task); `end_image` pins the last frame. LocalAI converts the upload to the binary PPM at the exact output canvas that the engine requires, using `ffmpeg`, so PNG and JPEG uploads work. When no `width`/`height` is given, the canvas is derived from the image's aspect on a 768-pixel short edge.
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/video \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"model\": \"minimax-h3-fl2va-q4\",
|
||||
\"prompt\": \"the subject turns toward the camera and starts speaking\",
|
||||
\"start_image\": \"$(base64 --wrap=0 portrait.png)\"
|
||||
}"
|
||||
```
|
||||
|
||||
### Partitions: what this checkpoint will and will not do
|
||||
|
||||
MiniMax-H3 ships as two DiT partitions, and the gallery entry installs **FL2VA**, which serves `t2va` (text only) and `fl2va` (first/last frame). Reference conditioning — a whole reference image, a reference clip, or reference audio — belongs to the separate **Ref2VA** checkpoint.
|
||||
|
||||
This matters because the mismatch does not fail cleanly upstream: a reference passed to an FL2VA DiT renders for hours and returns a coloured lattice over the frame. The backend refuses the combination up front instead, naming the partition. The community quantisations strip the release metadata and the two DiTs are byte-structurally identical, so the partition is *declared* in the model config (`video_partition`) rather than detected.
|
||||
|
||||
### ffmpeg is required on the host
|
||||
|
||||
The engine writes frames and a WAV and composes the `ffmpeg` command line; LocalAI runs it. That process boundary is deliberate upstream, so the backend image ships no `ffmpeg`: install one on the host, or point `options: [ffmpeg:/path/to/ffmpeg]` at a binary. Without it, generation succeeds and the mux fails with a message saying so.
|
||||
|
||||
### MiniMax-H3 model configuration
|
||||
|
||||
```yaml
|
||||
name: minimax-h3-fl2va-q4
|
||||
backend: vllm-cpp
|
||||
cuda: true
|
||||
known_usecases:
|
||||
- video
|
||||
known_input_modalities:
|
||||
- text
|
||||
- image
|
||||
known_output_modalities:
|
||||
- video
|
||||
options:
|
||||
- video_encoder:minimax-h3/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
|
||||
- video_tokenizer:minimax-h3/tokenizer.json
|
||||
- video_vae:minimax-h3/video_vae.safetensors
|
||||
- video_vae_config:minimax-h3/video_vae_config.json
|
||||
- audio_vae:minimax-h3/audio_vae.safetensors
|
||||
- audio_vae_config:minimax-h3/audio_vae_config.json
|
||||
- video_partition:fl2va
|
||||
- video_device:cuda
|
||||
- video_dequant_bf16:true
|
||||
- video_width:1344
|
||||
- video_height:768
|
||||
- video_num_frames:124
|
||||
parameters:
|
||||
model: minimax-h3/MiniMax-H3-FL2VA-Q4_K_M.gguf
|
||||
```
|
||||
|
||||
`parameters.model` is the DiT. H3 is a checkpoint *set* rather than one model directory, so the encoder, the tokenizer and the two VAEs are named in `options`. Relative paths resolve against the models directory.
|
||||
|
||||
#### Load options
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `video_encoder` | — | H3 text encoder (GGUF or a bf16 shard directory). Required unless `video_prompt_embeds` is set |
|
||||
| `video_tokenizer` | — | `tokenizer.json` for the encoder |
|
||||
| `video_vae` | — | Video VAE weights (`.safetensors`). Required |
|
||||
| `video_vae_config` | `config.json` beside the weights | Carries `latents_mean` / `latents_std` and `clip_length` / `token_drop`; the decode is wrong without it |
|
||||
| `audio_vae` | — | Audio VAE weights. Required |
|
||||
| `audio_vae_config` | `config.json` beside the weights | As above, for audio |
|
||||
| `video_prompt_embeds` | — | Pre-computed f32 conditioning, as an alternative to an encoder |
|
||||
| `video_partition` | `fl2va` | `fl2va` or `ref2va`; must match the DiT you installed |
|
||||
| `video_device` | `cpu`, or `cuda` when the config sets `cuda: true` | `cpu` or `cuda` |
|
||||
| `video_dequant_bf16` | `false` | Dequantise and stream the DiT as bf16; what the Q4_K_M GGUF arm wants |
|
||||
| `video_fp4_resident` | `false` | NVFP4 on CUDA: keep FP4 packed and use the Marlin W4A16 GEMM |
|
||||
| `video_width` / `video_height` | 1344 / 768 in the gallery entry | Default canvas when the request omits it |
|
||||
| `video_num_frames` | 124 in the gallery entry | Default clip length when the request omits it |
|
||||
| `video_steps` | engine default (50) | Default denoise steps when the request omits it |
|
||||
| `video_workdir` | a temporary directory | Where `frame_%06d.ppm` and `audio.wav` land. Set it to keep every run's frames |
|
||||
| `video_crf` | 18 | x264 CRF for the mux |
|
||||
| `ffmpeg` | `ffmpeg` from `PATH` | The mux binary |
|
||||
|
||||
#### Per-request parameters
|
||||
|
||||
The `/video` request's `params` object accepts string values. Unknown keys are rejected rather than ignored, so a typo does not cost you a multi-hour render of the wrong thing.
|
||||
|
||||
| Parameter | Description |
|
||||
|-----------|-------------|
|
||||
| `noise_aug` | Keyframe pinning strength; the default is 1.0 |
|
||||
| `ref_image` | `ref2va` only: one whole reference image, as a binary PPM |
|
||||
| `ref_video` | `ref2va` only: a directory of `frame_%06d.ppm` |
|
||||
| `crf` | Per-request x264 CRF override |
|
||||
|
||||
`negative_prompt`, `cfg_scale` and `fps` have no MiniMax-H3 equivalent: H3 has no negative prompt or CFG scale, and it renders at a fixed frame rate that the audio track is synchronised to. Setting them is logged and ignored rather than silently honoured.
|
||||
|
||||
### MiniMax-H3 troubleshooting
|
||||
|
||||
- **`ffmpeg not found`**: install ffmpeg on the host or set `options: [ffmpeg:<path>]`. The frames and WAV are already rendered; only the mux failed.
|
||||
- **`the FL2VA checkpoint serves t2va and fl2va only`**: you passed a reference image, clip or audio to the FL2VA DiT. Use `start_image` for first-frame conditioning, or install a Ref2VA checkpoint.
|
||||
- **`video_partition must be "fl2va" or "ref2va"`**: the config declares something else.
|
||||
- **`unknown params key`**: `params` accepts only the four keys above.
|
||||
- **Text inside the frame comes out malformed**: this is the model's weakest area. Composite logos and signage in afterwards.
|
||||
|
||||
## Error Responses
|
||||
|
||||
| Status Code | Description |
|
||||
|
||||
@@ -8497,6 +8497,101 @@
|
||||
- vae/**
|
||||
parameters:
|
||||
model: meituan-longcat/LongCat-Video-Avatar-1.5
|
||||
- name: minimax-h3-fl2va-q4
|
||||
url: github:mudler/LocalAI/gallery/virtual.yaml@master
|
||||
urls:
|
||||
- https://huggingface.co/MiniMaxAI/MiniMax-H3
|
||||
- https://huggingface.co/realrebelai/MiniMax-H3_GGUFs
|
||||
- https://huggingface.co/lilcheaty/MiniMax-H3-NVFP4
|
||||
- https://github.com/mudler/vllm.cpp
|
||||
description: |
|
||||
MiniMax-H3 served by vllm.cpp, LocalAI's own C++ port of vLLM. It generates
|
||||
video AND audio jointly from a text prompt, so a clip comes back as an MP4
|
||||
with a real soundtrack rather than a silent render: ask for speech in the
|
||||
prompt and the model lip-syncs it.
|
||||
|
||||
This is the Q4_K_M quantisation of the FL2VA partition, which serves
|
||||
text-to-video (t2va) and first/last-frame conditioning (fl2va). Reference
|
||||
conditioning (ref2va) is a different checkpoint and is refused by this one.
|
||||
|
||||
Roughly 40 GB of weights across five files, plus the two VAE configs that
|
||||
carry the latent statistics. The default canvas is 1344x768
|
||||
at 124 frames and 24 fps, about 5.2 seconds. Generation is slow — measured
|
||||
at roughly 176 s per denoise step at that canvas on a 20-SM device, so the
|
||||
50-step default is a multi-hour job. Muxing the finished frames needs
|
||||
ffmpeg on the host.
|
||||
license: other
|
||||
icon: https://huggingface.co/MiniMaxAI/MiniMax-H3/resolve/main/assets/minimax-h3.png
|
||||
tags:
|
||||
- text-to-video
|
||||
- image-to-video
|
||||
- video-generation
|
||||
- audio-video-generation
|
||||
- minimax-h3
|
||||
- vllm-cpp
|
||||
- cuda
|
||||
- gpu
|
||||
- dgx-spark
|
||||
last_checked: "2026-08-09"
|
||||
overrides:
|
||||
backend: vllm-cpp
|
||||
# The video engine has no auto device slot; this is what selects CUDA over
|
||||
# the CPU queue, which for a multi-hour render is not a small difference.
|
||||
cuda: true
|
||||
known_usecases:
|
||||
- video
|
||||
known_input_modalities:
|
||||
- text
|
||||
- image
|
||||
known_output_modalities:
|
||||
- video
|
||||
options:
|
||||
# MiniMax-H3 is a checkpoint SET, not one model directory: parameters.model
|
||||
# is the DiT and the rest of the set is named here.
|
||||
- video_encoder:minimax-h3/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
|
||||
- video_tokenizer:minimax-h3/tokenizer.json
|
||||
- video_vae:minimax-h3/video_vae.safetensors
|
||||
- video_vae_config:minimax-h3/video_vae_config.json
|
||||
- audio_vae:minimax-h3/audio_vae.safetensors
|
||||
- audio_vae_config:minimax-h3/audio_vae_config.json
|
||||
# The community GGUF strips the release metadata and the FL2VA and Ref2VA
|
||||
# DiTs are byte-structurally identical, so the partition must be declared.
|
||||
# This is the FL2VA checkpoint: t2va and fl2va, never ref2va.
|
||||
- video_partition:fl2va
|
||||
- video_device:cuda
|
||||
# Q4_K_M streams up as bf16; keep-quant is for the NVFP4 arm.
|
||||
- video_dequant_bf16:true
|
||||
# H3's trained canvas and clip length. Frame counts sit on a 17n+5 grid.
|
||||
- video_width:1344
|
||||
- video_height:768
|
||||
- video_num_frames:124
|
||||
parameters:
|
||||
model: minimax-h3/MiniMax-H3-FL2VA-Q4_K_M.gguf
|
||||
files:
|
||||
- filename: minimax-h3/MiniMax-H3-FL2VA-Q4_K_M.gguf
|
||||
sha256: 5e8fa6e960d5fbd547390ceec63fcead275435d8f3bd2466a8a2cbd8c2e361e3
|
||||
uri: huggingface://realrebelai/MiniMax-H3_GGUFs/MiniMax-H3-FL2VA-Q4_K_M.gguf
|
||||
- filename: minimax-h3/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
|
||||
sha256: 1bf75e038c5895b97b6ea16cc1e3d32076254b06ec3df10657650d86dc82279e
|
||||
uri: huggingface://realrebelai/MiniMax-H3_GGUFs/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf
|
||||
- filename: minimax-h3/video_vae.safetensors
|
||||
sha256: 7c1f131492e7eddacaac9069a61b81bdd39de5cc96561e677c5eab1cdce5e522
|
||||
uri: huggingface://lilcheaty/MiniMax-H3-NVFP4/vae/minimax_h3_video_vae_fp16.safetensors
|
||||
- filename: minimax-h3/audio_vae.safetensors
|
||||
sha256: 37dddc2f3e6d5d5139d823d5ea283bbf304dadcb885b1ccda818aa13dade5ea2
|
||||
uri: huggingface://MiniMaxAI/MiniMax-H3/FL2VA/audio_vae/model.safetensors
|
||||
# Each VAE config carries its per-channel latents_mean / latents_std and the
|
||||
# temporal clip_length / token_drop. The decode is wrong without them, so
|
||||
# they ship with the weights rather than being optional extras.
|
||||
- filename: minimax-h3/video_vae_config.json
|
||||
sha256: 3edd2cdd1ebc823c868be55ef917e1b3b8a398fde4d3150dae44a3bf05d9f627
|
||||
uri: huggingface://MiniMaxAI/MiniMax-H3/FL2VA/video_vae/config.json
|
||||
- filename: minimax-h3/audio_vae_config.json
|
||||
sha256: d8f3bcc62e23c7e9806970fa63cca6139c06faa3797cf9c94034f60db8512771
|
||||
uri: huggingface://MiniMaxAI/MiniMax-H3/FL2VA/audio_vae/config.json
|
||||
- filename: minimax-h3/tokenizer.json
|
||||
sha256: a5d85b6dcc535e6b93115a9ef287e6132fdbf30270da6218194ba742261173c7
|
||||
uri: huggingface://MiniMaxAI/MiniMax-H3/FL2VA/tokenizer/tokenizer.json
|
||||
- name: vllm-omni-qwen3-omni-30b
|
||||
url: github:mudler/LocalAI/gallery/virtual.yaml@master
|
||||
urls:
|
||||
|
||||
Reference in new issue
Block a user