mirror of
https://github.com/mudler/LocalAI.git
synced 2026-11-04 11:33:22 -05:00
mudler-agentandEttore Di Giacinto
e28d5009d9
* feat(stablediffusion-ggml): run MiniMax-H3 video with reference conditioning Bump stable-diffusion.cpp to 2988060 and follow its tiling rename (tile_size_x/y -> tile_size_w/h, rel_size_x/y -> rel_size_w/h). GenerateVideo now passes params["ref_image"] and the request audio to the MiniMax-H3 Ref2VA inputs (ref_images, ref_audios). The audio must be a WAV file. params["ref_video"] is refused, since the proto cannot carry a frame sequence yet. stable-diffusion.cpp cannot tell the FL2VA and REF2VA DiTs apart, and a reference input on FL2VA renders a wrong video without an error. The backend now reads the video_partition option, or the model file name, and fails early on that mix. Keyframes plus references are refused too. Add gallery entries for the FL2VA and REF2VA Q4_K_M sets. They do not enable any bf16 dequantisation. Assisted-by: Claude:claude-sonnet-5-5 go vet gofmt * feat(stablediffusion-ggml): take several reference images, videos and audio clips Phase 1 passed one reference image and one reference audio clip to stable-diffusion.cpp and refused ref_video. Add repeated reference fields to GenerateVideoRequest (ref_images, ref_videos, ref_audios) so the MiniMax-H3 Ref2VA path can use as many references as upstream accepts. A reference video is a directory of frames with an fps and an optional paired audio clip. The /video endpoint stages the new inputs through the same path as start_image, end_image and audio. Frames of one reference video are written to a private directory under zero-padded names so the backend can read them in order. The input count and frame count per request are bounded. The backend merges the new fields with params.ref_image, params.ref_video and audio, which keep working. The FL2VA partition refuses any reference input, REF2VA accepts all of them, and keyframes stay exclusive with references. Reference audio is still WAV only. Assisted-by: Claude:claude-sonnet-5-5 [Bash] * feat(gallery): offer MiniMax-H3 by partition, DiT layout and format Name each MiniMax-H3 entry after its partition (fl2va or ref2va), DiT layout (unpruned or pruned) and format. Add the vllm.cpp builds that its model page lists: pruned Q8_0 (renders), pruned Q6_K, Q5_0, Q4_K, Q3_K and Q2_K (load only), and the pruned FL2VA NVFP4 file (loads only). The earlier names minimax-h3-fl2va-q4 and minimax-h3-ref2va-q4 stay as aliases. The two stable-diffusion.cpp entries become minimax-h3-fl2va-unpruned-q4-ggml and minimax-h3-ref2va-pruned-q4-ggml, and the latter now lists video input. Fix two bad sources. The stable-diffusion.cpp entries took the VAEs from a path that does not exist in the GGUF repository; they now use the Comfy-Org copies. The vllm.cpp entries paired the fp16 video VAE weights with a config from a different directory; they now use the official weights and config from FL2VA/video_vae/source/. Drop video_dequant_bf16 from every vllm.cpp entry. Expanding a quantised DiT to bf16 is the suspected cause of a host out-of-memory error, so the quantised weights stay resident. The model page records its renders with the expansion on, and each description says so. Assisted-by: Claude:claude-sonnet-5-5 [Bash] * feat(gallery): add pruned FL2VA Q4 MiniMax-H3 for sd.cpp Add minimax-h3-fl2va-pruned-q4-ggml, the 11.4 GB pruned FL2VA counterpart of the existing pruned REF2VA entry. Assisted-by: Claude Code:claude-sonnet-5-5 [rc] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
39 lines
2.3 KiB
C++
39 lines
2.3 KiB
C++
#include <cstdint>
|
|
#include "stable-diffusion.h"
|
|
|
|
#ifdef __cplusplus
|
|
extern "C" {
|
|
#endif
|
|
|
|
// Opaque ESRGAN context. Run returns 0, or argument/decode/inference/output/write
|
|
// errors (1..5); all image allocations remain native.
|
|
void* gosd_upscaler_create(const char* path, bool direct, int threads, int tile_size, const char* backend, const char* params_backend);
|
|
int gosd_upscaler_scale(void* handle);
|
|
int gosd_upscaler_run(void* handle, const char* src, const char* dst, int scale);
|
|
void gosd_upscaler_destroy(void* handle);
|
|
|
|
void sd_tiling_params_set_enabled(sd_tiling_params_t *params, bool enabled);
|
|
void sd_tiling_params_set_tile_sizes(sd_tiling_params_t *params, int tile_size_w, int tile_size_h);
|
|
void sd_tiling_params_set_rel_sizes(sd_tiling_params_t *params, float rel_size_w, float rel_size_h);
|
|
void sd_tiling_params_set_target_overlap(sd_tiling_params_t *params, float target_overlap);
|
|
sd_tiling_params_t* sd_img_gen_params_get_vae_tiling_params(sd_img_gen_params_t *params);
|
|
|
|
sd_img_gen_params_t* sd_img_gen_params_new(void);
|
|
void sd_img_gen_params_set_prompts(sd_img_gen_params_t *params, const char *prompt, const char *negative_prompt);
|
|
void sd_img_gen_params_set_dimensions(sd_img_gen_params_t *params, int width, int height);
|
|
void sd_img_gen_params_set_seed(sd_img_gen_params_t *params, int64_t seed);
|
|
void sd_img_gen_params_set_custom_sigmas(sd_img_gen_params_t *params, float *sigmas, int count);
|
|
|
|
int load_model(const char *model, char *model_path, char* options[], int threads, int diffusionModel);
|
|
int gen_image(sd_img_gen_params_t *p, int steps, char *dst, float cfg_scale, char *src_image, float strength, char *mask_image, char* ref_images[], int ref_images_count);
|
|
|
|
sd_vid_gen_params_t* sd_vid_gen_params_new(void);
|
|
void sd_vid_gen_params_set_prompts(sd_vid_gen_params_t *params, const char *prompt, const char *negative_prompt);
|
|
void sd_vid_gen_params_set_dimensions(sd_vid_gen_params_t *params, int width, int height);
|
|
void sd_vid_gen_params_set_seed(sd_vid_gen_params_t *params, int64_t seed);
|
|
void sd_vid_gen_params_set_video_frames(sd_vid_gen_params_t *params, int n);
|
|
int gen_video(sd_vid_gen_params_t *p, int steps, char *dst, float cfg_scale, int fps, char *init_image, char *end_image, char *ref_images, char *ref_videos, char *ref_audios);
|
|
#ifdef __cplusplus
|
|
}
|
|
#endif
|