mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-31 10:28:43 -04:00
b00422e45f
* feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
45 lines
2.3 KiB
Markdown
45 lines
2.3 KiB
Markdown
# LongCat Video backend
|
|
|
|
This backend serves Meituan's `LongCat-Video` and
|
|
`LongCat-Video-Avatar-1.5` checkpoints through LocalAI's `GenerateVideo`
|
|
RPC. It supports:
|
|
|
|
- text-to-video and image-to-video with `LongCat-Video`;
|
|
- audio + text-to-avatar and portrait + audio-to-avatar with Avatar 1.5;
|
|
- multi-segment avatar continuation for speech longer than one segment;
|
|
- PyTorch SDPA when FlashAttention is unavailable, including CUDA 13 ARM64
|
|
systems such as NVIDIA DGX Spark.
|
|
|
|
Install the `longcat-video` or `longcat-video-avatar-1.5` recipe from the
|
|
LocalAI Model Gallery. See the [LongCat user guide](../../../docs/content/features/longcat-video.md)
|
|
for Studio and API examples, hardware requirements, and manual configuration.
|
|
|
|
The upstream source is pinned in `Makefile` and patched at build time. The
|
|
patch adds only the missing SDPA attention branches; model and source licenses
|
|
remain MIT.
|
|
|
|
## Model options
|
|
|
|
| Option | Default | Description |
|
|
| --- | --- | --- |
|
|
| `attention_backend` | `sdpa` | `sdpa`, `auto`, `flash2`, `flash3`, or `xformers`. The packaged backend guarantees only `sdpa`. |
|
|
| `use_distill` | `true` for Avatar, `false` for base | Loads the checkpoint's fast distillation LoRA. |
|
|
| `use_int8` | `false` | Loads Avatar 1.5's INT8 DiT. BF16 has a lower load-time peak on unified-memory systems. |
|
|
| `base_model` | `meituan-longcat/LongCat-Video` | Base components used by Avatar 1.5. |
|
|
| `max_segments` | `8` | Maximum avatar continuation segments accepted per request. |
|
|
| `resolution` | `480p` | Image-conditioned generation resolution (`480p` or `720p`). |
|
|
|
|
Per-request `params` may set `num_segments`, `audio_guidance_scale`,
|
|
`offload_kv_cache`, `ref_img_index`, `mask_frame_range`, and `resolution`.
|
|
|
|
Gallery and imported configs declare `known_input_modalities` and
|
|
`known_output_modalities`. Keep those declarations in manual configs as well;
|
|
they let model discovery distinguish base image-conditioned video from Avatar
|
|
audio conditioning without inspecting the backend or checkpoint name.
|
|
|
|
LongCat is CUDA-only and very large. Avatar 1.5 also loads tokenizer,
|
|
text-encoder, and VAE components from the base checkpoint. Keep ample unified
|
|
memory and storage available; no CPU or macOS backend image is published. The
|
|
initial backend supports one GPU per process; tensor parallel sizes above one
|
|
are rejected explicitly.
|