mirror of
https://github.com/mudler/LocalAI.git
synced 2026-02-13 08:03:18 -05:00
cff972094c
* add experimental support for sd_embed-style prompt embedding Signed-off-by: Austen Dicken <cvpcsm@gmail.com> * add doc equivalent to compel Signed-off-by: Austen Dicken <cvpcsm@gmail.com> * need to use flux1 embedding function for flux model Signed-off-by: Austen Dicken <cvpcsm@gmail.com> --------- Signed-off-by: Austen Dicken <cvpcsm@gmail.com>
137 lines
4.7 KiB
Markdown
137 lines
4.7 KiB
Markdown
# LocalAI Diffusers Backend
|
|
|
|
This backend provides gRPC access to Hugging Face diffusers pipelines with dynamic pipeline loading.
|
|
|
|
## Creating a separate environment for the diffusers project
|
|
|
|
```
|
|
make diffusers
|
|
```
|
|
|
|
## Dynamic Pipeline Loader
|
|
|
|
The diffusers backend includes a dynamic pipeline loader (`diffusers_dynamic_loader.py`) that automatically discovers and loads diffusers pipelines at runtime. This eliminates the need for per-pipeline conditional statements - new pipelines added to diffusers become available automatically without code changes.
|
|
|
|
### How It Works
|
|
|
|
1. **Pipeline Discovery**: On first use, the loader scans the `diffusers` package to find all classes that inherit from `DiffusionPipeline`.
|
|
|
|
2. **Registry Caching**: Discovery results are cached for the lifetime of the process to avoid repeated scanning.
|
|
|
|
3. **Task Aliases**: The loader automatically derives task aliases from class names (e.g., "text-to-image", "image-to-image", "inpainting") without hardcoding.
|
|
|
|
4. **Multiple Resolution Methods**: Pipelines can be resolved by:
|
|
- Exact class name (e.g., `StableDiffusionPipeline`)
|
|
- Task alias (e.g., `text-to-image`, `img2img`)
|
|
- Model ID (uses HuggingFace Hub to infer pipeline type)
|
|
|
|
### Usage Examples
|
|
|
|
```python
|
|
from diffusers_dynamic_loader import (
|
|
load_diffusers_pipeline,
|
|
get_available_pipelines,
|
|
get_available_tasks,
|
|
resolve_pipeline_class,
|
|
discover_diffusers_classes,
|
|
get_available_classes,
|
|
)
|
|
|
|
# List all available pipelines
|
|
pipelines = get_available_pipelines()
|
|
print(f"Available pipelines: {pipelines[:10]}...")
|
|
|
|
# List all task aliases
|
|
tasks = get_available_tasks()
|
|
print(f"Available tasks: {tasks}")
|
|
|
|
# Resolve a pipeline class by name
|
|
cls = resolve_pipeline_class(class_name="StableDiffusionPipeline")
|
|
|
|
# Resolve by task alias
|
|
cls = resolve_pipeline_class(task="stable-diffusion")
|
|
|
|
# Load and instantiate a pipeline
|
|
pipe = load_diffusers_pipeline(
|
|
class_name="StableDiffusionPipeline",
|
|
model_id="runwayml/stable-diffusion-v1-5",
|
|
torch_dtype=torch.float16
|
|
)
|
|
|
|
# Load from single file
|
|
pipe = load_diffusers_pipeline(
|
|
class_name="StableDiffusionPipeline",
|
|
model_id="/path/to/model.safetensors",
|
|
from_single_file=True,
|
|
torch_dtype=torch.float16
|
|
)
|
|
|
|
# Discover other diffusers classes (schedulers, models, etc.)
|
|
schedulers = discover_diffusers_classes("SchedulerMixin")
|
|
print(f"Available schedulers: {list(schedulers.keys())[:5]}...")
|
|
|
|
# Get list of available scheduler classes
|
|
scheduler_list = get_available_classes("SchedulerMixin")
|
|
```
|
|
|
|
### Generic Class Discovery
|
|
|
|
The dynamic loader can discover not just pipelines but any class type from diffusers:
|
|
|
|
```python
|
|
# Discover all scheduler classes
|
|
schedulers = discover_diffusers_classes("SchedulerMixin")
|
|
|
|
# Discover all model classes
|
|
models = discover_diffusers_classes("ModelMixin")
|
|
|
|
# Get a sorted list of available classes
|
|
scheduler_names = get_available_classes("SchedulerMixin")
|
|
```
|
|
|
|
### Special Pipeline Handling
|
|
|
|
Most pipelines are loaded dynamically through `load_diffusers_pipeline()`. Only pipelines requiring truly custom initialization logic are handled explicitly:
|
|
|
|
- `FluxTransformer2DModel`: Requires quantization and custom transformer loading (cannot use dynamic loader)
|
|
- `WanPipeline` / `WanImageToVideoPipeline`: Uses dynamic loader with special VAE (float32 dtype)
|
|
- `SanaPipeline`: Uses dynamic loader with post-load dtype conversion for VAE/text encoder
|
|
- `StableVideoDiffusionPipeline`: Uses dynamic loader with CPU offload handling
|
|
- `VideoDiffusionPipeline`: Alias for DiffusionPipeline with video flags
|
|
|
|
All other pipelines (StableDiffusionPipeline, StableDiffusionXLPipeline, FluxPipeline, etc.) are loaded purely through the dynamic loader.
|
|
|
|
### Error Handling
|
|
|
|
When a pipeline cannot be resolved, the loader provides helpful error messages listing available pipelines and tasks:
|
|
|
|
```
|
|
ValueError: Unknown pipeline class 'NonExistentPipeline'.
|
|
Available pipelines: AnimateDiffPipeline, AnimateDiffVideoToVideoPipeline, ...
|
|
```
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `COMPEL` | `0` | Enable Compel for prompt weighting |
|
|
| `SD_EMBED` | `0` | Enable sd_embed for prompt weighting |
|
|
| `XPU` | `0` | Enable Intel XPU support |
|
|
| `CLIPSKIP` | `1` | Enable CLIP skip support |
|
|
| `SAFETENSORS` | `1` | Use safetensors format |
|
|
| `CHUNK_SIZE` | `8` | Decode chunk size for video |
|
|
| `FPS` | `7` | Video frames per second |
|
|
| `DISABLE_CPU_OFFLOAD` | `0` | Disable CPU offload |
|
|
| `FRAMES` | `64` | Number of video frames |
|
|
| `BFL_REPO` | `ChuckMcSneed/FLUX.1-dev` | Flux base repo |
|
|
| `PYTHON_GRPC_MAX_WORKERS` | `1` | Max gRPC workers |
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
./test.sh
|
|
```
|
|
|
|
The test suite includes:
|
|
- Unit tests for the dynamic loader (`test_dynamic_loader.py`)
|
|
- Integration tests for the gRPC backend (`test.py`) |