LocalAI/docs/content/features/image-generation.md

+++
disableToc = false
title = "Image Generation"
weight = 50
url = "/features/image-generation/"
+++

![anime_girl](https://github.com/go-skynet/LocalAI/assets/2420543/8aaca62a-e864-4011-98ae-dcc708103928)
(Generated with [AnimagineXL](https://huggingface.co/Linaqruf/animagine-xl))

LocalAI supports generating images with Stable diffusion, running on CPU using C++ and Python implementations.

## Usage

OpenAI docs: https://platform.openai.com/docs/api-reference/images/create

To generate an image you can send a POST request to the `/v1/images/generations` endpoint with the instruction as the request body:

```bash
curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "A cute baby sea otter",
  "size": "256x256"
}'
```

Available additional parameters: `mode`, `step`, `negative_prompt`.

To set a negative prompt you can either pass a separate `negative_prompt` field or split the prompt with `|`, e.g. `a cute baby sea otter|malformed`. When both are present they are joined with a comma (`negative_prompt` first, then the `|`-suffixed tags), so a global negative prompt can live in `negative_prompt` while per-image negatives are appended after `|`.

```bash
curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "prompt": "floating hair, portrait, ((loli)), ((one girl)), cute face, hidden hands, asymmetrical bangs, beautiful detailed eyes, eye shadow, hair ornament, ribbons, bowties, buttons, pleated skirt, (((masterpiece))), ((best quality)), colorful|((part of the head)), ((((mutated hands and fingers)))), deformed, blurry, bad anatomy, disfigured, poorly drawn face, mutation, mutated, extra limb, ugly, poorly drawn hands, missing limb, blurry, floating limbs, disconnected limbs, malformed hands, blur, out of focus, long neck, long body, Octane renderer, lowres, bad anatomy, bad hands, text",
  "size": "256x256"
}'
```

## Backends

### stablediffusion-ggml

This backend is based on [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp). Every model supported by that backend is supported indeed with LocalAI.


#### Setup

There are already several models in the gallery that are available to install and get up and running with this backend, you can for example run flux by searching it in the Model gallery (`flux.1-dev-ggml`) or start LocalAI with `run`:

```bash
local-ai run flux.1-dev-ggml
```

To use a custom model, you can follow these steps:

1. Create a model file `stablediffusion.yaml` in the models folder:

```yaml
name: stablediffusion
backend: stablediffusion-ggml
parameters:
  model: gguf_model.gguf
step: 25
cfg_scale: 4.5
options:
- "clip_l_path:clip_l.safetensors"
- "clip_g_path:clip_g.safetensors"
- "t5xxl_path:t5xxl-Q5_0.gguf"
- "sampler:euler"
```

2. Download the required assets to the `models` repository
3. Start LocalAI

#### Memory and device placement options

When a model does not fit entirely in VRAM, the following `options:` control where weights and computation are placed. They map directly to the upstream stable-diffusion.cpp options.

| Option | Example | Description |
|--------|---------|-------------|
| `backend` | `backend:clip=cpu,vae=cuda0,diffusion=vulkan0` | Runtime (compute) backend assignment per component. Use `cpu` to place a component's compute on the CPU. Component keys include `te` (text encoder / CLIP), `vae`, `diffusion`, `controlnet`. |
| `params_backend` | `params_backend:diffusion=disk,clip=cpu` | Where parameters (weights) are stored. Supports `cpu`, `disk` (mmap weights from disk to save RAM/VRAM), or per-component specs. |
| `max_vram` | `max_vram:8` or `max_vram:-1` | Optional per-device VRAM budget (in GiB) for managed weights and automatic graph-cut execution. `0` uses live free VRAM without an explicit cap; a negative value reserves that many GiB of free VRAM. Also accepts per-backend budgets. |
| `stream_layers` | `stream_layers:true` | Deprecated compatibility option. Segmented weight streaming is now selected automatically, so this value is ignored. |
| `rpc_servers` | `rpc_servers:localhost:50052,192.168.1.3:50052` | Comma-separated list of `host:port` RPC servers to offload compute to. |
| `pulid_weights_path` | `pulid_weights_path:pulid.safetensors` | Path to PuLID-Flux weights for identity injection. |

The following convenience booleans are still accepted and are translated into the `backend` / `params_backend` specs above:

| Option | Equivalent spec |
|--------|-----------------|
| `offload_params_to_cpu:true` | `params_backend` += `*=cpu` |
| `keep_clip_on_cpu:true` | `backend` += `te=cpu` |
| `keep_vae_on_cpu:true` | `backend` += `vae=cpu` |
| `keep_control_net_on_cpu:true` | `backend` += `controlnet=cpu` |

For example, to mmap the diffusion weights from disk while keeping the text encoder on the CPU:

```yaml
options:
- "diffusion_model"
- "sampler:euler"
- "params_backend:diffusion=disk"
- "keep_clip_on_cpu:true"
```

{{% notice note %}}
`vae_decode_only` is still accepted for backwards compatibility but is now a no-op: upstream removed the flag and the model decides automatically.
{{% /notice %}}

#### VAE tiling

Sampling runs in a small latent space, but the final VAE decode expands it to full resolution and needs one large compute buffer. At 1024x1024 that buffer can exceed 8 GB, which is enough to fail on a GPU that has plenty of memory overall:

```
ggml_vulkan: Requested buffer size exceeds device buffer size limit: ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] stable-diffusion.cpp - decode_first_stage failed for latent 1
```

Tiling decodes the latent in overlapping tiles, so peak memory scales with the tile instead of the image. It costs a little time and can leave faint seams, so it is off unless you ask for it.

| Option | Example | Description |
|--------|---------|-------------|
| `vae_tiling` | `vae_tiling:true` | Enable tiled VAE decoding. The bare form `vae_tiling` also works. |
| `vae_tile_size` | `vae_tile_size:512` or `vae_tile_size:512x384` | Tile dimensions in pixels. One number means a square tile. Omit to use the upstream default. |
| `vae_tile_overlap` | `vae_tile_overlap:0.25` | Overlap between neighbouring tiles. More overlap hides seams better and costs more time. Omit to use the upstream default. |

```yaml
options:
- "diffusion_model"
- "sampler:euler"
- "vae_tiling:true"
- "vae_tile_size:512"
```

Reach for this when the decode fails but sampling completed, which the backend log shows as `sampling completed` followed by a VAE allocation error. Two cases hit it: cards without the VRAM for a full-frame decode, and drivers that cap a single allocation regardless of free memory (Mesa RADV reports a 4 GiB `maxMemoryAllocationSize`, so a large decode fails there even with tens of GB free). Lowering the output resolution avoids it too, at the cost of the resolution.

#### ESRGAN image upscaling

The `stablediffusion-ggml` backend supports standalone ESRGAN upscaling through
`POST /v1/images/upscale`. This is a separate, upscale-only model, not a diffusion
pipeline. Image generation does **not** automatically upscale its results.

Install the gallery model:

```bash
local-ai models install realesrgan-x4plus-anime-6b
```

This uses the official [RealESRGAN_x4plus_anime_6B.pth release](https://github.com/xinntao/Real-ESRGAN/releases/tag/v0.2.2.4),
a six-block model for anime images with a native **4x** scale. The PyTorch `.pth`
checkpoint is supported directly by [stable-diffusion.cpp](https://github.com/leejet/stable-diffusion.cpp/blob/f89d9b13d730eabeede7314ce49dacd18d3c90c2/docs/esrgan.md);
no GGUF conversion is needed. The upstream project is licensed under
[BSD-3-Clause](https://github.com/xinntao/Real-ESRGAN/blob/v0.2.2.4/LICENSE).

For manual installation, download that checkpoint into your models directory and
save the following as `realesrgan-x4plus-anime-6b.yaml` alongside it (the gallery
installer configures these settings for you):

```yaml
name: realesrgan-x4plus-anime-6b
backend: stablediffusion-ggml
parameters:
  model: RealESRGAN_x4plus_anime_6B.pth
known_usecases: [upscale]
options:
  - upscale_scale:4
  - upscale_tile_size:128
```

`upscale_scale` and `upscale_tile_size` are backend options, expressed as
`key:value` strings in `options`, not root-level configuration fields.
`upscale_scale` must be a positive integer. Keep `known_usecases` explicitly
set to `[upscale]`: do not add `image` or `diffusion_model` options to this
configuration. Options are forwarded unchanged; the backend rejects invalid
or duplicate recognized settings.

The multipart fields `model`, `image`, and **`scale` are required**. The HTTP
endpoint accepts only integer scales from **1 through 16**, rejecting missing,
malformed, zero, negative, or larger values with HTTP 400. This range is not a
promise that each model supports every scale: for ESRGAN, the configured
`upscale_scale`, the loaded model's detected native scale, and the request's
`scale` must all match exactly. Mismatches fail rather than silently resizing or
running multiple passes. For this model, always request `scale=4`.

```bash
curl http://localhost:8080/v1/images/upscale \
  -F model=realesrgan-x4plus-anime-6b \
  -F scale=4 \
  -F image=@input.png
```

The response follows the image-generation response format, with the output URL
in `data[0].url` under `/generated-images`. A successful 4x upscale multiplies
both width and height by four.

`upscale_tile_size` controls ESRGAN tiling in pixels: omit it or set it to
`0` for stable-diffusion.cpp full-frame processing; set a positive value (such
as the gallery entry's `128`) to enable tiling. Smaller tiles reduce working
memory at the cost of more tile work. Full-frame processing can use much more
memory. `upscale_direct` is a separate convolution-algorithm flag, not a tile
control. Tile settings apply to ESRGAN, not the diffusion model's separate VAE
tiling settings.

In **Studio**, open an existing image, choose **Upscale**, select an installed
upscaler, and submit. The scale is read-only and comes from the selected model's
configured scale. A successful result becomes an **Upscale child node** of the
source image, leaving the original intact. Open that child to continue the
workflow with the available image actions, including another upscale or image
editing; each upscale is an explicit operation, not a post-generation default.

Unlike the Diffusers upscaling described below, this path uses ESRGAN inference
and does not fall back to Lanczos resizing.

#### Distributed inference (RPC workers)

The `stablediffusion-ggml` backend can offload computation to remote `ggml` RPC workers, sharding a model that does not fit on a single machine. It reuses the **same backend-agnostic `rpc-server` workers as the llama.cpp backend**, so one worker pool can serve both.

**Manual:** point the model at running workers with the `rpc_servers` option:

```yaml
options:
- "rpc_servers:192.168.1.10:50052,192.168.1.11:50052"
```

Start a worker on each remote machine the same way you would for llama.cpp:

```bash
local-ai worker llama-cpp-rpc --llama-cpp-args="--host 0.0.0.0 --port 50052"
```

**Automatic (peer-to-peer):** when LocalAI runs in [p2p worker mode]({{%relref "features/distributed_inferencing" %}}), discovered workers are published in the `LLAMACPP_GRPC_SERVERS` environment variable. The image-generation backend reads that variable automatically (when `rpc_servers` is not set), so the same `local-ai worker p2p-llama-cpp-rpc` workers used for text generation also accelerate image generation - no per-model configuration needed.

By default the RPC devices join the pool and participate in placement; combine with the `backend` / `params_backend` options above to pin specific components to them (e.g. `backend:diffusion=rpc0`).


### Diffusers

[Diffusers](https://huggingface.co/docs/diffusers/index) is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. LocalAI has a diffusers backend which allows image generation using the `diffusers` library.

![anime_girl](https://github.com/go-skynet/LocalAI/assets/2420543/8aaca62a-e864-4011-98ae-dcc708103928)
(Generated with [AnimagineXL](https://huggingface.co/Linaqruf/animagine-xl))

#### Image upscaling

The Diffusers backend can upscale an uploaded image through
`POST /v1/images/upscale`. The multipart `scale` field is required and must be an
integer from 1 through 16 at the HTTP boundary; actual scaling behavior depends
on the selected pipeline. Install the included Stable Diffusion x4 upscaler
gallery model first:

```bash
local-ai models install stable-diffusion-x4-upscaler
```

Then send the model name, scale, and image as multipart form fields:

```bash
curl http://localhost:8080/v1/images/upscale \
  -F model=stable-diffusion-x4-upscaler \
  -F scale=4 \
  -F image=@input.png
```

The response uses the same format as image generation and returns the generated
image under `/generated-images`. The `diffusers` backend uses a loaded
`StableDiffusionUpscalePipeline` or `StableDiffusionLatentUpscalePipeline` when
configured. Other diffusers pipelines fall back to Lanczos resizing. Upscale
routes select models with `known_usecases: [upscale]`, not ordinary image
generation models. If you installed the x4 upscaler before it declared this
usecase, update its model YAML to `known_usecases: [upscale]` or reinstall the
gallery entry. Custom Diffusers configurations intended for upscaling should
also explicitly declare the `upscale` usecase.

#### Model setup

The models will be downloaded the first time you use the backend from `huggingface` automatically.

Create a model configuration file in the `models` directory, for instance to use `Linaqruf/animagine-xl` with CPU:

```yaml
name: animagine-xl
parameters:
  model: Linaqruf/animagine-xl
backend: diffusers

f16: false
diffusers:
  cuda: false # Enable for GPU usage (CUDA)
  scheduler_type: euler_a
```

#### Dependencies

This is an extra backend - in the container is already available and there is nothing to do for the setup. Do not use *core* images (ending with `-core`). If you are building manually, see the [build instructions]({{%relref "getting-started/build" %}}).

#### Model setup

The models will be downloaded the first time you use the backend from `huggingface` automatically.

Create a model configuration file in the `models` directory, for instance to use `Linaqruf/animagine-xl` with CPU:

```yaml
name: animagine-xl
parameters:
  model: Linaqruf/animagine-xl
backend: diffusers
cuda: true
f16: true
diffusers:
  scheduler_type: euler_a
```

#### Local models

You can also use local models, or modify some parameters like `clip_skip`, `scheduler_type`, for instance:

```yaml
name: stablediffusion
parameters:
  model: toonyou_beta6.safetensors
backend: diffusers
step: 30
f16: true
cuda: true
diffusers:
  pipeline_type: StableDiffusionPipeline
  enable_parameters: "negative_prompt,num_inference_steps,clip_skip"
  scheduler_type: "k_dpmpp_sde"
  clip_skip: 11

cfg_scale: 8
```

For an offline single-file checkpoint that needs its original Diffusers configuration, keep both files in the mounted models directory:

```yaml
name: offline-stable-diffusion
parameters:
  model: model.safetensors
backend: diffusers
diffusers:
  pipeline_type: StableDiffusionPipeline
  original_config_file: /models/v1-inference.yaml
```

#### Configuration parameters

The following parameters are available in the configuration file:

| Parameter | Description | Default |
| --- | --- | --- |
| `f16` | Force the usage of `float16` instead of `float32` | `false` |
| `step` | Number of steps to run the model for | `30` |
| `cuda` | Enable CUDA acceleration | `false` |
| `enable_parameters` | Parameters to enable for the model | `negative_prompt,num_inference_steps,clip_skip` |
| `scheduler_type` | Scheduler type | `k_dpp_sde` |
| `cfg_scale` | Configuration scale | `8` |
| `clip_skip` | Clip skip | None |
| `pipeline_type` | Pipeline type | `AutoPipelineForText2Image` |
| `lora_adapters` | A list of lora adapters (file names relative to model directory) to apply | None |
| `lora_scales` | A list of lora scales (floats) to apply | None |


There are available several types of schedulers:

| Scheduler | Description |
| --- | --- |
| `ddim` | DDIM |
| `pndm` | PNDM |
| `heun` | Heun |
| `unipc` | UniPC |
| `euler` | Euler |
| `euler_a` | Euler a |
| `lms` | LMS |
| `k_lms` | LMS Karras |
| `dpm_2` | DPM2 |
| `k_dpm_2` | DPM2 Karras |
| `dpm_2_a` | DPM2 a |
| `k_dpm_2_a` | DPM2 a Karras |
| `dpmpp_2m` | DPM++ 2M |
| `k_dpmpp_2m` | DPM++ 2M Karras |
| `dpmpp_sde` | DPM++ SDE |
| `k_dpmpp_sde` | DPM++ SDE Karras |
| `dpmpp_2m_sde` | DPM++ 2M SDE |
| `k_dpmpp_2m_sde` | DPM++ 2M SDE Karras |

Pipelines types available:

| Pipeline type | Description |
| --- | --- |
| `StableDiffusionPipeline` | Stable diffusion pipeline |
| `StableDiffusionImg2ImgPipeline` | Stable diffusion image to image pipeline |
| `StableDiffusionDepth2ImgPipeline` | Stable diffusion depth to image pipeline |
| `DiffusionPipeline` | Diffusion pipeline |
| `StableDiffusionXLPipeline` | Stable diffusion XL pipeline |
| `StableVideoDiffusionPipeline` | Stable video diffusion pipeline |
| `AutoPipelineForText2Image` | Automatic detection pipeline for text to image |
| `VideoDiffusionPipeline` | Video diffusion pipeline |
| `StableDiffusion3Pipeline` | Stable diffusion 3 pipeline |
| `FluxPipeline` | Flux pipeline |
| `FluxTransformer2DModel` | Flux transformer 2D model |
| `SanaPipeline` | Sana pipeline |

##### Advanced: Additional parameters

Additional arbitrary parameters can be specified in the option field in key/value separated by `:`:

```yaml
name: animagine-xl
options:
- "cfg_scale:6"
```

**Note**: There is no complete parameter list. Any parameter can be passed arbitrarily and is passed to the model directly as argument to the pipeline. Different pipelines/implementations support different parameters.

The example above, will result in the following python code when generating images:

```python
pipe(
    prompt="A cute baby sea otter", # Options passed via API
    size="256x256", # Options passed via API
    cfg_scale=6 # Additional parameter passed via configuration file
)
```

#### Usage

#### Text to Image
Use the `image` generation endpoint with the `model` name from the configuration file:

```bash
curl http://localhost:8080/v1/images/generations \
    -H "Content-Type: application/json" \
    -d '{
      "prompt": "<positive prompt>|<negative prompt>", 
      "model": "animagine-xl", 
      "step": 51,
      "size": "1024x1024" 
    }'
```

#### Image to Image

https://huggingface.co/docs/diffusers/using-diffusers/img2img

An example model (GPU):
```yaml
name: stablediffusion-edit
parameters:
  model: nitrosocke/Ghibli-Diffusion
backend: diffusers
step: 25
cuda: true
f16: true
diffusers:
  pipeline_type: StableDiffusionImg2ImgPipeline
  enable_parameters: "negative_prompt,num_inference_steps,image"
```

```bash
IMAGE_PATH=/path/to/your/image
(echo -n '{"file": "'; base64 $IMAGE_PATH; echo '", "prompt": "a sky background","size": "512x512","model":"stablediffusion-edit"}') |
curl -H "Content-Type: application/json" -d @-  http://localhost:8080/v1/images/generations
```

##### 🖼️ Flux kontext with `stable-diffusion.cpp`

LocalAI supports Flux Kontext and can be used to edit images via the API:

Install with:

```local-ai run flux.1-kontext-dev```

To test:

```
curl http://localhost:8080/v1/images/generations -H "Content-Type: application/json" -d '{
  "model": "flux.1-kontext-dev",
  "prompt": "change 'flux.cpp' to 'LocalAI'",
  "size": "256x256",
  "ref_images": [
  	"https://raw.githubusercontent.com/leejet/stable-diffusion.cpp/master/assets/flux/flux1-dev-q8_0.png"
  ]
}'
```

#### Depth to Image

https://huggingface.co/docs/diffusers/using-diffusers/depth2img

```yaml
name: stablediffusion-depth
parameters:
  model: stabilityai/stable-diffusion-2-depth
backend: diffusers
step: 50
f16: true
cuda: true
diffusers:
  pipeline_type: StableDiffusionDepth2ImgPipeline
  enable_parameters: "negative_prompt,num_inference_steps,image"

cfg_scale: 6
```

```bash
(echo -n '{"file": "'; base64 ~/path/to/image.jpeg; echo '", "prompt": "a sky background","size": "512x512","model":"stablediffusion-depth"}') |
curl -H "Content-Type: application/json" -d @-  http://localhost:8080/v1/images/generations
```

#### img2vid


```yaml
name: img2vid
parameters:
  model: stabilityai/stable-video-diffusion-img2vid
backend: diffusers
step: 25
f16: true
cuda: true
diffusers:
  pipeline_type: StableVideoDiffusionPipeline
```

```bash
(echo -n '{"file": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png?download=true","size": "512x512","model":"img2vid"}') |
curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8080/v1/images/generations
```

#### txt2vid

```yaml
name: txt2vid
parameters:
  model: damo-vilab/text-to-video-ms-1.7b
backend: diffusers
step: 25
f16: true
cuda: true
diffusers:
  pipeline_type: VideoDiffusionPipeline
  cuda: true
```

```bash
(echo -n '{"prompt": "spiderman surfing","size": "512x512","model":"txt2vid"}') |
curl -H "Content-Type: application/json" -X POST -d @- http://localhost:8080/v1/images/generations
```