LocalAI/docs/content/features/quantization.md

+++ disableToc = false title = "Model Quantization" weight = 83 url = '/features/quantization/' +++

LocalAI supports model quantization directly through the API and Web UI. Quantization converts HuggingFace models to GGUF format and compresses them to smaller sizes for efficient inference with llama.cpp.

{{% notice note %}} This feature is experimental and may change in future releases. {{% /notice %}}

Supported Backends

Backend	Description	Quantization Types	Platforms
llama-cpp-quantization	Downloads HF models, converts to GGUF, and quantizes using llama.cpp	q2_k, q3_k_s, q3_k_m, q3_k_l, q4_0, q4_k_s, q4_k_m, q5_0, q5_k_s, q5_k_m, q6_k, q8_0, f16	CPU (Linux, macOS)

Quick Start

1. Start a quantization job

curl -X POST http://localhost:8080/api/quantization/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unsloth/functiongemma-270m-it",
    "quantization_type": "q4_k_m"
  }'

2. Monitor progress (SSE stream)

curl -N http://localhost:8080/api/quantization/jobs/{job_id}/progress

3. Download the quantized model

curl -o model.gguf http://localhost:8080/api/quantization/jobs/{job_id}/download

4. Or import it directly into LocalAI

curl -X POST http://localhost:8080/api/quantization/jobs/{job_id}/import \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-quantized-model"
  }'

API Reference

Endpoints

Method	Path	Description
POST	/api/quantization/jobs	Start a quantization job
GET	/api/quantization/jobs	List all jobs
GET	/api/quantization/jobs/:id	Get job details
POST	/api/quantization/jobs/:id/stop	Stop a running job
DELETE	/api/quantization/jobs/:id	Delete a job and its data
GET	/api/quantization/jobs/:id/progress	SSE progress stream
POST	/api/quantization/jobs/:id/import	Import quantized model into LocalAI
GET	/api/quantization/jobs/:id/download	Download quantized GGUF file
GET	/api/quantization/backends	List available quantization backends

Job Request Fields

Field	Type	Description
model	string	HuggingFace model ID or local path (required)
backend	string	Backend name (default: llama-cpp-quantization)
quantization_type	string	Quantization format (default: q4_k_m)
extra_options	map	Backend-specific options (see below)

Extra Options

Key	Description
hf_token	HuggingFace token for gated models

Import Request Fields

Field	Type	Description
name	string	Model name for LocalAI (auto-generated if empty)

Job Status Values

Status	Description
queued	Job created, waiting to start
downloading	Downloading model from HuggingFace
converting	Converting model to f16 GGUF
quantizing	Running quantization
completed	Quantization finished successfully
failed	Job failed (check message for details)
stopped	Job stopped by user

Progress Stream

The GET /api/quantization/jobs/:id/progress endpoint returns Server-Sent Events (SSE) with JSON payloads:

{
  "job_id": "abc-123",
  "progress_percent": 65.0,
  "status": "quantizing",
  "message": "[ 234/ 567] quantizing blk.5.attn_k.weight ...",
  "output_file": "",
  "extra_metrics": {}
}

When the job completes, output_file contains the path to the quantized GGUF file and extra_metrics includes file_size_mb.

Quantization Types

Type	Size	Quality	Description
q2_k	Smallest	Lowest	2-bit quantization
q3_k_s	Very small	Low	3-bit small
q3_k_m	Very small	Low	3-bit medium
q3_k_l	Small	Low-medium	3-bit large
q4_0	Small	Medium	4-bit legacy
q4_k_s	Small	Medium	4-bit small
q4_k_m	Small	Good	4-bit medium (recommended)
q5_0	Medium	Good	5-bit legacy
q5_k_s	Medium	Good	5-bit small
q5_k_m	Medium	Very good	5-bit medium
q6_k	Large	Excellent	6-bit
q8_0	Large	Near-lossless	8-bit
f16	Largest	Lossless	16-bit (no quantization, GGUF conversion only)

The UI also supports entering a custom quantization type string for any format supported by llama-quantize.

Web UI

Build → Quantize uses the same page pattern as fine-tuning: set up, check, run, result.

1. Set up - Choose the model and the quantization type (a preset, or a custom type). The backend and the Hugging Face token are under More options.
2. Check before you start - Whether a model and type are set, whether a quantization backend is installed, the free RAM and the free space on the models disk. The server reports no model size before the job starts, so the page does not estimate one.
3. Run - Percent, the stage (downloading, converting, quantizing), and a log of what the job reported since the page opened. Stop ends the job.
4. Result - A failed job shows the server's message. A finished job shows the output file, downloads it, or imports it into LocalAI under a name you choose, then links to a chat with it and to Models.

Earlier jobs are listed below the form, with Reuse and Delete (after a confirmation).

A user needs the quantization permission. Without it, the page says the account cannot quantize.

Quantization is an add-on

The Quantize page is the quantization add-on. It does not ship with LocalAI. An administrator installs it from Browse add-ons in the drawer and starts it. Once it runs, it opens from the drawer at /app/addons/quantization/jobs and can be pinned. The Build hub no longer has a Quantize tab.

The old link /app/quantize keeps working. It opens the add-on page while the add-on is installed and usable, and it opens the same page in the core web UI otherwise.

The API does not change. The page calls /api/quantization/* with the signed-in user's own session, and the server checks the quantization permission on every call.

The add-on also offers tools to the chat: list jobs, propose a job, read the status of a job, stop a job and import the result as a model. A tool can only control a job that a tool started. A job started in the page is listed, but a tool cannot stop or import it. A tool never starts, stops or imports anything by itself: it proposes the action and a person confirms it in an approval card. A model given as a local path is refused. The tools call the engine through the host operations (see Host operations), which the host setting LOCALAI_ADDON_HOST_OPS grants to this add-on by the quantize. prefix, and they run with the permissions of the person in the chat.

Architecture

Quantization uses the same gRPC backend architecture as fine-tuning:

1. Proto layer: QuantizationRequest, QuantizationProgress (streaming), StopQuantization
2. Python backend: Downloads model, runs convert_hf_to_gguf.py and llama-quantize
3. Go service: Manages job lifecycle, state persistence, async import
4. REST API: HTTP endpoints with SSE progress streaming
5. React UI: Configuration form, real-time progress monitor, download/import panel