eb32cd9073
Realtime sessions previously lazy-loaded each pipeline sub-model (VAD,
transcription, LLM, TTS) on first use, so every cold session paid a
per-request model-load stall and load errors only surfaced mid-stream.
Warm the whole pipeline eagerly and blockingly at session start
(including the voice-gate speaker-recognition model, which an enforced
gate blocks each utterance on; compaction's summary_model stays lazy
since it only runs off the response path):
- Add backend.PreloadModel / PreloadModelByName as the single load path
for every modality (no transcription special-case; backend-omitted
configs are deprecated).
- The realtime session blocks on Model.Warmup and returns a
model_load_error to the client if any stage fails to load;
updateSession warms in the background. Opt out per pipeline with
pipeline.disable_warmup, exposed as a UI toggle via the
config-metadata registry.
Add a LocalAI-native POST /backend/load (and /v1/backend/load) that
pre-loads a model -- expanding realtime pipelines into their sub-models
-- as the inverse of /backend/shutdown. There is one preload engine
(backend.PreloadStages): the realtime Warmup methods, /backend/load and
the --load-to-memory startup flag all use it, so --load-to-memory now
also expands pipeline models and records load-failure traces. Pipeline
sub-model alias resolution is likewise shared
(ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint
everywhere an admin manages models:
- MCP admin tool load_model (httpapi + inproc clients, safety/catalog
prompts, catalog/dispatch tests).
- "Load into memory" action in the React models UI.
- Swagger regenerated; docs moved to the general backend-monitor page
since it is not realtime-specific.
Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet
RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized
to drop non-finite floats before marshaling. The sanitizer is
copy-on-write -- it runs on every RecordBackendTrace, so containers are
only re-allocated on the paths that actually changed.
Migrate core/http/openresponses_test.go onto the prebuilt mock-backend
the rest of the http suite already uses -- it was the last spec still
pointing at a real HuggingFace model, so it 404'd wherever no vision
backend was built -- and fix its item_reference specs to send the
spec's "id" field instead of "item_id", which the handler never
accepted.
Assisted-by: Claude:claude-opus-4-8 Claude Code
Signed-off-by: Richard Palethorpe <io@richiejp.com>
4.3 KiB
+++ disableToc = false title = "Backend Monitor" weight = 20 url = "/features/backend-monitor/" +++
LocalAI provides endpoints to monitor and manage running backends. The /backend/monitor endpoint reports the status and resource usage of loaded models, /backend/load pre-loads a model into memory, and /backend/shutdown allows stopping a model's backend process.
All three are admin-only.
Monitor API
- Method:
GET - Endpoints:
/backend/monitor,/v1/backend/monitor
Request
The model to monitor is passed as a query parameter:
| Parameter | Type | Required | Location | Description |
|---|---|---|---|---|
model |
string |
Yes | query | Name of the model to monitor |
For backwards compatibility, a JSON body with the same field is still accepted when the model query parameter is not set, but new clients should use the query parameter.
Response
Returns a JSON object with the backend status:
| Field | Type | Description |
|---|---|---|
state |
int |
Backend state: 0 = uninitialized, 1 = busy, 2 = ready, -1 = error |
memory |
object |
Memory usage information |
memory.total |
uint64 |
Total memory usage in bytes |
memory.breakdown |
object |
Per-component memory breakdown (key-value pairs) |
If the gRPC status call fails, the endpoint falls back to local process metrics:
| Field | Type | Description |
|---|---|---|
memory_info |
object |
Process memory info (RSS, VMS) |
memory_percent |
float |
Memory usage percentage |
cpu_percent |
float |
CPU usage percentage |
Usage
curl "http://localhost:8080/backend/monitor?model=my-model"
Example response
{
"state": 2,
"memory": {
"total": 1073741824,
"breakdown": {
"weights": 536870912,
"kv_cache": 268435456
}
}
}
Load API
Pre-loads a model into memory ahead of its first request, so that request pays no cold-start load cost. It is the inverse of the Shutdown API and works for any model, not just realtime pipelines.
- Method:
POST - Endpoints:
/backend/load,/v1/backend/load
Request
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string |
Yes | Name of the model to load |
Behavior
- For a regular model, its own backend is loaded.
- For a realtime pipeline model (a config with a
pipeline:block), every configured sub-model (VAD, transcription, LLM, TTS, sound_detection, voice_recognition) is loaded concurrently instead of the pipeline stub, which has no backend of its own.
The call blocks until loading finishes and reports which model names became resident, so partial failures are visible.
Usage
curl -X POST http://localhost:8080/backend/load \
-H "Content-Type: application/json" \
-d '{"model": "my-model"}'
Example response
{ "loaded": ["my-model"], "message": "model loaded" }
On failure the call returns 500 with loaded listing whichever sub-models did load and message naming the failures.
Shutdown API
- Method:
POST - Endpoints:
/backend/shutdown,/v1/backend/shutdown
Request
| Parameter | Type | Required | Description |
|---|---|---|---|
model |
string |
Yes | Name of the model to shut down |
Usage
curl -X POST http://localhost:8080/backend/shutdown \
-H "Content-Type: application/json" \
-d '{"model": "my-model"}'
Response
Returns 200 OK with the shutdown confirmation message on success.
Error Responses
| Status Code | Description |
|---|---|
| 400 | Invalid or missing model name |
| 500 | Backend error or model not loaded |