mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs(config): document grpc.attempts timing + tuning guidance (#10868)

Browse Source 
The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.

Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
  LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
  vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
  140, attempts_sleep_time 5), with a note that inference-timeouts and
  the watchdog are unaffected.
This commit is contained in:
pos-ei-don
2026-07-16 22:18:47 +02:00
committed by GitHub
parent e62221b020
commit 06b4a29387
1 changed files with 33 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
docs/content/advanced/model-configuration.md
View File

@@ -914,12 +914,40 @@ Define pipelines for audio-to-audio processing and the [Realtime API]({{%relref
## gRPC Configuration
Backend gRPC communication settings:
Backend gRPC communication settings. These control the readiness handshake
between LocalAI and a freshly spawned backend process — LocalAI polls the
backend's `Health` gRPC method up to `grpc.attempts` times, sleeping
`grpc.attempts_sleep_time` seconds between polls, before giving up and
terminating the backend as unresponsive.
| Field | Type | Description |
|-------|------|-------------|
| `grpc.attempts` | int | Number of retry attempts |
| `grpc.attempts_sleep_time` | int | Sleep time between retries (seconds) |
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `grpc.attempts` | int | 20 | Number of health-check attempts before the backend is killed as unresponsive |
| `grpc.attempts_sleep_time` | int | 2 | Sleep time between health-check attempts (seconds) |
**Total load window ≈ `grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)`.**
The default of `20 × 2 s ≈ 40 s` is fine for typical backends but is too
short for large models that need substantial time to become gRPC-ready
after the process starts — for example NVFP4 / FP8 models whose shard
loading and CUDA-graph capture can take several minutes, or slow storage
backends. If the backend keeps getting killed while still legitimately
loading (visible as `exitCode=120` + `rpc error: code = Canceled desc =
context canceled` in the LocalAI log, while the backend's own stderr
shows continued forward progress), raise these values.
Example configuration for a model that needs up to ~10 minutes to become
gRPC-ready (large NVFP4 model, cold shard load + CUDA-graph capture):
```yaml
grpc:
attempts: 140
attempts_sleep_time: 5
```
This gives a ~700 s window while keeping health-check polling frequent
enough to detect real backend crashes quickly. The values only affect
the initial readiness handshake — inference-request timeouts and the
watchdog are unchanged.
## Overrides