From 06b4a29387b5abf17c7472eb85cdb2381b96bdc5 Mon Sep 17 00:00:00 2001
From: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Date: Thu, 16 Jul 2026 22:18:47 +0200
Subject: [PATCH] docs(config): document grpc.attempts timing + tuning guidance
 (#10868)

The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.

Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
  LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
  vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
  140, attempts_sleep_time 5), with a note that inference-timeouts and
  the watchdog are unaffected.
---
 docs/content/advanced/model-configuration.md | 38 +++++++++++++++++---
 1 file changed, 33 insertions(+), 5 deletions(-)

diff --git a/docs/content/advanced/model-configuration.md b/docs/content/advanced/model-configuration.md
index 0662d7287..fabda406a 100644
--- a/docs/content/advanced/model-configuration.md
+++ b/docs/content/advanced/model-configuration.md
@@ -914,12 +914,40 @@ Define pipelines for audio-to-audio processing and the [Realtime API]({{%relref
 
 ## gRPC Configuration
 
-Backend gRPC communication settings:
+Backend gRPC communication settings. These control the readiness handshake
+between LocalAI and a freshly spawned backend process — LocalAI polls the
+backend's `Health` gRPC method up to `grpc.attempts` times, sleeping
+`grpc.attempts_sleep_time` seconds between polls, before giving up and
+terminating the backend as unresponsive.
 
-| Field | Type | Description |
-|-------|------|-------------|
-| `grpc.attempts` | int | Number of retry attempts |
-| `grpc.attempts_sleep_time` | int | Sleep time between retries (seconds) |
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `grpc.attempts` | int | 20 | Number of health-check attempts before the backend is killed as unresponsive |
+| `grpc.attempts_sleep_time` | int | 2 | Sleep time between health-check attempts (seconds) |
+
+**Total load window ≈ `grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)`.**
+The default of `20 × 2 s ≈ 40 s` is fine for typical backends but is too
+short for large models that need substantial time to become gRPC-ready
+after the process starts — for example NVFP4 / FP8 models whose shard
+loading and CUDA-graph capture can take several minutes, or slow storage
+backends. If the backend keeps getting killed while still legitimately
+loading (visible as `exitCode=120` + `rpc error: code = Canceled desc =
+context canceled` in the LocalAI log, while the backend's own stderr
+shows continued forward progress), raise these values.
+
+Example configuration for a model that needs up to ~10 minutes to become
+gRPC-ready (large NVFP4 model, cold shard load + CUDA-graph capture):
+
+```yaml
+grpc:
+  attempts: 140
+  attempts_sleep_time: 5
+```
+
+This gives a ~700 s window while keeping health-check polling frequent
+enough to detect real backend crashes quickly. The values only affect
+the initial readiness handshake — inference-request timeouts and the
+watchdog are unchanged.
 
 ## Overrides