mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-30 09:57:57 -04:00
Code Issues Packages Projects Releases Wiki Activity

docs(proxy): cover long inference timeouts (#11065)

Browse Source 
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog.

Assisted-by: Codex:gpt-5 [Codex]

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
This commit is contained in:
localai-org-maint-bot
2026-07-29 16:37:03 +02:00
committed by GitHub
parent 9058a2bb46
commit ecdb32193d
1 changed files with 52 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
docs/content/advanced/reverse-proxy-tls.md
View File

@@ -108,25 +108,48 @@ server {
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Set the X-Forwarded-Proto header
proxy_set_header X-Forwarded-Proto $scheme;
# Pass the original host
proxy_set_header X-Forwarded-Host $host;
# If serving under a sub-path
# proxy_set_header X-Forwarded-Prefix /localai;
# Other proxy settings
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
# Allow multimodal requests with images or other large inputs.
client_max_body_size 50m;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_http_version 1.1;
# Set the externally visible scheme and host.
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header Host $host;
# If serving under a sub-path
# proxy_set_header X-Forwarded-Prefix /localai;
# Local inference can take much longer than common proxy defaults.
proxy_connect_timeout 60s;
proxy_send_timeout 60m;
proxy_read_timeout 60m;
# Stream responses as LocalAI emits them. Disabling request buffering is
# optional and can help when clients upload large multimodal inputs.
proxy_buffering off;
proxy_request_buffering off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
}
}
```
Adjust the `60m` values for the slowest request you expect to serve. The
important setting for long non-streaming requests is `proxy_read_timeout`: if
the proxy stops waiting before LocalAI finishes, clients receive a proxy-generated
`504 Gateway Time-out` even though inference may still be running. OpenResty,
Nginx Proxy Manager, Caddy, Traefik, and HAProxy have equivalent upstream
response timeout settings.
For bulk jobs on a trusted private network, you can also bypass the public
reverse proxy and connect directly to LocalAI.
## Serving Under a Sub-Path
If you serve LocalAI under a sub-path (e.g., `https://your-domain.com/localai`), you need to:
@@ -165,6 +188,19 @@ proxy_set_header X-Forwarded-Prefix /localai;
- Check that your proxy is not adding duplicate headers
- Verify `X-Forwarded-Proto` is not being set to both `http` and `https`
### 504 Gateway Time-out During Inference
An HTML `504 Gateway Time-out` page branded by Nginx or OpenResty means the
reverse proxy stopped waiting for LocalAI. Increase the proxy's upstream read
and send timeouts beyond the worst-case inference time. Also check its request
body limit when sending images, audio, or other large inputs.
This is separate from LocalAI's optional busy watchdog. If busy watchdog checks
are enabled, set `LOCALAI_WATCHDOG_BUSY_TIMEOUT` beyond the longest expected
request or disable the busy check for workloads that legitimately run longer.
The default busy timeout is 5 minutes, but it is only enforced when the busy
watchdog is enabled.
## Security Note
When using reverse proxies, ensure your proxy only accepts connections from trusted sources and properly validates SSL certificates. Never expose LocalAI directly to the internet without TLS termination.