docs(vllm,sglang): note that omitted max_tokens is not capped

The vLLM and SGLang backends no longer stop at 200 tokens when a request
has no max_tokens. Generation is now bounded by the context window.

Assisted-by: Claude:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto committed 2026-10-25 08:50:18 +00:00
1 parent ac5a0f3947
commit 3302c76964
1 file changed
+2
+2
View File
@@ -31,6 +31,8 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso
Available additional parameters: `top_p`, `top_k`, `max_tokens`
When a request omits `max_tokens`, the vLLM and SGLang backends do not cap generation at 200 tokens. Generation is bounded only by the context window of the model.
Reasoning models return their thinking in the `reasoning` field. When a model reasons and calls a tool in the same turn, see [Interleaved Thinking with Tool Calls]({{%relref "features/interleaved-thinking" %}}).
When `stream: true` is set and the llama.cpp backend fails before the first chunk, for example because the prompt exceeds the context size, the request fails with an HTTP error. The error message is not streamed as assistant content. An error after streaming has started is reported inside the stream.