mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
pos-ei-don
d254ef5408
Both Python backends start from a fixed default (sglang max_new_tokens=200,
vLLM max_tokens=200) and only replace it when the request carries Tokens.
OpenAI clients that leave max_tokens out (several agent frameworks do) were
silently truncated after 200 tokens, and the response still said
finish_reason "stop": the Go side only reports "length" against a budget
the client requested, so it cannot see a backend-side cap.
Use None instead, which allows generation up to the remaining context.
That is what sglang's own OpenAI server does when max_tokens is omitted
(the scheduler turns None into the remaining request capacity; dropping
the key would select SamplingParams' own default of 128), and vLLM treats
max_tokens=None the same way. A requested max_tokens still wins.
Tests check both cases and that the None default passes each library's own
parameter validation.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5bdab54d97)