mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-29 01:28:43 -04:00
pos-ei-don
5bdab54d97
Both Python backends start from a fixed default (sglang max_new_tokens=200, vLLM max_tokens=200) and only replace it when the request carries Tokens. OpenAI clients that leave max_tokens out (several agent frameworks do) were silently truncated after 200 tokens, and the response still said finish_reason "stop": the Go side only reports "length" against a budget the client requested, so it cannot see a backend-side cap. Use None instead, which allows generation up to the remaining context. That is what sglang's own OpenAI server does when max_tokens is omitted (the scheduler turns None into the remaining request capacity; dropping the key would select SamplingParams' own default of 128), and vLLM treats max_tokens=None the same way. A requested max_tokens still wins. Tests check both cases and that the None default passes each library's own parameter validation. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>