mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
pos-ei-don
651c8da4d6
With a reasoning parser configured, the streaming path sent every raw
text delta as ChatDelta.content. LocalAI uses the backend's ChatDeltas as
they are, so with thinking enabled the reasoning arrived as
delta.content, and the final chunk then repeated the whole answer and
the reasoning a second time. Non-streaming requests were split correctly.
Use vLLM's composed parser (ParserManager.get_parser, vLLM >= 0.30), the
one vLLM's own OpenAI server streams with: one instance per request,
built with the request's chat_template_kwargs (enable_thinking), fed
each delta via parse_delta. Reasoning, content and tool calls go out in
their own fields as they are produced; the final chunk carries only
token counts and logprobs.
If the parser raises mid-stream, the rest of the output is buffered and
the final chunk sends what the full-text parse has beyond what was
already streamed (suffixes per field and per tool-call index). Where the
full parse disagrees with what went out, nothing is guessed and the
mismatch is logged.
Only active when a reasoning_parser is set; without one, or on a vLLM
without ParserManager, streaming is unchanged.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 5392eb813e)
Creating a separate environment for the vllm project
make vllm