mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
pos-ei-don
5392eb813e
With a reasoning parser configured, the streaming path sent every raw text delta as ChatDelta.content. LocalAI uses the backend's ChatDeltas as they are, so with thinking enabled the reasoning arrived as delta.content, and the final chunk then repeated the whole answer and the reasoning a second time. Non-streaming requests were split correctly. Use vLLM's composed parser (ParserManager.get_parser, vLLM >= 0.30), the one vLLM's own OpenAI server streams with: one instance per request, built with the request's chat_template_kwargs (enable_thinking), fed each delta via parse_delta. Reasoning, content and tool calls go out in their own fields as they are produced; the final chunk carries only token counts and logprobs. If the parser raises mid-stream, the rest of the output is buffered and the final chunk sends what the full-text parse has beyond what was already streamed (suffixes per field and per tool-call index). Where the full parse disagrees with what went out, nothing is guessed and the mismatch is logged. Only active when a reasoning_parser is set; without one, or on a vLLM without ParserManager, streaming is unchanged. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Creating a separate environment for the vllm project
make vllm