mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
pos-ei-don
50ba04681a
_build_sampling_params constructed SamplingParams and then assigned the
request values with setattr. protobuf repeated fields (StopPrompts) stay
RepeatedScalarContainer that way, and __post_init__ never sees them:
- vLLM's argument checks assert `isinstance(self.stop, list)` and the
msgpack encoder that ships the params to the engine core cannot
serialise the container, so every request with `stop` failed
(HTTP 500, EngineGenerateError), streaming or not.
- the stop-string hold-back buffer (output_text_buffer_length) was never
set, so the start of a multi-character stop string could be streamed;
seed -1 was not mapped to None.
Collect the values, turn repeated fields into lists and construct
SamplingParams once, as vLLM's OpenAI server does. Values vLLM rejects
(e.g. a penalty out of range, an empty stop string) now fail with vLLM's
validation error instead of being set unchecked.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 48d11fcf74)
Creating a separate environment for the vllm project
make vllm