mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 17:18:49 -04:00
pos-ei-don
48d11fcf74
_build_sampling_params constructed SamplingParams and then assigned the request values with setattr. protobuf repeated fields (StopPrompts) stay RepeatedScalarContainer that way, and __post_init__ never sees them: - vLLM's argument checks assert `isinstance(self.stop, list)` and the msgpack encoder that ships the params to the engine core cannot serialise the container, so every request with `stop` failed (HTTP 500, EngineGenerateError), streaming or not. - the stop-string hold-back buffer (output_text_buffer_length) was never set, so the start of a multi-character stop string could be streamed; seed -1 was not mapped to None. Collect the values, turn repeated fields into lists and construct SamplingParams once, as vLLM's OpenAI server does. Values vLLM rejects (e.g. a penalty out of range, an empty stop string) now fail with vLLM's validation error instead of being set unchecked. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Creating a separate environment for the vllm project
make vllm