mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 00:59:28 -04:00
docs(vllm): explain sampling validation
Describe stop strings, greedy decoding, and seed handling after sampling
parameters are validated during construction.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
(cherry picked from commit 05af7b072c)
This commit is contained in:
1 parent
50ba04681a
commit
a1e0faa9f0
1 file changed
+6
@@ -833,6 +833,12 @@ The fields below come from upstream llama.cpp's **CLI completion tool** and are
|
||||
|
||||
These options apply when using the `vllm` backend:
|
||||
|
||||
The `vllm` backend validates sampling parameters before generation. Stop strings
|
||||
work with streaming and non-streaming requests; an empty stop string is invalid.
|
||||
Presence and frequency penalties must be between -2 and 2. A temperature of `0`
|
||||
selects greedy decoding, which ignores `top_p` and `top_k`. A seed of `-1` uses
|
||||
vLLM's default random seed behavior.
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `gpu_memory_utilization` | float32 | GPU memory utilization (0.0-1.0, default 0.9) |
|
||||
|
||||
Reference in new issue
Block a user