mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-06-21 23:29:04 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(vllm): structured outputs silently ignored on vLLM >= 0.23 (GuidedDecodingParams removed) (#10343)

Browse Source 
fix(vllm): structured outputs silently ignored on vLLM >= 0.23

vLLM >= 0.23 removed GuidedDecodingParams (now StructuredOutputsParams) and
renamed the SamplingParams field guided_decoding -> structured_outputs. The
import failed, HAS_GUIDED_DECODING became False, and the whole guided-decoding
block was skipped, so response_format / grammar constraints were silently
ignored. Adapt the existing request.Grammar path to the new class/field.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
This commit is contained in:
pos-ei-don
2026-06-21 17:02:31 +02:00
committed by GitHub
parent 3e96d811b7
commit c6303104c7
1 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
backend/python/vllm/backend.py
View File

@@ -48,8 +48,10 @@ try:
except ImportError:
HAS_REASONING_PARSERS = False
# vLLM >= 0.23 renamed GuidedDecodingParams -> StructuredOutputsParams and the
# SamplingParams field guided_decoding -> structured_outputs.
try:
from vllm.sampling_params import GuidedDecodingParams
from vllm.sampling_params import StructuredOutputsParams
HAS_GUIDED_DECODING = True
except ImportError:
HAS_GUIDED_DECODING = False
@@ -536,13 +538,13 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
if value not in (None, 0, [], False):
setattr(sampling_params, param_field, value)
# Guided decoding: use Grammar field to pass JSON schema or BNF
# Structured-output decoding: use Grammar field to pass JSON schema or BNF
if HAS_GUIDED_DECODING and request.Grammar:
try:
json.loads(request.Grammar) # valid JSON = JSON schema
sampling_params.guided_decoding = GuidedDecodingParams(json=request.Grammar)
sampling_params.structured_outputs = StructuredOutputsParams(json=request.Grammar)
except json.JSONDecodeError:
sampling_params.guided_decoding = GuidedDecodingParams(grammar=request.Grammar)
sampling_params.structured_outputs = StructuredOutputsParams(grammar=request.Grammar)
# Extract image paths and process images
prompt = request.Prompt