mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-31 10:28:43 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(vllm): apply Options[] engine flags before engine init (#11147)

Browse Source 
fix(vllm): apply Options[] engine flags before engine init (#11130)

CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.

Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.

Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.

The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.


Assisted-by: Claude:claude-opus-5 golangci-lint

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
This commit is contained in:
mudler's LocalAI [bot]
2026-07-30 12:13:00 +02:00
committed by GitHub
parent 9c85cacfe3
commit 47097041ff
9 changed files with 481 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.github/workflows/lint.yml vendored
View File

@@ -69,3 +69,9 @@ jobs:
node-version: '20'
- name: run CI script tests
run: make test-ci-scripts
# The shared python backend helpers (Options[] parsing, engine-arg
# mapping, model reference resolution) are stdlib-only, so their tests
# ride along here instead of waiting on a multi-GB backend image build.
- name: run shared python backend helper tests
run: make test-python-helpers