mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
47097041ff
fix(vllm): apply Options[] engine flags before engine init (#11130) CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`, `--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could still reach the engine. Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine is constructed. Names are normalized the way vLLM's CLI spells them (`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to the target field's type (bare flag -> True for booleans), and unknown or uncoercible flags warn and are skipped instead of failing the load, since Options[] is a bag shared with backend-level settings. Field types come from the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not mistaken for a float. Precedence is typed proto fields -> `options:` -> `engine_args:`. The production engine_args defaults seeded in hooks_vllm.go therefore skip any key the user already set as an option, otherwise the later engine_args pass would silently override it. Parser lookups now accept both spellings, so `--reasoning-parser:qwen3` selects LocalAI's parser as well. The helper's tests are stdlib-only and run in the lint workflow's dependency-light job via `make test-python-helpers`. Assisted-by: Claude:claude-opus-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>