mirror of
https://github.com/mudler/LocalAI.git
synced 2026-03-31 21:25:59 -04:00
031a36c995
* feat: wire min_p Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: inferencing defaults Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(refactor): re-use iterative parser Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: generate automatically inference defaults from unsloth Instead of trying to re-invent the wheel and maintain here the inference defaults, prefer to consume unsloth ones, and contribute there as necessary. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: apply defaults also to models installed via gallery Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: be consistent and apply fallback to all endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
gen_inference_defaults
This tool fetches per-model-family inference parameter defaults from unsloth's inference_defaults.json, validates the data, remaps field names to LocalAI conventions, and writes core/config/inference_defaults.json.
What it does
- Fetches the latest
inference_defaults.jsonfrom unsloth's repo - Validates that every entry has required fields (
temperature,top_p,top_k) - Validates that every pattern references an existing family
- Warns if pattern ordering would cause shorter prefixes to shadow longer ones
- Remaps
repetition_penalty→repeat_penalty(LocalAI naming) - Filters to allowed fields only:
temperature,top_p,top_k,min_p,repeat_penalty,presence_penalty - Writes the validated JSON to
core/config/inference_defaults.json
Usage
# Only regenerate if the file is missing (runs during make build)
make generate
# Force re-fetch from unsloth
make generate-force
# Or directly via go generate
go generate ./core/config/...
Automation
The GitHub Actions workflow .github/workflows/bump-inference-defaults.yml runs make generate-force daily and opens a PR if the upstream data changed.