mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-07-31 10:28:43 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
feat/vllm-cpp-engine-args
LocalAI/core/config/meta/registry.go
Richard Palethorpe 49ef40a187 feat(classifier/VAD): support voice control on low power devices (#10804) 
* feat(llama-cpp): route Score through the slot loop

Score previously bypassed the slot loop with a direct llama_decode: a
conflict guard aborted the whole process if scoring raced generation, the
config validator had to reject score alongside chat/completion/embeddings,
and every candidate re-decoded the full shared prompt.

Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks
are scheduled like any other slot work: generation and scoring serialize
naturally, the shared prompt is decoded once per call, and the slot's
prompt cache carries the conversation prefix across calls. Context
checkpoints at the score boundary and at the cache-divergence point keep
SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole
prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s
to ~0.5s on a desktop CPU.

The conflict guard and the validation split are removed; declaring score
with generation usecases on one config is now supported and shares the
slot cache.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier wire types and pipeline config

Wire types and YAML config for realtime classifier mode: sessions carry a
localai_classifier extension (options with canned replies/tool calls,
softmax threshold, normalization, history trimming, fallback modes, and a
deterministic wake-word address gate), mirrored by pipeline.classifier in
the model YAML and surfaced in the config-meta registry. The
localai.classifier.result server event reports the full score distribution
per turn.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier response flow

Classifier-mode responses: instead of autoregressive generation, each user
turn is prefill-scored against the option list (router.ScoreClassifier
prompt/candidate shapes over the Score primitive) and the winning option's
canned reply and tool call are emitted through the existing response
machinery. Below-threshold turns take the configured fallback (none /
canned reply / generate); empty transcripts and unaddressed turns (wake
word not mentioned) skip scoring entirely. The scoring probe defaults to
the latest user message only — small scorers echo canned replies from
prior turns back as the top option otherwise.

Built for hardware that can afford prompt processing but not decode: with
slot-based Score the option list stays KV-cached across turns, so a turn
costs roughly one forward pass over the new words.

session_update_error events now carry the validation cause instead of a
generic message.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): bound the VAD tick's scan window and buffer retention

The VAD tick loop re-scanned the entire input buffer every 300ms and only
trimmed it on zero-segment ticks or commits. Audio that keeps producing
segments without a committing pause (steady noise a mic pipeline lets
through, music, continuous speech) grew the buffer toward the 100MB cap
with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero
per buffered second: past ~90s retained, ticks run back to back and pin
~4 cores until the stream stops.

Silero's recurrent state only carries a few hundred ms of context, so
rescanning old audio buys nothing. Clip the slice handed to the VAD to the
largest silence the commit test can need to measure (server_vad silence
window or the semantic eagerness fallback) plus a warm-up margin, and
rebase the returned segment times so every downstream consumer keeps
whole-buffer coordinates. An open turn whose clipped window is all silence
now commits (the silence outran the window) instead of being discarded as
no-speech. Independently, retain at most 90s of raw buffer, rebasing the
live-feed and EOU cursors on trim — this also bounds the previously
unbounded VAD-error path. Turn boundaries are otherwise unchanged: no
forced commits, no new coordinator states.

pipeline.turn_detection.vad_window_sec can widen the scan window; values
below the automatic floor are ignored. The tick body is extracted into
vadTick so specs can drive turn detection synchronously (same shape as
classifySoundWindow); the babble reproduction that pinned 4 cores now
plateaus under 10% of one core.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(backend): let per-model threads override the global default

ModelOptions overrode a set per-model threads value with the app-level
--threads whenever the latter was non-zero — and WithThreads defaults it
to the physical core count, so it always was. The YAML threads: knob has
been dead config: a tiny VAD model could never opt down from the global
pool size.

SetDefaults already fills an unset per-model value from the app config,
which is the intended precedence; resolve threads through a helper that
honors it (explicit threads: 0 still means unset).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(gallery): single-thread the silero VAD

Silero is a ~2MB recurrent model with no exploitable graph parallelism:
measured per-call latency is identical at 1 and 10 ORT threads, while
every extra pool thread just spin-waits between the realtime loop's
frequent tiny inferences.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* docs(realtime): classifier mode, VAD scan window, threads precedence

Document the realtime classifier mode (options, threshold guidance,
wake-word address gate, empty-transcript handling), the VAD scan window
and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the
per-model threads precedence, and the M3 classifier note in the realtime
state-machine design doc.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): score all candidates in one batched decode

One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot
decodes the shared prefix (prompt + longest common candidate token
prefix) once, then forks one sequence per candidate off it
(metadata-only for the unified KV cache, copy-on-write for recurrent
state) and decodes every candidate's unique tail in one llama_decode.
Previously each candidate was its own task that restored the boundary
checkpoint and re-decoded its full tail sequentially, paying
per-candidate task and decode overhead.

The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and
recurrent-state cells) beyond the parallel slots via the new
common_params::n_seq_score_forks. Forking requires the unified KV cache
(already this backend's default) since per-sequence streams would shrink
n_ctx_seq; an explicit kv_unified:false disables forking and Score calls
that need it fail cleanly. Candidates beyond the fork/output budget
decode in successive chunks.

Wire contract and scores are unchanged: per-token logprobs are stitched
from the shared region and the forked tails. Verified bitwise
deterministic call-to-call and independent of candidate order (no
cross-fork leakage via equal-length candidate swap); ranking matches the
per-candidate implementation on the drone battery (winner softmax
0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and
empty candidates all pass.

Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm
realtime classifier turns 196-303ms. The 9-candidate drone turn decodes
~17 unique tail tokens in one batch instead of nine sequential ~220ms
checkpoint-restore tasks.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): gate scoring capacity by model usecase

Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity.

Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step

The builder-prebuilt path installs gcc-14 with apt directly and ignored
the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already
honors, so an ubuntu mirror outage broke every arm64 backend build. Pass
the args into the stage and run apt-mirror.sh (already in the build
context via COPY . /LocalAI) before the apt step.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): classifier argument slots via constrained completion

Hybrid classify-then-complete: a classifier option's canned tool call can
declare typed argument slots (number | enum | string, with defaults and
prompt hints) referenced as "{{name}}" in the arguments template. When
the option wins, the slots are filled by a short grammar-constrained
completion that continues the exact scoring prompt — rendered by the same
cached ScoreClassifier, so the llama.cpp prompt cache is already warm —
with the chosen route JSON re-opened at the first slot field. A GBNF
grammar pins the field skeleton and frees only the values; temperature 0,
a couple dozen tokens at most (~300ms on a desktop CPU for two slots).

Slot declarations and hints ride the option descriptions in the shared
system prompt, informing scoring and the fill alike at no per-turn token
cost. The localai.classifier.result event carries the final arguments and
a fill_latency_ms. On inference failure the slots' defaults apply; a slot
without a default fails the response (or falls through with
fallback.mode: generate). Slot filling requires completion alongside
score in the scoring model's known_usecases.

Verified end-to-end on the Pi drone demo: "fly forward three meters" in
distance mode classifies forward and infers {"distance": 3, "units":
"meters"} in ~310ms, and the drone flies exactly 3 units.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): splice filled slot values into classifier replies

A classifier option's spoken reply can now reference its tool's argument
slots ("Going forward {{distance}} {{units}}."): the values inferred by
the slot-fill completion — or the recovery defaults — are spliced into
the reply as plain text before it is emitted, so what the assistant says
confirms what it actually inferred. Placeholders without a value stay
literal, and options without slots are untouched.

FillToolArguments now returns the raw slot values alongside the spliced
arguments JSON to make the reply templating possible.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): harden classifier slot completion

Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(realtime): prewarm the classifier scoring prompt on registration

Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.

Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.

Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix

Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new
small models are headed) cannot rewind their state, so any prompt-cache
reuse that needs a rewind falls back to a full re-prefill. For classifier
scoring that meant every probe change re-processed the whole option-list
prompt: the server's checkpoints were placed reactively (at wherever the
previous task happened to diverge), so a checkpoint past the next
divergence was erased rather than restored — measured as intermittent
2-10s turns on prompts with a 95%+ common prefix.

The classifier now computes the probe-invariant prompt prefix once (the
byte-wise common prefix of two synthetic probe renders) and declares its
length with every Score request; the server maps it to a token boundary
and forces a KV checkpoint exactly there on each score prefill. That
checkpoint sits at or before every future divergence under the same
option list, so it always survives and always restores — repeat scoring
costs probe+candidates regardless of how the probe changes.

Also:
- prewarm reruns on every option-list registration instead of memoizing
  per list: with boundary checkpoints a redundant rewarm costs two
  probe-sized decodes, while skipping one after a slot eviction (three
  lists sharing fewer slots evict in LRU cascades) silently moves a full
  re-prefill onto the user's next turn
- new llama.cpp backend option rs_seq:N exposes bounded recurrent-state
  rollback outside speculative decoding; measured impractical for
  deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap
  insurance for small-state models
- docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot
  similarity threshold funnels distinct lists onto one slot)

Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady
state: every turn 285-421ms including mode switches, vs 2.4s post-switch
and intermittent 1.3-2.9s re-prefills before.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(realtime): align classifier cache guidance

Document the single-score prewarm behavior and clean the vendored score patch formatting.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(llama-cpp): guard score task for fork backends

TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields.

Assisted-by: Codex:gpt-5 [gh]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(dev): generate gRPC code before commit lint

The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
2026-07-29 12:50:22 +02:00

1061 lines
45 KiB
Go
Raw Permalink Blame History

package meta
import "github.com/mudler/LocalAI/core/services/routing/piipattern"
// builtinPatternOptions turns the piipattern built-in catalogue into select
// options for the editor's built-in-patterns checklist, keeping the catalogue
// the single source of truth.
func builtinPatternOptions() []FieldOption {
cat := piipattern.BuiltinCatalogue()
out := make([]FieldOption, 0, len(cat))
for _, b := range cat {
out = append(out, FieldOption{Value: b.Name, Label: b.Name + " — " + b.Description})
}
return out
}
// DefaultRegistry returns enrichment overrides for the ~30 most commonly used
// config fields. Fields not listed here still appear with auto-generated
// labels and type-inferred components.
func DefaultRegistry() map[string]FieldMetaOverride {
f64 := func(v float64) *float64 { return &v }
return map[string]FieldMetaOverride{
// --- General ---
"name": {
Section: "general",
Label: "Model Name",
Description: "Unique identifier for this model configuration",
Component: "input",
Order: 0,
},
"backend": {
Section: "general",
Label: "Backend",
Description: "The inference backend to use (e.g. llama-cpp, vllm, diffusers)",
Component: "select",
AutocompleteProvider: ProviderBackends,
Order: 1,
},
"description": {
Section: "general",
Label: "Description",
Description: "Human-readable description of what this model does",
Component: "textarea",
Order: 2,
},
"usage": {
Section: "general",
Label: "Usage",
Description: "Usage instructions or notes",
Component: "textarea",
Advanced: true,
Order: 3,
},
"cuda": {
Section: "general",
Label: "CUDA",
Description: "Explicitly enable CUDA acceleration",
Order: 5,
},
"known_usecases": {
Section: "general",
Label: "Known Use Cases",
Description: "Capabilities this model supports",
Component: "string-list",
Options: UsecaseOptions,
Order: 6,
},
"known_input_modalities": {
Section: "general",
Label: "Known Input Modalities",
Description: "Explicit input types this model accepts when use cases alone are not specific enough",
Component: "string-list",
Options: ModalityOptions,
Order: 7,
},
"known_output_modalities": {
Section: "general",
Label: "Known Output Modalities",
Description: "Explicit output types this model produces when use cases alone are not specific enough",
Component: "string-list",
Options: ModalityOptions,
Order: 8,
},
"artifacts": {
Section: "general",
Label: "Managed Model Artifacts",
Description: "Controller-managed model sources. LocalAI resolves, downloads, verifies, and binds these sources before a backend loads.",
Component: "json-editor",
Advanced: true,
Order: 9,
},
// --- LLM ---
"context_size": {
Section: "llm",
Label: "Context Size",
Description: "Maximum context window in tokens",
Component: "number",
VRAMImpact: true,
Order: 10,
},
"gpu_layers": {
Section: "llm",
Label: "GPU Layers",
Description: "Number of layers to offload to GPU (-1 = all)",
Component: "number",
Min: f64(-1),
VRAMImpact: true,
Order: 11,
},
"threads": {
Section: "llm",
Label: "Threads",
Description: "Number of CPU threads for inference",
Component: "number",
Min: f64(1),
Order: 12,
},
"f16": {
Section: "llm",
Label: "F16",
Description: "Use 16-bit floating point for key/value cache",
Order: 13,
},
"mmap": {
Section: "llm",
Label: "Memory Map",
Description: "Use memory-mapped files for model loading",
Order: 14,
},
"mmlock": {
Section: "llm",
Label: "Memory Lock",
Description: "Lock model memory to prevent swapping",
Advanced: true,
Order: 15,
},
"low_vram": {
Section: "llm",
Label: "Low VRAM",
Description: "Optimize for systems with limited GPU memory",
VRAMImpact: true,
Order: 16,
},
"embeddings": {
Section: "llm",
Label: "Embeddings",
Description: "Enable embedding generation mode",
Order: 17,
},
"quantization": {
Section: "llm",
Label: "Quantization",
Description: "Quantization method (e.g. q4_0, q5_1, q8_0)",
Component: "select",
Options: QuantizationOptions,
Advanced: true,
Order: 20,
},
"flash_attention": {
Section: "llm",
Label: "Flash Attention",
Description: "Enable flash attention for faster inference",
Component: "input",
Advanced: true,
Order: 21,
},
"reasoning_effort": {
Section: "llm",
Label: "Reasoning Effort",
Description: "Default reasoning effort, forwarded to the backend as the reasoning_effort chat_template_kwarg (jinja models like gpt-oss / LFM2.5 honor it). A per-request reasoning_effort overrides it. 'none' also turns thinking off.",
Component: "select",
Options: []FieldOption{
{Value: "", Label: "Unset (model default)"},
{Value: "none", Label: "none (disable thinking)"},
{Value: "minimal", Label: "minimal"},
{Value: "low", Label: "low"},
{Value: "medium", Label: "medium"},
{Value: "high", Label: "high"},
},
Advanced: true,
Order: 22,
},
"cache_type_k": {
Section: "llm",
Label: "KV Cache Type (K)",
Description: "Quantization type for key cache (e.g. f16, q8_0, q4_0)",
Component: "select",
Options: CacheTypeOptions,
VRAMImpact: true,
Advanced: true,
Order: 22,
},
"cache_type_v": {
Section: "llm",
Label: "KV Cache Type (V)",
Description: "Quantization type for value cache",
Component: "select",
Options: CacheTypeOptions,
VRAMImpact: true,
Advanced: true,
Order: 23,
},
// --- Parameters ---
"parameters.temperature": {
Section: "parameters",
Label: "Temperature",
Description: "Sampling temperature (higher = more creative, lower = more deterministic)",
Component: "slider",
Min: f64(0),
Max: f64(2),
Step: f64(0.05),
Order: 30,
},
"parameters.top_p": {
Section: "parameters",
Label: "Top P",
Description: "Nucleus sampling threshold",
Component: "slider",
Min: f64(0),
Max: f64(1),
Step: f64(0.01),
Order: 31,
},
"parameters.top_k": {
Section: "parameters",
Label: "Top K",
Description: "Top-K sampling: consider only the K most likely tokens",
Component: "number",
Min: f64(0),
Order: 32,
},
"parameters.max_tokens": {
Section: "parameters",
Label: "Max Tokens",
Description: "Maximum number of tokens to generate (0 = unlimited)",
Component: "number",
Min: f64(0),
Order: 33,
},
"parameters.repeat_penalty": {
Section: "parameters",
Label: "Repeat Penalty",
Description: "Penalize repeated tokens (1.0 = no penalty)",
Component: "number",
Min: f64(0),
Advanced: true,
Order: 34,
},
"parameters.seed": {
Section: "parameters",
Label: "Seed",
Description: "Random seed (-1 = random)",
Component: "number",
Advanced: true,
Order: 35,
},
// --- Templates ---
"template.chat": {
Section: "templates",
Label: "Chat Template",
Description: "Go template for chat completion requests",
Component: "code-editor",
Language: "gotemplate",
Order: 40,
},
"template.chat_message": {
Section: "templates",
Label: "Chat Message Template",
Description: "Go template for individual chat messages",
Component: "code-editor",
Language: "gotemplate",
Order: 41,
},
"template.completion": {
Section: "templates",
Label: "Completion Template",
Description: "Go template for completion requests",
Component: "code-editor",
Language: "gotemplate",
Order: 42,
},
"template.function": {
Section: "templates",
Label: "Functions Template",
Description: "Go template applied when tools/functions are present in the request",
Component: "code-editor",
Language: "gotemplate",
Order: 43,
},
"template.use_tokenizer_template": {
Section: "templates",
Label: "Use Tokenizer Template",
Description: "Use the chat template from the model's tokenizer config",
Order: 44,
},
// Router section template — kept in the templates UI section
// (rather than the router section under "other") so operators
// editing prompt shapes find all template-typed fields in one
// place, mirroring how chat / chat_message are grouped.
"router.classifier_system_template": {
Section: "templates",
Label: "Router Classifier System Prompt",
Description: "Go text/template (with sprig functions) for the routing system prompt the score classifier feeds to its classifier_model. Executed with `.Policies` ([]{Label, Description}). Empty falls back to the built-in Arch-Router-shaped prompt (route-listing block + JSON output schema). Override when the classifier model was trained on a different schema or you need the routing instructions in a different language. The candidate format scored against the model is fixed at `{\"route\": \"<label>\"}` — keep your override's output schema instruction matching that.",
Component: "code-editor",
Language: "gotemplate",
Order: 45,
},
// --- Alias ---
"alias": {
Section: "alias",
Label: "Alias target",
Description: "Redirect all traffic for this model to another configured model. When set, every other field on this config is ignored and requests are served by the target model.",
Component: "model-select",
Order: 0,
},
// --- Pipeline ---
"pipeline.llm": {
Section: "pipeline",
Label: "LLM Model",
Description: "Model to use for LLM inference in the pipeline",
Component: "model-select",
AutocompleteProvider: ProviderModelsChat,
Order: 60,
},
"pipeline.tts": {
Section: "pipeline",
Label: "TTS Model",
Description: "Model to use for text-to-speech in the pipeline",
Component: "model-select",
AutocompleteProvider: ProviderModelsTTS,
Order: 61,
},
"pipeline.transcription": {
Section: "pipeline",
Label: "Transcription Model",
Description: "Model to use for speech-to-text in the pipeline",
Component: "model-select",
AutocompleteProvider: ProviderModelsTranscript,
Order: 62,
},
"pipeline.vad": {
Section: "pipeline",
Label: "VAD Model",
Description: "Model to use for voice activity detection in the pipeline",
Component: "model-select",
AutocompleteProvider: ProviderModelsVAD,
Order: 63,
},
"pipeline.sound_detection": {
Section: "pipeline",
Label: "Sound Detection Model",
Description: "Model to use for sound-event classification (audio tagging, e.g. ced) in the pipeline. When set, committed realtime audio is also classified and the scored AudioSet tags are emitted as a conversation.item.sound_detection event.",
Component: "model-select",
AutocompleteProvider: ProviderModels,
Order: 64,
},
"pipeline.sound_detection_window_ms": {
Section: "pipeline",
Label: "Sound Detection Window (ms)",
Description: "Server-side windowing for a sound-only realtime session: length in ms of the audio window classified each hop. 0 = client-driven (the client commits windows).",
Component: "number",
Min: f64(0),
Order: 65,
},
"pipeline.sound_detection_hop_ms": {
Section: "pipeline",
Label: "Sound Detection Hop (ms)",
Description: "Server-side windowing hop in ms: how often the server classifies the last window. 0 = client-driven.",
Component: "number",
Min: f64(0),
Order: 66,
},
"pipeline.reasoning_effort": {
Section: "pipeline",
Label: "Reasoning Effort",
Description: "Reasoning effort for the pipeline's LLM, forwarded to the backend as the reasoning_effort chat_template_kwarg (jinja models like gpt-oss / LFM2.5 honor it). Overrides the LLM model's own reasoning_effort. 'none' also turns thinking off.",
Component: "select",
Options: []FieldOption{
{Value: "", Label: "Default (model config)"},
{Value: "none", Label: "none (disable thinking)"},
{Value: "minimal", Label: "minimal"},
{Value: "low", Label: "low"},
{Value: "medium", Label: "medium"},
{Value: "high", Label: "high"},
},
Order: 64,
},
"pipeline.disable_thinking": {
Section: "pipeline",
Label: "Disable Thinking",
Description: "Suppress reasoning/thinking output from the pipeline LLM (sets enable_thinking=false on the underlying model). Use for models that emit <think> blocks you don't want spoken or streamed back to the realtime client.",
Component: "toggle",
Order: 65,
},
"pipeline.streaming.llm": {
Section: "pipeline",
Label: "Stream LLM",
Description: "Stream LLM tokens to the realtime client as they are generated instead of waiting for the full response. Emits incremental response.output_audio_transcript.delta / text deltas.",
Component: "toggle",
Order: 66,
},
"pipeline.streaming.tts": {
Section: "pipeline",
Label: "Stream TTS",
Description: "Stream synthesized audio chunks to the realtime client as they are produced (requires a TTS backend that implements TTSStream). Falls back to unary synthesis otherwise.",
Component: "toggle",
Order: 67,
},
"pipeline.streaming.transcription": {
Section: "pipeline",
Label: "Stream Transcription",
Description: "Stream partial transcription text to the realtime client as the STT backend produces it (requires a transcription backend that implements AudioTranscriptionStream). Falls back to unary transcription otherwise.",
Component: "toggle",
Order: 68,
},
"pipeline.streaming.clause_chunking": {
Section: "pipeline",
Label: "Clause Chunking",
Description: "Split the streamed reply into speakable clauses and synthesize each as soon as it completes, instead of buffering the whole message before TTS — lower time-to-first-audio. Script-aware (handles CJK 。！？ and Thai/Lao spaces), so it does not whitespace-split. Requires Stream LLM; off buffers the whole message.",
Component: "toggle",
Order: 69,
},
"pipeline.voice_recognition.model": {
Section: "pipeline",
Label: "Voice Recognition Model",
Description: "Speaker-recognition backend model used to gate the pipeline behind speaker verification. Leave empty to disable the voice gate.",
Component: "model-select",
AutocompleteProvider: ProviderModels,
Order: 70,
},
"pipeline.voice_recognition.mode": {
Section: "pipeline",
Label: "Voice Gate Mode",
Description: "How callers are authorized: 'identify' matches the speaker 1:N against the voice registry; 'verify' matches 1:few against the configured reference audios.",
Component: "select",
Options: []FieldOption{
{Value: "identify", Label: "identify (registry)"},
{Value: "verify", Label: "verify (references)"},
},
Order: 71,
},
"pipeline.voice_recognition.threshold": {
Section: "pipeline",
Label: "Voice Gate Threshold",
Description: "Maximum cosine distance between the caller and an authorized speaker that still counts as a match. Lower is stricter. Default 0.25 is tuned for the ECAPA-TDNN encoder on VoxCeleb.",
Component: "slider",
Min: f64(0.01),
Max: f64(2),
Step: f64(0.01),
Order: 72,
},
"pipeline.voice_recognition.when": {
Section: "pipeline",
Label: "Voice Gate When",
Description: "How often to verify the speaker: 'every' checks each utterance; 'first' verifies once and then trusts the session.",
Component: "select",
Options: []FieldOption{
{Value: "every", Label: "every utterance"},
{Value: "first", Label: "first only"},
},
Order: 73,
},
"pipeline.voice_recognition.on_reject": {
Section: "pipeline",
Label: "Voice Gate On Reject",
Description: "What to do with an unauthorized utterance: 'drop_event' drops it and emits an error event to the client; 'drop_silent' drops it quietly.",
Component: "select",
Options: []FieldOption{
{Value: "drop_event", Label: "drop + error event"},
{Value: "drop_silent", Label: "drop silently"},
},
Order: 74,
},
"pipeline.voice_recognition.anti_spoofing": {
Section: "pipeline",
Label: "Voice Gate Anti-Spoofing",
Description: "Enable the backend liveness/anti-spoofing check (verify mode only) to reject replayed or synthesized audio.",
Component: "toggle",
Order: 75,
},
"pipeline.voice_recognition.allow.names": {
Section: "pipeline",
Label: "Voice Gate Allowed Names",
Description: "Identify mode: authorize only registry identities whose name matches one of these exactly. Empty allows any registered identity.",
Component: "string-list",
Order: 76,
},
"pipeline.voice_recognition.allow.labels": {
Section: "pipeline",
Label: "Voice Gate Allowed Labels",
Description: "Identify mode: authorize any registry identity carrying one of these label keys. Empty allows any registered identity.",
Component: "string-list",
Order: 77,
},
"pipeline.voice_recognition.references": {
Section: "pipeline",
Label: "Voice Gate References",
Description: "Verify mode: the authorized reference speakers, each with a name and an audio file path the caller's voice is matched against.",
Component: "json-editor",
Order: 78,
},
"pipeline.voice_recognition.enforce": {
Section: "pipeline",
Label: "Voice Gate Enforce",
Description: "Whether the gate rejects unauthorized speakers. Enabled (default) drops unauthorized utterances before the LLM. Disabled still resolves and surfaces the speaker (for the conversation.item.speaker event and personalization) but never drops a turn.",
Component: "toggle",
Order: 80,
},
"pipeline.voice_recognition.identity.announce": {
Section: "pipeline",
Label: "Speaker Identity Announce",
Description: "Emit a conversation.item.speaker event to the client naming the recognized speaker. When set, identity is resolved on every turn even if 'when' is 'first'.",
Component: "toggle",
Order: 81,
},
"pipeline.voice_recognition.identity.announce_unknown": {
Section: "pipeline",
Label: "Speaker Identity Announce Unknown",
Description: "Also emit the conversation.item.speaker event (with matched=false) when no confident match is found. Default only announces on a match.",
Component: "toggle",
Order: 82,
},
"pipeline.voice_recognition.identity.personalize": {
Section: "pipeline",
Label: "Speaker Identity Personalize",
Description: "Inform the LLM who is speaking so it can tailor replies. Enables the name and system-note injection below.",
Component: "toggle",
Order: 83,
},
"pipeline.voice_recognition.identity.inject_name": {
Section: "pipeline",
Label: "Speaker Identity Inject Name",
Description: "Personalization: set the per-message OpenAI 'name' field on each user turn to the recognized speaker.",
Component: "toggle",
Order: 84,
},
"pipeline.voice_recognition.identity.inject_system_note": {
Section: "pipeline",
Label: "Speaker Identity Inject System Note",
Description: "Personalization: append a 'The current speaker is <name>.' note to the system message reflecting the latest speaker.",
Component: "toggle",
Order: 85,
},
"pipeline.voice_recognition.identity.note_unknown": {
Section: "pipeline",
Label: "Speaker Identity Note Unknown",
Description: "Personalization: when the speaker is unidentified, append 'The current speaker is unknown.' to the system message so the model can ask who it is talking to.",
Component: "toggle",
Order: 86,
},
"pipeline.max_history_items": {
Section: "pipeline",
Label: "Max History Items",
Description: "Cap how many trailing conversation items are fed to the LLM each realtime turn (0 = unlimited, rely on the LLM's context window). Set it on a composed pipeline (VAD+STT+LLM+TTS) so a long-running session doesn't grow until the context fills. Unset uses the per-model-type default.",
Component: "number",
Order: 79,
},
"pipeline.compaction.enabled": {
Section: "pipeline",
Label: "Compaction Enabled",
Description: "Fold conversation items that age out of the live window (Max History Items) into a rolling summary instead of dropping them, so long realtime sessions stay cheap without losing earlier context. Off by default.",
Component: "toggle",
Order: 80,
},
"pipeline.compaction.trigger_items": {
Section: "pipeline",
Label: "Compaction Trigger Items",
Description: "High-water mark: once the live conversation exceeds this many items, the overflow above Max History Items is summarized and evicted. Must be greater than Max History Items; defaults to twice it. The gap controls how often summarization runs.",
Component: "number",
Order: 81,
},
"pipeline.compaction.summary_model": {
Section: "pipeline",
Label: "Compaction Summary Model",
Description: "Optional smaller/cheaper model used to produce the rolling summary. Empty reuses the pipeline's own LLM. On CPU, a tiny model here keeps compaction from competing with the conversation LLM.",
Component: "input",
Advanced: true,
Order: 82,
},
"pipeline.compaction.max_summary_tokens": {
Section: "pipeline",
Label: "Compaction Max Summary Tokens",
Description: "Advisory cap on the rolling summary length (fed to the summarizer prompt). Defaults to 512.",
Component: "number",
Advanced: true,
Order: 83,
},
"pipeline.turn_detection.type": {
Section: "pipeline",
Label: "Turn Detection",
Description: "Default turn-detection mode for realtime sessions on this pipeline. server_vad commits after a fixed silence window; semantic_vad lets the transcription model's end-of-utterance token drive a dynamic window (fast commit after the token, long eagerness fallback without it). semantic_vad requires a streaming-EOU transcription model (e.g. parakeet-cpp-realtime_eou_120m-v1) and degrades to silence-only otherwise. Clients can override per session via session.update.",
Component: "select",
Options: []FieldOption{
{Value: "", Label: "Default (server_vad)"},
{Value: "server_vad", Label: "server_vad (silence-based)"},
{Value: "semantic_vad", Label: "semantic_vad (end-of-utterance token)"},
},
Order: 87,
},
"pipeline.turn_detection.eagerness": {
Section: "pipeline",
Label: "Eagerness",
Description: "semantic_vad fallback silence window used when no end-of-utterance token was seen: low waits 8s, medium/auto 4s, high 2s.",
Component: "select",
Options: []FieldOption{
{Value: "", Label: "Default (auto)"},
{Value: "low", Label: "low (8s)"},
{Value: "medium", Label: "medium (4s)"},
{Value: "high", Label: "high (2s)"},
},
Order: 88,
},
"pipeline.turn_detection.retranscribe": {
Section: "pipeline",
Label: "Retranscribe on Commit",
Description: "Cross-check every semantic_vad commit with an offline decode of the buffered turn: commit only proceeds when the batch decode also ends in the end-of-utterance token, and its transcript is used. Logs a streamed-vs-batch comparison — useful to gauge streaming/batch alignment — at the cost of one extra decode per turn.",
Component: "toggle",
Order: 89,
},
"pipeline.turn_detection.vad_window_sec": {
Section: "pipeline",
Label: "VAD Window (s)",
Description: "Widen the slice of recent audio the VAD rescans each turn-detection tick. Sized automatically from the commit silence threshold (server_vad silence window, or the semantic eagerness fallback) plus a warm-up margin — set only to widen it; values below the automatic floor are ignored.",
Component: "number",
Order: 90,
},
"pipeline.disable_warmup": {
Section: "pipeline",
Label: "Disable Warmup",
Description: "Turn off eager pre-loading of the pipeline's sub-models at realtime session start. By default LocalAI loads every configured sub-model backend (VAD, transcription, LLM, TTS, sound detection, voice recognition) before the session starts and blocks until they are ready, so the first turn pays no cold-start cost and a model that fails to load is reported at session start instead of mid-call. Enable this to restore the lazy 'load on first use' behavior — session start no longer waits on loading and load errors surface on the first turn instead. Useful to keep idle sessions from holding model memory they may never use.",
Component: "toggle",
Order: 90,
},
"pipeline.classifier.enabled": {
Section: "pipeline",
Label: "Classifier Mode",
Description: "Replace autoregressive generation with prefill-only option selection: each user turn is scored against the option list via the Score primitive and the winning option's canned reply / tool call is emitted. Built for hardware that can afford prompt processing but not decode (e.g. a Raspberry Pi).",
Component: "toggle",
Order: 91,
},
"pipeline.classifier.options": {
Section: "pipeline",
Label: "Classifier Options",
Description: "The intents the classifier scores each turn against. Each option has an id (also the scored route label — keep it short), a description of when it applies, an optional canned spoken reply, and an optional canned tool call {name, arguments}. A tool may also declare slots ([{name, type: number|enum|string, values, default, hint}]) whose \"{{name}}\" placeholders in arguments (and, optionally, the reply) are filled by a short grammar-constrained completion when the option wins — the hybrid between prefill-only classification and full generation (requires completion in the scoring model's known_usecases). Clients can replace the list per session via session.update localai_classifier.",
Component: "json-editor",
Order: 92,
},
"pipeline.classifier.threshold": {
Section: "pipeline",
Label: "Classifier Threshold",
Description: "Softmax-probability floor the best option must clear; below it the fallback applies. 0 always picks the argmax.",
Component: "slider",
Min: f64(0),
Max: f64(0.99),
Step: f64(0.01),
Order: 93,
},
"pipeline.classifier.fallback.mode": {
Section: "pipeline",
Label: "Classifier Fallback",
Description: "What happens when no option clears the threshold: complete with no output, speak the canned fallback reply, or fall through to normal (slow) generation.",
Component: "select",
Options: []FieldOption{
{Value: "none", Label: "none (empty response)"},
{Value: "reply", Label: "canned reply"},
{Value: "generate", Label: "generate"},
},
Order: 94,
},
"pipeline.classifier.fallback.reply": {
Section: "pipeline",
Label: "Classifier Fallback Reply",
Description: "The canned reply spoken when the fallback mode is 'reply' and no option clears the threshold.",
Component: "text",
Order: 95,
},
"pipeline.classifier.normalization": {
Section: "pipeline",
Label: "Classifier Normalization",
Description: "How option scores feed the softmax: 'raw' compares joint log-probs (default); 'mean' divides by token count, which is fairer when option ids have very different lengths.",
Component: "select",
Options: []FieldOption{
{Value: "raw", Label: "raw (joint log-prob)"},
{Value: "mean", Label: "mean (per-token)"},
},
Order: 96,
},
"pipeline.classifier.history_items": {
Section: "pipeline",
Label: "Classifier History Items",
Description: "What gets scored: 0 or -1 (default) score only the latest user message; a positive N includes the trailing N conversation messages, role-labeled. Prior turns echo option names and can dominate small scoring models — only opt in with a larger scorer.",
Component: "number",
Order: 97,
},
"pipeline.classifier.model": {
Section: "pipeline",
Label: "Classifier Scoring Model",
Description: "Optionally score on a different model config. Empty uses the pipeline LLM — scoring runs through the same llama.cpp slot as generation and shares its prompt cache, so a separate model is rarely needed.",
Component: "model-select",
AutocompleteProvider: ProviderModels,
Order: 98,
},
"pipeline.classifier.address.names": {
Section: "pipeline",
Label: "Classifier Address Names",
Description: "Wake-word gate: only act on turns that mention one of these names as a whole word ('Drone go up', not just 'go up'). Matching is deterministic on the transcript; unaddressed turns skip scoring entirely.",
Component: "string-list",
Order: 99,
},
"pipeline.classifier.address.mode": {
Section: "pipeline",
Label: "Classifier Address Mode",
Description: "What to do with unaddressed turns: 'ignore' completes silently (right for ambient conversation), 'reply' speaks the address reply.",
Component: "select",
Options: []FieldOption{
{Value: "ignore", Label: "ignore (stay silent)"},
{Value: "reply", Label: "reply (speak the address reply)"},
},
Order: 100,
},
"pipeline.classifier.address.reply": {
Section: "pipeline",
Label: "Classifier Address Reply",
Description: "Spoken when an unaddressed turn arrives in 'reply' mode.",
Order: 101,
},
// --- Functions ---
"function.grammar.parallel_calls": {
Section: "functions",
Label: "Parallel Calls",
Description: "Allow the LLM to return multiple function calls in one response",
Order: 70,
},
"function.grammar.mixed_mode": {
Section: "functions",
Label: "Mixed Mode",
Description: "Allow the LLM to return both text and function calls",
Order: 71,
},
"function.grammar.disable": {
Section: "functions",
Label: "Disable Grammar",
Description: "Disable grammar-constrained generation for function calls",
Advanced: true,
Order: 72,
},
// --- TTS ---
"tts.voice_cloning": {
Section: "tts",
Label: "Voice Cloning",
Description: "Override automatic Voice Library profile support detection for this model. Leave unset to infer support from the backend and model variant.",
Component: "toggle",
Advanced: true,
Order: 89,
},
"tts.voice": {
Section: "tts",
Label: "Voice",
Description: "Default voice for TTS output",
Component: "input",
Order: 90,
},
"tts.audio_path": {
Section: "tts",
Label: "Reference Audio Path",
Description: "Default reference audio for voice cloning. A per-request voice or saved Voice Library profile takes precedence.",
Component: "input",
Order: 91,
},
// --- Diffusers ---
"diffusers.pipeline_type": {
Section: "diffusers",
Label: "Pipeline Type",
Description: "Diffusers pipeline type (e.g. StableDiffusionPipeline)",
Component: "select",
Options: DiffusersPipelineOptions,
Order: 80,
},
"diffusers.scheduler_type": {
Section: "diffusers",
Label: "Scheduler Type",
Description: "Noise scheduler type",
Component: "select",
Options: DiffusersSchedulerOptions,
Order: 81,
},
"diffusers.cuda": {
Section: "diffusers",
Label: "CUDA",
Description: "Enable CUDA for diffusers",
Order: 82,
},
// --- PII filtering (per-model) ---
"pii.enabled": {
Section: "pii",
Label: "PII Filtering Enabled",
Description: "Enable PII redaction middleware for this model. Unset means use the default (off for local backends, on for proxy-* / cloud-hosted backends).",
Component: "toggle",
Order: 200,
},
"pii.detectors": {
Section: "pii",
Label: "PII Detector Models",
Description: "Token-classification (NER) models that scan this model's requests for PII. The detection policy (which entities, what action, min score) lives on each detector model's own PII Detection block. Multiple detectors union their hits.",
Component: "model-multi-select",
AutocompleteProvider: "models:token_classify",
Order: 201,
},
// --- PII detection policy (on a token_classify detector model) ---
"pii_detection.min_score": {
Section: "pii",
Label: "Detector Min Score",
Description: "When this model is used as a PII detector, drop detections scored below this confidence before they are acted on. 0 keeps every detection.",
Component: "slider",
Min: f64(0),
Max: f64(1),
Step: f64(0.01),
Order: 210,
},
"pii_detection.default_action": {
Section: "pii",
Label: "Detector Default Action",
Description: "Action applied to detected entity groups with no explicit per-entity override. Defaults to mask — the safe-by-default policy for a PII filter.",
Component: "select",
Options: []FieldOption{
{Value: "mask", Label: "mask (redact the span)"},
{Value: "block", Label: "block (reject the request)"},
{Value: "allow", Label: "allow (detect & log only)"},
},
Default: "mask",
Order: 211,
},
"pii_detection.entity_actions": {
Section: "pii",
Label: "Detector Entity Actions",
Description: "Per-entity-group action policy for this detector model (e.g. PASSWORD → block, EMAIL → mask). Groups without an entry use the default action.",
Component: "entity-action-list",
Order: 212,
},
"pii_detection.builtins": {
Section: "pii",
Label: "Built-in Secret Patterns",
Description: "Built-in regex patterns for common credentials (API keys, tokens, private keys). Turning any on makes this a pattern detector — it matches high-entropy secrets the NER tier can't, in-process with no model load.",
Component: "pii-builtins-select",
Options: builtinPatternOptions(),
Order: 213,
},
"pii_detection.patterns": {
Section: "pii",
Label: "Custom Secret Patterns",
Description: "Operator-defined patterns in a restricted regex subset (e.g. \"sk-prefix-\\w+\"). Each must contain a fixed literal anchor of ≥3 chars; open-ended shapes like emails are rejected (leave those to NER). Matches report under the pattern name as the entity group.",
Component: "pii-pattern-list",
Order: 214,
},
// --- Cloud passthrough proxy ---
// These only have an effect when Backend is set to
// "cloud-proxy". When the upstream URL is empty, the model
// fails closed — the chat handler does NOT silently fall back
// to the local gRPC pipeline.
"proxy.mode": {
Section: "proxy",
Label: "Proxy Mode",
Description: "passthrough forwards the client's OpenAI body verbatim — point upstream_url at an OpenAI-compatible endpoint (incl. Anthropic's /v1/chat/completions compat layer). translate converts OpenAI ↔ Anthropic Messages so you can target a native API (/v1/messages); tool_calls and usage tokens survive the round-trip.",
Component: "select",
Options: []FieldOption{
{Value: "passthrough", Label: "passthrough (raw forward)"},
{Value: "translate", Label: "translate (OpenAI ↔ native)"},
},
Default: "passthrough",
Order: 208,
},
"proxy.provider": {
Section: "proxy",
Label: "Proxy Provider",
Description: "Upstream API family. Drives auth header shape (Bearer vs x-api-key + anthropic-version) and, in translate mode, which request/response codec is used.",
Component: "select",
Options: []FieldOption{
{Value: "openai", Label: "OpenAI"},
{Value: "anthropic", Label: "Anthropic"},
},
Default: "openai",
Order: 209,
},
"proxy.upstream_url": {
Section: "proxy",
Label: "Proxy Upstream URL",
Description: "Full POST endpoint of the upstream provider (e.g. https://api.openai.com/v1/chat/completions). Only used when Backend is cloud-proxy.",
Component: "input",
Order: 210,
},
"proxy.api_key_env": {
Section: "proxy",
Label: "Proxy API Key Env Var",
Description: "Name of the environment variable holding the upstream API key. Reading from env keeps the secret out of the YAML and the admin UI.",
Component: "input",
Order: 211,
},
"proxy.upstream_model": {
Section: "proxy",
Label: "Proxy Upstream Model",
Description: "Model name sent to the upstream. Leave empty to forward the client's model field unchanged. Useful when the LocalAI alias differs from the upstream's canonical name.",
Component: "input",
Order: 212,
},
"proxy.request_timeout_seconds": {
Section: "proxy",
Label: "Proxy Request Timeout (seconds)",
Description: "Caps the upstream HTTP request duration. 0 disables the deadline; the request still ends when the client disconnects.",
Component: "number",
Min: f64(0),
Order: 213,
},
"proxy.cache_prompt": {
Section: "proxy",
Label: "Proxy Anthropic Prompt Cache",
Description: "Inject Anthropic prompt-cache breakpoints (cache_control: ephemeral) on the stable prefix (system, tools, last message) when mode is translate and provider is anthropic. Serves the repeated prefix at the cache-read rate on multi-turn/agentic calls. No effect otherwise.",
Component: "checkbox",
Order: 214,
},
// --- MITM intercept hosts ---
// Each host listed here is claimed by this model config; the
// cloudproxy MITM listener (see Middleware → MITM Proxy) uses
// THIS config's pii: settings to filter the intercepted traffic.
// A host claimed by two configs is a critical error — the
// listener refuses to start until resolved.
"mitm.hosts": {
Section: "mitm",
Label: "MITM Intercept Hosts",
Description: "Hostnames the cloudproxy MITM proxy terminates TLS for on behalf of this model config. PII filtering and pattern overrides flow from this model when the host is intercepted. Each host must be unique across all configs.",
Component: "string-list",
Order: 220,
},
// --- Router ---
// Routing turns this model config into a dispatcher: the
// classifier scores every policy label as a continuation of
// the routing prompt and picks the first candidate whose
// labels are a superset of the active set. The Routing tab of
// the middleware admin page surfaces every model with a router
// block.
"router.classifier": {
Section: "router",
Label: "Classifier",
Description: "Picks a candidate by scoring every policy label against the prompt. Only \"score\" is shipped today; it asks the classifier_model to rank each label and reads off the softmax. Empty defaults to \"score\".",
Component: "select",
Options: []FieldOption{
{Value: "score", Label: "Score (Arch-Router-style)"},
},
Order: 230,
},
"router.classifier_model": {
Section: "router",
Label: "Classifier Model",
Description: "Loaded LocalAI model the score classifier asks to rank each policy label as a continuation. Must support the Score gRPC primitive (today: llama-cpp, vLLM) and use the ChatML template. Arch-Router-1.5B Q4_K_M is the canonical choice; any small ChatML instruct model also works at a higher activation_threshold.",
Component: "model-select",
AutocompleteProvider: ProviderModelsScore,
Order: 231,
},
"router.fallback": {
Section: "router",
Label: "Fallback Model",
Description: "Model used when no candidate's labels cover the classifier's active label set, or when the classifier errors. Empty means router failures bubble up as HTTP 500 — fail-fast, not silent-bypass.",
Component: "model-select",
AutocompleteProvider: ProviderModelsChat,
Order: 232,
},
"router.activation_threshold": {
Section: "router",
Label: "Activation Threshold",
Description: "Softmax-probability floor a policy must clear to join the active label set for a request. Higher → single-label dominant routes; lower → more multi-label activations. 0 picks the package default (0.15). On Arch-Router-1.5B a value around 0.40 keeps the dominant label clean without losing genuine compound activations.",
Component: "slider",
Min: f64(0),
Max: f64(1),
Step: f64(0.05),
Order: 233,
},
"router.classifier_cache_size": {
Section: "router",
Label: "Classifier L1 Cache Size",
Description: "Bounded LRU keyed on (case-folded, whitespace-trimmed) prompt — amortises the classifier round-trip across verbatim repeats common in agent loops. 0 here means \"use the default\" (1024); the cache cannot be disabled from YAML.",
Component: "number",
Min: f64(0),
Order: 234,
},
"router.policies": {
Section: "router",
Label: "Policies",
Description: "Label vocabulary the classifier scores over. Each policy has a label and a short natural-language description fed verbatim to the classifier model. Short action-oriented sentences work best (\"writing or debugging code\"; \"small talk\").",
Component: "router-policies",
Order: 235,
},
"router.candidates": {
Section: "router",
Label: "Candidates",
Description: "Routing table: each entry binds a downstream model to a set of policy labels it can serve. Order matters — the middleware picks the FIRST candidate whose labels are a superset of the active set, so list candidates smallest → largest.",
Component: "router-candidates",
Order: 236,
},
"router.score_normalization": {
Section: "router",
Label: "Score Normalization",
Description: "How the score classifier collapses per-candidate joint log-probs into the softmax input. \"raw\" (default) feeds joint log-prob as-is — on-distribution for Arch-Router (the route the model would actually emit if decoded freely). \"mean\" divides by candidate token count — fairer to long labels but off-distribution for models trained to emit fixed-format outputs.",
Component: "select",
Options: []FieldOption{
{Value: "", Label: "Raw (default)"},
{Value: "raw", Label: "Raw"},
{Value: "mean", Label: "Mean (length-normalised)"},
},
Order: 240,
},
"router.embedding_cache.embedding_model": {
Section: "router",
Label: "L2 Cache: Embedding Model",
Description: "Embedding model used by the L2 decision cache. Embeds incoming probes and looks them up in the per-router local-store collection. Empty disables the cache entirely. nomic-embed-text-v1.5 is the recommended default.",
Component: "model-select",
AutocompleteProvider: ProviderModels,
Order: 237,
},
"router.embedding_cache.similarity_threshold": {
Section: "router",
Label: "L2 Cache: Similarity Threshold",
Description: "Cosine-similarity floor a cache candidate must clear to count as a hit. 0 picks the package default (0.80). Re-tune per embedding model — the histogram on the Routing tab shows where the cosine distribution actually sits.",
Component: "slider",
Min: f64(0),
Max: f64(1),
Step: f64(0.01),
Order: 238,
},
"router.embedding_cache.confidence_threshold": {
Section: "router",
Label: "L2 Cache: Confidence Threshold",
Description: "Minimum top-label probability a classifier decision must have to be inserted into the cache. 0 picks the package default (0.60). Uncertain decisions are skipped so they can't poison future paraphrases.",
Component: "slider",
Min: f64(0),
Max: f64(1),
Step: f64(0.05),
Order: 239,
},
"router.embedding_cache.store_name": {
Section: "router",
Label: "L2 Cache: Store Name",
Description: "Optional override for the local-store collection used by this router's cache. Empty defaults to \"router-cache-<router-model-name>\". Two routers sharing a store_name share their cache (rare).",
Component: "input",
Order: 240,
},
}
}
Reference in New Issue View Git Blame Copy Permalink