mirror/LocalAI
1
0
Fork 0
mirror of https://github.com/mudler/LocalAI.git synced 2026-06-13 11:21:42 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104)

Browse Source 
* fix(router): score classifier production-readiness

Conversation trimming runs through the classifier model's chat template
and trims by exact token count, sized to the model's n_batch which is
now scaled to context so long probes can't crash the backend. Missing
chat_message templates are a hard error at router build time. Router-
facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve
ModelConfig per call so a model installed post-startup doesn't bind a
stub Backend="" config and silently fall into the loader's auto-
iterate path.

New 'vector_store' backend trace recorded inside localVectorStore on
every Search/Insert — including the backend-load-failure path that
previously vanished into an xlog.Warn — with outcome tagging
(hit/miss/empty_store/backend_load_error/find_error/insert_error/ok).
Companion cleanup drops misleading similarity:0 and input_tokens_count:0
from non-hit and text-mode traces.

Gallery local-store-development aliases to 'local-store' so the master
image satisfies pkg/model.LocalStoreBackend lookups from the embedding
cache.

Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key
(the original bug); ModelTokenize nil-guard; non-fatal mitm proxy
startup; PII 'route_local' renamed to 'allow' with docs/UI in sync;
model-editor footer no longer eats the edit area on small screens;
several config-editor template/dropdown/section fixes.

Tests: e2e router specs (casual/code-hint + long-conversation trim),
vector_store trace specs, lazy-factory specs, gallery dev-alias
resolution, Playwright trace badge + scroll regression.

Assisted-by: Claude:claude-opus-4-7 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(backend): auto-size batch to context for embedding and rerank models

Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins.

Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse.

Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch.

Assisted-by: claude-code:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(gallery): raise arch-router scoring output cap via parallel:64

Scoring decodes the whole prompt+candidate in a single llama_decode and
reads one logit row per candidate token. The vendored llama.cpp server
caps causal output rows at n_parallel, so the default of 1 aborts with
GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route
labels. Set options: [parallel:64] on both arch-router quant entries to
lift the cap; kv_unified (the grpc-server default) keeps the full context
per sequence, so this does not split the KV cache.

Assisted-by: claude-code:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
This commit is contained in:
Richard Palethorpe
2026-06-12 15:21:15 +01:00
committed by GitHub
parent 56cc4f63fc
commit 085fc53bbc
86 changed files with 2305 additions and 387 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
core/schema/localai.go
View File

@@ -499,7 +499,7 @@ type RouterDecideResponse struct {
// inspects the text and returns findings + a suggested action; it
// does NOT mutate the input, record an audit event, or rewrite any
// downstream request. The caller composes the decision with its own
// policy (mask, block, route to local-only backends, allow).
// policy (mask, block, or allow).
type PIIDecideRequest struct {
// Text is the user-visible content to inspect. Required.
Text string `json:"text"`
@@ -507,19 +507,20 @@ type PIIDecideRequest struct {
// PIIDecideResponse carries the redactor's findings.
// SuggestedAction is derived from the action ordering used by the
// internal redactor (block > route_local > mask > allow) so callers
// don't need to replicate that logic.
// internal redactor (block > mask > allow) so callers don't need to
// replicate that logic.
type PIIDecideResponse struct {
// Findings is one entry per matched span — pattern id, byte
// range, and audit-safe hash prefix (never the matched value).
Findings []PIIFinding `json:"findings"`
// SuggestedAction is the strongest action across all findings:
// "block", "route_local", "mask", or "allow" (no findings).
// "block", "mask", or "allow" (no findings, or all findings
// resolved to the allow action).
SuggestedAction string `json:"suggested_action"`
// RedactedPreview is the input with mask-action spans replaced
// by their placeholders. Identical to Text when no findings or
// when the strongest action is block/route_local (which don't
// rewrite content).
// when the strongest action is block/allow (which don't rewrite
// content).
RedactedPreview string `json:"redacted_preview"`
}