Files
LocalAI/pkg/mcp/localaitools/coverage_test.go
T
Richard Palethorpe d10374f849 feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus

Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.

Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.

Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.

Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.

Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.

Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(router): name consulted corpus neighbours in knn decisions

Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste

Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.

Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
  router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
  resolution and seed validation; the REST endpoints and the assistant
  MCP client are thin transport adapters over them, with sentinel
  errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
  once for all five entry points (OpenAI, Anthropic, realtime, decide,
  corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
  unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
  their buildClassifier arms; the knn arm owns its embedding_cache
  opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
  middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).

Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
  JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
  crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
  longer takes the manager mutex, so the 5s status poll stops parsing
  vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
  builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
  dropped); test helper dead branch removed.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(mcp): align corpus tool prompts and the mutating-tool safety list

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(proto,backend): report embedding shape from the llama-cpp backend

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* chore(middleware): name the failing fields when post-merge validation 400s

An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix(embeddings): scheme override must not inherit the config's half-life

A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.

Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* fix embedding pooling validation and router bounds

Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

* ci: run local-store integration tests

Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>

---------

Signed-off-by: Richard Palethorpe <io@richiejp.com>
2026-08-18 09:37:43 +02:00

110 lines
5.0 KiB
Go

package localaitools
import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
// toolToHTTPRoute is the canonical mapping between MCP tools and the
// LocalAI admin REST endpoints they wrap. The httpapi.Client MUST hit the
// listed route for the tool; the inproc.Client may bypass HTTP and call
// services directly, but the on-the-wire shape is documented here so the
// two sides stay aligned.
//
// Updating the map is REQUIRED when:
// - You add a Tool* constant (tools.go).
// - You change which REST endpoint the httpapi.Client calls.
//
// The TestToolHTTPRouteMappingComplete spec below FAILS until every Tool*
// is in the map. That is the drift detector — see
// .agents/localai-assistant-mcp.md for the contributor contract.
//
// "(none)" is a deliberate sentinel for tools whose data is not exposed
// over a single REST endpoint (e.g. system_info aggregates data the
// inproc client picks up directly from services). The httpapi.Client may
// approximate via the welcome JSON; the test still requires an entry so
// the contributor explicitly acknowledges the asymmetry.
var toolToHTTPRoute = map[string]string{
// Read-only tools.
ToolGallerySearch: "GET /models/available",
ToolListInstalledModels: "GET / (welcome JSON, ModelsConfig field)",
ToolListGalleries: "GET /models/galleries",
ToolGetJobStatus: "GET /models/jobs/:uuid",
ToolGetModelConfig: "(none) — no JSON-only REST yet; httpapi.Client returns a documented stub",
ToolListBackends: "GET /backends",
ToolListKnownBackends: "GET /backends/known",
ToolSystemInfo: "GET / (welcome JSON)",
ToolListNodes: "GET /api/nodes",
ToolListScheduling: "GET /api/nodes/scheduling",
ToolGetScheduling: "GET /api/nodes/scheduling/:model",
ToolVRAMEstimate: "POST /api/models/vram-estimate",
ToolGetBranding: "GET /api/branding",
ToolGetUsageStats: "GET /api/usage (or /api/usage/all when all=true)",
ToolGetPIIEvents: "GET /api/pii/events",
ToolGetMiddlewareStatus: "GET /api/middleware/status",
ToolGetRouterDecisions: "GET /api/router/decisions",
ToolGetRouterCorpusStats: "GET /api/router/:name/corpus/stats",
ToolListAliases: "GET /api/aliases",
ToolListVoiceProfiles: "GET /api/voice-profiles",
// Mutating tools.
ToolInstallModel: "POST /models/apply",
ToolImportModelURI: "POST /models/import-uri",
ToolDeleteModel: "POST /models/delete/:name",
ToolEditModelConfig: "PATCH /api/models/config-json/:name",
ToolReloadModels: "POST /models/reload",
ToolLoadModel: "POST /backend/load",
ToolInstallBackend: "POST /backends/apply",
ToolUpgradeBackend: "POST /backends/upgrade/:name",
ToolToggleModelState: "PUT /models/toggle-state/:name/:action",
ToolToggleModelPinned: "PUT /models/toggle-pinned/:name/:action",
ToolSetBranding: "POST /api/settings (instance_name, instance_tagline)",
ToolSetAlias: "PATCH /api/models/config-json/:name (swap) or POST /models/import (create)",
ToolSeedRouterCorpus: "POST /api/router/:name/corpus",
ToolClearRouterCorpus: "DELETE /api/router/:name/corpus",
ToolCreateVoiceProfile: "POST /api/voice-profiles",
ToolDeleteVoiceProfile: "DELETE /api/voice-profiles/:id",
ToolSetNodeVRAMBudget: "PUT /api/nodes/:id/vram-budget",
ToolSetScheduling: "POST /api/nodes/scheduling",
ToolDeleteScheduling: "DELETE /api/nodes/scheduling/:model",
}
// allKnownTools is the union of expectedFullCatalog (defined in
// server_test.go). Keeping a single source of truth — the slice from
// server_test — and asserting the route map covers every entry catches
// the case "you added a Tool* but forgot to register it as MCP" indirectly
// (it'd be missing from expectedFullCatalog, which has its own assertion
// in TestServerRegistersExpectedToolCatalog).
var _ = Describe("Tool ↔ HTTP route coverage map", func() {
It("has an entry for every Tool* in the published catalog", func() {
for _, name := range expectedFullCatalog {
_, ok := toolToHTTPRoute[name]
Expect(ok).To(BeTrue(),
"Tool %q is in expectedFullCatalog but not in toolToHTTPRoute. "+
"When adding an MCP tool, update toolToHTTPRoute in coverage_test.go "+
"with the REST endpoint the httpapi.Client calls (or '(none)' with a reason).",
name)
}
})
It("does not document tools that no longer exist in the catalog", func() {
catalog := map[string]struct{}{}
for _, name := range expectedFullCatalog {
catalog[name] = struct{}{}
}
for name := range toolToHTTPRoute {
_, ok := catalog[name]
Expect(ok).To(BeTrue(),
"toolToHTTPRoute documents %q but the tool is not registered. "+
"Remove the stale entry.",
name)
}
})
// Deliberate non-test: we don't enumerate admin REST routes here. That
// would require booting Application or parsing core/http/routes/localai.go,
// both of which are brittle. The contract for "new admin REST endpoint
// → MCP tool" is enforced by the PR checklist in
// .agents/api-endpoints-and-auth.md, not by this test.
})