mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-28 17:15:02 -04:00
feat(failover): add MCP tools to list chains and pin targets
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
Ettore Di Giacinto committed
1 parent
ea55e2ffc3
commit
191d2d6301
19 files changed
+536
-20
No files matched your search
@@ -0,0 +1,137 @@
|
||||
# Task 12 report: MCP admin tools for failover chains
|
||||
|
||||
## What I implemented
|
||||
|
||||
Three MCP admin tools mirroring the Task 9 REST endpoints:
|
||||
|
||||
- `list_failover_chains` (read-only) → `GET /api/failover`
|
||||
- `pin_failover_target` (mutating) → `POST /api/failover/:chain/pin`
|
||||
- `unpin_failover_target` (mutating) → `DELETE /api/failover/:chain/pin`
|
||||
|
||||
Files changed, by layer:
|
||||
|
||||
- **Tool identity**: `pkg/mcp/localaitools/tools.go` — added `ToolListFailoverChains` (read-only block), `ToolPinFailoverTarget`/`ToolUnpinFailoverTarget` (mutating block + `mutatingToolNames`).
|
||||
- **DTOs**: `pkg/mcp/localaitools/dto.go` — added `FailoverTargetInfo` and `FailoverChainInfo` (LLM-facing subset of `failover.TargetStatus`/`failover.ChainStatus`).
|
||||
- **Interface**: `pkg/mcp/localaitools/client.go` — added `ListFailoverChains`, `PinFailoverTarget`, `UnpinFailoverTarget` to `LocalAIClient`.
|
||||
- **In-process impl**: `pkg/mcp/localaitools/inproc/client.go` — added `Failover *failover.Manager` field and the three methods (nil-safe: `ListFailoverChains` returns `[]`, Pin/Unpin return `"failover is not running"`).
|
||||
- **HTTP impl**: `pkg/mcp/localaitools/httpapi/client.go` + `httpapi/routes.go` — added `routeFailover = "/api/failover"` and the three methods, using `c.do`.
|
||||
- **Tool registration**: new file `pkg/mcp/localaitools/tools_failover.go` (`registerFailoverTools`), wired into `pkg/mcp/localaitools/server.go`.
|
||||
- **Prompts**: `prompts/20_tools.md` (read-only + mutating one-liners) and `prompts/10_safety.md` (both mutating names added to the confirmation-rule list).
|
||||
- **Wiring the manager into the in-process client**: `core/application/application.go` and `core/application/startup.go` — see "Adaptation: startup ordering" below; this was the one place I had to deviate from a literal reading of the brief.
|
||||
- **Tests**: `coverage_test.go`, `server_test.go`, `fakes_test.go`, `inproc/client_test.go`, `httpapi/client_test.go` — all as specified, plus `core/http/endpoints/mcp/localai_assistant_test.go` (see adaptations).
|
||||
|
||||
`parity_test.go` was left untouched — the brief didn't specify a new parity spec for failover, and the file's existing specs are all hand-picked equality checks for specific methods (ListGalleries, GallerySearch, ImportModelURI, SystemInfo); it isn't a generic "every method" loop, so nothing there needed updating for the build to stay green.
|
||||
|
||||
## Adaptations from the brief (read the real code, deviated where it disagreed)
|
||||
|
||||
1. **`jsonResult`/`errorResult` return one value, not three.** The brief's `tools_failover.go` snippet writes `return jsonResult(chains)` as if it were the handler's whole 3-tuple return. The real helpers in `pkg/mcp/localaitools/errors.go` are:
|
||||
```go
|
||||
func errorResult(err error) *mcp.CallToolResult
|
||||
func jsonResult(v any) *mcp.CallToolResult
|
||||
```
|
||||
Matching `tools_aliases.go`'s actual pattern, every handler returns `jsonResult(x), nil, nil` / `errorResult(err), nil, nil` — three explicit values. I used that form throughout `tools_failover.go`.
|
||||
|
||||
2. **Mutating tool descriptions reference safety rule 1.** `.agents/localai-assistant-mcp.md`'s checklist says "Mutating tools must reference safety rule 1 in the description," and `tools_aliases.go`'s `set_alias` does this ("Requires user confirmation per safety rule 1."). The brief's descriptions for `pin_failover_target`/`unpin_failover_target` didn't include this phrase, so I added it to match the established convention and the checklist.
|
||||
|
||||
3. **Startup ordering: `assistantClient.Failover` can't be set where the brief implies.** The brief says to set the field "where the client is constructed (from `application.FailoverManager()`)." I found that construction site (`core/application/application.go`'s `start()`, called from `core/application/startup.go`'s `New()` at line 74) — but `application.failoverManager` is only built later in the same `New()` function, at line 259, **after** `start()` (and therefore the assistant-client construction) has already returned. Calling `a.FailoverManager()` inside `start()` would have captured a permanent `nil`.
|
||||
|
||||
Fix: added an `assistantClient *localaiInproc.Client` field to `Application` (set in `start()` when the assistant client is built), then in `startup.go`, right after `application.failoverManager = failover.New(...)`, added:
|
||||
```go
|
||||
if application.assistantClient != nil {
|
||||
application.assistantClient.Failover = application.failoverManager
|
||||
}
|
||||
```
|
||||
This is safe because `assistantClient` is a pointer already captured by value inside the `LocalAIClient` interface passed to `holder.Initialize()` — mutating a field on it after the fact is visible through the interface. Verified with `go test ./core/application/...` and `go test ./core/http/endpoints/mcp/...`.
|
||||
|
||||
4. **`stubClient` in `core/http/endpoints/mcp/localai_assistant_test.go`** implements `localaitools.LocalAIClient` for that package's own tests and isn't in the brief's file list, but the interface change broke its build. Added the three stub methods (empty list, nil errors) to keep it compiling — same pattern as its existing stubs for `GetRouterCorpusStats` etc.
|
||||
|
||||
5. **inproc failover test fixture**: the brief says to "build a `failover.Manager` over a small in-memory source with one chain." `failover.ConfigSource` (`GetModelConfig`/`GetAllModelsConfigs`) is exported, but the concrete fake used by `core/services/failover`'s own tests (`fakeSource`, `chainCfg`, `t()`) is unexported and package-local, so I wrote a minimal `fakeFailoverSource` directly in `inproc/client_test.go` implementing the same two-method interface over a `map[string]config.ModelConfig`, seeded with a `chain` config (`config.FailoverConfig{Targets: [...]}`) plus two local targets `a`/`b`.
|
||||
|
||||
## TDD evidence
|
||||
|
||||
**RED** — after writing all test-file changes (`coverage_test.go`, `server_test.go`, `fakes_test.go`, plus new specs in `inproc/client_test.go` and `httpapi/client_test.go` were added later, see below), I temporarily reverted the implementation files (`git stash push` on `tools.go`, `dto.go`, `client.go`, `server.go`, `inproc/client.go`, `httpapi/client.go`, `httpapi/routes.go`, prompts, and the two `core/application` files; moved `tools_failover.go` out of the package) and ran:
|
||||
|
||||
```
|
||||
$ go test ./pkg/mcp/localaitools/... 2>&1 | tail -30
|
||||
# github.com/mudler/LocalAI/pkg/mcp/localaitools [github.com/mudler/LocalAI/pkg/mcp/localaitools.test]
|
||||
pkg/mcp/localaitools/fakes_test.go:62:32: undefined: FailoverChainInfo
|
||||
pkg/mcp/localaitools/fakes_test.go:400:63: undefined: FailoverChainInfo
|
||||
pkg/mcp/localaitools/fakes_test.go:405:11: undefined: FailoverChainInfo
|
||||
pkg/mcp/localaitools/coverage_test.go:49:2: undefined: ToolListFailoverChains
|
||||
pkg/mcp/localaitools/coverage_test.go:71:2: undefined: ToolPinFailoverTarget
|
||||
pkg/mcp/localaitools/coverage_test.go:72:2: undefined: ToolUnpinFailoverTarget
|
||||
pkg/mcp/localaitools/server_test.go:95:2: undefined: ToolListFailoverChains
|
||||
pkg/mcp/localaitools/server_test.go:159:4: undefined: ToolListFailoverChains
|
||||
pkg/mcp/localaitools/server_test.go:160:4: undefined: ToolPinFailoverTarget
|
||||
pkg/mcp/localaitools/server_test.go:161:4: undefined: ToolUnpinFailoverTarget
|
||||
pkg/mcp/localaitools/server_test.go:161:4: too many errors
|
||||
FAIL github.com/mudler/LocalAI/pkg/mcp/localaitools [build failed]
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/httpapi 0.035s
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/inproc 0.167s
|
||||
```
|
||||
This matches the brief's Step 1 expectation ("Expected: compile failure"). I then `git stash pop` and restored `tools_failover.go` to get back to the implemented state.
|
||||
|
||||
**GREEN** — after restoring the implementation and adding the remaining `inproc/client_test.go` / `httpapi/client_test.go` specs:
|
||||
|
||||
```
|
||||
$ go test ./pkg/mcp/localaitools/... -count=1 -v 2>&1 | grep -E "SUCCESS|FAIL|ok "
|
||||
SUCCESS! -- 53 Passed | 0 Failed | 0 Pending | 0 Skipped
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools 0.133s
|
||||
SUCCESS! -- 24 Passed | 0 Failed | 0 Pending | 0 Skipped
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/httpapi 0.037s
|
||||
SUCCESS! -- 16 Passed | 0 Failed | 0 Pending | 0 Skipped
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/inproc 0.171s
|
||||
```
|
||||
|
||||
Additional verification (build scope per implementer-rules, plus the two packages touched indirectly by the interface change):
|
||||
|
||||
```
|
||||
$ go build ./core/... ./pkg/mcp/... ./tests/...
|
||||
(clean, no output)
|
||||
|
||||
$ go test ./pkg/mcp/localaitools/... ./core/application/... ./core/http/endpoints/mcp/... -count=1
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools 0.155s
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/httpapi 0.045s
|
||||
ok github.com/mudler/LocalAI/pkg/mcp/localaitools/inproc 0.176s
|
||||
ok github.com/mudler/LocalAI/core/application 0.265s
|
||||
ok github.com/mudler/LocalAI/core/http/endpoints/mcp 0.237s
|
||||
|
||||
$ gofmt -l <all touched .go files>
|
||||
(empty — clean)
|
||||
|
||||
$ go vet ./core/application/... ./pkg/mcp/...
|
||||
(clean, no output)
|
||||
```
|
||||
|
||||
## Files changed
|
||||
|
||||
- `pkg/mcp/localaitools/tools.go`
|
||||
- `pkg/mcp/localaitools/dto.go`
|
||||
- `pkg/mcp/localaitools/client.go`
|
||||
- `pkg/mcp/localaitools/server.go`
|
||||
- `pkg/mcp/localaitools/tools_failover.go` (new)
|
||||
- `pkg/mcp/localaitools/inproc/client.go`
|
||||
- `pkg/mcp/localaitools/httpapi/client.go`
|
||||
- `pkg/mcp/localaitools/httpapi/routes.go`
|
||||
- `pkg/mcp/localaitools/prompts/20_tools.md`
|
||||
- `pkg/mcp/localaitools/prompts/10_safety.md`
|
||||
- `pkg/mcp/localaitools/coverage_test.go`
|
||||
- `pkg/mcp/localaitools/server_test.go`
|
||||
- `pkg/mcp/localaitools/fakes_test.go`
|
||||
- `pkg/mcp/localaitools/inproc/client_test.go`
|
||||
- `pkg/mcp/localaitools/httpapi/client_test.go`
|
||||
- `core/application/application.go`
|
||||
- `core/application/startup.go`
|
||||
- `core/http/endpoints/mcp/localai_assistant_test.go`
|
||||
|
||||
## Self-review
|
||||
|
||||
- Completeness: all three tools registered, gated correctly (`ToolListFailoverChains` in the read-only catalog; both mutating tools skipped when `Options.DisableMutating`), both client implementations covered, prompts updated, safety-rule coverage test (`TestPromptsContainSafetyAnchors`'s "names every mutating tool" spec) passes automatically since it reads `mutatingToolNames`.
|
||||
- Quality/YAGNI: DTOs intentionally drop `ConsecutiveOK`/`LastProbe`/`ActiveSince` — internal probe bookkeeping the LLM has no use for when deciding to pin/unpin; documented why in the doc comment.
|
||||
- Nil-safety: both inproc failover methods and the httpapi Pin/Unpin exercise the "no failover configured" path in tests (inproc has explicit specs for it; httpapi's behavior when unconfigured is identical to any other client error — REST returns 404, `c.do` surfaces `*HTTPError`, no new code path needed there).
|
||||
- Existing patterns followed: constant grouping/comments, `errorResult`/`jsonResult` triple-return, `c.do` signature, `url.PathEscape` on the chain path segment, fake-client recording pattern, Ginkgo `Describe`/`It` structure matching the alias specs.
|
||||
- Pristine output: `gofmt -l` and `go vet` clean across all touched files.
|
||||
|
||||
## Concerns
|
||||
|
||||
- None blocking. The startup-ordering fix (adaptation 3) is the only piece that goes beyond a single-file, mechanical change — it touches two `core/application` files instead of the "one field set inline" the brief describes. I verified it with both `core/application` and `core/http/endpoints/mcp` package tests, and confirmed via read of `startup.go` that `New()` is the sole caller of `start()` and that `failoverManager` is not read anywhere between `start()` and its own assignment, so there's no other place relying on it being nil momentarily.
|
||||
@@ -101,6 +101,13 @@ type Application struct {
|
||||
// is set; otherwise initialised in start() after galleryService.
|
||||
localAIAssistant *mcpTools.LocalAIAssistantHolder
|
||||
|
||||
// assistantClient is the concrete inproc client backing localAIAssistant.
|
||||
// start() constructs it before failoverManager exists (see New() in
|
||||
// startup.go), so New() sets assistantClient.Failover once the manager
|
||||
// is built, using this field to reach back into the already-registered
|
||||
// MCP tool set. nil when DisableLocalAIAssistant is set.
|
||||
assistantClient *localaiInproc.Client
|
||||
|
||||
// startupComplete flips to true once New() has finished its whole startup
|
||||
// sequence. It backs the /readyz probe.
|
||||
//
|
||||
@@ -598,6 +605,12 @@ func (a *Application) start() error {
|
||||
assistantClient.RouterEmbedder = a.Embedder
|
||||
assistantClient.RouterEmbedderFingerprint = a.EmbedderFingerprint
|
||||
assistantClient.RouterVectorStore = a.VectorStore
|
||||
// Failover chains: failoverManager does not exist yet at this point
|
||||
// in startup (New() in startup.go builds it after start() returns),
|
||||
// so it can't be wired here like the fields above. New() sets
|
||||
// assistantClient.Failover directly once the manager is built;
|
||||
// stash the client so it can reach back into it.
|
||||
a.assistantClient = assistantClient
|
||||
if err := holder.Initialize(a.applicationConfig.Context, assistantClient, localaitools.Options{}); err != nil {
|
||||
// Why log+continue instead of fail: the assistant is an optional
|
||||
// feature; a failure here must not take down the whole server.
|
||||
|
||||
@@ -262,6 +262,12 @@ func New(opts ...config.AppOption) (*Application, error) {
|
||||
})),
|
||||
failover.WithOnWarmChanged(application.applyFailoverWarmTargets),
|
||||
)
|
||||
// The assistant client was built in start() (above), before this
|
||||
// manager existed; wire it now so list_failover_chains /
|
||||
// pin_failover_target / unpin_failover_target see real chains.
|
||||
if application.assistantClient != nil {
|
||||
application.assistantClient.Failover = application.failoverManager
|
||||
}
|
||||
|
||||
// Subsystem 5: admission control. Limiter is always wired so a
|
||||
// model that gains a limits: block via gallery install or YAML
|
||||
|
||||
@@ -214,3 +214,11 @@ func (stubClient) SeedRouterCorpus(_ context.Context, req localaitools.RouterCor
|
||||
func (stubClient) ClearRouterCorpus(_ context.Context, routerModel string) (*localaitools.RouterCorpusClearResult, error) {
|
||||
return &localaitools.RouterCorpusClearResult{Router: routerModel}, nil
|
||||
}
|
||||
|
||||
func (stubClient) ListFailoverChains(_ context.Context) ([]localaitools.FailoverChainInfo, error) {
|
||||
return []localaitools.FailoverChainInfo{}, nil
|
||||
}
|
||||
|
||||
func (stubClient) PinFailoverTarget(_ context.Context, _, _ string) error { return nil }
|
||||
|
||||
func (stubClient) UnpinFailoverTarget(_ context.Context, _ string) error { return nil }
|
||||
@@ -130,4 +130,15 @@ type LocalAIClient interface {
|
||||
// ClearRouterCorpus wipes a knn router's corpus — file and live
|
||||
// index.
|
||||
ClearRouterCorpus(ctx context.Context, routerModel string) (*RouterCorpusClearResult, error)
|
||||
|
||||
// ---- Failover chains ----
|
||||
// ListFailoverChains reports every configured failover chain, its
|
||||
// currently active target, and the health of each target.
|
||||
ListFailoverChains(ctx context.Context) ([]FailoverChainInfo, error)
|
||||
// PinFailoverTarget forces chain to serve every request from target,
|
||||
// regardless of health, until unpinned.
|
||||
PinFailoverTarget(ctx context.Context, chain, target string) error
|
||||
// UnpinFailoverTarget removes chain's pin so health decides the
|
||||
// active target again.
|
||||
UnpinFailoverTarget(ctx context.Context, chain string) error
|
||||
}
|
||||
@@ -46,27 +46,30 @@ var toolToHTTPRoute = map[string]string{
|
||||
ToolGetRouterCorpusStats: "GET /api/router/:name/corpus/stats",
|
||||
ToolListAliases: "GET /api/aliases",
|
||||
ToolListVoiceProfiles: "GET /api/voice-profiles",
|
||||
ToolListFailoverChains: "GET /api/failover",
|
||||
|
||||
// Mutating tools.
|
||||
ToolInstallModel: "POST /models/apply",
|
||||
ToolImportModelURI: "POST /models/import-uri",
|
||||
ToolDeleteModel: "POST /models/delete/:name",
|
||||
ToolEditModelConfig: "PATCH /api/models/config-json/:name",
|
||||
ToolReloadModels: "POST /models/reload",
|
||||
ToolLoadModel: "POST /backend/load",
|
||||
ToolInstallBackend: "POST /backends/apply",
|
||||
ToolUpgradeBackend: "POST /backends/upgrade/:name",
|
||||
ToolToggleModelState: "PUT /models/toggle-state/:name/:action",
|
||||
ToolToggleModelPinned: "PUT /models/toggle-pinned/:name/:action",
|
||||
ToolSetBranding: "POST /api/settings (instance_name, instance_tagline)",
|
||||
ToolSetAlias: "PATCH /api/models/config-json/:name (swap) or POST /models/import (create)",
|
||||
ToolSeedRouterCorpus: "POST /api/router/:name/corpus",
|
||||
ToolClearRouterCorpus: "DELETE /api/router/:name/corpus",
|
||||
ToolCreateVoiceProfile: "POST /api/voice-profiles",
|
||||
ToolDeleteVoiceProfile: "DELETE /api/voice-profiles/:id",
|
||||
ToolSetNodeVRAMBudget: "PUT /api/nodes/:id/vram-budget",
|
||||
ToolSetScheduling: "POST /api/nodes/scheduling",
|
||||
ToolDeleteScheduling: "DELETE /api/nodes/scheduling/:model",
|
||||
ToolInstallModel: "POST /models/apply",
|
||||
ToolImportModelURI: "POST /models/import-uri",
|
||||
ToolDeleteModel: "POST /models/delete/:name",
|
||||
ToolEditModelConfig: "PATCH /api/models/config-json/:name",
|
||||
ToolReloadModels: "POST /models/reload",
|
||||
ToolLoadModel: "POST /backend/load",
|
||||
ToolInstallBackend: "POST /backends/apply",
|
||||
ToolUpgradeBackend: "POST /backends/upgrade/:name",
|
||||
ToolToggleModelState: "PUT /models/toggle-state/:name/:action",
|
||||
ToolToggleModelPinned: "PUT /models/toggle-pinned/:name/:action",
|
||||
ToolSetBranding: "POST /api/settings (instance_name, instance_tagline)",
|
||||
ToolSetAlias: "PATCH /api/models/config-json/:name (swap) or POST /models/import (create)",
|
||||
ToolSeedRouterCorpus: "POST /api/router/:name/corpus",
|
||||
ToolClearRouterCorpus: "DELETE /api/router/:name/corpus",
|
||||
ToolCreateVoiceProfile: "POST /api/voice-profiles",
|
||||
ToolDeleteVoiceProfile: "DELETE /api/voice-profiles/:id",
|
||||
ToolSetNodeVRAMBudget: "PUT /api/nodes/:id/vram-budget",
|
||||
ToolSetScheduling: "POST /api/nodes/scheduling",
|
||||
ToolDeleteScheduling: "DELETE /api/nodes/scheduling/:model",
|
||||
ToolPinFailoverTarget: "POST /api/failover/:chain/pin",
|
||||
ToolUnpinFailoverTarget: "DELETE /api/failover/:chain/pin",
|
||||
}
|
||||
|
||||
// allKnownTools is the union of expectedFullCatalog (defined in
|
||||
|
||||
@@ -413,3 +413,26 @@ type VRAMEstimateRequest struct {
|
||||
GPULayers int `json:"gpu_layers,omitempty" jsonschema:"Number of layers to offload to GPU. -1 for all."`
|
||||
KVQuantBits int `json:"kv_quant_bits,omitempty" jsonschema:"KV cache quantization bits (e.g. 4, 8, 16)."`
|
||||
}
|
||||
|
||||
// FailoverTargetInfo is the LLM-facing view of one failover chain target's
|
||||
// health. It mirrors failover.TargetStatus but drops ConsecutiveOK and
|
||||
// LastProbe — internal probing detail the LLM doesn't need to decide
|
||||
// whether to pin or unpin a target.
|
||||
type FailoverTargetInfo struct {
|
||||
Model string `json:"model"`
|
||||
Kind string `json:"kind"`
|
||||
Warm bool `json:"warm"`
|
||||
State string `json:"state"`
|
||||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// FailoverChainInfo is the LLM-facing view of one failover chain: its
|
||||
// current state, the target serving it now, an optional pin, and every
|
||||
// target's health.
|
||||
type FailoverChainInfo struct {
|
||||
Name string `json:"name"`
|
||||
State string `json:"state"`
|
||||
Active string `json:"active"`
|
||||
Pinned string `json:"pinned,omitempty"`
|
||||
Targets []FailoverTargetInfo `json:"targets"`
|
||||
}
|
||||
@@ -59,6 +59,9 @@ type fakeClient struct {
|
||||
getPIIEvents func(PIIEventsQuery) ([]PIIEvent, error)
|
||||
getMiddlewareStatus func() (*MiddlewareStatus, error)
|
||||
getRouterDecisions func(RouterDecisionsQuery) ([]RouterDecision, error)
|
||||
listFailoverChains func() ([]FailoverChainInfo, error)
|
||||
pinFailoverTarget func(string, string) error
|
||||
unpinFailoverTarget func(string) error
|
||||
}
|
||||
|
||||
type fakeCall struct {
|
||||
@@ -393,3 +396,27 @@ func (f *fakeClient) ClearRouterCorpus(_ context.Context, routerModel string) (*
|
||||
f.record("ClearRouterCorpus", routerModel)
|
||||
return &RouterCorpusClearResult{Router: routerModel}, nil
|
||||
}
|
||||
|
||||
func (f *fakeClient) ListFailoverChains(_ context.Context) ([]FailoverChainInfo, error) {
|
||||
f.record("ListFailoverChains", nil)
|
||||
if f.listFailoverChains != nil {
|
||||
return f.listFailoverChains()
|
||||
}
|
||||
return []FailoverChainInfo{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeClient) PinFailoverTarget(_ context.Context, chain, target string) error {
|
||||
f.record("PinFailoverTarget", []any{chain, target})
|
||||
if f.pinFailoverTarget != nil {
|
||||
return f.pinFailoverTarget(chain, target)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeClient) UnpinFailoverTarget(_ context.Context, chain string) error {
|
||||
f.record("UnpinFailoverTarget", chain)
|
||||
if f.unpinFailoverTarget != nil {
|
||||
return f.unpinFailoverTarget(chain)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -829,3 +829,23 @@ func (c *Client) ClearRouterCorpus(ctx context.Context, routerModel string) (*lo
|
||||
}
|
||||
return &out, nil
|
||||
}
|
||||
|
||||
// ---- Failover chains ----
|
||||
|
||||
func (c *Client) ListFailoverChains(ctx context.Context) ([]localaitools.FailoverChainInfo, error) {
|
||||
var out struct {
|
||||
Chains []localaitools.FailoverChainInfo `json:"chains"`
|
||||
}
|
||||
if err := c.do(ctx, http.MethodGet, routeFailover, nil, &out); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out.Chains, nil
|
||||
}
|
||||
|
||||
func (c *Client) PinFailoverTarget(ctx context.Context, chain, target string) error {
|
||||
return c.do(ctx, http.MethodPost, routeFailover+"/"+url.PathEscape(chain)+"/pin", map[string]string{"target": target}, nil)
|
||||
}
|
||||
|
||||
func (c *Client) UnpinFailoverTarget(ctx context.Context, chain string) error {
|
||||
return c.do(ctx, http.MethodDelete, routeFailover+"/"+url.PathEscape(chain)+"/pin", nil, nil)
|
||||
}
|
||||
@@ -365,6 +365,68 @@ var _ = Describe("Model aliases", func() {
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("Failover chains", func() {
|
||||
Describe("ListFailoverChains", func() {
|
||||
It("issues GET /api/failover and unwraps the chains array", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
Expect(r.Method).To(Equal(http.MethodGet))
|
||||
Expect(r.URL.Path).To(Equal("/api/failover"))
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"chains": []map[string]any{
|
||||
{
|
||||
"name": "chain",
|
||||
"state": "primary",
|
||||
"active": "a",
|
||||
"pinned": nil,
|
||||
"targets": []map[string]any{
|
||||
{"model": "a", "kind": "local", "warm": false, "state": "healthy"},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
}))
|
||||
DeferCleanup(srv.Close)
|
||||
|
||||
out, err := New(srv.URL, "").ListFailoverChains(context.Background())
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out).To(HaveLen(1))
|
||||
Expect(out[0].Name).To(Equal("chain"))
|
||||
Expect(out[0].Active).To(Equal("a"))
|
||||
Expect(out[0].Pinned).To(BeEmpty())
|
||||
Expect(out[0].Targets).To(ConsistOf(localaitools.FailoverTargetInfo{Model: "a", Kind: "local", State: "healthy"}))
|
||||
})
|
||||
})
|
||||
|
||||
Describe("PinFailoverTarget", func() {
|
||||
It("issues POST /api/failover/:chain/pin with the target in the body", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
Expect(r.Method).To(Equal(http.MethodPost))
|
||||
Expect(r.URL.Path).To(Equal("/api/failover/chain/pin"))
|
||||
var body map[string]string
|
||||
Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed())
|
||||
Expect(body).To(HaveKeyWithValue("target", "b"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
DeferCleanup(srv.Close)
|
||||
|
||||
Expect(New(srv.URL, "").PinFailoverTarget(context.Background(), "chain", "b")).To(Succeed())
|
||||
})
|
||||
})
|
||||
|
||||
Describe("UnpinFailoverTarget", func() {
|
||||
It("issues DELETE /api/failover/:chain/pin", func() {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
Expect(r.Method).To(Equal(http.MethodDelete))
|
||||
Expect(r.URL.Path).To(Equal("/api/failover/chain/pin"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
DeferCleanup(srv.Close)
|
||||
|
||||
Expect(New(srv.URL, "").UnpinFailoverTarget(context.Background(), "chain")).To(Succeed())
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("ErrHTTPNotFound", func() {
|
||||
Context("on a clean 404 status", func() {
|
||||
var (
|
||||
|
||||
@@ -34,6 +34,7 @@ const (
|
||||
routeMiddleware = "/api/middleware/status"
|
||||
routeRouterDecisions = "/api/router/decisions"
|
||||
routeVoiceProfiles = "/api/voice-profiles"
|
||||
routeFailover = "/api/failover"
|
||||
)
|
||||
|
||||
func routeJobStatus(jobID string) string {
|
||||
|
||||
@@ -21,6 +21,7 @@ import (
|
||||
"github.com/mudler/LocalAI/core/gallery/importers"
|
||||
"github.com/mudler/LocalAI/core/http/auth"
|
||||
"github.com/mudler/LocalAI/core/schema"
|
||||
"github.com/mudler/LocalAI/core/services/failover"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/core/services/modeladmin"
|
||||
"github.com/mudler/LocalAI/core/services/nodes"
|
||||
@@ -83,6 +84,12 @@ type Client struct {
|
||||
RouterEmbedderFingerprint func(modelName string) (string, error)
|
||||
RouterVectorStore func(storeName string) backend.VectorStore
|
||||
|
||||
// Failover backs list_failover_chains / pin_failover_target /
|
||||
// unpin_failover_target. nil makes the tools report "failover is not
|
||||
// running" — the same as a deployment with no failover chains
|
||||
// configured.
|
||||
Failover *failover.Manager
|
||||
|
||||
modelAdmin *modeladmin.ConfigService
|
||||
}
|
||||
|
||||
@@ -1103,3 +1110,39 @@ func (c *Client) ClearRouterCorpus(ctx context.Context, routerModel string) (*lo
|
||||
}
|
||||
return &localaitools.RouterCorpusClearResult{Router: cfg.Name, Cleared: cleared}, nil
|
||||
}
|
||||
|
||||
// ---- Failover chains ----
|
||||
|
||||
func (c *Client) ListFailoverChains(_ context.Context) ([]localaitools.FailoverChainInfo, error) {
|
||||
out := []localaitools.FailoverChainInfo{}
|
||||
if c.Failover == nil {
|
||||
return out, nil
|
||||
}
|
||||
for _, ch := range c.Failover.Status() {
|
||||
info := localaitools.FailoverChainInfo{Name: ch.Name, State: string(ch.State), Active: ch.Active}
|
||||
if ch.Pinned != nil {
|
||||
info.Pinned = *ch.Pinned
|
||||
}
|
||||
for _, t := range ch.Targets {
|
||||
info.Targets = append(info.Targets, localaitools.FailoverTargetInfo{
|
||||
Model: t.Model, Kind: string(t.Kind), Warm: t.Warm, State: string(t.State), LastError: t.LastError,
|
||||
})
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *Client) PinFailoverTarget(_ context.Context, chain, target string) error {
|
||||
if c.Failover == nil {
|
||||
return errors.New("failover is not running")
|
||||
}
|
||||
return c.Failover.Pin(chain, target)
|
||||
}
|
||||
|
||||
func (c *Client) UnpinFailoverTarget(_ context.Context, chain string) error {
|
||||
if c.Failover == nil {
|
||||
return errors.New("failover is not running")
|
||||
}
|
||||
return c.Failover.Unpin(chain)
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
|
||||
"github.com/mudler/LocalAI/core/config"
|
||||
"github.com/mudler/LocalAI/core/gallery"
|
||||
"github.com/mudler/LocalAI/core/services/failover"
|
||||
"github.com/mudler/LocalAI/core/services/galleryop"
|
||||
"github.com/mudler/LocalAI/core/services/nodes"
|
||||
localaitools "github.com/mudler/LocalAI/pkg/mcp/localaitools"
|
||||
@@ -129,6 +130,95 @@ var _ = Describe("inproc.Client model aliases", func() {
|
||||
})
|
||||
})
|
||||
|
||||
// fakeFailoverSource is a minimal failover.ConfigSource over an in-memory
|
||||
// map, so these specs don't need a real ModelConfigLoader + on-disk YAML.
|
||||
type fakeFailoverSource struct {
|
||||
cfgs map[string]config.ModelConfig
|
||||
}
|
||||
|
||||
func (s *fakeFailoverSource) GetModelConfig(name string) (config.ModelConfig, bool) {
|
||||
c, ok := s.cfgs[name]
|
||||
return c, ok
|
||||
}
|
||||
|
||||
func (s *fakeFailoverSource) GetAllModelsConfigs() []config.ModelConfig {
|
||||
out := make([]config.ModelConfig, 0, len(s.cfgs))
|
||||
for _, c := range s.cfgs {
|
||||
out = append(out, c)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var _ = Describe("inproc.Client failover chains", func() {
|
||||
var (
|
||||
ctx context.Context
|
||||
c *Client
|
||||
fm *failover.Manager
|
||||
)
|
||||
|
||||
BeforeEach(func() {
|
||||
ctx = context.Background()
|
||||
src := &fakeFailoverSource{cfgs: map[string]config.ModelConfig{
|
||||
"a": {Name: "a", Backend: "llama-cpp"},
|
||||
"b": {Name: "b", Backend: "llama-cpp"},
|
||||
"chain": {Name: "chain", Failover: &config.FailoverConfig{
|
||||
Targets: []config.FailoverTarget{{Model: "a"}, {Model: "b"}},
|
||||
}},
|
||||
}}
|
||||
fm = failover.New(src)
|
||||
c = &Client{Failover: fm}
|
||||
})
|
||||
|
||||
It("ListFailoverChains reports the chain, its active target, and target health", func() {
|
||||
out, err := c.ListFailoverChains(ctx)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out).To(HaveLen(1))
|
||||
Expect(out[0].Name).To(Equal("chain"))
|
||||
Expect(out[0].Active).To(Equal("a"))
|
||||
Expect(out[0].Pinned).To(BeEmpty())
|
||||
Expect(out[0].Targets).To(HaveLen(2))
|
||||
Expect(out[0].Targets[0].Model).To(Equal("a"))
|
||||
Expect(out[0].Targets[0].Kind).To(Equal("local"))
|
||||
Expect(out[0].Targets[0].State).To(Equal("healthy"))
|
||||
})
|
||||
|
||||
It("returns an empty slice, not an error, when no failover manager is wired", func() {
|
||||
c = &Client{}
|
||||
out, err := c.ListFailoverChains(ctx)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("PinFailoverTarget pins the chain and ListFailoverChains reflects it", func() {
|
||||
Expect(c.PinFailoverTarget(ctx, "chain", "b")).To(Succeed())
|
||||
|
||||
out, err := c.ListFailoverChains(ctx)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out[0].Pinned).To(Equal("b"))
|
||||
})
|
||||
|
||||
It("PinFailoverTarget errors when the failover manager is unavailable", func() {
|
||||
c = &Client{}
|
||||
err := c.PinFailoverTarget(ctx, "chain", "b")
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
|
||||
It("UnpinFailoverTarget clears a pin", func() {
|
||||
Expect(c.PinFailoverTarget(ctx, "chain", "b")).To(Succeed())
|
||||
Expect(c.UnpinFailoverTarget(ctx, "chain")).To(Succeed())
|
||||
|
||||
out, err := c.ListFailoverChains(ctx)
|
||||
Expect(err).ToNot(HaveOccurred())
|
||||
Expect(out[0].Pinned).To(BeEmpty())
|
||||
})
|
||||
|
||||
It("UnpinFailoverTarget errors when the failover manager is unavailable", func() {
|
||||
c = &Client{}
|
||||
err := c.UnpinFailoverTarget(ctx, "chain")
|
||||
Expect(err).To(HaveOccurred())
|
||||
})
|
||||
})
|
||||
|
||||
var _ = Describe("inproc.Client model scheduling", func() {
|
||||
var (
|
||||
ctx context.Context
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
These rules are non-negotiable. The user trusts you to operate their server without unintended changes.
|
||||
|
||||
1. **Confirm before mutating.** Before calling any of these tools — `install_model`, `import_model_uri`, `delete_model`, `install_backend`, `upgrade_backend`, `edit_model_config`, `reload_models`, `load_model`, `toggle_model_state`, `toggle_model_pinned`, `set_branding`, `set_alias`, `seed_router_corpus`, `clear_router_corpus`, `create_voice_profile`, `delete_voice_profile`, `set_node_vram_budget`, `set_scheduling`, `delete_scheduling` — first state in plain language what you are about to do (which tool, which target, which arguments) and wait for the user's explicit confirmation in the next turn. "Yes", "do it", "go ahead", "proceed" all count as confirmation. Anything else does not.
|
||||
1. **Confirm before mutating.** Before calling any of these tools — `install_model`, `import_model_uri`, `delete_model`, `install_backend`, `upgrade_backend`, `edit_model_config`, `reload_models`, `load_model`, `toggle_model_state`, `toggle_model_pinned`, `set_branding`, `set_alias`, `seed_router_corpus`, `clear_router_corpus`, `create_voice_profile`, `delete_voice_profile`, `set_node_vram_budget`, `set_scheduling`, `delete_scheduling`, `pin_failover_target`, `unpin_failover_target` — first state in plain language what you are about to do (which tool, which target, which arguments) and wait for the user's explicit confirmation in the next turn. "Yes", "do it", "go ahead", "proceed" all count as confirmation. Anything else does not.
|
||||
|
||||
2. **Disambiguate before mutating.** If the user's request is ambiguous (several gallery candidates match, the model name has multiple installed versions, the backend has variants), present the candidates as a numbered list and ask the user to pick before calling any mutating tool.
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@ The MCP `tools/list` endpoint also exposes the full input schema for each of the
|
||||
- `get_router_decisions` — Inspect recent router decisions and classifier signals.
|
||||
- `get_router_corpus_stats` — Inspect a KNN router corpus by count and label only; exemplar texts are never returned.
|
||||
- `list_aliases` — List configured model aliases and their targets.
|
||||
- `list_failover_chains` — List failover chains, their active target and target health.
|
||||
|
||||
## Mutating (require user confirmation per safety rule 1)
|
||||
|
||||
@@ -46,3 +47,5 @@ The MCP `tools/list` endpoint also exposes the full input schema for each of the
|
||||
- `set_node_vram_budget` — Set or clear a federated node's VRAM budget override.
|
||||
- `set_scheduling` — Create or update a distributed per-model scheduling config.
|
||||
- `delete_scheduling` — Remove a distributed per-model scheduling config.
|
||||
- `pin_failover_target` — Force a failover chain to one target.
|
||||
- `unpin_failover_target` — Remove a failover pin.
|
||||
@@ -54,6 +54,7 @@ func NewServer(client LocalAIClient, opts Options) *mcp.Server {
|
||||
registerUsageTools(srv, client, opts)
|
||||
registerPIITools(srv, client, opts)
|
||||
registerMiddlewareTools(srv, client, opts)
|
||||
registerFailoverTools(srv, client, opts)
|
||||
|
||||
return srv
|
||||
}
|
||||
@@ -92,6 +92,7 @@ var expectedReadOnlyCatalog = sortedStrings(
|
||||
ToolListVoiceProfiles,
|
||||
ToolSystemInfo,
|
||||
ToolVRAMEstimate,
|
||||
ToolListFailoverChains,
|
||||
)
|
||||
|
||||
// expectedFullCatalog derives from the read-only catalog plus the canonical
|
||||
@@ -155,6 +156,9 @@ var _ = Describe("Tool dispatch", func() {
|
||||
{ToolListAliases, struct{}{}, "ListAliases"},
|
||||
{ToolCreateVoiceProfile, CreateVoiceProfileRequest{Name: "Narrator", Transcript: "Reference words", AudioBase64: "UklGRg==", ConsentConfirmed: true}, "CreateVoiceProfile"},
|
||||
{ToolDeleteVoiceProfile, DeleteVoiceProfileRequest{ID: "00000000-0000-0000-0000-000000000001"}, "DeleteVoiceProfile"},
|
||||
{ToolListFailoverChains, map[string]any{}, "ListFailoverChains"},
|
||||
{ToolPinFailoverTarget, map[string]any{"chain": "c", "target": "b"}, "PinFailoverTarget"},
|
||||
{ToolUnpinFailoverTarget, map[string]any{"chain": "c"}, "UnpinFailoverTarget"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
|
||||
@@ -49,10 +49,19 @@ const (
|
||||
ToolSetNodeVRAMBudget = "set_node_vram_budget"
|
||||
ToolSetScheduling = "set_scheduling"
|
||||
ToolDeleteScheduling = "delete_scheduling"
|
||||
// ToolPinFailoverTarget and ToolUnpinFailoverTarget live here (rather
|
||||
// than grouped with ToolListFailoverChains below) so mutatingToolNames
|
||||
// stays a contiguous scan of this block.
|
||||
ToolPinFailoverTarget = "pin_failover_target"
|
||||
ToolUnpinFailoverTarget = "unpin_failover_target"
|
||||
|
||||
// ToolListAliases is read-only but lives here so the alias tools stay
|
||||
// grouped; the catalog tests assert its read-only placement.
|
||||
ToolListAliases = "list_aliases"
|
||||
|
||||
// ToolListFailoverChains is read-only but lives here so the failover
|
||||
// tools stay grouped; the catalog tests assert its read-only placement.
|
||||
ToolListFailoverChains = "list_failover_chains"
|
||||
)
|
||||
|
||||
// DefaultServerName is the MCP Implementation.Name surfaced when
|
||||
@@ -83,4 +92,6 @@ var mutatingToolNames = []string{
|
||||
ToolSetNodeVRAMBudget,
|
||||
ToolSetScheduling,
|
||||
ToolDeleteScheduling,
|
||||
ToolPinFailoverTarget,
|
||||
ToolUnpinFailoverTarget,
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package localaitools
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
// registerFailoverTools wires the conversational failover-chain tools.
|
||||
// list_failover_chains reports the health of every chain, pin_failover_target
|
||||
// forces a chain to one target, and unpin_failover_target hands control back
|
||||
// to health-based selection.
|
||||
func registerFailoverTools(s *mcp.Server, client LocalAIClient, opts Options) {
|
||||
mcp.AddTool(s, &mcp.Tool{
|
||||
Name: ToolListFailoverChains,
|
||||
Description: "List model failover chains, the target serving each one now, and the health of every target.",
|
||||
}, func(ctx context.Context, _ *mcp.CallToolRequest, _ struct{}) (*mcp.CallToolResult, any, error) {
|
||||
chains, err := client.ListFailoverChains(ctx)
|
||||
if err != nil {
|
||||
return errorResult(err), nil, nil
|
||||
}
|
||||
return jsonResult(chains), nil, nil
|
||||
})
|
||||
|
||||
if opts.DisableMutating {
|
||||
return
|
||||
}
|
||||
|
||||
mcp.AddTool(s, &mcp.Tool{
|
||||
Name: ToolPinFailoverTarget,
|
||||
Description: "Force a failover chain to serve every request from one target, regardless of health, until it is unpinned. Requires user confirmation per safety rule 1.",
|
||||
}, func(ctx context.Context, _ *mcp.CallToolRequest, args struct {
|
||||
Chain string `json:"chain" jsonschema:"failover chain name"`
|
||||
Target string `json:"target" jsonschema:"target model to pin"`
|
||||
}) (*mcp.CallToolResult, any, error) {
|
||||
if err := client.PinFailoverTarget(ctx, args.Chain, args.Target); err != nil {
|
||||
return errorResult(err), nil, nil
|
||||
}
|
||||
return jsonResult(map[string]string{"chain": args.Chain, "pinned": args.Target}), nil, nil
|
||||
})
|
||||
|
||||
mcp.AddTool(s, &mcp.Tool{
|
||||
Name: ToolUnpinFailoverTarget,
|
||||
Description: "Remove the pin from a failover chain so health decides the target again. Requires user confirmation per safety rule 1.",
|
||||
}, func(ctx context.Context, _ *mcp.CallToolRequest, args struct {
|
||||
Chain string `json:"chain" jsonschema:"failover chain name"`
|
||||
}) (*mcp.CallToolResult, any, error) {
|
||||
if err := client.UnpinFailoverTarget(ctx, args.Chain); err != nil {
|
||||
return errorResult(err), nil, nil
|
||||
}
|
||||
return jsonResult(map[string]string{"chain": args.Chain, "pinned": ""}), nil, nil
|
||||
})
|
||||
}
|
||||
Reference in new issue
Block a user