From b56b903c9283ec45665c57aeb67d9993454f8fdf Mon Sep 17 00:00:00 2001 From: Stefan Walcz Date: Mon, 19 Oct 2026 23:42:40 +0200 Subject: [PATCH] feat(config): let model aliases carry an opt-in profile block (system prompt, MCP, client tools) (#12661) * feat(config): let model aliases carry an opt-in profile block An alias can now act as a profile of its target through an explicit `profile:` block carrying system_prompt, mcp (remote/stdio servers, plus the new mcp.auto) and the new client_tools. ResolveAlias applies only that block on top of a copy of the target, while Name, Model, Backend and the persisted revision stay the target's. The loader key is therefore unchanged and every profile runs on the target's backend process; today a persona needs a separate model config, which loads the weights a second time. The block is opt-in so existing aliases never change: top-level fields on an alias stay ignored, and an alias without a profile block resolves to exactly the target. A profile block on a non-alias config is rejected. A profile with its own MCP servers caches its sessions under the alias name (MCPSessionKey), because the cache slot keyed on the target's name is invalidated whenever its config hash changes and two server sets would tear each other's sessions down. mcp.auto enables all configured servers on /v1/chat/completions when the request does not name any in metadata.mcp_servers, which plain OpenAI clients cannot send. client_tools: drop discards client tools and tool_choice before the request is merged (chat completions, responses, Anthropic messages), so only the model's own MCP tools are offered. Both keys work on any model and inside an alias's profile block. Refs #12548 Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz * fix(openai): only auto-enable model MCP when MCP is enabled and permitted mcp.auto turns a plain /v1/chat/completions request into a server-side MCP request. It now applies the same controls as the dedicated MCP routes before any server is selected: the instance switch (LOCALAI_DISABLE_MCP) and, with auth on, the caller's "mcp" feature, checked with auth.HasFeatureAccess against the auth database that the chat route now passes to ChatEndpoint. A denied request is answered as a plain chat, without contacting a server or adding tools, rather than rejected: the client did not ask for MCP, it only reached a model that offers it. Endpoint specs drive ChatEndpoint against a counting MCP fixture server: MCP disabled and a user with chat but without mcp see no tools and no connection; MCP enabled and a user with mcp get the fixture tool. The user specs need the auth build tag (SQLite auth database). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz * fix(mcp): resolve profile aliases in the MCP discovery endpoints GET /v1/mcp/servers/:model, /v1/mcp/prompts/:model and /v1/mcp/resources/:model looked the model up with GetModelConfig, which returns the alias stub. A profile alias carries its servers in its profile block, so the stub has none of its own and the endpoints reported an empty list for an alias whose chat requests do use tools. Resolve the alias the way the request middleware does before reading the MCP config, and open the sessions under MCPSessionKey so discovery shares the profile's session cache slot instead of tearing down the target's. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz * feat(config): opt-in evidence check for profile aliases A profile with server-side MCP tools (profile.mcp, mcp.auto) runs its tools on the server and returns only the answer text. On the next turn the client sends the history back without those tool calls and results, and the model then tends to skip the tool and answer from nothing. In a measurement on a follow-up question, 0 of 12 turns called the tool and the answers carried invented numbers (e.g. "0" where the true value was 9); with the tool trail kept in the history the tool was called 12 of 12 times. A prompt instruction does not reliably prevent this, so the check sits at the output. New profile keys: profile: evidence_check: numbers: true # enables the check note: "..." # optional, defaults to an English note Rule, per request and stateless: when the check is on, no MCP tool call succeeded on the server during this request (a call that returned an error delivers no evidence and does not count), and the final answer contains a number (\b\d+([.,]\d+)?\b) that occurs in none of the request's input (the profile's system prompt and every message as received, before compression), the note is appended after a blank line. "3,5" and "3.5" count as the same number. In a streamed response the note is one extra content delta before the finish chunk. Choices that hand tool calls to the client are not checked. The setting is only read from a profile block, so plain models and aliases behave as before; a note without numbers: true is rejected at validation. Tests (Ginkgo): the number rule (invented number, number from the input, no numbers, comma/dot decimals, system prompt as evidence, tool ran), the overlay into the resolved config with plain alias, target and a top-level key as counter-checks, validation, and ChatEndpoint specs on a stubbed backend for both paths: note on an invented number, none for a number from the request, none for a profile without the check, none after a successful MCP tool call against an in-process MCP server, and the note again after a failed tool call (arguments that are not JSON). Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz --------- Signed-off-by: Stefan Walcz --- core/config/meta/registry.go | 70 +++- core/config/model_config.go | 168 +++++++++- core/config/model_config_loader.go | 5 +- .../config/model_config_profile_alias_test.go | 239 +++++++++++++ core/http/endpoints/anthropic/messages.go | 9 +- core/http/endpoints/localai/mcp_prompts.go | 14 +- core/http/endpoints/localai/mcp_resources.go | 14 +- core/http/endpoints/localai/mcp_tools.go | 33 +- .../localai/mcp_tools_profile_alias_test.go | 102 ++++++ core/http/endpoints/openai/chat.go | 84 ++++- .../openai/chat_mcp_auto_gate_auth_test.go | 57 ++++ .../openai/chat_mcp_auto_gate_test.go | 117 +++++++ .../endpoints/openai/chat_mcp_auto_test.go | 41 +++ core/http/endpoints/openai/evidence.go | 112 +++++++ core/http/endpoints/openai/evidence_test.go | 315 ++++++++++++++++++ .../http/endpoints/openresponses/responses.go | 4 +- core/http/middleware/request.go | 12 + .../middleware/request_profile_alias_test.go | 131 ++++++++ core/http/routes/openai.go | 2 +- docs/content/features/mcp.md | 35 ++ docs/content/features/model-aliases.md | 90 ++++- 21 files changed, 1623 insertions(+), 31 deletions(-) create mode 100644 core/config/model_config_profile_alias_test.go create mode 100644 core/http/endpoints/localai/mcp_tools_profile_alias_test.go create mode 100644 core/http/endpoints/openai/chat_mcp_auto_gate_auth_test.go create mode 100644 core/http/endpoints/openai/chat_mcp_auto_gate_test.go create mode 100644 core/http/endpoints/openai/chat_mcp_auto_test.go create mode 100644 core/http/endpoints/openai/evidence.go create mode 100644 core/http/endpoints/openai/evidence_test.go create mode 100644 core/http/middleware/request_profile_alias_test.go diff --git a/core/config/meta/registry.go b/core/config/meta/registry.go index f89c0d3f9..5900600c5 100644 --- a/core/config/meta/registry.go +++ b/core/config/meta/registry.go @@ -414,11 +414,64 @@ func DefaultRegistry() map[string]FieldMetaOverride { "alias": { Section: "alias", Label: "Alias target", - Description: "Redirect all traffic for this model to another configured model. When set, every other field on this config is ignored and requests are served by the target model.", + Description: "Redirect all traffic for this model to another configured model. When set, every other field on this config is ignored, except the optional profile block, and requests are served by the target model.", Component: "model-select", Order: 0, }, + "profile.system_prompt": { + Section: "alias", + Label: "Profile system prompt", + Description: "Alias only. Replaces the target's system prompt for requests to this alias; the target's backend process is reused.", + Component: "textarea", + Order: 1, + }, + "profile.mcp.remote": { + Section: "alias", + Label: "Profile remote MCP servers", + Description: "Alias only. Replaces the target's remote MCP servers (same mcpServers format as mcp.remote).", + Component: "code-editor", + Language: "yaml", + Order: 2, + }, + "profile.mcp.stdio": { + Section: "alias", + Label: "Profile MCP STDIO servers", + Description: "Alias only. Replaces the target's MCP STDIO servers (same mcpServers format as mcp.stdio).", + Component: "code-editor", + Language: "yaml", + Order: 3, + }, + "profile.mcp.auto": { + Section: "alias", + Label: "Profile: enable MCP servers automatically", + Description: "Alias only. Same as mcp.auto, for requests to this alias.", + Component: "toggle", + Order: 4, + }, + "profile.client_tools": { + Section: "alias", + Label: "Profile client tools", + Description: "Alias only. Same as client_tools, for requests to this alias.", + Component: "select", + Options: []FieldOption{{Value: "", Label: "Keep"}, {Value: "drop", Label: "Drop"}}, + Order: 5, + }, + "profile.evidence_check.numbers": { + Section: "alias", + Label: "Profile: mark unverified numbers", + Description: "Alias only. When no MCP tool ran in the request and the answer contains a number found in no input message, append a note marking it as not verified.", + Component: "toggle", + Order: 6, + }, + "profile.evidence_check.note": { + Section: "alias", + Label: "Profile: unverified-number note", + Description: "Alias only. Text appended by profile.evidence_check.numbers. Empty uses the built-in English note.", + Component: "input", + Order: 7, + }, + // --- Failover --- "failover.targets": { Section: "failover", @@ -909,6 +962,21 @@ func DefaultRegistry() map[string]FieldMetaOverride { Language: "yaml", Order: 131, }, + "mcp.auto": { + Section: "mcp", + Label: "Enable MCP servers automatically", + Description: "Enable every configured MCP server on /v1/chat/completions when the request does not select servers via metadata.mcp_servers, so plain OpenAI clients get the server-side tools.", + Component: "toggle", + Order: 132, + }, + "client_tools": { + Section: "mcp", + Label: "Client tools", + Description: "What to do with tools sent by the client. Empty keeps them; \"drop\" discards them (and tool_choice) so only the model's own MCP tools are offered.", + Component: "select", + Options: []FieldOption{{Value: "", Label: "Keep"}, {Value: "drop", Label: "Drop"}}, + Order: 133, + }, // --- TTS --- "tts.voice_cloning": { diff --git a/core/config/model_config.go b/core/config/model_config.go index 18c45df78..3b56213ea 100644 --- a/core/config/model_config.go +++ b/core/config/model_config.go @@ -60,6 +60,9 @@ type TTSVoice struct { type ModelConfig struct { modelConfigFile string `yaml:"-" json:"-"` modelTemplate string `yaml:"-" json:"-"` + // evidenceCheck is set only from a profile alias's block, so a plain + // model config cannot switch it on by a top-level key. + evidenceCheck EvidenceCheckConfig `yaml:"-" json:"-"` // persistedConfigRevision is the revision of this model's persisted // configuration, stamped when the loader materializes it and therefore // before any per-request override is merged in. The request pipeline @@ -68,17 +71,31 @@ type ModelConfig struct { // the only way the controller sees one revision per configuration rather // than one per request body. Unexported, so it never enters the hash it // describes and never reaches YAML or JSON. - persistedConfigRevision string `yaml:"-" json:"-"` + persistedConfigRevision string `yaml:"-" json:"-"` + // mcpSessionKey keys the MCP session cache when it must differ from Name. + // A profile alias that brings its own MCP servers resolves to its target's + // config (so Name, and with it the backend process, stays the target's), + // but its sessions must not share a cache slot with the target's: the + // slot is invalidated on every config change, so two server sets under one + // key would tear each other's sessions down. Unexported: never persisted. + mcpSessionKey string `yaml:"-" json:"-"` schema.PredictionOptions `yaml:"parameters,omitempty" json:"parameters,omitempty"` Name string `yaml:"name,omitempty" json:"name,omitempty"` Artifacts []modelartifacts.Spec `yaml:"artifacts,omitempty" json:"artifacts,omitempty"` // Alias, when set, makes this config a pure redirect: every request for - // Name is served by the model named here. All other fields are ignored. - // The target must be an existing, non-alias model (enforced at load and - // at create/swap time). See docs/content for Model Aliases. + // Name is served by the model named here. All other fields are ignored, + // except the opt-in Profile block. The target must be an existing, + // non-alias model (enforced at load and at create/swap time). See + // docs/content for Model Aliases. Alias string `yaml:"alias,omitempty" json:"alias,omitempty"` + // Profile is only valid on an alias. Its fields are applied on top of the + // target's config for requests to the alias, while the request still runs + // on the target's backend process. Explicit block rather than honouring + // top-level fields so existing aliases keep behaving as pure redirects. + Profile ProfileConfig `yaml:"profile,omitempty" json:"profile,omitempty"` + // Failover makes this config a failover chain over other models. Like an // alias it has no backend of its own. Failover *FailoverConfig `yaml:"failover,omitempty" json:"failover,omitempty"` @@ -169,7 +186,13 @@ type ModelConfig struct { MCP MCPConfig `yaml:"mcp,omitempty" json:"mcp,omitempty"` Agent AgentConfig `yaml:"agent,omitempty" json:"agent,omitempty"` - PII PIIConfig `yaml:"pii,omitempty" json:"pii,omitempty"` + + // ClientTools controls tools and functions sent by the client. Empty keeps + // them (default). "drop" discards them, together with tool_choice, before + // the request is merged, so only the model's own MCP tools are offered. + ClientTools string `yaml:"client_tools,omitempty" json:"client_tools,omitempty"` + + PII PIIConfig `yaml:"pii,omitempty" json:"pii,omitempty"` // PIIDetection is the detection policy when THIS model is used as a // PII detector (a token_classify model named in another model's // pii.detectors). Ignored on models that aren't referenced as @@ -598,6 +621,84 @@ func (c *ModelConfig) HasRouter() bool { // Value receiver so it is callable on non-addressable config values too. func (c ModelConfig) IsAlias() bool { return c.Alias != "" } +// ClientToolsDrop is the client_tools value that discards client-sent tools. +const ClientToolsDrop = "drop" + +// DropsClientTools reports whether tools sent by the client are discarded. +func (c ModelConfig) DropsClientTools() bool { return c.ClientTools == ClientToolsDrop } + +// MCPSessionKey is the name the MCP session cache uses for this config. It is +// Name unless a profile alias replaced the MCP servers of its target. +func (c ModelConfig) MCPSessionKey() string { + if c.mcpSessionKey != "" { + return c.mcpSessionKey + } + return c.Name +} + +// @Description Profile overlay of a model alias +type ProfileConfig struct { + // SystemPrompt replaces the target's system prompt. + SystemPrompt string `yaml:"system_prompt,omitempty" json:"system_prompt,omitempty"` + // MCP replaces the target's MCP servers when it lists any; MCP.Auto + // enables automatic activation either way. + MCP MCPConfig `yaml:"mcp,omitempty" json:"mcp,omitempty"` + // ClientTools overrides the target's client_tools policy. + ClientTools string `yaml:"client_tools,omitempty" json:"client_tools,omitempty"` + // EvidenceCheck marks answers whose numbers have no evidence in the request. + EvidenceCheck EvidenceCheckConfig `yaml:"evidence_check,omitempty" json:"evidence_check,omitempty"` +} + +// DefaultEvidenceNote is appended when evidence_check.note is empty. +const DefaultEvidenceNote = "_Note: this answer contains numbers that no tool returned in this turn — not verified._" + +// @Description Output check of a profile alias +type EvidenceCheckConfig struct { + // Numbers enables the check for numbers in the final answer. + Numbers bool `yaml:"numbers,omitempty" json:"numbers,omitempty"` + // Note replaces DefaultEvidenceNote. + Note string `yaml:"note,omitempty" json:"note,omitempty"` +} + +// EvidenceCheck returns the evidence check a profile alias applied to this +// resolved config; it is zero for every other config. +func (c ModelConfig) EvidenceCheck() EvidenceCheckConfig { return c.evidenceCheck } + +// EvidenceNote returns the configured note or the default one. +func (e EvidenceCheckConfig) EvidenceNote() string { + if e.Note != "" { + return e.Note + } + return DefaultEvidenceNote +} + +// IsZero reports whether the block sets nothing. +func (p ProfileConfig) IsZero() bool { return p == ProfileConfig{} } + +// applyProfileOverlay merges an alias's profile block into a copy of its +// target. Name, Model, Backend and the persisted revision stay the target's: +// they form the loader key, so the alias keeps sharing the target's backend +// process instead of loading the weights a second time. Only the profile block +// is read; any other field on the alias stays ignored, as for a plain alias. +func (c *ModelConfig) applyProfileOverlay(alias *ModelConfig) { + p := alias.Profile + if p.SystemPrompt != "" { + c.SystemPrompt = p.SystemPrompt + } + if p.MCP.HasMCPServers() { + c.MCP.Servers = p.MCP.Servers + c.MCP.Stdio = p.MCP.Stdio + c.mcpSessionKey = alias.Name + } + if p.MCP.Auto { + c.MCP.Auto = true + } + if p.ClientTools != "" { + c.ClientTools = p.ClientTools + } + c.evidenceCheck = p.EvidenceCheck +} + // @Description PII filtering configuration. PII redaction is per-model so // that local models don't pay the latency or behaviour change of regex // scanning, while cloud-bound traffic (cloud-proxy backend) can default to @@ -800,6 +901,11 @@ func (c *ModelConfig) IsPatternDetector() bool { type MCPConfig struct { Servers string `yaml:"remote,omitempty" json:"remote,omitempty"` Stdio string `yaml:"stdio,omitempty" json:"stdio,omitempty"` + // Auto enables every configured server on /v1/chat/completions when the + // request does not select servers via metadata.mcp_servers. Off by + // default: plain OpenAI clients cannot send that metadata, so without it + // they never reach the model's server-side tools. + Auto bool `yaml:"auto,omitempty" json:"auto,omitempty"` } // @Description Agent configuration @@ -821,6 +927,24 @@ func (c MCPConfig) HasMCPServers() bool { return c.Servers != "" || c.Stdio != "" } +// ServerNames returns the names of all configured remote and stdio servers, +// sorted so the enabled set does not depend on map iteration order. +func (c *MCPConfig) ServerNames() ([]string, error) { + remote, stdio, err := c.MCPConfigFromYAML() + if err != nil { + return nil, err + } + names := make([]string, 0, len(remote.Servers)+len(stdio.Servers)) + for name := range remote.Servers { + names = append(names, name) + } + for name := range stdio.Servers { + names = append(names, name) + } + slices.Sort(names) + return names, nil +} + func (c *MCPConfig) MCPConfigFromYAML() (MCPGenericConfig[MCPRemoteServers], MCPGenericConfig[MCPSTDIOServers], error) { var remote MCPGenericConfig[MCPRemoteServers] var stdio MCPGenericConfig[MCPSTDIOServers] @@ -1867,9 +1991,16 @@ func (c *ModelConfig) Validate() (bool, error) { if c.Backend != "" || c.Model != "" { return false, fmt.Errorf("alias config %q must not set backend or parameters.model: an alias is a pure redirect", c.Name) } + if err := c.validateProfileFields(); err != nil { + return false, err + } return true, nil } + if err := c.validateProfileFields(); err != nil { + return false, err + } + downloadedFileNames := []string{} for _, f := range c.DownloadFiles { downloadedFileNames = append(downloadedFileNames, f.Filename) @@ -2118,6 +2249,33 @@ func (c *ModelConfig) GetModelConfigFile() string { return c.modelConfigFile } +// validateProfileFields checks client_tools and the profile block. It runs for +// aliases too, which otherwise skip validation: their profile is used. +func (c *ModelConfig) validateProfileFields() error { + for key, value := range map[string]string{"client_tools": c.ClientTools, "profile.client_tools": c.Profile.ClientTools} { + switch value { + case "", ClientToolsDrop: + default: + return fmt.Errorf("%s: unknown value %q (allowed: %q)", key, value, ClientToolsDrop) + } + } + if c.Profile.IsZero() { + return nil + } + if !c.IsAlias() { + return fmt.Errorf("profile is only valid on an alias config (model %q has no alias)", c.Name) + } + if c.Profile.MCP.HasMCPServers() { + if _, _, err := c.Profile.MCP.MCPConfigFromYAML(); err != nil { + return fmt.Errorf("profile: invalid MCP configuration: %w", err) + } + } + if ec := c.Profile.EvidenceCheck; ec.Note != "" && !ec.Numbers { + return fmt.Errorf("profile.evidence_check.note has no effect without numbers: true") + } + return nil +} + // PersistedConfigRevision returns the revision stamped when this configuration // was loaded, or "" when it was never stamped (a config synthesized outside the // loader). Callers that need a revision for a request must prefer this over diff --git a/core/config/model_config_loader.go b/core/config/model_config_loader.go index b6554f937..79371a32c 100644 --- a/core/config/model_config_loader.go +++ b/core/config/model_config_loader.go @@ -572,7 +572,9 @@ func (bcl *ModelConfigLoader) UpdateModelConfig(m string, updater func(*ModelCon // ResolveAlias follows a one-hop alias to its target config. Returns // (resolved, wasAlias, err). Non-alias configs return (cfg, false, nil) // unchanged. Strict: the target must exist and must not itself be an alias -// (chains are rejected). The returned config is a copy of the target. +// (chains are rejected). The returned config is a copy of the target with the +// alias's opt-in profile block applied on top; an alias without a profile +// block yields a plain copy of the target. func (bcl *ModelConfigLoader) ResolveAlias(cfg *ModelConfig) (*ModelConfig, bool, error) { if cfg == nil || !cfg.IsAlias() { return cfg, false, nil @@ -584,6 +586,7 @@ func (bcl *ModelConfigLoader) ResolveAlias(cfg *ModelConfig) (*ModelConfig, bool if target.IsAlias() { return nil, true, fmt.Errorf("alias %q points to another alias %q (chains are not allowed)", cfg.Name, cfg.Alias) } + target.applyProfileOverlay(cfg) return &target, true, nil } diff --git a/core/config/model_config_profile_alias_test.go b/core/config/model_config_profile_alias_test.go new file mode 100644 index 000000000..350b2bc17 --- /dev/null +++ b/core/config/model_config_profile_alias_test.go @@ -0,0 +1,239 @@ +package config_test + +import ( + "os" + "path/filepath" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/mudler/LocalAI/core/config" +) + +// A profile alias carries an opt-in profile block (system_prompt, mcp, +// client_tools) that is applied on top of its target while the target's loader +// identity is kept, so several assistant personas share one loaded model +// instead of each loading the weights again. +var _ = Describe("Profile alias overlay", func() { + const target = `name: base-llm +backend: llama-cpp +system_prompt: base prompt +parameters: + model: base-llm.gguf +mcp: + remote: | + {"mcpServers": {"base-tools": {"url": "http://base.invalid/mcp"}}} +` + const profile = `name: researcher +alias: base-llm +profile: + system_prompt: You are a research assistant. + client_tools: drop + mcp: + auto: true + remote: | + {"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}} + stdio: | + {"mcpServers": {"notes": {"command": "notes-mcp"}}} +` + const plain = `name: gpt-4 +alias: base-llm +` + // Top-level fields on an alias were always ignored; without a profile + // block they must stay ignored, so existing aliases never change. + const legacy = `name: legacy +alias: base-llm +system_prompt: ignored prompt +client_tools: drop +mcp: + auto: true + remote: | + {"mcpServers": {"ignored": {"url": "http://ignored.invalid/mcp"}}} +` + + var cl *config.ModelConfigLoader + + BeforeEach(func() { + dir := GinkgoT().TempDir() + Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "researcher.yaml"), []byte(profile), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "gpt-4.yaml"), []byte(plain), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "legacy.yaml"), []byte(legacy), 0644)).To(Succeed()) + cl = config.NewModelConfigLoader(dir) + Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed()) + }) + + resolve := func(name string) *config.ModelConfig { + cfg, ok := cl.GetModelConfig(name) + Expect(ok).To(BeTrue(), name) + resolved, wasAlias, err := cl.ResolveAlias(&cfg) + Expect(err).NotTo(HaveOccurred()) + Expect(wasAlias).To(BeTrue()) + return resolved + } + + It("applies the alias's system prompt, MCP servers and tool policy", func() { + resolved := resolve("researcher") + + Expect(resolved.SystemPrompt).To(Equal("You are a research assistant.")) + Expect(resolved.DropsClientTools()).To(BeTrue()) + Expect(resolved.MCP.Auto).To(BeTrue()) + names, err := resolved.MCP.ServerNames() + Expect(err).NotTo(HaveOccurred()) + // The alias's servers replace the target's; nothing of the target's leaks in. + Expect(names).To(Equal([]string{"notes", "web"})) + // Sessions are cached apart from the target's, which has other servers. + Expect(resolved.MCPSessionKey()).To(Equal("researcher")) + }) + + It("keeps the target's loader identity, so no second model load happens", func() { + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + resolved := resolve("researcher") + + // ModelID, Backend, Model and the persisted revision are what the + // model loader and the controller key a backend process on. + Expect(resolved.ModelID()).To(Equal("base-llm")) + Expect(resolved.Backend).To(Equal(base.Backend)) + Expect(resolved.Model).To(Equal(base.Model)) + Expect(resolved.PersistedConfigRevision()).NotTo(BeEmpty()) + Expect(resolved.PersistedConfigRevision()).To(Equal(base.PersistedConfigRevision())) + Expect(resolve("gpt-4").ModelID()).To(Equal(resolved.ModelID())) + }) + + It("does not leak the overlay into the stored target", func() { + _ = resolve("researcher") + + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + Expect(base.SystemPrompt).To(Equal("base prompt")) + Expect(base.DropsClientTools()).To(BeFalse()) + Expect(base.MCP.Auto).To(BeFalse()) + Expect(base.MCPSessionKey()).To(Equal("base-llm")) + names, err := base.MCP.ServerNames() + Expect(err).NotTo(HaveOccurred()) + Expect(names).To(Equal([]string{"base-tools"})) + }) + + It("resolves an alias without overlay fields to exactly the target", func() { + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + + resolved := resolve("gpt-4") + Expect(*resolved).To(Equal(base)) + Expect(resolved.MCPSessionKey()).To(Equal("base-llm")) + }) + + It("ignores top-level system_prompt, mcp and client_tools on an alias, as before", func() { + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + + resolved := resolve("legacy") + Expect(*resolved).To(Equal(base)) + Expect(resolved.SystemPrompt).To(Equal("base prompt")) + Expect(resolved.DropsClientTools()).To(BeFalse()) + Expect(resolved.MCP.Auto).To(BeFalse()) + Expect(resolved.MCPSessionKey()).To(Equal("base-llm")) + }) + + It("keeps the target's servers and session key when the profile only sets mcp.auto", func() { + dir := GinkgoT().TempDir() + Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "auto.yaml"), []byte("name: auto\nalias: base-llm\nprofile:\n mcp:\n auto: true\n"), 0644)).To(Succeed()) + cl = config.NewModelConfigLoader(dir) + Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed()) + + resolved := resolve("auto") + Expect(resolved.MCP.Auto).To(BeTrue()) + names, err := resolved.MCP.ServerNames() + Expect(err).NotTo(HaveOccurred()) + Expect(names).To(Equal([]string{"base-tools"})) + Expect(resolved.MCPSessionKey()).To(Equal("base-llm")) + }) + + It("carries evidence_check into the resolved config and nowhere else", func() { + dir := GinkgoT().TempDir() + Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "checked.yaml"), []byte("name: checked\nalias: base-llm\nprofile:\n evidence_check:\n numbers: true\n note: custom note\n"), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "default-note.yaml"), []byte("name: default-note\nalias: base-llm\nprofile:\n evidence_check:\n numbers: true\n"), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "gpt-4.yaml"), []byte(plain), 0644)).To(Succeed()) + cl = config.NewModelConfigLoader(dir) + Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed()) + + checked := resolve("checked").EvidenceCheck() + Expect(checked.Numbers).To(BeTrue()) + Expect(checked.EvidenceNote()).To(Equal("custom note")) + Expect(resolve("default-note").EvidenceCheck().EvidenceNote()).To(Equal(config.DefaultEvidenceNote)) + + // A plain alias and the target itself stay unchecked. + Expect(resolve("gpt-4").EvidenceCheck().Numbers).To(BeFalse()) + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + Expect(base.EvidenceCheck().Numbers).To(BeFalse()) + }) + + It("ignores a top-level evidence_check on a model, which only a profile can set", func() { + dir := GinkgoT().TempDir() + Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target+"evidence_check:\n numbers: true\n"), 0644)).To(Succeed()) + cl = config.NewModelConfigLoader(dir) + Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed()) + + base, ok := cl.GetModelConfig("base-llm") + Expect(ok).To(BeTrue()) + Expect(base.EvidenceCheck().Numbers).To(BeFalse()) + }) + + Context("validation", func() { + It("rejects an unknown profile.client_tools value", func() { + cfg := config.ModelConfig{Name: "p", Alias: "base-llm"} + cfg.Profile.ClientTools = "ignore" + ok, err := cfg.Validate() + Expect(ok).To(BeFalse()) + Expect(err).To(MatchError(ContainSubstring("profile.client_tools"))) + }) + + It("rejects a profile whose MCP config does not parse", func() { + cfg := config.ModelConfig{Name: "p", Alias: "base-llm"} + cfg.Profile.MCP.Servers = "{not yaml: [" + ok, err := cfg.Validate() + Expect(ok).To(BeFalse()) + Expect(err).To(MatchError(ContainSubstring("invalid MCP configuration"))) + }) + + It("rejects an evidence_check note without numbers: true", func() { + cfg := config.ModelConfig{Name: "p", Alias: "base-llm"} + cfg.Profile.EvidenceCheck.Note = "unverified" + ok, err := cfg.Validate() + Expect(ok).To(BeFalse()) + Expect(err).To(MatchError(ContainSubstring("profile.evidence_check.note"))) + + cfg.Profile.EvidenceCheck.Numbers = true + ok, err = cfg.Validate() + Expect(err).NotTo(HaveOccurred()) + Expect(ok).To(BeTrue()) + }) + + It("rejects a profile block on a model that is not an alias", func() { + cfg := config.ModelConfig{Name: "m", Backend: "llama-cpp"} + cfg.Profile.SystemPrompt = "persona" + ok, err := cfg.Validate() + Expect(ok).To(BeFalse()) + Expect(err).To(MatchError(ContainSubstring("only valid on an alias"))) + }) + + It("accepts a well-formed profile alias and a plain alias", func() { + cfg := config.ModelConfig{Name: "p", Alias: "base-llm"} + cfg.Profile.SystemPrompt = "persona" + cfg.Profile.ClientTools = config.ClientToolsDrop + cfg.Profile.MCP.Servers = `{"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}}` + ok, err := cfg.Validate() + Expect(err).NotTo(HaveOccurred()) + Expect(ok).To(BeTrue()) + + plainCfg := config.ModelConfig{Name: "gpt-4", Alias: "base-llm"} + ok, err = plainCfg.Validate() + Expect(err).NotTo(HaveOccurred()) + Expect(ok).To(BeTrue()) + }) + }) +}) diff --git a/core/http/endpoints/anthropic/messages.go b/core/http/endpoints/anthropic/messages.go index 7c5fd7e68..0604c9ef5 100644 --- a/core/http/endpoints/anthropic/messages.go +++ b/core/http/endpoints/anthropic/messages.go @@ -49,6 +49,11 @@ func MessagesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evalu xlog.Debug("Anthropic Messages endpoint configuration read", "config", cfg) + if cfg.DropsClientTools() { + input.Tools = nil + input.ToolChoice = nil + } + // Cloud-proxy bail. Same shape as the OpenAI chat endpoint — // forwards via the cloud-proxy gRPC backend. if cfg.IsCloudProxyBackendPassthrough() { @@ -70,10 +75,10 @@ func MessagesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evalu if (len(mcpServers) > 0 || mcpPromptName != "" || len(mcpResourceURIs) > 0) && (cfg.MCP.Servers != "" || cfg.MCP.Stdio != "") { remote, stdio, mcpErr := cfg.MCP.MCPConfigFromYAML() if mcpErr == nil { - mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.Name, remote, stdio, mcpServers) + mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio, mcpServers) // Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode) - namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, mcpServers) + namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, mcpServers) if sessErr == nil && len(namedSessions) > 0 { mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs) if mcpCtx != nil { diff --git a/core/http/endpoints/localai/mcp_prompts.go b/core/http/endpoints/localai/mcp_prompts.go index 8f04ee6c8..96da27f83 100644 --- a/core/http/endpoints/localai/mcp_prompts.go +++ b/core/http/endpoints/localai/mcp_prompts.go @@ -17,10 +17,13 @@ func MCPPromptsEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat return echo.ErrBadRequest } - cfg, exists := cl.GetModelConfig(modelName) + cfg, exists, err := mcpModelConfig(cl, modelName) if !exists { return fmt.Errorf("model %q not found", modelName) } + if err != nil { + return err + } if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" { return c.JSON(200, []any{}) @@ -31,7 +34,7 @@ func MCPPromptsEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat return fmt.Errorf("failed to parse MCP config: %w", err) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } @@ -86,10 +89,13 @@ func MCPGetPromptEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic return echo.ErrBadRequest } - cfg, exists := cl.GetModelConfig(modelName) + cfg, exists, err := mcpModelConfig(cl, modelName) if !exists { return fmt.Errorf("model %q not found", modelName) } + if err != nil { + return err + } if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" { return fmt.Errorf("no MCP servers configured for model %q", modelName) @@ -107,7 +113,7 @@ func MCPGetPromptEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic return fmt.Errorf("failed to parse MCP config: %w", err) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } diff --git a/core/http/endpoints/localai/mcp_resources.go b/core/http/endpoints/localai/mcp_resources.go index 0cacec3c1..9d4e877fa 100644 --- a/core/http/endpoints/localai/mcp_resources.go +++ b/core/http/endpoints/localai/mcp_resources.go @@ -17,10 +17,13 @@ func MCPResourcesEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic return echo.ErrBadRequest } - cfg, exists := cl.GetModelConfig(modelName) + cfg, exists, err := mcpModelConfig(cl, modelName) if !exists { return fmt.Errorf("model %q not found", modelName) } + if err != nil { + return err + } if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" { return c.JSON(200, []any{}) @@ -31,7 +34,7 @@ func MCPResourcesEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic return fmt.Errorf("failed to parse MCP config: %w", err) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } @@ -73,10 +76,13 @@ func MCPReadResourceEndpoint(cl *config.ModelConfigLoader, appConfig *config.App return echo.ErrBadRequest } - cfg, exists := cl.GetModelConfig(modelName) + cfg, exists, err := mcpModelConfig(cl, modelName) if !exists { return fmt.Errorf("model %q not found", modelName) } + if err != nil { + return err + } if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" { return fmt.Errorf("no MCP servers configured for model %q", modelName) @@ -94,7 +100,7 @@ func MCPReadResourceEndpoint(cl *config.ModelConfigLoader, appConfig *config.App return fmt.Errorf("failed to parse MCP config: %w", err) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } diff --git a/core/http/endpoints/localai/mcp_tools.go b/core/http/endpoints/localai/mcp_tools.go index 8839cd74d..381ad2be9 100644 --- a/core/http/endpoints/localai/mcp_tools.go +++ b/core/http/endpoints/localai/mcp_tools.go @@ -20,7 +20,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat return echo.ErrBadRequest } - cfg, exists := cl.GetModelConfig(modelName) + cfg, exists, err := mcpModelConfig(cl, modelName) if !exists { return c.JSON(http.StatusNotFound, map[string]any{ "model": modelName, @@ -28,6 +28,13 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat "error": fmt.Sprintf("model %q not found", modelName), }) } + if err != nil { + return c.JSON(http.StatusBadRequest, map[string]any{ + "model": modelName, + "servers": []any{}, + "error": err.Error(), + }) + } if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" { return c.JSON(200, map[string]any{ @@ -48,7 +55,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat // In distributed mode, route discovery through NATS to an agent worker // that can actually connect to the MCP servers. if agentControl != nil { - resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.Name, remote, stdio) + resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio) if err != nil { return c.JSON(http.StatusOK, map[string]any{ "model": modelName, @@ -61,7 +68,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat }) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } @@ -105,7 +112,7 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H // In distributed mode, route discovery through NATS to an agent worker. if agentControl != nil { - resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.Name, remote, stdio) + resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio) if err != nil { return c.JSON(http.StatusOK, map[string]any{ "model": cfg.Name, @@ -118,7 +125,7 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H }) } - namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil) + namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil) if err != nil { return fmt.Errorf("failed to get MCP sessions: %w", err) } @@ -135,6 +142,22 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H } } +// mcpModelConfig returns the config whose MCP servers a chat request for name +// would use. A profile alias carries its servers in its profile block, so the +// raw alias stub has none of its own; resolving it the way the request +// middleware does keeps these endpoints in line with what chat actually offers. +func mcpModelConfig(cl *config.ModelConfigLoader, name string) (*config.ModelConfig, bool, error) { + cfg, exists := cl.GetModelConfig(name) + if !exists { + return nil, false, nil + } + resolved, _, err := cl.ResolveAlias(&cfg) + if err != nil { + return nil, true, err + } + return resolved, true, nil +} + func unavailableMCPServers( remote config.MCPGenericConfig[config.MCPRemoteServers], stdio config.MCPGenericConfig[config.MCPSTDIOServers], diff --git a/core/http/endpoints/localai/mcp_tools_profile_alias_test.go b/core/http/endpoints/localai/mcp_tools_profile_alias_test.go new file mode 100644 index 000000000..14c0ebc84 --- /dev/null +++ b/core/http/endpoints/localai/mcp_tools_profile_alias_test.go @@ -0,0 +1,102 @@ +package localai_test + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + + "github.com/labstack/echo/v4" + "github.com/modelcontextprotocol/go-sdk/mcp" + "github.com/mudler/LocalAI/core/config" + . "github.com/mudler/LocalAI/core/http/endpoints/localai" + mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("MCPServersEndpoint with profile aliases", func() { + var ( + tempDir string + fixture *httptest.Server + app *echo.Echo + ) + + BeforeEach(func() { + var err error + tempDir, err = os.MkdirTemp("", "localai-mcp-profile-test") + Expect(err).ToNot(HaveOccurred()) + + srv := mcp.NewServer(&mcp.Implementation{Name: "fixture", Version: "v0.0.1"}, nil) + srv.AddTool(&mcp.Tool{ + Name: "list_collections", + Description: "fixture tool", + InputSchema: json.RawMessage(`{"type":"object","properties":{}}`), + }, func(context.Context, *mcp.CallToolRequest) (*mcp.CallToolResult, error) { + return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: "ok"}}}, nil + }) + fixture = httptest.NewServer(mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return srv }, nil)) + + write := func(name, body string) { + Expect(os.WriteFile(filepath.Join(tempDir, name+".yaml"), []byte(body), 0644)).To(Succeed()) + } + write("real", "name: real\nbackend: llama-cpp\nmodel: foo\n") + write("plain-alias", "name: plain-alias\nalias: real\n") + write("intern", fmt.Sprintf(`name: intern +alias: real +profile: + mcp: + auto: true + remote: | + {"mcpServers": {"intern-set": {"url": %q}}} +`, fixture.URL)) + + loader := config.NewModelConfigLoader(tempDir) + Expect(loader.LoadModelConfigsFromPath(tempDir)).To(Succeed()) + + app = echo.New() + app.GET("/v1/mcp/servers/:model", MCPServersEndpoint(loader, &config.ApplicationConfig{}, nil)) + }) + + AfterEach(func() { + // The endpoint caches the session under the profile's name and keeps its + // stream open; close it first or the fixture waits for it on shutdown. + mcpTools.CloseMCPSessions("intern") + fixture.Close() + _ = os.RemoveAll(tempDir) + }) + + type serverInfo struct { + Name string `json:"name"` + Tools []string `json:"tools"` + Error string `json:"error"` + } + servers := func(model string) []serverInfo { + rec := httptest.NewRecorder() + app.ServeHTTP(rec, httptest.NewRequest("GET", "/v1/mcp/servers/"+model, nil)) + Expect(rec.Code).To(Equal(http.StatusOK), rec.Body.String()) + var body struct { + Model string `json:"model"` + Servers []serverInfo `json:"servers"` + } + Expect(json.Unmarshal(rec.Body.Bytes(), &body)).To(Succeed()) + Expect(body.Model).To(Equal(model)) + return body.Servers + } + + It("lists the servers and tools a profile alias brings, as chat would use them", func() { + got := servers("intern") + Expect(got).To(HaveLen(1)) + Expect(got[0].Name).To(Equal("intern-set")) + Expect(got[0].Error).To(BeEmpty()) + Expect(got[0].Tools).To(ConsistOf("list_collections")) + }) + + It("does not attach the profile's servers to the target or to a plain alias", func() { + Expect(servers("real")).To(BeEmpty()) + Expect(servers("plain-alias")).To(BeEmpty()) + }) +}) diff --git a/core/http/endpoints/openai/chat.go b/core/http/endpoints/openai/chat.go index 638be16a9..7f82415f3 100644 --- a/core/http/endpoints/openai/chat.go +++ b/core/http/endpoints/openai/chat.go @@ -11,6 +11,7 @@ import ( "github.com/labstack/echo/v4" "github.com/mudler/LocalAI/core/backend" "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/auth" mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" "github.com/mudler/LocalAI/core/http/middleware" "github.com/mudler/LocalAI/core/schema" @@ -23,6 +24,7 @@ import ( "github.com/mudler/LocalAI/pkg/model" "github.com/mudler/xlog" + "gorm.io/gorm" ) // messageText returns the textual content of a message, preferring the @@ -212,13 +214,56 @@ func applyAutoparserOverride( return []schema.Choice{newChoice} } +// selectMCPServers returns the MCP servers a chat request enables: the ones +// named in metadata.mcp_servers or, with mcp.auto, every configured server when +// the request names none. mcp.auto exists because plain OpenAI clients cannot +// send that metadata and would otherwise never reach server-side tools. The +// assistant modality brings its own tools and is left alone. autoPermitted is +// consulted only when the automatic path would apply, so plain chat requests +// on models without mcp.auto pay no permission lookup. +func selectMCPServers(cfg *config.ModelConfig, requested []string, assistantMode bool, autoPermitted func() bool) []string { + if len(requested) > 0 || assistantMode || !cfg.MCP.Auto || !cfg.MCP.HasMCPServers() { + return requested + } + if autoPermitted != nil && !autoPermitted() { + return requested + } + names, err := cfg.MCP.ServerNames() + if err != nil { + xlog.Error("Failed to parse MCP config", "error", err) + return requested + } + return names +} + +// mcpAutoPermitted reports whether mcp.auto may turn this chat request into a +// server-side MCP request. It applies the same controls as the dedicated MCP +// routes: the instance switch (LOCALAI_DISABLE_MCP) and, when auth is on, the +// caller's "mcp" feature. A denied request is answered as a plain chat rather +// than refused, because the client did not ask for MCP; it only reached a +// model that offers it. Without an auth database only the instance switch +// applies, matching auth.RequireFeature. +func mcpAutoPermitted(c echo.Context, appConfig *config.ApplicationConfig, authDB *gorm.DB) bool { + if appConfig != nil && appConfig.DisableMCP { + return false + } + if authDB == nil { + return true + } + return auth.HasFeatureAccess(authDB, auth.GetUser(c), auth.FeatureMCP) +} + // ChatEndpoint is the OpenAI Completion API endpoint https://platform.openai.com/docs/api-reference/chat/create // @Summary Generate a chat completions for a given prompt and model. // @Tags inference // @Param request body schema.OpenAIRequest true "query params" // @Success 200 {object} schema.OpenAIResponse "Response" // @Router /v1/chat/completions [post] -func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, startupOptions *config.ApplicationConfig, agentControl mcpTools.AgentControl, assistantHolder *mcpTools.LocalAIAssistantHolder, compressor middleware.ChatCompressor) echo.HandlerFunc { +func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, startupOptions *config.ApplicationConfig, agentControl mcpTools.AgentControl, assistantHolder *mcpTools.LocalAIAssistantHolder, compressor middleware.ChatCompressor, authDB ...*gorm.DB) echo.HandlerFunc { + var db *gorm.DB + if len(authDB) > 0 { + db = authDB[0] + } return func(c echo.Context) error { var textContentToReturn string id := uuid.New().String() @@ -313,6 +358,10 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator xlog.Debug("LocalAI Assistant tools injected", "count", len(mcpFuncs)) } + mcpServers = selectMCPServers(config, mcpServers, assistantMode, func() bool { + return mcpAutoPermitted(c, startupOptions, db) + }) + // MCP prompt and resource injection (extracted before tool injection) mcpPromptName, mcpPromptArgs := mcpTools.MCPPromptFromMetadata(input.Metadata) mcpResourceURIs := mcpTools.MCPResourcesFromMetadata(input.Metadata) @@ -320,10 +369,10 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator if (len(mcpServers) > 0 || mcpPromptName != "" || len(mcpResourceURIs) > 0) && (config.MCP.Servers != "" || config.MCP.Stdio != "") { remote, stdio, mcpErr := config.MCP.MCPConfigFromYAML() if mcpErr == nil { - mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, config.Name, remote, stdio, mcpServers) + mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, config.MCPSessionKey(), remote, stdio, mcpServers) // Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode) - namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(config.Name, remote, stdio, mcpServers) + namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(config.MCPSessionKey(), remote, stdio, mcpServers) if sessErr == nil && len(namedSessions) > 0 { mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs) if mcpCtx != nil { @@ -350,6 +399,8 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator xlog.Error("Failed to parse MCP config", "error", mcpErr) } } + // Before compression, which may summarise away numbers the client sent. + evidence := newEvidenceGate(config, input.Messages) if err := middleware.CompressChatRequest(c, compressor); err != nil { return err } @@ -586,7 +637,7 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator // We collect the RAW (unfiltered) content so the model's tool-call // markup keeps parsing correctly even when PII redaction would mask // substrings. - if (hasMCPToolsStream || config.FunctionsConfig.AutomaticToolParsingFallback) && haveContent { + if (hasMCPToolsStream || config.FunctionsConfig.AutomaticToolParsingFallback || evidence != nil) && haveContent { collectedContent += rawContent } respData, err := json.Marshal(ev) @@ -683,6 +734,9 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator if toolErr != nil { xlog.Error("MCP tool execution failed", "tool", tc.FunctionCall.Name, "error", toolErr) toolResult = fmt.Sprintf("Error: %v", toolErr) + } else { + // A failed call delivers no evidence for the answer. + evidence.markToolExecuted() } input.Messages = append(input.Messages, schema.Message{ Role: "tool", @@ -746,6 +800,23 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator } } + // The answer has been streamed already, so the note can only + // follow it as one more content delta. + if !toolsCalled { + if note := evidence.suffix(collectedContent); note != "" { + noteChunk := schema.OpenAIResponse{ + ID: id, + Created: created, + Model: input.Model, + Choices: []schema.Choice{{Index: 0, Delta: &schema.Message{Content: ¬e}}}, + Object: "chat.completion.chunk", + } + noteData, _ := json.Marshal(noteChunk) + fmt.Fprintf(c.Response().Writer, "data: %s\n\n", noteData) + c.Response().Flush() + } + } + // No MCP tools to execute, send final stop message finishReason := FinishReasonStop if toolsCalled && len(input.Tools) > 0 { @@ -1113,6 +1184,9 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator if toolErr != nil { xlog.Error("MCP tool execution failed", "tool", tc.FunctionCall.Name, "error", toolErr) toolResult = fmt.Sprintf("Error: %v", toolErr) + } else { + // A failed call delivers no evidence for the answer. + evidence.markToolExecuted() } input.Messages = append(input.Messages, schema.Message{ Role: "tool", @@ -1143,6 +1217,8 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator } } + evidence.apply(result) + // No MCP tools to execute (or no MCP tools configured), return response usage := schema.OpenAIUsage{ PromptTokens: tokenUsage.Prompt, diff --git a/core/http/endpoints/openai/chat_mcp_auto_gate_auth_test.go b/core/http/endpoints/openai/chat_mcp_auto_gate_auth_test.go new file mode 100644 index 000000000..a2a9f16e2 --- /dev/null +++ b/core/http/endpoints/openai/chat_mcp_auto_gate_auth_test.go @@ -0,0 +1,57 @@ +//go:build auth + +package openai + +import ( + "github.com/google/uuid" + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/auth" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gorm.io/gorm" +) + +// With auth on, mcp.auto must honour the caller's "mcp" feature like the +// dedicated MCP routes do; having "chat" alone is not enough. +var _ = Describe("ChatEndpoint mcp.auto user gate", func() { + var ( + db *gorm.DB + appConfig *config.ApplicationConfig + ) + + BeforeEach(func() { + var err error + db, err = auth.InitDB(":memory:") + Expect(err).ToNot(HaveOccurred()) + appConfig = &config.ApplicationConfig{} + appConfig.Auth.Enabled = true + }) + + userWith := func(perms auth.PermissionMap) *auth.User { + user := &auth.User{ + ID: uuid.NewString(), + Email: uuid.NewString() + "@example.com", + Provider: auth.ProviderLocal, + Subject: uuid.NewString(), + Role: auth.RoleUser, + Status: auth.StatusActive, + } + Expect(db.Create(user).Error).ToNot(HaveOccurred()) + Expect(auth.UpdateUserPermissions(db, user.ID, perms)).To(Succeed()) + return user + } + + It("stays a plain chat for a user who has chat but not mcp", func() { + fx := newMCPAutoFixture() + user := userWith(auth.PermissionMap{auth.FeatureChat: true, auth.FeatureMCP: false}) + Expect(fx.chat(appConfig, db, user)).To(BeEmpty()) + Expect(fx.requests.Load()).To(BeZero(), "no MCP server may be contacted") + }) + + It("enables the model's servers for a user who has mcp", func() { + fx := newMCPAutoFixture() + user := userWith(auth.PermissionMap{auth.FeatureChat: true, auth.FeatureMCP: true}) + Expect(fx.chat(appConfig, db, user)).To(ContainElement("lookup")) + Expect(fx.requests.Load()).To(BeNumerically(">", 0)) + }) +}) diff --git a/core/http/endpoints/openai/chat_mcp_auto_gate_test.go b/core/http/endpoints/openai/chat_mcp_auto_gate_test.go new file mode 100644 index 000000000..1308ce6dd --- /dev/null +++ b/core/http/endpoints/openai/chat_mcp_auto_gate_test.go @@ -0,0 +1,117 @@ +package openai + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + + "github.com/google/uuid" + "github.com/labstack/echo/v4" + "github.com/modelcontextprotocol/go-sdk/mcp" + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/http/auth" + mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" + "github.com/mudler/LocalAI/core/http/middleware" + "github.com/mudler/LocalAI/core/schema" + compressionservice "github.com/mudler/LocalAI/core/services/compression" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "gorm.io/gorm" +) + +// errStopAfterTools ends the chat handler right after the MCP block: context +// compression runs after tool injection and before any backend is touched, so +// the specs can read the final tool list without loading a model. +var errStopAfterTools = errors.New("stop after tool injection") + +type stopCompressor struct{} + +func (stopCompressor) Transform(context.Context, config.CompressionConfig, int, string, []schema.Message) ([]schema.Message, *compressionservice.Metadata, error) { + return nil, nil, errStopAfterTools +} + +// mcpAutoFixture is a remote MCP server with one tool that counts every HTTP +// request it receives, so a spec can prove no connection was attempted. +type mcpAutoFixture struct { + url string + requests atomic.Int32 +} + +func newMCPAutoFixture() *mcpAutoFixture { + fx := &mcpAutoFixture{} + srv := mcp.NewServer(&mcp.Implementation{Name: "fixture", Version: "v0.0.1"}, nil) + srv.AddTool(&mcp.Tool{ + Name: "lookup", + Description: "fixture tool", + InputSchema: json.RawMessage(`{"type":"object","properties":{}}`), + }, func(context.Context, *mcp.CallToolRequest) (*mcp.CallToolResult, error) { + return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: "ok"}}}, nil + }) + handler := mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return srv }, nil) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + fx.requests.Add(1) + handler.ServeHTTP(w, r) + })) + DeferCleanup(server.Close) + fx.url = server.URL + return fx +} + +// chat sends a plain chat request (no MCP metadata) through ChatEndpoint to a +// model with mcp.auto and returns the tool names that reached the backend stage. +func (fx *mcpAutoFixture) chat(appConfig *config.ApplicationConfig, db *gorm.DB, user *auth.User) []string { + modelName := "mcp-auto-gate-" + uuid.NewString() + DeferCleanup(func() { mcpTools.CloseMCPSessions(modelName) }) + + cfg := &config.ModelConfig{Name: modelName} + cfg.MCP.Servers = `{"mcpServers": {"fixture": {"url": "` + fx.url + `"}}}` + cfg.MCP.Auto = true + cfg.Compression.Enabled = true + + input := &schema.OpenAIRequest{ + PredictionOptions: schema.PredictionOptions{BasicModelRequest: schema.BasicModelRequest{Model: modelName}}, + Messages: []schema.Message{{Role: "user", Content: "hi", StringContent: "hi"}}, + } + + e := echo.New() + req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader("{}")) + c := e.NewContext(req, httptest.NewRecorder()) + c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, input) + c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, cfg) + if user != nil { + c.Set("auth_user", user) + } + + var dbs []*gorm.DB + if db != nil { + dbs = append(dbs, db) + } + handler := ChatEndpoint(nil, nil, nil, appConfig, nil, nil, stopCompressor{}, dbs...) + Expect(handler(c)).To(MatchError(ContainSubstring(errStopAfterTools.Error()))) + + var names []string + for _, t := range input.Tools { + names = append(names, t.Function.Name) + } + return names +} + +// mcp.auto turns a plain chat request into a server-side MCP request, so it +// must honour the instance switch the dedicated MCP routes honour. +var _ = Describe("ChatEndpoint mcp.auto instance gate", func() { + It("enables the model's servers when MCP is enabled", func() { + fx := newMCPAutoFixture() + Expect(fx.chat(&config.ApplicationConfig{}, nil, nil)).To(ContainElement("lookup")) + Expect(fx.requests.Load()).To(BeNumerically(">", 0)) + }) + + It("stays a plain chat when MCP is disabled on the instance", func() { + fx := newMCPAutoFixture() + Expect(fx.chat(&config.ApplicationConfig{DisableMCP: true}, nil, nil)).To(BeEmpty()) + Expect(fx.requests.Load()).To(BeZero(), "no MCP server may be contacted") + }) +}) diff --git a/core/http/endpoints/openai/chat_mcp_auto_test.go b/core/http/endpoints/openai/chat_mcp_auto_test.go new file mode 100644 index 000000000..3fdbe9109 --- /dev/null +++ b/core/http/endpoints/openai/chat_mcp_auto_test.go @@ -0,0 +1,41 @@ +package openai + +import ( + "github.com/mudler/LocalAI/core/config" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// mcp.auto lets plain OpenAI clients, which cannot send metadata.mcp_servers, +// reach the model's server-side tools on /v1/chat/completions. +var _ = Describe("selectMCPServers", func() { + withServers := func(auto bool) *config.ModelConfig { + cfg := &config.ModelConfig{} + cfg.MCP.Servers = `{"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}}` + cfg.MCP.Stdio = `{"mcpServers": {"notes": {"command": "notes-mcp"}}}` + cfg.MCP.Auto = auto + return cfg + } + + It("enables every configured server when mcp.auto is set and the request names none", func() { + Expect(selectMCPServers(withServers(true), nil, false, nil)).To(Equal([]string{"notes", "web"})) + }) + + It("keeps the servers the request selected", func() { + Expect(selectMCPServers(withServers(true), []string{"web"}, false, nil)).To(Equal([]string{"web"})) + }) + + It("enables nothing without mcp.auto, as before", func() { + Expect(selectMCPServers(withServers(false), nil, false, nil)).To(BeEmpty()) + }) + + It("leaves the assistant modality alone", func() { + Expect(selectMCPServers(withServers(true), nil, true, nil)).To(BeEmpty()) + }) + + It("enables nothing when no servers are configured", func() { + cfg := &config.ModelConfig{} + cfg.MCP.Auto = true + Expect(selectMCPServers(cfg, nil, false, nil)).To(BeEmpty()) + }) +}) diff --git a/core/http/endpoints/openai/evidence.go b/core/http/endpoints/openai/evidence.go new file mode 100644 index 000000000..4602f6c46 --- /dev/null +++ b/core/http/endpoints/openai/evidence.go @@ -0,0 +1,112 @@ +package openai + +import ( + "regexp" + "strings" + + "github.com/mudler/LocalAI/core/config" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/xlog" +) + +var numberToken = regexp.MustCompile(`\b\d+(?:[.,]\d+)?\b`) + +// numberKey makes "3,5" and "3.5" the same number, so an answer is not +// flagged for writing a decimal with the other separator than its source. +func numberKey(token string) string { + return strings.ReplaceAll(token, ",", ".") +} + +// evidenceNumbers collects the numbers of the request as received. It is +// taken before the request is compressed or extended with tool results, so +// it holds only what the client and the configuration supplied. +func evidenceNumbers(systemPrompt string, messages []schema.Message) map[string]struct{} { + numbers := map[string]struct{}{} + add := func(text string) { + for _, token := range numberToken.FindAllString(text, -1) { + numbers[numberKey(token)] = struct{}{} + } + } + add(systemPrompt) + for _, m := range messages { + add(messageText(m)) + } + return numbers +} + +// hasUnsupportedNumber reports whether answer contains a number that occurs in +// no input message. A successful server-side tool run counts as evidence for +// the whole answer: its results are in the model's context but never reach the +// client, so they cannot be checked against the request. +func hasUnsupportedNumber(answer string, evidence map[string]struct{}, toolExecuted bool) bool { + if toolExecuted { + return false + } + for _, token := range numberToken.FindAllString(answer, -1) { + if _, ok := evidence[numberKey(token)]; !ok { + return true + } + } + return false +} + +// evidenceGate carries the per-request state of a profile's evidence check. +// A nil gate is disabled, so callers need no separate enabled check. +type evidenceGate struct { + note string + evidence map[string]struct{} + toolExecuted bool +} + +func newEvidenceGate(cfg *config.ModelConfig, messages []schema.Message) *evidenceGate { + check := cfg.EvidenceCheck() + if !check.Numbers { + return nil + } + return &evidenceGate{note: check.EvidenceNote(), evidence: evidenceNumbers(cfg.SystemPrompt, messages)} +} + +func (g *evidenceGate) markToolExecuted() { + if g != nil { + g.toolExecuted = true + } +} + +// suffix returns the text to append to answer, or "" when it needs none. +func (g *evidenceGate) suffix(answer string) string { + if g == nil || !hasUnsupportedNumber(answer, g.evidence, g.toolExecuted) { + return "" + } + xlog.Debug("Evidence check: answer has numbers without evidence in the request, appending note") + return "\n\n" + g.note +} + +// apply appends the note to every final answer choice. Choices that hand tool +// calls to the client are not final answers and stay untouched. +func (g *evidenceGate) apply(choices []schema.Choice) { + if g == nil { + return + } + for i := range choices { + msg := choices[i].Message + if msg == nil || len(msg.ToolCalls) > 0 || msg.FunctionCall != nil { + continue + } + var content string + switch v := msg.Content.(type) { + case string: + content = v + case *string: + if v == nil { + continue + } + content = *v + default: + continue + } + if s := g.suffix(content); s != "" { + content += s + msg.Content = &content + } + } +} diff --git a/core/http/endpoints/openai/evidence_test.go b/core/http/endpoints/openai/evidence_test.go new file mode 100644 index 000000000..fd756b65f --- /dev/null +++ b/core/http/endpoints/openai/evidence_test.go @@ -0,0 +1,315 @@ +package openai + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + + "github.com/google/uuid" + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/backend" + "github.com/mudler/LocalAI/core/config" + mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp" + "github.com/mudler/LocalAI/core/http/middleware" + "github.com/mudler/LocalAI/core/schema" + pb "github.com/mudler/LocalAI/pkg/grpc/proto" + "github.com/mudler/LocalAI/pkg/model" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func userMessages(texts ...string) []schema.Message { + var msgs []schema.Message + for _, t := range texts { + msgs = append(msgs, schema.Message{Role: "user", Content: t, StringContent: t}) + } + return msgs +} + +var _ = Describe("hasUnsupportedNumber", func() { + evidence := func(texts ...string) map[string]struct{} { + return evidenceNumbers("", userMessages(texts...)) + } + + It("flags a number that occurs in no input message", func() { + Expect(hasUnsupportedNumber("There are 0 open invoices.", evidence("How many open invoices are there?"), false)).To(BeTrue()) + }) + + It("does not flag a number the request already contains", func() { + Expect(hasUnsupportedNumber("Yes, all 9 are open.", evidence("Are the 9 invoices open?"), false)).To(BeFalse()) + }) + + It("does not flag an answer without numbers", func() { + Expect(hasUnsupportedNumber("I cannot tell without looking it up.", evidence("How many?"), false)).To(BeFalse()) + }) + + It("matches decimals regardless of the separator", func() { + Expect(hasUnsupportedNumber("The rate is 3,5 percent.", evidence("Is the rate 3.5%?"), false)).To(BeFalse()) + Expect(hasUnsupportedNumber("The rate is 3.5 percent.", evidence("Ist der Satz 3,5 %?"), false)).To(BeFalse()) + }) + + It("flags a decimal whose digits differ from the input", func() { + Expect(hasUnsupportedNumber("The rate is 3,7 percent.", evidence("Is the rate 3.5%?"), false)).To(BeTrue()) + }) + + It("does not treat a number inside a longer one as evidence", func() { + Expect(hasUnsupportedNumber("There are 9.", evidence("Invoice 19 is open."), false)).To(BeTrue()) + Expect(hasUnsupportedNumber("Invoice 19 is open.", evidence("Invoice 19?"), false)).To(BeFalse()) + }) + + It("takes the system prompt as evidence", func() { + ev := evidenceNumbers("The office opens at 8.", userMessages("When does it open?")) + Expect(hasUnsupportedNumber("At 8.", ev, false)).To(BeFalse()) + Expect(hasUnsupportedNumber("At 9.", ev, false)).To(BeTrue()) + }) + + It("does not flag anything once a tool ran in the request", func() { + Expect(hasUnsupportedNumber("There are 7 open invoices.", evidence("How many open invoices?"), true)).To(BeFalse()) + Expect(hasUnsupportedNumber("There are 7 open invoices.", evidence("How many open invoices?"), false)).To(BeTrue()) + }) +}) + +var _ = Describe("evidenceGate.apply", func() { + gate := func() *evidenceGate { + return &evidenceGate{note: "NOTE", evidence: evidenceNumbers("", userMessages("How many?"))} + } + answer := func(content string) []schema.Choice { + return []schema.Choice{{Message: &schema.Message{Role: "assistant", Content: &content}}} + } + contentOf := func(choices []schema.Choice) string { + return *(choices[0].Message.Content.(*string)) + } + + It("appends the note to an answer with an invented number", func() { + choices := answer("There are 0.") + gate().apply(choices) + Expect(contentOf(choices)).To(Equal("There are 0.\n\nNOTE")) + }) + + It("leaves an answer without numbers alone", func() { + choices := answer("Let me check.") + gate().apply(choices) + Expect(contentOf(choices)).To(Equal("Let me check.")) + }) + + It("leaves tool calls for the client alone", func() { + content := "Calling 2 tools." + choices := []schema.Choice{{Message: &schema.Message{Content: &content, ToolCalls: []schema.ToolCall{{ID: "x"}}}}} + gate().apply(choices) + Expect(contentOf(choices)).To(Equal("Calling 2 tools.")) + }) + + It("does nothing when the check is off", func() { + var g *evidenceGate + choices := answer("There are 0.") + g.apply(choices) + Expect(contentOf(choices)).To(Equal("There are 0.")) + }) +}) + +// The endpoint specs run the real ChatEndpoint against a stubbed backend and a +// profile alias resolved by the real loader, so they cover the wiring from the +// alias YAML to the response the client sees. +var _ = Describe("ChatEndpoint evidence check", func() { + const note = "UNVERIFIED" + + var origInference modelInferenceFunc + + BeforeEach(func() { origInference = backend.ModelInferenceFunc }) + AfterEach(func() { backend.ModelInferenceFunc = origInference }) + + // stubBackend answers the n-th inference call with replies[n] (the last one + // repeats), streaming the text through the token callback as well. + stubBackend := func(replies ...backend.LLMResponse) { + call := 0 + backend.ModelInferenceFunc = func( + ctx context.Context, s string, messages schema.Messages, + images, videos, audios []string, + loader *model.ModelLoader, c *config.ModelConfig, cl *config.ModelConfigLoader, + o *config.ApplicationConfig, + tokenCallback func(string, backend.TokenUsage) bool, + tools, toolChoice string, + logprobs, topLogprobs *int, + logitBias map[string]float64, + metadata map[string]string, + ) (func() (backend.LLMResponse, error), error) { + reply := replies[min(call, len(replies)-1)] + call++ + return func() (backend.LLMResponse, error) { + if tokenCallback != nil && reply.Response != "" { + tokenCallback(reply.Response, backend.TokenUsage{}) + } + return reply, nil + }, nil + } + } + + // resolve writes a target and an alias to a fresh model dir and returns the + // alias resolved the way the request middleware resolves it. + resolve := func(aliasProfile, mcpURL string) *config.ModelConfig { + dir := GinkgoT().TempDir() + name := "evidence-" + uuid.NewString() + target := "name: " + name + "-base\nbackend: llama-cpp\nparameters:\n model: base.gguf\ntemplate:\n use_tokenizer_template: true\n" + alias := "name: " + name + "\nalias: " + name + "-base\n" + aliasProfile + if mcpURL != "" { + alias += " mcp:\n auto: true\n remote: |\n {\"mcpServers\": {\"fixture\": {\"url\": \"" + mcpURL + "\"}}}\n" + } + Expect(os.WriteFile(filepath.Join(dir, "base.yaml"), []byte(target), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(dir, "alias.yaml"), []byte(alias), 0644)).To(Succeed()) + cl := config.NewModelConfigLoader(dir) + Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed()) + cfg, ok := cl.GetModelConfig(name) + Expect(ok).To(BeTrue()) + resolved, _, err := cl.ResolveAlias(&cfg) + Expect(err).NotTo(HaveOccurred()) + DeferCleanup(func() { mcpTools.CloseMCPSessions(resolved.MCPSessionKey()) }) + return resolved + } + + const withCheck = "profile:\n evidence_check:\n numbers: true\n note: " + note + "\n" + const withoutCheck = "profile:\n system_prompt: plain persona\n" + + run := func(cfg *config.ModelConfig, stream bool, question string) *httptest.ResponseRecorder { + ctx, cancel := context.WithCancel(context.Background()) + DeferCleanup(cancel) + input := &schema.OpenAIRequest{ + PredictionOptions: schema.PredictionOptions{BasicModelRequest: schema.BasicModelRequest{Model: "alias"}}, + Messages: userMessages(question), + Stream: stream, + Context: ctx, + Cancel: cancel, + } + e := echo.New() + rec := httptest.NewRecorder() + c := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader("{}")), rec) + c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, input) + c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, cfg) + Expect(ChatEndpoint(nil, nil, nil, config.NewApplicationConfig(), nil, nil, nil)(c)).To(Succeed()) + return rec + } + + answerOf := func(rec *httptest.ResponseRecorder) string { + var resp schema.OpenAIResponse + Expect(json.Unmarshal(rec.Body.Bytes(), &resp)).To(Succeed(), rec.Body.String()) + Expect(resp.Choices).To(HaveLen(1)) + s, ok := resp.Choices[0].Message.Content.(string) + Expect(ok).To(BeTrue()) + return s + } + + // streamedAnswer joins the content deltas of an SSE body. + streamedAnswer := func(rec *httptest.ResponseRecorder) string { + var b strings.Builder + for _, line := range strings.Split(rec.Body.String(), "\n") { + data, ok := strings.CutPrefix(line, "data: ") + if !ok || data == "[DONE]" { + continue + } + var chunk schema.OpenAIResponse + if json.Unmarshal([]byte(data), &chunk) != nil || len(chunk.Choices) == 0 || chunk.Choices[0].Delta == nil { + continue + } + if s, ok := chunk.Choices[0].Delta.Content.(string); ok { + b.WriteString(s) + } + } + Expect(rec.Body.String()).To(HaveSuffix("data: [DONE]\n\n")) + return b.String() + } + + Context("non-streaming", func() { + It("marks an invented number", func() { + stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."}) + Expect(answerOf(run(resolve(withCheck, ""), false, "How many open invoices?"))). + To(Equal("There are 0 open invoices.\n\n" + note)) + }) + + It("leaves a number from the request unmarked", func() { + stubBackend(backend.LLMResponse{Response: "Yes, all 9 are open."}) + Expect(answerOf(run(resolve(withCheck, ""), false, "Are the 9 invoices open?"))). + To(Equal("Yes, all 9 are open.")) + }) + + It("leaves a profile without the check unchanged", func() { + stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."}) + Expect(answerOf(run(resolve(withoutCheck, ""), false, "How many open invoices?"))). + To(Equal("There are 0 open invoices.")) + }) + + It("leaves the answer unmarked when an MCP tool ran in this request", func() { + fx := newMCPAutoFixture() + stubBackend( + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{}"}}}}}, + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}}, + ) + Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))). + To(Equal("There are 7 open invoices.")) + Expect(fx.requests.Load()).To(BeNumerically(">", 0)) + }) + + It("marks an invented number when the MCP tool call failed", func() { + fx := newMCPAutoFixture() + // Arguments that are not JSON make the real executor fail the call. + stubBackend( + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{not json"}}}}}, + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}}, + ) + Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))). + To(Equal("There are 7 open invoices.\n\n" + note)) + }) + + It("marks the answer when MCP tools were offered but none ran", func() { + fx := newMCPAutoFixture() + stubBackend(backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 0 open invoices."}}}) + Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))). + To(Equal("There are 0 open invoices.\n\n" + note)) + }) + }) + + Context("streaming", func() { + It("sends the note as a last content delta", func() { + stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."}) + Expect(streamedAnswer(run(resolve(withCheck, ""), true, "How many open invoices?"))). + To(Equal("There are 0 open invoices.\n\n" + note)) + }) + + It("sends no note for a number from the request", func() { + stubBackend(backend.LLMResponse{Response: "Yes, all 9 are open."}) + Expect(streamedAnswer(run(resolve(withCheck, ""), true, "Are the 9 invoices open?"))). + To(Equal("Yes, all 9 are open.")) + }) + + It("sends no note when an MCP tool ran in this request", func() { + fx := newMCPAutoFixture() + stubBackend( + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{}"}}}}}, + backend.LLMResponse{Response: "There are 7 open invoices.", ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}}, + ) + rec := run(resolve(withCheck, fx.url), true, "How many open invoices?") + Expect(rec.Body.String()).To(ContainSubstring("mcp_tool_result")) + Expect(streamedAnswer(rec)).To(ContainSubstring("There are 7 open invoices.")) + Expect(streamedAnswer(rec)).NotTo(ContainSubstring(note)) + }) + + It("sends the note when the MCP tool call failed", func() { + fx := newMCPAutoFixture() + stubBackend( + backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{not json"}}}}}, + backend.LLMResponse{Response: "There are 7 open invoices.", ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}}, + ) + rec := run(resolve(withCheck, fx.url), true, "How many open invoices?") + Expect(rec.Body.String()).To(ContainSubstring("mcp_tool_result")) + Expect(streamedAnswer(rec)).To(HaveSuffix("There are 7 open invoices.\n\n" + note)) + }) + + It("sends no note for a profile without the check", func() { + stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."}) + Expect(streamedAnswer(run(resolve(withoutCheck, ""), true, "How many open invoices?"))). + To(Equal("There are 0 open invoices.")) + }) + }) +}) diff --git a/core/http/endpoints/openresponses/responses.go b/core/http/endpoints/openresponses/responses.go index 71164e291..3ca4727c9 100644 --- a/core/http/endpoints/openresponses/responses.go +++ b/core/http/endpoints/openresponses/responses.go @@ -108,11 +108,11 @@ func ResponsesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, eval if !hasMCPRequest { enabledServers = nil // backward compat: auto-activate all servers } - mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.Name, remote, stdio, enabledServers) + mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio, enabledServers) // Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode) if hasMCPRequest { - namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, mcpServers) + namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, mcpServers) if sessErr == nil && len(namedSessions) > 0 { mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs) if mcpCtx != nil { diff --git a/core/http/middleware/request.go b/core/http/middleware/request.go index 5c0806e8c..e60aa3393 100644 --- a/core/http/middleware/request.go +++ b/core/http/middleware/request.go @@ -392,6 +392,14 @@ func extractToolChoiceFunctionName(m map[string]any) string { } func mergeOpenAIRequestAndModelConfig(config *config.ModelConfig, input *schema.OpenAIRequest) error { + // Discard client tools before anything below derives functions or a forced + // tool_choice from them; the model's own MCP tools are injected later. + if config.DropsClientTools() { + input.Tools = nil + input.Functions = nil + input.ToolsChoice = nil + input.FunctionCall = nil + } if input.Echo { config.Echo = input.Echo } @@ -835,6 +843,10 @@ func (re *RequestExtractor) SetOpenResponsesRequest(c echo.Context) (err error) // MergeOpenResponsesConfig merges request parameters into the model configuration. func MergeOpenResponsesConfig(config *config.ModelConfig, input *schema.OpenResponsesRequest) error { + if config.DropsClientTools() { + input.Tools = nil + input.ToolChoice = nil + } // Temperature if input.Temperature != nil { config.Temperature = input.Temperature diff --git a/core/http/middleware/request_profile_alias_test.go b/core/http/middleware/request_profile_alias_test.go new file mode 100644 index 000000000..4030a2169 --- /dev/null +++ b/core/http/middleware/request_profile_alias_test.go @@ -0,0 +1,131 @@ +package middleware_test + +import ( + "net/http" + "os" + "path/filepath" + + "github.com/labstack/echo/v4" + "github.com/mudler/LocalAI/core/config" + . "github.com/mudler/LocalAI/core/http/middleware" + "github.com/mudler/LocalAI/core/schema" + "github.com/mudler/LocalAI/pkg/model" + "github.com/mudler/LocalAI/pkg/system" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// A profile alias resolves to its target's backend process but serves the +// request with the alias's own system prompt and tool policy. These specs drive +// the production middleware chain of /v1/chat/completions. +var _ = Describe("Profile alias through the chat middleware", func() { + var ( + app *echo.Echo + modelDir string + capturedConfig *config.ModelConfig + capturedInput *schema.OpenAIRequest + ) + + BeforeEach(func() { + var err error + modelDir, err = os.MkdirTemp("", "localai-profile-alias-*") + Expect(err).ToNot(HaveOccurred()) + + Expect(os.WriteFile(filepath.Join(modelDir, "base.yaml"), + []byte("name: base\nbackend: llama-cpp\nsystem_prompt: base prompt\n"), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(modelDir, "researcher.yaml"), + []byte("name: researcher\nalias: base\nprofile:\n system_prompt: persona prompt\n client_tools: drop\n"), 0644)).To(Succeed()) + Expect(os.WriteFile(filepath.Join(modelDir, "plain.yaml"), + []byte("name: plain\nalias: base\nsystem_prompt: ignored\nclient_tools: drop\n"), 0644)).To(Succeed()) + + ss := &system.SystemState{Model: system.Model{ModelsPath: modelDir}} + appConfig := config.NewApplicationConfig() + appConfig.SystemState = ss + mcl := config.NewModelConfigLoader(modelDir) + Expect(mcl.LoadModelConfigsFromPath(modelDir)).To(Succeed()) + re := NewRequestExtractor(mcl, model.NewModelLoader(ss), appConfig) + + capturedConfig, capturedInput = nil, nil + app = echo.New() + app.POST("/v1/chat/completions", + func(c echo.Context) error { + capturedConfig, _ = c.Get(CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig) + capturedInput, _ = c.Get(CONTEXT_LOCALS_KEY_LOCALAI_REQUEST).(*schema.OpenAIRequest) + return c.String(http.StatusOK, "ok") + }, + re.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.OpenAIRequest) }), + func(next echo.HandlerFunc) echo.HandlerFunc { + return func(c echo.Context) error { + if err := re.SetOpenAIRequest(c); err != nil { + return err + } + return next(c) + } + }, + ) + }) + + AfterEach(func() { + _ = os.RemoveAll(modelDir) + }) + + withTools := func(model string) string { + return `{"model":"` + model + `",` + + `"messages":[{"role":"user","content":"hi"}],` + + `"tools":[{"type":"function","function":{"name":"client_tool"}}],` + + `"tool_choice":{"type":"function","function":{"name":"client_tool"}}}` + } + + It("serves the target process with the alias's prompt and drops client tools", func() { + rec := postJSON(app, "/v1/chat/completions", withTools("researcher")) + + Expect(rec.Code).To(Equal(http.StatusOK)) + Expect(capturedConfig).ToNot(BeNil()) + Expect(capturedConfig.ModelID()).To(Equal("base")) + Expect(capturedConfig.SystemPrompt).To(Equal("persona prompt")) + Expect(capturedInput.Tools).To(BeEmpty()) + Expect(capturedInput.Functions).To(BeEmpty()) + // The client's forced tool_choice must not survive either. + Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse()) + // The response still echoes the alias name. + Expect(capturedInput.Model).To(Equal("researcher")) + }) + + It("leaves an alias without profile block exactly as before, even with top-level fields", func() { + rec := postJSON(app, "/v1/chat/completions", withTools("plain")) + + Expect(rec.Code).To(Equal(http.StatusOK)) + Expect(capturedConfig.ModelID()).To(Equal("base")) + Expect(capturedConfig.SystemPrompt).To(Equal("base prompt")) + Expect(capturedInput.Functions).To(HaveLen(1)) + Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue()) + Expect(capturedConfig.FunctionToCall()).To(Equal("client_tool")) + }) +}) + +var _ = Describe("MergeOpenResponsesConfig with client_tools", func() { + request := func() *schema.OpenResponsesRequest { + return &schema.OpenResponsesRequest{ + Tools: []schema.ORFunctionTool{{Type: "function", Name: "client_tool"}}, + ToolChoice: "required", + } + } + + It("drops client tools and tool_choice when the config says drop", func() { + cfg := &config.ModelConfig{ClientTools: config.ClientToolsDrop} + req := request() + Expect(MergeOpenResponsesConfig(cfg, req)).To(Succeed()) + + Expect(req.Tools).To(BeEmpty()) + Expect(req.ToolChoice).To(BeNil()) + }) + + It("keeps client tools by default", func() { + cfg := &config.ModelConfig{} + req := request() + Expect(MergeOpenResponsesConfig(cfg, req)).To(Succeed()) + + Expect(req.Tools).To(HaveLen(1)) + Expect(req.ToolChoice).To(Equal("required")) + }) +}) diff --git a/core/http/routes/openai.go b/core/http/routes/openai.go index d6af7158f..ccded2c52 100644 --- a/core/http/routes/openai.go +++ b/core/http/routes/openai.go @@ -51,7 +51,7 @@ func RegisterOpenAIRoutes(app *echo.Echo, compressionservice.CounterFunc(tokens.CountMessages), compressionservice.NewInferenceSummarizer(application.ModelConfigLoader(), application.ModelLoader(), application.ApplicationConfig()), ) - chatHandler := openai.ChatEndpoint(application.ModelConfigLoader(), application.ModelLoader(), application.TemplatesEvaluator(), application.ApplicationConfig(), agentControl, application.LocalAIAssistant(), chatCompressor) + chatHandler := openai.ChatEndpoint(application.ModelConfigLoader(), application.ModelLoader(), application.TemplatesEvaluator(), application.ApplicationConfig(), agentControl, application.LocalAIAssistant(), chatCompressor, application.AuthDB()) chatMiddleware := []echo.MiddlewareFunc{ nodeHeaderMiddleware, usageMiddleware, diff --git a/docs/content/features/mcp.md b/docs/content/features/mcp.md index 9310c2b5b..e8b452b0a 100644 --- a/docs/content/features/mcp.md +++ b/docs/content/features/mcp.md @@ -153,6 +153,41 @@ Configure local command-based MCP servers: - **`args`**: Array of command-line arguments - **`env`**: Environment variables (optional) +#### Automatic activation (`auto`) + +`/v1/chat/completions` only runs a model's MCP servers when the request selects +them through `metadata.mcp_servers`. Plain OpenAI clients cannot send that field. +Set `auto: true` in the `mcp:` block to enable every configured server when the +request names none: + +```yaml +mcp: + auto: true + remote: | + {"mcpServers": {"weather-api": {"url": "https://weather.example.com/mcp"}}} +``` + +A request that does send `metadata.mcp_servers` still gets exactly the servers it +names. `/v1/responses` already enables all servers when the request sends no +`tools`, and `/mcp/v1/chat/completions` always does; `auto` does not change them. + +`auto` only applies when MCP is enabled on the instance (not disabled with +`LOCALAI_DISABLE_MCP`) and, with authentication on, the caller has the `mcp` +permission. Otherwise the request is answered as a plain chat: no server is +contacted and no tools are added. It is not rejected, because the client did not +ask for MCP. + +#### Client tools (`client_tools`) + +`client_tools: drop` (a top-level model config key) discards the `tools`, +`functions` and `tool_choice` a client sends, so the model is offered only its own +MCP tools. Use it when a front end attaches tools of its own that should not +reach this model. Empty (the default) keeps client tools. + +On a [model alias]({{%relref "features/model-aliases" %}}#profile-aliases) these +keys go inside the alias's `profile` block, which lets several assistants with +different tools share one loaded model. + #### Agent Configuration (`agent`) - **`max_iterations`**: Maximum number of MCP tool execution loop iterations (default: 10). Each iteration allows the model to call tools and receive results before generating the next response. diff --git a/docs/content/features/model-aliases.md b/docs/content/features/model-aliases.md index 292c05374..5a334ef2f 100644 --- a/docs/content/features/model-aliases.md +++ b/docs/content/features/model-aliases.md @@ -22,7 +22,8 @@ alias: my-llama-3 ``` That is the whole config: a `name` (the alias clients call) and an `alias` key -(the target that actually serves the request). +(the target that actually serves the request). An alias may add an +optional [profile](#profile-aliases) block. ## Rules and behavior @@ -41,6 +42,90 @@ That is the whole config: a `name` (the alias clients call) and an `alias` key To serve a name from several models with automatic fallback, use a [failover chain]({{%relref "features/model-failover" %}}). +## Profile aliases + +An alias can also act as a **profile** of its target: a different assistant on +the same loaded model. Add an optional `profile` block; its fields are applied on +top of the target's config for requests to the alias: + +| Field | Effect | +|-------|--------| +| `profile.system_prompt` | Replaces the target's system prompt. A system message sent by the client still takes precedence, as for any model. | +| `profile.mcp` (`remote`, `stdio`) | Replaces the target's MCP servers. | +| `profile.mcp.auto` | Enables the MCP servers on `/v1/chat/completions` without `metadata.mcp_servers`, when MCP is enabled and the caller has the `mcp` permission. See [MCP]({{%relref "features/mcp" %}}). | +| `profile.client_tools: drop` | Discards tools and `tool_choice` sent by the client. | +| `profile.evidence_check.numbers` | Marks answers that contain numbers without evidence in the request. See [Evidence check](#evidence-check). | +| `profile.evidence_check.note` | Text appended by the evidence check. Defaults to an English note. | + +```yaml +name: researcher +alias: my-llama-3 +profile: + system_prompt: You are a careful research assistant. Cite your sources. + client_tools: drop + mcp: + auto: true + remote: | + {"mcpServers": {"search": {"url": "http://search.internal/mcp"}}} +``` + +Requests to `researcher` run on the **same backend process** as `my-llama-3`: +the model is not loaded a second time, however many profiles point at it. A +separate model config with its own `backend` and model file would load the +weights again. Fields the profile does not set are taken from the target. + +The block is opt-in. Only fields inside `profile` are applied: top-level fields +on an alias (`system_prompt`, `mcp`, sampling parameters, templates, ...) are +still ignored, so an alias without a `profile` block behaves exactly as before. +A `profile` block on a model that is not an alias is rejected. + +MCP sessions of a profile with its own servers are kept apart from the target's, +so profiles with different tool sets do not interfere with each other. +`GET /v1/mcp/servers/` (and the `prompts` and `resources` endpoints) report +the servers a chat request for the alias would use, so a profile's tool set can +be checked without calling the model. + +### Evidence check + +A profile with server-side MCP tools returns only the final answer to the +client; the tool calls and their results stay on the server. When the client +sends the conversation back on the next turn, that tool trail is missing, and a +model may answer a follow-up question with a number it never looked up instead +of calling the tool again. The evidence check marks such answers at the output, +independently of the prompt: + +```yaml +name: bookkeeper +alias: my-llama-3 +profile: + mcp: + auto: true + remote: | + {"mcpServers": {"ledger": {"url": "http://ledger.internal/mcp"}}} + evidence_check: + numbers: true + # optional, defaults to the note below + note: "_Note: this answer contains numbers that no tool returned in this turn — not verified._" +``` + +The rule is deterministic and applies to each request on its own: + +1. If an MCP tool call succeeded on the server during this request, the answer + is left unchanged. A call that failed (the tool could not be called or + returned an error to LocalAI) delivers no evidence and does not count. +2. Otherwise every number in the final answer (a digit sequence, optionally with + one `.` or `,` decimal part) is looked up among the numbers of the request as + received: the profile's system prompt and the content of every system, user, + assistant and tool message. `3,5` and `3.5` count as the same number. +3. If at least one number of the answer is not found, the note is appended after + a blank line. In a streamed response it arrives as one additional content + chunk before the final chunk. + +Answers that hand tool calls to the client are not checked. The check is off +unless `numbers: true` is set, and `note` without `numbers: true` is rejected. +It flags, it does not prove: a number the client happened to mention is +accepted even if the answer uses it wrongly. + ## Managing aliases You can create, swap, and remove aliases from any of the management surfaces. @@ -96,6 +181,7 @@ time. See [Scheduling a model alias]({{%relref "features/distributed-mode" %}}#s ## Limits -Aliases are a static 1:1 redirect. For classifier-based or load-balanced +Aliases are a static 1:1 redirect; a profile block changes only the fields listed +above, not the model or its loading options. For classifier-based or load-balanced selection across several downstream models, use the intelligent router in the [Middleware]({{%relref "operations/middleware" %}}) feature instead.