feat(config): let model aliases carry an opt-in profile block (system prompt, MCP, client tools) (#12661)

* feat(config): let model aliases carry an opt-in profile block

An alias can now act as a profile of its target through an explicit
`profile:` block carrying system_prompt, mcp (remote/stdio servers, plus
the new mcp.auto) and the new client_tools. ResolveAlias applies only
that block on top of a copy of the target, while Name, Model, Backend
and the persisted revision stay the target's. The loader key is
therefore unchanged and every profile runs on the target's backend
process; today a persona needs a separate model config, which loads the
weights a second time.

The block is opt-in so existing aliases never change: top-level fields
on an alias stay ignored, and an alias without a profile block resolves
to exactly the target. A profile block on a non-alias config is
rejected.

A profile with its own MCP servers caches its sessions under the alias
name (MCPSessionKey), because the cache slot keyed on the target's name
is invalidated whenever its config hash changes and two server sets
would tear each other's sessions down.

mcp.auto enables all configured servers on /v1/chat/completions when the
request does not name any in metadata.mcp_servers, which plain OpenAI
clients cannot send. client_tools: drop discards client tools and
tool_choice before the request is merged (chat completions, responses,
Anthropic messages), so only the model's own MCP tools are offered.
Both keys work on any model and inside an alias's profile block.

Refs #12548

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

* fix(openai): only auto-enable model MCP when MCP is enabled and permitted

mcp.auto turns a plain /v1/chat/completions request into a server-side
MCP request. It now applies the same controls as the dedicated MCP
routes before any server is selected: the instance switch
(LOCALAI_DISABLE_MCP) and, with auth on, the caller's "mcp" feature,
checked with auth.HasFeatureAccess against the auth database that the
chat route now passes to ChatEndpoint.

A denied request is answered as a plain chat, without contacting a
server or adding tools, rather than rejected: the client did not ask
for MCP, it only reached a model that offers it.

Endpoint specs drive ChatEndpoint against a counting MCP fixture server:
MCP disabled and a user with chat but without mcp see no tools and no
connection; MCP enabled and a user with mcp get the fixture tool. The
user specs need the auth build tag (SQLite auth database).

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

* fix(mcp): resolve profile aliases in the MCP discovery endpoints

GET /v1/mcp/servers/:model, /v1/mcp/prompts/:model and
/v1/mcp/resources/:model looked the model up with GetModelConfig, which
returns the alias stub. A profile alias carries its servers in its
profile block, so the stub has none of its own and the endpoints
reported an empty list for an alias whose chat requests do use tools.

Resolve the alias the way the request middleware does before reading the
MCP config, and open the sessions under MCPSessionKey so discovery shares
the profile's session cache slot instead of tearing down the target's.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

* feat(config): opt-in evidence check for profile aliases

A profile with server-side MCP tools (profile.mcp, mcp.auto) runs its
tools on the server and returns only the answer text. On the next turn
the client sends the history back without those tool calls and results,
and the model then tends to skip the tool and answer from nothing. In a
measurement on a follow-up question, 0 of 12 turns called the tool and
the answers carried invented numbers (e.g. "0" where the true value was
9); with the tool trail kept in the history the tool was called 12 of
12 times. A prompt instruction does not reliably prevent this, so the
check sits at the output.

New profile keys:

  profile:
    evidence_check:
      numbers: true   # enables the check
      note: "..."     # optional, defaults to an English note

Rule, per request and stateless: when the check is on, no MCP tool call
succeeded on the server during this request (a call that returned an
error delivers no evidence and does not count), and the final answer contains a
number (\b\d+([.,]\d+)?\b) that occurs in none of the request's input
(the profile's system prompt and every message as received, before
compression), the note is appended after a blank line. "3,5" and "3.5"
count as the same number. In a streamed response the note is one extra
content delta before the finish chunk. Choices that hand tool calls to
the client are not checked. The setting is only read from a profile
block, so plain models and aliases behave as before; a note without
numbers: true is rejected at validation.

Tests (Ginkgo): the number rule (invented number, number from the input,
no numbers, comma/dot decimals, system prompt as evidence, tool ran),
the overlay into the resolved config with plain alias, target and a
top-level key as counter-checks, validation, and ChatEndpoint specs on a
stubbed backend for both paths: note on an invented number, none for a
number from the request, none for a profile without the check, none
after a successful MCP tool call against an in-process MCP server, and
the note again after a failed tool call (arguments that are not JSON).

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

---------

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
This commit is contained in:
Stefan Walcz authored and GitHub committed 2026-10-19 23:42:40 +02:00
1 parent d768f1f973
commit b56b903c92
21 files changed
+1623 -31

No files matched your search

+69 -1
View File
@@ -414,11 +414,64 @@ func DefaultRegistry() map[string]FieldMetaOverride {
"alias": {
Section: "alias",
Label: "Alias target",
Description: "Redirect all traffic for this model to another configured model. When set, every other field on this config is ignored and requests are served by the target model.",
Description: "Redirect all traffic for this model to another configured model. When set, every other field on this config is ignored, except the optional profile block, and requests are served by the target model.",
Component: "model-select",
Order: 0,
},
"profile.system_prompt": {
Section: "alias",
Label: "Profile system prompt",
Description: "Alias only. Replaces the target's system prompt for requests to this alias; the target's backend process is reused.",
Component: "textarea",
Order: 1,
},
"profile.mcp.remote": {
Section: "alias",
Label: "Profile remote MCP servers",
Description: "Alias only. Replaces the target's remote MCP servers (same mcpServers format as mcp.remote).",
Component: "code-editor",
Language: "yaml",
Order: 2,
},
"profile.mcp.stdio": {
Section: "alias",
Label: "Profile MCP STDIO servers",
Description: "Alias only. Replaces the target's MCP STDIO servers (same mcpServers format as mcp.stdio).",
Component: "code-editor",
Language: "yaml",
Order: 3,
},
"profile.mcp.auto": {
Section: "alias",
Label: "Profile: enable MCP servers automatically",
Description: "Alias only. Same as mcp.auto, for requests to this alias.",
Component: "toggle",
Order: 4,
},
"profile.client_tools": {
Section: "alias",
Label: "Profile client tools",
Description: "Alias only. Same as client_tools, for requests to this alias.",
Component: "select",
Options: []FieldOption{{Value: "", Label: "Keep"}, {Value: "drop", Label: "Drop"}},
Order: 5,
},
"profile.evidence_check.numbers": {
Section: "alias",
Label: "Profile: mark unverified numbers",
Description: "Alias only. When no MCP tool ran in the request and the answer contains a number found in no input message, append a note marking it as not verified.",
Component: "toggle",
Order: 6,
},
"profile.evidence_check.note": {
Section: "alias",
Label: "Profile: unverified-number note",
Description: "Alias only. Text appended by profile.evidence_check.numbers. Empty uses the built-in English note.",
Component: "input",
Order: 7,
},
// --- Failover ---
"failover.targets": {
Section: "failover",
@@ -909,6 +962,21 @@ func DefaultRegistry() map[string]FieldMetaOverride {
Language: "yaml",
Order: 131,
},
"mcp.auto": {
Section: "mcp",
Label: "Enable MCP servers automatically",
Description: "Enable every configured MCP server on /v1/chat/completions when the request does not select servers via metadata.mcp_servers, so plain OpenAI clients get the server-side tools.",
Component: "toggle",
Order: 132,
},
"client_tools": {
Section: "mcp",
Label: "Client tools",
Description: "What to do with tools sent by the client. Empty keeps them; \"drop\" discards them (and tool_choice) so only the model's own MCP tools are offered.",
Component: "select",
Options: []FieldOption{{Value: "", Label: "Keep"}, {Value: "drop", Label: "Drop"}},
Order: 133,
},
// --- TTS ---
"tts.voice_cloning": {
+163 -5
View File
@@ -60,6 +60,9 @@ type TTSVoice struct {
type ModelConfig struct {
modelConfigFile string `yaml:"-" json:"-"`
modelTemplate string `yaml:"-" json:"-"`
// evidenceCheck is set only from a profile alias's block, so a plain
// model config cannot switch it on by a top-level key.
evidenceCheck EvidenceCheckConfig `yaml:"-" json:"-"`
// persistedConfigRevision is the revision of this model's persisted
// configuration, stamped when the loader materializes it and therefore
// before any per-request override is merged in. The request pipeline
@@ -68,17 +71,31 @@ type ModelConfig struct {
// the only way the controller sees one revision per configuration rather
// than one per request body. Unexported, so it never enters the hash it
// describes and never reaches YAML or JSON.
persistedConfigRevision string `yaml:"-" json:"-"`
persistedConfigRevision string `yaml:"-" json:"-"`
// mcpSessionKey keys the MCP session cache when it must differ from Name.
// A profile alias that brings its own MCP servers resolves to its target's
// config (so Name, and with it the backend process, stays the target's),
// but its sessions must not share a cache slot with the target's: the
// slot is invalidated on every config change, so two server sets under one
// key would tear each other's sessions down. Unexported: never persisted.
mcpSessionKey string `yaml:"-" json:"-"`
schema.PredictionOptions `yaml:"parameters,omitempty" json:"parameters,omitempty"`
Name string `yaml:"name,omitempty" json:"name,omitempty"`
Artifacts []modelartifacts.Spec `yaml:"artifacts,omitempty" json:"artifacts,omitempty"`
// Alias, when set, makes this config a pure redirect: every request for
// Name is served by the model named here. All other fields are ignored.
// The target must be an existing, non-alias model (enforced at load and
// at create/swap time). See docs/content for Model Aliases.
// Name is served by the model named here. All other fields are ignored,
// except the opt-in Profile block. The target must be an existing,
// non-alias model (enforced at load and at create/swap time). See
// docs/content for Model Aliases.
Alias string `yaml:"alias,omitempty" json:"alias,omitempty"`
// Profile is only valid on an alias. Its fields are applied on top of the
// target's config for requests to the alias, while the request still runs
// on the target's backend process. Explicit block rather than honouring
// top-level fields so existing aliases keep behaving as pure redirects.
Profile ProfileConfig `yaml:"profile,omitempty" json:"profile,omitempty"`
// Failover makes this config a failover chain over other models. Like an
// alias it has no backend of its own.
Failover *FailoverConfig `yaml:"failover,omitempty" json:"failover,omitempty"`
@@ -169,7 +186,13 @@ type ModelConfig struct {
MCP MCPConfig `yaml:"mcp,omitempty" json:"mcp,omitempty"`
Agent AgentConfig `yaml:"agent,omitempty" json:"agent,omitempty"`
PII PIIConfig `yaml:"pii,omitempty" json:"pii,omitempty"`
// ClientTools controls tools and functions sent by the client. Empty keeps
// them (default). "drop" discards them, together with tool_choice, before
// the request is merged, so only the model's own MCP tools are offered.
ClientTools string `yaml:"client_tools,omitempty" json:"client_tools,omitempty"`
PII PIIConfig `yaml:"pii,omitempty" json:"pii,omitempty"`
// PIIDetection is the detection policy when THIS model is used as a
// PII detector (a token_classify model named in another model's
// pii.detectors). Ignored on models that aren't referenced as
@@ -598,6 +621,84 @@ func (c *ModelConfig) HasRouter() bool {
// Value receiver so it is callable on non-addressable config values too.
func (c ModelConfig) IsAlias() bool { return c.Alias != "" }
// ClientToolsDrop is the client_tools value that discards client-sent tools.
const ClientToolsDrop = "drop"
// DropsClientTools reports whether tools sent by the client are discarded.
func (c ModelConfig) DropsClientTools() bool { return c.ClientTools == ClientToolsDrop }
// MCPSessionKey is the name the MCP session cache uses for this config. It is
// Name unless a profile alias replaced the MCP servers of its target.
func (c ModelConfig) MCPSessionKey() string {
if c.mcpSessionKey != "" {
return c.mcpSessionKey
}
return c.Name
}
// @Description Profile overlay of a model alias
type ProfileConfig struct {
// SystemPrompt replaces the target's system prompt.
SystemPrompt string `yaml:"system_prompt,omitempty" json:"system_prompt,omitempty"`
// MCP replaces the target's MCP servers when it lists any; MCP.Auto
// enables automatic activation either way.
MCP MCPConfig `yaml:"mcp,omitempty" json:"mcp,omitempty"`
// ClientTools overrides the target's client_tools policy.
ClientTools string `yaml:"client_tools,omitempty" json:"client_tools,omitempty"`
// EvidenceCheck marks answers whose numbers have no evidence in the request.
EvidenceCheck EvidenceCheckConfig `yaml:"evidence_check,omitempty" json:"evidence_check,omitempty"`
}
// DefaultEvidenceNote is appended when evidence_check.note is empty.
const DefaultEvidenceNote = "_Note: this answer contains numbers that no tool returned in this turn — not verified._"
// @Description Output check of a profile alias
type EvidenceCheckConfig struct {
// Numbers enables the check for numbers in the final answer.
Numbers bool `yaml:"numbers,omitempty" json:"numbers,omitempty"`
// Note replaces DefaultEvidenceNote.
Note string `yaml:"note,omitempty" json:"note,omitempty"`
}
// EvidenceCheck returns the evidence check a profile alias applied to this
// resolved config; it is zero for every other config.
func (c ModelConfig) EvidenceCheck() EvidenceCheckConfig { return c.evidenceCheck }
// EvidenceNote returns the configured note or the default one.
func (e EvidenceCheckConfig) EvidenceNote() string {
if e.Note != "" {
return e.Note
}
return DefaultEvidenceNote
}
// IsZero reports whether the block sets nothing.
func (p ProfileConfig) IsZero() bool { return p == ProfileConfig{} }
// applyProfileOverlay merges an alias's profile block into a copy of its
// target. Name, Model, Backend and the persisted revision stay the target's:
// they form the loader key, so the alias keeps sharing the target's backend
// process instead of loading the weights a second time. Only the profile block
// is read; any other field on the alias stays ignored, as for a plain alias.
func (c *ModelConfig) applyProfileOverlay(alias *ModelConfig) {
p := alias.Profile
if p.SystemPrompt != "" {
c.SystemPrompt = p.SystemPrompt
}
if p.MCP.HasMCPServers() {
c.MCP.Servers = p.MCP.Servers
c.MCP.Stdio = p.MCP.Stdio
c.mcpSessionKey = alias.Name
}
if p.MCP.Auto {
c.MCP.Auto = true
}
if p.ClientTools != "" {
c.ClientTools = p.ClientTools
}
c.evidenceCheck = p.EvidenceCheck
}
// @Description PII filtering configuration. PII redaction is per-model so
// that local models don't pay the latency or behaviour change of regex
// scanning, while cloud-bound traffic (cloud-proxy backend) can default to
@@ -800,6 +901,11 @@ func (c *ModelConfig) IsPatternDetector() bool {
type MCPConfig struct {
Servers string `yaml:"remote,omitempty" json:"remote,omitempty"`
Stdio string `yaml:"stdio,omitempty" json:"stdio,omitempty"`
// Auto enables every configured server on /v1/chat/completions when the
// request does not select servers via metadata.mcp_servers. Off by
// default: plain OpenAI clients cannot send that metadata, so without it
// they never reach the model's server-side tools.
Auto bool `yaml:"auto,omitempty" json:"auto,omitempty"`
}
// @Description Agent configuration
@@ -821,6 +927,24 @@ func (c MCPConfig) HasMCPServers() bool {
return c.Servers != "" || c.Stdio != ""
}
// ServerNames returns the names of all configured remote and stdio servers,
// sorted so the enabled set does not depend on map iteration order.
func (c *MCPConfig) ServerNames() ([]string, error) {
remote, stdio, err := c.MCPConfigFromYAML()
if err != nil {
return nil, err
}
names := make([]string, 0, len(remote.Servers)+len(stdio.Servers))
for name := range remote.Servers {
names = append(names, name)
}
for name := range stdio.Servers {
names = append(names, name)
}
slices.Sort(names)
return names, nil
}
func (c *MCPConfig) MCPConfigFromYAML() (MCPGenericConfig[MCPRemoteServers], MCPGenericConfig[MCPSTDIOServers], error) {
var remote MCPGenericConfig[MCPRemoteServers]
var stdio MCPGenericConfig[MCPSTDIOServers]
@@ -1867,9 +1991,16 @@ func (c *ModelConfig) Validate() (bool, error) {
if c.Backend != "" || c.Model != "" {
return false, fmt.Errorf("alias config %q must not set backend or parameters.model: an alias is a pure redirect", c.Name)
}
if err := c.validateProfileFields(); err != nil {
return false, err
}
return true, nil
}
if err := c.validateProfileFields(); err != nil {
return false, err
}
downloadedFileNames := []string{}
for _, f := range c.DownloadFiles {
downloadedFileNames = append(downloadedFileNames, f.Filename)
@@ -2118,6 +2249,33 @@ func (c *ModelConfig) GetModelConfigFile() string {
return c.modelConfigFile
}
// validateProfileFields checks client_tools and the profile block. It runs for
// aliases too, which otherwise skip validation: their profile is used.
func (c *ModelConfig) validateProfileFields() error {
for key, value := range map[string]string{"client_tools": c.ClientTools, "profile.client_tools": c.Profile.ClientTools} {
switch value {
case "", ClientToolsDrop:
default:
return fmt.Errorf("%s: unknown value %q (allowed: %q)", key, value, ClientToolsDrop)
}
}
if c.Profile.IsZero() {
return nil
}
if !c.IsAlias() {
return fmt.Errorf("profile is only valid on an alias config (model %q has no alias)", c.Name)
}
if c.Profile.MCP.HasMCPServers() {
if _, _, err := c.Profile.MCP.MCPConfigFromYAML(); err != nil {
return fmt.Errorf("profile: invalid MCP configuration: %w", err)
}
}
if ec := c.Profile.EvidenceCheck; ec.Note != "" && !ec.Numbers {
return fmt.Errorf("profile.evidence_check.note has no effect without numbers: true")
}
return nil
}
// PersistedConfigRevision returns the revision stamped when this configuration
// was loaded, or "" when it was never stamped (a config synthesized outside the
// loader). Callers that need a revision for a request must prefer this over
+4 -1
View File
@@ -572,7 +572,9 @@ func (bcl *ModelConfigLoader) UpdateModelConfig(m string, updater func(*ModelCon
// ResolveAlias follows a one-hop alias to its target config. Returns
// (resolved, wasAlias, err). Non-alias configs return (cfg, false, nil)
// unchanged. Strict: the target must exist and must not itself be an alias
// (chains are rejected). The returned config is a copy of the target.
// (chains are rejected). The returned config is a copy of the target with the
// alias's opt-in profile block applied on top; an alias without a profile
// block yields a plain copy of the target.
func (bcl *ModelConfigLoader) ResolveAlias(cfg *ModelConfig) (*ModelConfig, bool, error) {
if cfg == nil || !cfg.IsAlias() {
return cfg, false, nil
@@ -584,6 +586,7 @@ func (bcl *ModelConfigLoader) ResolveAlias(cfg *ModelConfig) (*ModelConfig, bool
if target.IsAlias() {
return nil, true, fmt.Errorf("alias %q points to another alias %q (chains are not allowed)", cfg.Name, cfg.Alias)
}
target.applyProfileOverlay(cfg)
return &target, true, nil
}
@@ -0,0 +1,239 @@
package config_test
import (
"os"
"path/filepath"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/mudler/LocalAI/core/config"
)
// A profile alias carries an opt-in profile block (system_prompt, mcp,
// client_tools) that is applied on top of its target while the target's loader
// identity is kept, so several assistant personas share one loaded model
// instead of each loading the weights again.
var _ = Describe("Profile alias overlay", func() {
const target = `name: base-llm
backend: llama-cpp
system_prompt: base prompt
parameters:
model: base-llm.gguf
mcp:
remote: |
{"mcpServers": {"base-tools": {"url": "http://base.invalid/mcp"}}}
`
const profile = `name: researcher
alias: base-llm
profile:
system_prompt: You are a research assistant.
client_tools: drop
mcp:
auto: true
remote: |
{"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}}
stdio: |
{"mcpServers": {"notes": {"command": "notes-mcp"}}}
`
const plain = `name: gpt-4
alias: base-llm
`
// Top-level fields on an alias were always ignored; without a profile
// block they must stay ignored, so existing aliases never change.
const legacy = `name: legacy
alias: base-llm
system_prompt: ignored prompt
client_tools: drop
mcp:
auto: true
remote: |
{"mcpServers": {"ignored": {"url": "http://ignored.invalid/mcp"}}}
`
var cl *config.ModelConfigLoader
BeforeEach(func() {
dir := GinkgoT().TempDir()
Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "researcher.yaml"), []byte(profile), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "gpt-4.yaml"), []byte(plain), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "legacy.yaml"), []byte(legacy), 0644)).To(Succeed())
cl = config.NewModelConfigLoader(dir)
Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed())
})
resolve := func(name string) *config.ModelConfig {
cfg, ok := cl.GetModelConfig(name)
Expect(ok).To(BeTrue(), name)
resolved, wasAlias, err := cl.ResolveAlias(&cfg)
Expect(err).NotTo(HaveOccurred())
Expect(wasAlias).To(BeTrue())
return resolved
}
It("applies the alias's system prompt, MCP servers and tool policy", func() {
resolved := resolve("researcher")
Expect(resolved.SystemPrompt).To(Equal("You are a research assistant."))
Expect(resolved.DropsClientTools()).To(BeTrue())
Expect(resolved.MCP.Auto).To(BeTrue())
names, err := resolved.MCP.ServerNames()
Expect(err).NotTo(HaveOccurred())
// The alias's servers replace the target's; nothing of the target's leaks in.
Expect(names).To(Equal([]string{"notes", "web"}))
// Sessions are cached apart from the target's, which has other servers.
Expect(resolved.MCPSessionKey()).To(Equal("researcher"))
})
It("keeps the target's loader identity, so no second model load happens", func() {
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
resolved := resolve("researcher")
// ModelID, Backend, Model and the persisted revision are what the
// model loader and the controller key a backend process on.
Expect(resolved.ModelID()).To(Equal("base-llm"))
Expect(resolved.Backend).To(Equal(base.Backend))
Expect(resolved.Model).To(Equal(base.Model))
Expect(resolved.PersistedConfigRevision()).NotTo(BeEmpty())
Expect(resolved.PersistedConfigRevision()).To(Equal(base.PersistedConfigRevision()))
Expect(resolve("gpt-4").ModelID()).To(Equal(resolved.ModelID()))
})
It("does not leak the overlay into the stored target", func() {
_ = resolve("researcher")
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
Expect(base.SystemPrompt).To(Equal("base prompt"))
Expect(base.DropsClientTools()).To(BeFalse())
Expect(base.MCP.Auto).To(BeFalse())
Expect(base.MCPSessionKey()).To(Equal("base-llm"))
names, err := base.MCP.ServerNames()
Expect(err).NotTo(HaveOccurred())
Expect(names).To(Equal([]string{"base-tools"}))
})
It("resolves an alias without overlay fields to exactly the target", func() {
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
resolved := resolve("gpt-4")
Expect(*resolved).To(Equal(base))
Expect(resolved.MCPSessionKey()).To(Equal("base-llm"))
})
It("ignores top-level system_prompt, mcp and client_tools on an alias, as before", func() {
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
resolved := resolve("legacy")
Expect(*resolved).To(Equal(base))
Expect(resolved.SystemPrompt).To(Equal("base prompt"))
Expect(resolved.DropsClientTools()).To(BeFalse())
Expect(resolved.MCP.Auto).To(BeFalse())
Expect(resolved.MCPSessionKey()).To(Equal("base-llm"))
})
It("keeps the target's servers and session key when the profile only sets mcp.auto", func() {
dir := GinkgoT().TempDir()
Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "auto.yaml"), []byte("name: auto\nalias: base-llm\nprofile:\n mcp:\n auto: true\n"), 0644)).To(Succeed())
cl = config.NewModelConfigLoader(dir)
Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed())
resolved := resolve("auto")
Expect(resolved.MCP.Auto).To(BeTrue())
names, err := resolved.MCP.ServerNames()
Expect(err).NotTo(HaveOccurred())
Expect(names).To(Equal([]string{"base-tools"}))
Expect(resolved.MCPSessionKey()).To(Equal("base-llm"))
})
It("carries evidence_check into the resolved config and nowhere else", func() {
dir := GinkgoT().TempDir()
Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "checked.yaml"), []byte("name: checked\nalias: base-llm\nprofile:\n evidence_check:\n numbers: true\n note: custom note\n"), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "default-note.yaml"), []byte("name: default-note\nalias: base-llm\nprofile:\n evidence_check:\n numbers: true\n"), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "gpt-4.yaml"), []byte(plain), 0644)).To(Succeed())
cl = config.NewModelConfigLoader(dir)
Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed())
checked := resolve("checked").EvidenceCheck()
Expect(checked.Numbers).To(BeTrue())
Expect(checked.EvidenceNote()).To(Equal("custom note"))
Expect(resolve("default-note").EvidenceCheck().EvidenceNote()).To(Equal(config.DefaultEvidenceNote))
// A plain alias and the target itself stay unchecked.
Expect(resolve("gpt-4").EvidenceCheck().Numbers).To(BeFalse())
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
Expect(base.EvidenceCheck().Numbers).To(BeFalse())
})
It("ignores a top-level evidence_check on a model, which only a profile can set", func() {
dir := GinkgoT().TempDir()
Expect(os.WriteFile(filepath.Join(dir, "base-llm.yaml"), []byte(target+"evidence_check:\n numbers: true\n"), 0644)).To(Succeed())
cl = config.NewModelConfigLoader(dir)
Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed())
base, ok := cl.GetModelConfig("base-llm")
Expect(ok).To(BeTrue())
Expect(base.EvidenceCheck().Numbers).To(BeFalse())
})
Context("validation", func() {
It("rejects an unknown profile.client_tools value", func() {
cfg := config.ModelConfig{Name: "p", Alias: "base-llm"}
cfg.Profile.ClientTools = "ignore"
ok, err := cfg.Validate()
Expect(ok).To(BeFalse())
Expect(err).To(MatchError(ContainSubstring("profile.client_tools")))
})
It("rejects a profile whose MCP config does not parse", func() {
cfg := config.ModelConfig{Name: "p", Alias: "base-llm"}
cfg.Profile.MCP.Servers = "{not yaml: ["
ok, err := cfg.Validate()
Expect(ok).To(BeFalse())
Expect(err).To(MatchError(ContainSubstring("invalid MCP configuration")))
})
It("rejects an evidence_check note without numbers: true", func() {
cfg := config.ModelConfig{Name: "p", Alias: "base-llm"}
cfg.Profile.EvidenceCheck.Note = "unverified"
ok, err := cfg.Validate()
Expect(ok).To(BeFalse())
Expect(err).To(MatchError(ContainSubstring("profile.evidence_check.note")))
cfg.Profile.EvidenceCheck.Numbers = true
ok, err = cfg.Validate()
Expect(err).NotTo(HaveOccurred())
Expect(ok).To(BeTrue())
})
It("rejects a profile block on a model that is not an alias", func() {
cfg := config.ModelConfig{Name: "m", Backend: "llama-cpp"}
cfg.Profile.SystemPrompt = "persona"
ok, err := cfg.Validate()
Expect(ok).To(BeFalse())
Expect(err).To(MatchError(ContainSubstring("only valid on an alias")))
})
It("accepts a well-formed profile alias and a plain alias", func() {
cfg := config.ModelConfig{Name: "p", Alias: "base-llm"}
cfg.Profile.SystemPrompt = "persona"
cfg.Profile.ClientTools = config.ClientToolsDrop
cfg.Profile.MCP.Servers = `{"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}}`
ok, err := cfg.Validate()
Expect(err).NotTo(HaveOccurred())
Expect(ok).To(BeTrue())
plainCfg := config.ModelConfig{Name: "gpt-4", Alias: "base-llm"}
ok, err = plainCfg.Validate()
Expect(err).NotTo(HaveOccurred())
Expect(ok).To(BeTrue())
})
})
})
+7 -2
View File
@@ -49,6 +49,11 @@ func MessagesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evalu
xlog.Debug("Anthropic Messages endpoint configuration read", "config", cfg)
if cfg.DropsClientTools() {
input.Tools = nil
input.ToolChoice = nil
}
// Cloud-proxy bail. Same shape as the OpenAI chat endpoint —
// forwards via the cloud-proxy gRPC backend.
if cfg.IsCloudProxyBackendPassthrough() {
@@ -70,10 +75,10 @@ func MessagesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evalu
if (len(mcpServers) > 0 || mcpPromptName != "" || len(mcpResourceURIs) > 0) && (cfg.MCP.Servers != "" || cfg.MCP.Stdio != "") {
remote, stdio, mcpErr := cfg.MCP.MCPConfigFromYAML()
if mcpErr == nil {
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.Name, remote, stdio, mcpServers)
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio, mcpServers)
// Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode)
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, mcpServers)
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, mcpServers)
if sessErr == nil && len(namedSessions) > 0 {
mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs)
if mcpCtx != nil {
+10 -4
View File
@@ -17,10 +17,13 @@ func MCPPromptsEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
return echo.ErrBadRequest
}
cfg, exists := cl.GetModelConfig(modelName)
cfg, exists, err := mcpModelConfig(cl, modelName)
if !exists {
return fmt.Errorf("model %q not found", modelName)
}
if err != nil {
return err
}
if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" {
return c.JSON(200, []any{})
@@ -31,7 +34,7 @@ func MCPPromptsEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
return fmt.Errorf("failed to parse MCP config: %w", err)
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
@@ -86,10 +89,13 @@ func MCPGetPromptEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic
return echo.ErrBadRequest
}
cfg, exists := cl.GetModelConfig(modelName)
cfg, exists, err := mcpModelConfig(cl, modelName)
if !exists {
return fmt.Errorf("model %q not found", modelName)
}
if err != nil {
return err
}
if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" {
return fmt.Errorf("no MCP servers configured for model %q", modelName)
@@ -107,7 +113,7 @@ func MCPGetPromptEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic
return fmt.Errorf("failed to parse MCP config: %w", err)
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
+10 -4
View File
@@ -17,10 +17,13 @@ func MCPResourcesEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic
return echo.ErrBadRequest
}
cfg, exists := cl.GetModelConfig(modelName)
cfg, exists, err := mcpModelConfig(cl, modelName)
if !exists {
return fmt.Errorf("model %q not found", modelName)
}
if err != nil {
return err
}
if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" {
return c.JSON(200, []any{})
@@ -31,7 +34,7 @@ func MCPResourcesEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applic
return fmt.Errorf("failed to parse MCP config: %w", err)
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
@@ -73,10 +76,13 @@ func MCPReadResourceEndpoint(cl *config.ModelConfigLoader, appConfig *config.App
return echo.ErrBadRequest
}
cfg, exists := cl.GetModelConfig(modelName)
cfg, exists, err := mcpModelConfig(cl, modelName)
if !exists {
return fmt.Errorf("model %q not found", modelName)
}
if err != nil {
return err
}
if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" {
return fmt.Errorf("no MCP servers configured for model %q", modelName)
@@ -94,7 +100,7 @@ func MCPReadResourceEndpoint(cl *config.ModelConfigLoader, appConfig *config.App
return fmt.Errorf("failed to parse MCP config: %w", err)
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
+28 -5
View File
@@ -20,7 +20,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
return echo.ErrBadRequest
}
cfg, exists := cl.GetModelConfig(modelName)
cfg, exists, err := mcpModelConfig(cl, modelName)
if !exists {
return c.JSON(http.StatusNotFound, map[string]any{
"model": modelName,
@@ -28,6 +28,13 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
"error": fmt.Sprintf("model %q not found", modelName),
})
}
if err != nil {
return c.JSON(http.StatusBadRequest, map[string]any{
"model": modelName,
"servers": []any{},
"error": err.Error(),
})
}
if cfg.MCP.Servers == "" && cfg.MCP.Stdio == "" {
return c.JSON(200, map[string]any{
@@ -48,7 +55,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
// In distributed mode, route discovery through NATS to an agent worker
// that can actually connect to the MCP servers.
if agentControl != nil {
resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.Name, remote, stdio)
resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio)
if err != nil {
return c.JSON(http.StatusOK, map[string]any{
"model": modelName,
@@ -61,7 +68,7 @@ func MCPServersEndpoint(cl *config.ModelConfigLoader, appConfig *config.Applicat
})
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
@@ -105,7 +112,7 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H
// In distributed mode, route discovery through NATS to an agent worker.
if agentControl != nil {
resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.Name, remote, stdio)
resp, err := mcpTools.DiscoverMCPToolsRemote(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio)
if err != nil {
return c.JSON(http.StatusOK, map[string]any{
"model": cfg.Name,
@@ -118,7 +125,7 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H
})
}
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, nil)
namedSessions, err := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, nil)
if err != nil {
return fmt.Errorf("failed to get MCP sessions: %w", err)
}
@@ -135,6 +142,22 @@ func MCPServersEndpointFromMiddleware(agentControl mcpTools.AgentControl) echo.H
}
}
// mcpModelConfig returns the config whose MCP servers a chat request for name
// would use. A profile alias carries its servers in its profile block, so the
// raw alias stub has none of its own; resolving it the way the request
// middleware does keeps these endpoints in line with what chat actually offers.
func mcpModelConfig(cl *config.ModelConfigLoader, name string) (*config.ModelConfig, bool, error) {
cfg, exists := cl.GetModelConfig(name)
if !exists {
return nil, false, nil
}
resolved, _, err := cl.ResolveAlias(&cfg)
if err != nil {
return nil, true, err
}
return resolved, true, nil
}
func unavailableMCPServers(
remote config.MCPGenericConfig[config.MCPRemoteServers],
stdio config.MCPGenericConfig[config.MCPSTDIOServers],
@@ -0,0 +1,102 @@
package localai_test
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"github.com/labstack/echo/v4"
"github.com/modelcontextprotocol/go-sdk/mcp"
"github.com/mudler/LocalAI/core/config"
. "github.com/mudler/LocalAI/core/http/endpoints/localai"
mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
var _ = Describe("MCPServersEndpoint with profile aliases", func() {
var (
tempDir string
fixture *httptest.Server
app *echo.Echo
)
BeforeEach(func() {
var err error
tempDir, err = os.MkdirTemp("", "localai-mcp-profile-test")
Expect(err).ToNot(HaveOccurred())
srv := mcp.NewServer(&mcp.Implementation{Name: "fixture", Version: "v0.0.1"}, nil)
srv.AddTool(&mcp.Tool{
Name: "list_collections",
Description: "fixture tool",
InputSchema: json.RawMessage(`{"type":"object","properties":{}}`),
}, func(context.Context, *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: "ok"}}}, nil
})
fixture = httptest.NewServer(mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return srv }, nil))
write := func(name, body string) {
Expect(os.WriteFile(filepath.Join(tempDir, name+".yaml"), []byte(body), 0644)).To(Succeed())
}
write("real", "name: real\nbackend: llama-cpp\nmodel: foo\n")
write("plain-alias", "name: plain-alias\nalias: real\n")
write("intern", fmt.Sprintf(`name: intern
alias: real
profile:
mcp:
auto: true
remote: |
{"mcpServers": {"intern-set": {"url": %q}}}
`, fixture.URL))
loader := config.NewModelConfigLoader(tempDir)
Expect(loader.LoadModelConfigsFromPath(tempDir)).To(Succeed())
app = echo.New()
app.GET("/v1/mcp/servers/:model", MCPServersEndpoint(loader, &config.ApplicationConfig{}, nil))
})
AfterEach(func() {
// The endpoint caches the session under the profile's name and keeps its
// stream open; close it first or the fixture waits for it on shutdown.
mcpTools.CloseMCPSessions("intern")
fixture.Close()
_ = os.RemoveAll(tempDir)
})
type serverInfo struct {
Name string `json:"name"`
Tools []string `json:"tools"`
Error string `json:"error"`
}
servers := func(model string) []serverInfo {
rec := httptest.NewRecorder()
app.ServeHTTP(rec, httptest.NewRequest("GET", "/v1/mcp/servers/"+model, nil))
Expect(rec.Code).To(Equal(http.StatusOK), rec.Body.String())
var body struct {
Model string `json:"model"`
Servers []serverInfo `json:"servers"`
}
Expect(json.Unmarshal(rec.Body.Bytes(), &body)).To(Succeed())
Expect(body.Model).To(Equal(model))
return body.Servers
}
It("lists the servers and tools a profile alias brings, as chat would use them", func() {
got := servers("intern")
Expect(got).To(HaveLen(1))
Expect(got[0].Name).To(Equal("intern-set"))
Expect(got[0].Error).To(BeEmpty())
Expect(got[0].Tools).To(ConsistOf("list_collections"))
})
It("does not attach the profile's servers to the target or to a plain alias", func() {
Expect(servers("real")).To(BeEmpty())
Expect(servers("plain-alias")).To(BeEmpty())
})
})
+80 -4
View File
@@ -11,6 +11,7 @@ import (
"github.com/labstack/echo/v4"
"github.com/mudler/LocalAI/core/backend"
"github.com/mudler/LocalAI/core/config"
"github.com/mudler/LocalAI/core/http/auth"
mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp"
"github.com/mudler/LocalAI/core/http/middleware"
"github.com/mudler/LocalAI/core/schema"
@@ -23,6 +24,7 @@ import (
"github.com/mudler/LocalAI/pkg/model"
"github.com/mudler/xlog"
"gorm.io/gorm"
)
// messageText returns the textual content of a message, preferring the
@@ -212,13 +214,56 @@ func applyAutoparserOverride(
return []schema.Choice{newChoice}
}
// selectMCPServers returns the MCP servers a chat request enables: the ones
// named in metadata.mcp_servers or, with mcp.auto, every configured server when
// the request names none. mcp.auto exists because plain OpenAI clients cannot
// send that metadata and would otherwise never reach server-side tools. The
// assistant modality brings its own tools and is left alone. autoPermitted is
// consulted only when the automatic path would apply, so plain chat requests
// on models without mcp.auto pay no permission lookup.
func selectMCPServers(cfg *config.ModelConfig, requested []string, assistantMode bool, autoPermitted func() bool) []string {
if len(requested) > 0 || assistantMode || !cfg.MCP.Auto || !cfg.MCP.HasMCPServers() {
return requested
}
if autoPermitted != nil && !autoPermitted() {
return requested
}
names, err := cfg.MCP.ServerNames()
if err != nil {
xlog.Error("Failed to parse MCP config", "error", err)
return requested
}
return names
}
// mcpAutoPermitted reports whether mcp.auto may turn this chat request into a
// server-side MCP request. It applies the same controls as the dedicated MCP
// routes: the instance switch (LOCALAI_DISABLE_MCP) and, when auth is on, the
// caller's "mcp" feature. A denied request is answered as a plain chat rather
// than refused, because the client did not ask for MCP; it only reached a
// model that offers it. Without an auth database only the instance switch
// applies, matching auth.RequireFeature.
func mcpAutoPermitted(c echo.Context, appConfig *config.ApplicationConfig, authDB *gorm.DB) bool {
if appConfig != nil && appConfig.DisableMCP {
return false
}
if authDB == nil {
return true
}
return auth.HasFeatureAccess(authDB, auth.GetUser(c), auth.FeatureMCP)
}
// ChatEndpoint is the OpenAI Completion API endpoint https://platform.openai.com/docs/api-reference/chat/create
// @Summary Generate a chat completions for a given prompt and model.
// @Tags inference
// @Param request body schema.OpenAIRequest true "query params"
// @Success 200 {object} schema.OpenAIResponse "Response"
// @Router /v1/chat/completions [post]
func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, startupOptions *config.ApplicationConfig, agentControl mcpTools.AgentControl, assistantHolder *mcpTools.LocalAIAssistantHolder, compressor middleware.ChatCompressor) echo.HandlerFunc {
func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, startupOptions *config.ApplicationConfig, agentControl mcpTools.AgentControl, assistantHolder *mcpTools.LocalAIAssistantHolder, compressor middleware.ChatCompressor, authDB ...*gorm.DB) echo.HandlerFunc {
var db *gorm.DB
if len(authDB) > 0 {
db = authDB[0]
}
return func(c echo.Context) error {
var textContentToReturn string
id := uuid.New().String()
@@ -313,6 +358,10 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
xlog.Debug("LocalAI Assistant tools injected", "count", len(mcpFuncs))
}
mcpServers = selectMCPServers(config, mcpServers, assistantMode, func() bool {
return mcpAutoPermitted(c, startupOptions, db)
})
// MCP prompt and resource injection (extracted before tool injection)
mcpPromptName, mcpPromptArgs := mcpTools.MCPPromptFromMetadata(input.Metadata)
mcpResourceURIs := mcpTools.MCPResourcesFromMetadata(input.Metadata)
@@ -320,10 +369,10 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
if (len(mcpServers) > 0 || mcpPromptName != "" || len(mcpResourceURIs) > 0) && (config.MCP.Servers != "" || config.MCP.Stdio != "") {
remote, stdio, mcpErr := config.MCP.MCPConfigFromYAML()
if mcpErr == nil {
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, config.Name, remote, stdio, mcpServers)
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, config.MCPSessionKey(), remote, stdio, mcpServers)
// Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode)
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(config.Name, remote, stdio, mcpServers)
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(config.MCPSessionKey(), remote, stdio, mcpServers)
if sessErr == nil && len(namedSessions) > 0 {
mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs)
if mcpCtx != nil {
@@ -350,6 +399,8 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
xlog.Error("Failed to parse MCP config", "error", mcpErr)
}
}
// Before compression, which may summarise away numbers the client sent.
evidence := newEvidenceGate(config, input.Messages)
if err := middleware.CompressChatRequest(c, compressor); err != nil {
return err
}
@@ -586,7 +637,7 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
// We collect the RAW (unfiltered) content so the model's tool-call
// markup keeps parsing correctly even when PII redaction would mask
// substrings.
if (hasMCPToolsStream || config.FunctionsConfig.AutomaticToolParsingFallback) && haveContent {
if (hasMCPToolsStream || config.FunctionsConfig.AutomaticToolParsingFallback || evidence != nil) && haveContent {
collectedContent += rawContent
}
respData, err := json.Marshal(ev)
@@ -683,6 +734,9 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
if toolErr != nil {
xlog.Error("MCP tool execution failed", "tool", tc.FunctionCall.Name, "error", toolErr)
toolResult = fmt.Sprintf("Error: %v", toolErr)
} else {
// A failed call delivers no evidence for the answer.
evidence.markToolExecuted()
}
input.Messages = append(input.Messages, schema.Message{
Role: "tool",
@@ -746,6 +800,23 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
}
}
// The answer has been streamed already, so the note can only
// follow it as one more content delta.
if !toolsCalled {
if note := evidence.suffix(collectedContent); note != "" {
noteChunk := schema.OpenAIResponse{
ID: id,
Created: created,
Model: input.Model,
Choices: []schema.Choice{{Index: 0, Delta: &schema.Message{Content: &note}}},
Object: "chat.completion.chunk",
}
noteData, _ := json.Marshal(noteChunk)
fmt.Fprintf(c.Response().Writer, "data: %s\n\n", noteData)
c.Response().Flush()
}
}
// No MCP tools to execute, send final stop message
finishReason := FinishReasonStop
if toolsCalled && len(input.Tools) > 0 {
@@ -1113,6 +1184,9 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
if toolErr != nil {
xlog.Error("MCP tool execution failed", "tool", tc.FunctionCall.Name, "error", toolErr)
toolResult = fmt.Sprintf("Error: %v", toolErr)
} else {
// A failed call delivers no evidence for the answer.
evidence.markToolExecuted()
}
input.Messages = append(input.Messages, schema.Message{
Role: "tool",
@@ -1143,6 +1217,8 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
}
}
evidence.apply(result)
// No MCP tools to execute (or no MCP tools configured), return response
usage := schema.OpenAIUsage{
PromptTokens: tokenUsage.Prompt,
@@ -0,0 +1,57 @@
//go:build auth
package openai
import (
"github.com/google/uuid"
"github.com/mudler/LocalAI/core/config"
"github.com/mudler/LocalAI/core/http/auth"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"gorm.io/gorm"
)
// With auth on, mcp.auto must honour the caller's "mcp" feature like the
// dedicated MCP routes do; having "chat" alone is not enough.
var _ = Describe("ChatEndpoint mcp.auto user gate", func() {
var (
db *gorm.DB
appConfig *config.ApplicationConfig
)
BeforeEach(func() {
var err error
db, err = auth.InitDB(":memory:")
Expect(err).ToNot(HaveOccurred())
appConfig = &config.ApplicationConfig{}
appConfig.Auth.Enabled = true
})
userWith := func(perms auth.PermissionMap) *auth.User {
user := &auth.User{
ID: uuid.NewString(),
Email: uuid.NewString() + "@example.com",
Provider: auth.ProviderLocal,
Subject: uuid.NewString(),
Role: auth.RoleUser,
Status: auth.StatusActive,
}
Expect(db.Create(user).Error).ToNot(HaveOccurred())
Expect(auth.UpdateUserPermissions(db, user.ID, perms)).To(Succeed())
return user
}
It("stays a plain chat for a user who has chat but not mcp", func() {
fx := newMCPAutoFixture()
user := userWith(auth.PermissionMap{auth.FeatureChat: true, auth.FeatureMCP: false})
Expect(fx.chat(appConfig, db, user)).To(BeEmpty())
Expect(fx.requests.Load()).To(BeZero(), "no MCP server may be contacted")
})
It("enables the model's servers for a user who has mcp", func() {
fx := newMCPAutoFixture()
user := userWith(auth.PermissionMap{auth.FeatureChat: true, auth.FeatureMCP: true})
Expect(fx.chat(appConfig, db, user)).To(ContainElement("lookup"))
Expect(fx.requests.Load()).To(BeNumerically(">", 0))
})
})
@@ -0,0 +1,117 @@
package openai
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"github.com/google/uuid"
"github.com/labstack/echo/v4"
"github.com/modelcontextprotocol/go-sdk/mcp"
"github.com/mudler/LocalAI/core/config"
"github.com/mudler/LocalAI/core/http/auth"
mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp"
"github.com/mudler/LocalAI/core/http/middleware"
"github.com/mudler/LocalAI/core/schema"
compressionservice "github.com/mudler/LocalAI/core/services/compression"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"gorm.io/gorm"
)
// errStopAfterTools ends the chat handler right after the MCP block: context
// compression runs after tool injection and before any backend is touched, so
// the specs can read the final tool list without loading a model.
var errStopAfterTools = errors.New("stop after tool injection")
type stopCompressor struct{}
func (stopCompressor) Transform(context.Context, config.CompressionConfig, int, string, []schema.Message) ([]schema.Message, *compressionservice.Metadata, error) {
return nil, nil, errStopAfterTools
}
// mcpAutoFixture is a remote MCP server with one tool that counts every HTTP
// request it receives, so a spec can prove no connection was attempted.
type mcpAutoFixture struct {
url string
requests atomic.Int32
}
func newMCPAutoFixture() *mcpAutoFixture {
fx := &mcpAutoFixture{}
srv := mcp.NewServer(&mcp.Implementation{Name: "fixture", Version: "v0.0.1"}, nil)
srv.AddTool(&mcp.Tool{
Name: "lookup",
Description: "fixture tool",
InputSchema: json.RawMessage(`{"type":"object","properties":{}}`),
}, func(context.Context, *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
return &mcp.CallToolResult{Content: []mcp.Content{&mcp.TextContent{Text: "ok"}}}, nil
})
handler := mcp.NewStreamableHTTPHandler(func(*http.Request) *mcp.Server { return srv }, nil)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fx.requests.Add(1)
handler.ServeHTTP(w, r)
}))
DeferCleanup(server.Close)
fx.url = server.URL
return fx
}
// chat sends a plain chat request (no MCP metadata) through ChatEndpoint to a
// model with mcp.auto and returns the tool names that reached the backend stage.
func (fx *mcpAutoFixture) chat(appConfig *config.ApplicationConfig, db *gorm.DB, user *auth.User) []string {
modelName := "mcp-auto-gate-" + uuid.NewString()
DeferCleanup(func() { mcpTools.CloseMCPSessions(modelName) })
cfg := &config.ModelConfig{Name: modelName}
cfg.MCP.Servers = `{"mcpServers": {"fixture": {"url": "` + fx.url + `"}}}`
cfg.MCP.Auto = true
cfg.Compression.Enabled = true
input := &schema.OpenAIRequest{
PredictionOptions: schema.PredictionOptions{BasicModelRequest: schema.BasicModelRequest{Model: modelName}},
Messages: []schema.Message{{Role: "user", Content: "hi", StringContent: "hi"}},
}
e := echo.New()
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader("{}"))
c := e.NewContext(req, httptest.NewRecorder())
c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, input)
c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, cfg)
if user != nil {
c.Set("auth_user", user)
}
var dbs []*gorm.DB
if db != nil {
dbs = append(dbs, db)
}
handler := ChatEndpoint(nil, nil, nil, appConfig, nil, nil, stopCompressor{}, dbs...)
Expect(handler(c)).To(MatchError(ContainSubstring(errStopAfterTools.Error())))
var names []string
for _, t := range input.Tools {
names = append(names, t.Function.Name)
}
return names
}
// mcp.auto turns a plain chat request into a server-side MCP request, so it
// must honour the instance switch the dedicated MCP routes honour.
var _ = Describe("ChatEndpoint mcp.auto instance gate", func() {
It("enables the model's servers when MCP is enabled", func() {
fx := newMCPAutoFixture()
Expect(fx.chat(&config.ApplicationConfig{}, nil, nil)).To(ContainElement("lookup"))
Expect(fx.requests.Load()).To(BeNumerically(">", 0))
})
It("stays a plain chat when MCP is disabled on the instance", func() {
fx := newMCPAutoFixture()
Expect(fx.chat(&config.ApplicationConfig{DisableMCP: true}, nil, nil)).To(BeEmpty())
Expect(fx.requests.Load()).To(BeZero(), "no MCP server may be contacted")
})
})
@@ -0,0 +1,41 @@
package openai
import (
"github.com/mudler/LocalAI/core/config"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
// mcp.auto lets plain OpenAI clients, which cannot send metadata.mcp_servers,
// reach the model's server-side tools on /v1/chat/completions.
var _ = Describe("selectMCPServers", func() {
withServers := func(auto bool) *config.ModelConfig {
cfg := &config.ModelConfig{}
cfg.MCP.Servers = `{"mcpServers": {"web": {"url": "http://web.invalid/mcp"}}}`
cfg.MCP.Stdio = `{"mcpServers": {"notes": {"command": "notes-mcp"}}}`
cfg.MCP.Auto = auto
return cfg
}
It("enables every configured server when mcp.auto is set and the request names none", func() {
Expect(selectMCPServers(withServers(true), nil, false, nil)).To(Equal([]string{"notes", "web"}))
})
It("keeps the servers the request selected", func() {
Expect(selectMCPServers(withServers(true), []string{"web"}, false, nil)).To(Equal([]string{"web"}))
})
It("enables nothing without mcp.auto, as before", func() {
Expect(selectMCPServers(withServers(false), nil, false, nil)).To(BeEmpty())
})
It("leaves the assistant modality alone", func() {
Expect(selectMCPServers(withServers(true), nil, true, nil)).To(BeEmpty())
})
It("enables nothing when no servers are configured", func() {
cfg := &config.ModelConfig{}
cfg.MCP.Auto = true
Expect(selectMCPServers(cfg, nil, false, nil)).To(BeEmpty())
})
})
+112
View File
@@ -0,0 +1,112 @@
package openai
import (
"regexp"
"strings"
"github.com/mudler/LocalAI/core/config"
"github.com/mudler/LocalAI/core/schema"
"github.com/mudler/xlog"
)
var numberToken = regexp.MustCompile(`\b\d+(?:[.,]\d+)?\b`)
// numberKey makes "3,5" and "3.5" the same number, so an answer is not
// flagged for writing a decimal with the other separator than its source.
func numberKey(token string) string {
return strings.ReplaceAll(token, ",", ".")
}
// evidenceNumbers collects the numbers of the request as received. It is
// taken before the request is compressed or extended with tool results, so
// it holds only what the client and the configuration supplied.
func evidenceNumbers(systemPrompt string, messages []schema.Message) map[string]struct{} {
numbers := map[string]struct{}{}
add := func(text string) {
for _, token := range numberToken.FindAllString(text, -1) {
numbers[numberKey(token)] = struct{}{}
}
}
add(systemPrompt)
for _, m := range messages {
add(messageText(m))
}
return numbers
}
// hasUnsupportedNumber reports whether answer contains a number that occurs in
// no input message. A successful server-side tool run counts as evidence for
// the whole answer: its results are in the model's context but never reach the
// client, so they cannot be checked against the request.
func hasUnsupportedNumber(answer string, evidence map[string]struct{}, toolExecuted bool) bool {
if toolExecuted {
return false
}
for _, token := range numberToken.FindAllString(answer, -1) {
if _, ok := evidence[numberKey(token)]; !ok {
return true
}
}
return false
}
// evidenceGate carries the per-request state of a profile's evidence check.
// A nil gate is disabled, so callers need no separate enabled check.
type evidenceGate struct {
note string
evidence map[string]struct{}
toolExecuted bool
}
func newEvidenceGate(cfg *config.ModelConfig, messages []schema.Message) *evidenceGate {
check := cfg.EvidenceCheck()
if !check.Numbers {
return nil
}
return &evidenceGate{note: check.EvidenceNote(), evidence: evidenceNumbers(cfg.SystemPrompt, messages)}
}
func (g *evidenceGate) markToolExecuted() {
if g != nil {
g.toolExecuted = true
}
}
// suffix returns the text to append to answer, or "" when it needs none.
func (g *evidenceGate) suffix(answer string) string {
if g == nil || !hasUnsupportedNumber(answer, g.evidence, g.toolExecuted) {
return ""
}
xlog.Debug("Evidence check: answer has numbers without evidence in the request, appending note")
return "\n\n" + g.note
}
// apply appends the note to every final answer choice. Choices that hand tool
// calls to the client are not final answers and stay untouched.
func (g *evidenceGate) apply(choices []schema.Choice) {
if g == nil {
return
}
for i := range choices {
msg := choices[i].Message
if msg == nil || len(msg.ToolCalls) > 0 || msg.FunctionCall != nil {
continue
}
var content string
switch v := msg.Content.(type) {
case string:
content = v
case *string:
if v == nil {
continue
}
content = *v
default:
continue
}
if s := g.suffix(content); s != "" {
content += s
msg.Content = &content
}
}
}
+315
View File
@@ -0,0 +1,315 @@
package openai
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"github.com/google/uuid"
"github.com/labstack/echo/v4"
"github.com/mudler/LocalAI/core/backend"
"github.com/mudler/LocalAI/core/config"
mcpTools "github.com/mudler/LocalAI/core/http/endpoints/mcp"
"github.com/mudler/LocalAI/core/http/middleware"
"github.com/mudler/LocalAI/core/schema"
pb "github.com/mudler/LocalAI/pkg/grpc/proto"
"github.com/mudler/LocalAI/pkg/model"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
func userMessages(texts ...string) []schema.Message {
var msgs []schema.Message
for _, t := range texts {
msgs = append(msgs, schema.Message{Role: "user", Content: t, StringContent: t})
}
return msgs
}
var _ = Describe("hasUnsupportedNumber", func() {
evidence := func(texts ...string) map[string]struct{} {
return evidenceNumbers("", userMessages(texts...))
}
It("flags a number that occurs in no input message", func() {
Expect(hasUnsupportedNumber("There are 0 open invoices.", evidence("How many open invoices are there?"), false)).To(BeTrue())
})
It("does not flag a number the request already contains", func() {
Expect(hasUnsupportedNumber("Yes, all 9 are open.", evidence("Are the 9 invoices open?"), false)).To(BeFalse())
})
It("does not flag an answer without numbers", func() {
Expect(hasUnsupportedNumber("I cannot tell without looking it up.", evidence("How many?"), false)).To(BeFalse())
})
It("matches decimals regardless of the separator", func() {
Expect(hasUnsupportedNumber("The rate is 3,5 percent.", evidence("Is the rate 3.5%?"), false)).To(BeFalse())
Expect(hasUnsupportedNumber("The rate is 3.5 percent.", evidence("Ist der Satz 3,5 %?"), false)).To(BeFalse())
})
It("flags a decimal whose digits differ from the input", func() {
Expect(hasUnsupportedNumber("The rate is 3,7 percent.", evidence("Is the rate 3.5%?"), false)).To(BeTrue())
})
It("does not treat a number inside a longer one as evidence", func() {
Expect(hasUnsupportedNumber("There are 9.", evidence("Invoice 19 is open."), false)).To(BeTrue())
Expect(hasUnsupportedNumber("Invoice 19 is open.", evidence("Invoice 19?"), false)).To(BeFalse())
})
It("takes the system prompt as evidence", func() {
ev := evidenceNumbers("The office opens at 8.", userMessages("When does it open?"))
Expect(hasUnsupportedNumber("At 8.", ev, false)).To(BeFalse())
Expect(hasUnsupportedNumber("At 9.", ev, false)).To(BeTrue())
})
It("does not flag anything once a tool ran in the request", func() {
Expect(hasUnsupportedNumber("There are 7 open invoices.", evidence("How many open invoices?"), true)).To(BeFalse())
Expect(hasUnsupportedNumber("There are 7 open invoices.", evidence("How many open invoices?"), false)).To(BeTrue())
})
})
var _ = Describe("evidenceGate.apply", func() {
gate := func() *evidenceGate {
return &evidenceGate{note: "NOTE", evidence: evidenceNumbers("", userMessages("How many?"))}
}
answer := func(content string) []schema.Choice {
return []schema.Choice{{Message: &schema.Message{Role: "assistant", Content: &content}}}
}
contentOf := func(choices []schema.Choice) string {
return *(choices[0].Message.Content.(*string))
}
It("appends the note to an answer with an invented number", func() {
choices := answer("There are 0.")
gate().apply(choices)
Expect(contentOf(choices)).To(Equal("There are 0.\n\nNOTE"))
})
It("leaves an answer without numbers alone", func() {
choices := answer("Let me check.")
gate().apply(choices)
Expect(contentOf(choices)).To(Equal("Let me check."))
})
It("leaves tool calls for the client alone", func() {
content := "Calling 2 tools."
choices := []schema.Choice{{Message: &schema.Message{Content: &content, ToolCalls: []schema.ToolCall{{ID: "x"}}}}}
gate().apply(choices)
Expect(contentOf(choices)).To(Equal("Calling 2 tools."))
})
It("does nothing when the check is off", func() {
var g *evidenceGate
choices := answer("There are 0.")
g.apply(choices)
Expect(contentOf(choices)).To(Equal("There are 0."))
})
})
// The endpoint specs run the real ChatEndpoint against a stubbed backend and a
// profile alias resolved by the real loader, so they cover the wiring from the
// alias YAML to the response the client sees.
var _ = Describe("ChatEndpoint evidence check", func() {
const note = "UNVERIFIED"
var origInference modelInferenceFunc
BeforeEach(func() { origInference = backend.ModelInferenceFunc })
AfterEach(func() { backend.ModelInferenceFunc = origInference })
// stubBackend answers the n-th inference call with replies[n] (the last one
// repeats), streaming the text through the token callback as well.
stubBackend := func(replies ...backend.LLMResponse) {
call := 0
backend.ModelInferenceFunc = func(
ctx context.Context, s string, messages schema.Messages,
images, videos, audios []string,
loader *model.ModelLoader, c *config.ModelConfig, cl *config.ModelConfigLoader,
o *config.ApplicationConfig,
tokenCallback func(string, backend.TokenUsage) bool,
tools, toolChoice string,
logprobs, topLogprobs *int,
logitBias map[string]float64,
metadata map[string]string,
) (func() (backend.LLMResponse, error), error) {
reply := replies[min(call, len(replies)-1)]
call++
return func() (backend.LLMResponse, error) {
if tokenCallback != nil && reply.Response != "" {
tokenCallback(reply.Response, backend.TokenUsage{})
}
return reply, nil
}, nil
}
}
// resolve writes a target and an alias to a fresh model dir and returns the
// alias resolved the way the request middleware resolves it.
resolve := func(aliasProfile, mcpURL string) *config.ModelConfig {
dir := GinkgoT().TempDir()
name := "evidence-" + uuid.NewString()
target := "name: " + name + "-base\nbackend: llama-cpp\nparameters:\n model: base.gguf\ntemplate:\n use_tokenizer_template: true\n"
alias := "name: " + name + "\nalias: " + name + "-base\n" + aliasProfile
if mcpURL != "" {
alias += " mcp:\n auto: true\n remote: |\n {\"mcpServers\": {\"fixture\": {\"url\": \"" + mcpURL + "\"}}}\n"
}
Expect(os.WriteFile(filepath.Join(dir, "base.yaml"), []byte(target), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(dir, "alias.yaml"), []byte(alias), 0644)).To(Succeed())
cl := config.NewModelConfigLoader(dir)
Expect(cl.LoadModelConfigsFromPath(dir)).To(Succeed())
cfg, ok := cl.GetModelConfig(name)
Expect(ok).To(BeTrue())
resolved, _, err := cl.ResolveAlias(&cfg)
Expect(err).NotTo(HaveOccurred())
DeferCleanup(func() { mcpTools.CloseMCPSessions(resolved.MCPSessionKey()) })
return resolved
}
const withCheck = "profile:\n evidence_check:\n numbers: true\n note: " + note + "\n"
const withoutCheck = "profile:\n system_prompt: plain persona\n"
run := func(cfg *config.ModelConfig, stream bool, question string) *httptest.ResponseRecorder {
ctx, cancel := context.WithCancel(context.Background())
DeferCleanup(cancel)
input := &schema.OpenAIRequest{
PredictionOptions: schema.PredictionOptions{BasicModelRequest: schema.BasicModelRequest{Model: "alias"}},
Messages: userMessages(question),
Stream: stream,
Context: ctx,
Cancel: cancel,
}
e := echo.New()
rec := httptest.NewRecorder()
c := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader("{}")), rec)
c.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, input)
c.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, cfg)
Expect(ChatEndpoint(nil, nil, nil, config.NewApplicationConfig(), nil, nil, nil)(c)).To(Succeed())
return rec
}
answerOf := func(rec *httptest.ResponseRecorder) string {
var resp schema.OpenAIResponse
Expect(json.Unmarshal(rec.Body.Bytes(), &resp)).To(Succeed(), rec.Body.String())
Expect(resp.Choices).To(HaveLen(1))
s, ok := resp.Choices[0].Message.Content.(string)
Expect(ok).To(BeTrue())
return s
}
// streamedAnswer joins the content deltas of an SSE body.
streamedAnswer := func(rec *httptest.ResponseRecorder) string {
var b strings.Builder
for _, line := range strings.Split(rec.Body.String(), "\n") {
data, ok := strings.CutPrefix(line, "data: ")
if !ok || data == "[DONE]" {
continue
}
var chunk schema.OpenAIResponse
if json.Unmarshal([]byte(data), &chunk) != nil || len(chunk.Choices) == 0 || chunk.Choices[0].Delta == nil {
continue
}
if s, ok := chunk.Choices[0].Delta.Content.(string); ok {
b.WriteString(s)
}
}
Expect(rec.Body.String()).To(HaveSuffix("data: [DONE]\n\n"))
return b.String()
}
Context("non-streaming", func() {
It("marks an invented number", func() {
stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."})
Expect(answerOf(run(resolve(withCheck, ""), false, "How many open invoices?"))).
To(Equal("There are 0 open invoices.\n\n" + note))
})
It("leaves a number from the request unmarked", func() {
stubBackend(backend.LLMResponse{Response: "Yes, all 9 are open."})
Expect(answerOf(run(resolve(withCheck, ""), false, "Are the 9 invoices open?"))).
To(Equal("Yes, all 9 are open."))
})
It("leaves a profile without the check unchanged", func() {
stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."})
Expect(answerOf(run(resolve(withoutCheck, ""), false, "How many open invoices?"))).
To(Equal("There are 0 open invoices."))
})
It("leaves the answer unmarked when an MCP tool ran in this request", func() {
fx := newMCPAutoFixture()
stubBackend(
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{}"}}}}},
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}},
)
Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))).
To(Equal("There are 7 open invoices."))
Expect(fx.requests.Load()).To(BeNumerically(">", 0))
})
It("marks an invented number when the MCP tool call failed", func() {
fx := newMCPAutoFixture()
// Arguments that are not JSON make the real executor fail the call.
stubBackend(
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{not json"}}}}},
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}},
)
Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))).
To(Equal("There are 7 open invoices.\n\n" + note))
})
It("marks the answer when MCP tools were offered but none ran", func() {
fx := newMCPAutoFixture()
stubBackend(backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{Content: "There are 0 open invoices."}}})
Expect(answerOf(run(resolve(withCheck, fx.url), false, "How many open invoices?"))).
To(Equal("There are 0 open invoices.\n\n" + note))
})
})
Context("streaming", func() {
It("sends the note as a last content delta", func() {
stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."})
Expect(streamedAnswer(run(resolve(withCheck, ""), true, "How many open invoices?"))).
To(Equal("There are 0 open invoices.\n\n" + note))
})
It("sends no note for a number from the request", func() {
stubBackend(backend.LLMResponse{Response: "Yes, all 9 are open."})
Expect(streamedAnswer(run(resolve(withCheck, ""), true, "Are the 9 invoices open?"))).
To(Equal("Yes, all 9 are open."))
})
It("sends no note when an MCP tool ran in this request", func() {
fx := newMCPAutoFixture()
stubBackend(
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{}"}}}}},
backend.LLMResponse{Response: "There are 7 open invoices.", ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}},
)
rec := run(resolve(withCheck, fx.url), true, "How many open invoices?")
Expect(rec.Body.String()).To(ContainSubstring("mcp_tool_result"))
Expect(streamedAnswer(rec)).To(ContainSubstring("There are 7 open invoices."))
Expect(streamedAnswer(rec)).NotTo(ContainSubstring(note))
})
It("sends the note when the MCP tool call failed", func() {
fx := newMCPAutoFixture()
stubBackend(
backend.LLMResponse{ChatDeltas: []*pb.ChatDelta{{ToolCalls: []*pb.ToolCallDelta{{Id: "call-1", Name: "lookup", Arguments: "{not json"}}}}},
backend.LLMResponse{Response: "There are 7 open invoices.", ChatDeltas: []*pb.ChatDelta{{Content: "There are 7 open invoices."}}},
)
rec := run(resolve(withCheck, fx.url), true, "How many open invoices?")
Expect(rec.Body.String()).To(ContainSubstring("mcp_tool_result"))
Expect(streamedAnswer(rec)).To(HaveSuffix("There are 7 open invoices.\n\n" + note))
})
It("sends no note for a profile without the check", func() {
stubBackend(backend.LLMResponse{Response: "There are 0 open invoices."})
Expect(streamedAnswer(run(resolve(withoutCheck, ""), true, "How many open invoices?"))).
To(Equal("There are 0 open invoices."))
})
})
})
@@ -108,11 +108,11 @@ func ResponsesEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, eval
if !hasMCPRequest {
enabledServers = nil // backward compat: auto-activate all servers
}
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.Name, remote, stdio, enabledServers)
mcpExecutor = mcpTools.NewToolExecutor(c.Request().Context(), agentControl, cfg.MCPSessionKey(), remote, stdio, enabledServers)
// Prompt and resource injection (pre-processing step — resolves locally regardless of distributed mode)
if hasMCPRequest {
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.Name, remote, stdio, mcpServers)
namedSessions, sessErr := mcpTools.NamedSessionsFromMCPConfig(cfg.MCPSessionKey(), remote, stdio, mcpServers)
if sessErr == nil && len(namedSessions) > 0 {
mcpCtx, _ := mcpTools.InjectMCPContext(c.Request().Context(), namedSessions, mcpPromptName, mcpPromptArgs, mcpResourceURIs)
if mcpCtx != nil {
+12
View File
@@ -392,6 +392,14 @@ func extractToolChoiceFunctionName(m map[string]any) string {
}
func mergeOpenAIRequestAndModelConfig(config *config.ModelConfig, input *schema.OpenAIRequest) error {
// Discard client tools before anything below derives functions or a forced
// tool_choice from them; the model's own MCP tools are injected later.
if config.DropsClientTools() {
input.Tools = nil
input.Functions = nil
input.ToolsChoice = nil
input.FunctionCall = nil
}
if input.Echo {
config.Echo = input.Echo
}
@@ -835,6 +843,10 @@ func (re *RequestExtractor) SetOpenResponsesRequest(c echo.Context) (err error)
// MergeOpenResponsesConfig merges request parameters into the model configuration.
func MergeOpenResponsesConfig(config *config.ModelConfig, input *schema.OpenResponsesRequest) error {
if config.DropsClientTools() {
input.Tools = nil
input.ToolChoice = nil
}
// Temperature
if input.Temperature != nil {
config.Temperature = input.Temperature
@@ -0,0 +1,131 @@
package middleware_test
import (
"net/http"
"os"
"path/filepath"
"github.com/labstack/echo/v4"
"github.com/mudler/LocalAI/core/config"
. "github.com/mudler/LocalAI/core/http/middleware"
"github.com/mudler/LocalAI/core/schema"
"github.com/mudler/LocalAI/pkg/model"
"github.com/mudler/LocalAI/pkg/system"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
)
// A profile alias resolves to its target's backend process but serves the
// request with the alias's own system prompt and tool policy. These specs drive
// the production middleware chain of /v1/chat/completions.
var _ = Describe("Profile alias through the chat middleware", func() {
var (
app *echo.Echo
modelDir string
capturedConfig *config.ModelConfig
capturedInput *schema.OpenAIRequest
)
BeforeEach(func() {
var err error
modelDir, err = os.MkdirTemp("", "localai-profile-alias-*")
Expect(err).ToNot(HaveOccurred())
Expect(os.WriteFile(filepath.Join(modelDir, "base.yaml"),
[]byte("name: base\nbackend: llama-cpp\nsystem_prompt: base prompt\n"), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(modelDir, "researcher.yaml"),
[]byte("name: researcher\nalias: base\nprofile:\n system_prompt: persona prompt\n client_tools: drop\n"), 0644)).To(Succeed())
Expect(os.WriteFile(filepath.Join(modelDir, "plain.yaml"),
[]byte("name: plain\nalias: base\nsystem_prompt: ignored\nclient_tools: drop\n"), 0644)).To(Succeed())
ss := &system.SystemState{Model: system.Model{ModelsPath: modelDir}}
appConfig := config.NewApplicationConfig()
appConfig.SystemState = ss
mcl := config.NewModelConfigLoader(modelDir)
Expect(mcl.LoadModelConfigsFromPath(modelDir)).To(Succeed())
re := NewRequestExtractor(mcl, model.NewModelLoader(ss), appConfig)
capturedConfig, capturedInput = nil, nil
app = echo.New()
app.POST("/v1/chat/completions",
func(c echo.Context) error {
capturedConfig, _ = c.Get(CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig)
capturedInput, _ = c.Get(CONTEXT_LOCALS_KEY_LOCALAI_REQUEST).(*schema.OpenAIRequest)
return c.String(http.StatusOK, "ok")
},
re.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.OpenAIRequest) }),
func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
if err := re.SetOpenAIRequest(c); err != nil {
return err
}
return next(c)
}
},
)
})
AfterEach(func() {
_ = os.RemoveAll(modelDir)
})
withTools := func(model string) string {
return `{"model":"` + model + `",` +
`"messages":[{"role":"user","content":"hi"}],` +
`"tools":[{"type":"function","function":{"name":"client_tool"}}],` +
`"tool_choice":{"type":"function","function":{"name":"client_tool"}}}`
}
It("serves the target process with the alias's prompt and drops client tools", func() {
rec := postJSON(app, "/v1/chat/completions", withTools("researcher"))
Expect(rec.Code).To(Equal(http.StatusOK))
Expect(capturedConfig).ToNot(BeNil())
Expect(capturedConfig.ModelID()).To(Equal("base"))
Expect(capturedConfig.SystemPrompt).To(Equal("persona prompt"))
Expect(capturedInput.Tools).To(BeEmpty())
Expect(capturedInput.Functions).To(BeEmpty())
// The client's forced tool_choice must not survive either.
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeFalse())
// The response still echoes the alias name.
Expect(capturedInput.Model).To(Equal("researcher"))
})
It("leaves an alias without profile block exactly as before, even with top-level fields", func() {
rec := postJSON(app, "/v1/chat/completions", withTools("plain"))
Expect(rec.Code).To(Equal(http.StatusOK))
Expect(capturedConfig.ModelID()).To(Equal("base"))
Expect(capturedConfig.SystemPrompt).To(Equal("base prompt"))
Expect(capturedInput.Functions).To(HaveLen(1))
Expect(capturedConfig.ShouldCallSpecificFunction()).To(BeTrue())
Expect(capturedConfig.FunctionToCall()).To(Equal("client_tool"))
})
})
var _ = Describe("MergeOpenResponsesConfig with client_tools", func() {
request := func() *schema.OpenResponsesRequest {
return &schema.OpenResponsesRequest{
Tools: []schema.ORFunctionTool{{Type: "function", Name: "client_tool"}},
ToolChoice: "required",
}
}
It("drops client tools and tool_choice when the config says drop", func() {
cfg := &config.ModelConfig{ClientTools: config.ClientToolsDrop}
req := request()
Expect(MergeOpenResponsesConfig(cfg, req)).To(Succeed())
Expect(req.Tools).To(BeEmpty())
Expect(req.ToolChoice).To(BeNil())
})
It("keeps client tools by default", func() {
cfg := &config.ModelConfig{}
req := request()
Expect(MergeOpenResponsesConfig(cfg, req)).To(Succeed())
Expect(req.Tools).To(HaveLen(1))
Expect(req.ToolChoice).To(Equal("required"))
})
})
+1 -1
View File
@@ -51,7 +51,7 @@ func RegisterOpenAIRoutes(app *echo.Echo,
compressionservice.CounterFunc(tokens.CountMessages),
compressionservice.NewInferenceSummarizer(application.ModelConfigLoader(), application.ModelLoader(), application.ApplicationConfig()),
)
chatHandler := openai.ChatEndpoint(application.ModelConfigLoader(), application.ModelLoader(), application.TemplatesEvaluator(), application.ApplicationConfig(), agentControl, application.LocalAIAssistant(), chatCompressor)
chatHandler := openai.ChatEndpoint(application.ModelConfigLoader(), application.ModelLoader(), application.TemplatesEvaluator(), application.ApplicationConfig(), agentControl, application.LocalAIAssistant(), chatCompressor, application.AuthDB())
chatMiddleware := []echo.MiddlewareFunc{
nodeHeaderMiddleware,
usageMiddleware,
+35
View File
@@ -153,6 +153,41 @@ Configure local command-based MCP servers:
- **`args`**: Array of command-line arguments
- **`env`**: Environment variables (optional)
#### Automatic activation (`auto`)
`/v1/chat/completions` only runs a model's MCP servers when the request selects
them through `metadata.mcp_servers`. Plain OpenAI clients cannot send that field.
Set `auto: true` in the `mcp:` block to enable every configured server when the
request names none:
```yaml
mcp:
auto: true
remote: |
{"mcpServers": {"weather-api": {"url": "https://weather.example.com/mcp"}}}
```
A request that does send `metadata.mcp_servers` still gets exactly the servers it
names. `/v1/responses` already enables all servers when the request sends no
`tools`, and `/mcp/v1/chat/completions` always does; `auto` does not change them.
`auto` only applies when MCP is enabled on the instance (not disabled with
`LOCALAI_DISABLE_MCP`) and, with authentication on, the caller has the `mcp`
permission. Otherwise the request is answered as a plain chat: no server is
contacted and no tools are added. It is not rejected, because the client did not
ask for MCP.
#### Client tools (`client_tools`)
`client_tools: drop` (a top-level model config key) discards the `tools`,
`functions` and `tool_choice` a client sends, so the model is offered only its own
MCP tools. Use it when a front end attaches tools of its own that should not
reach this model. Empty (the default) keeps client tools.
On a [model alias]({{%relref "features/model-aliases" %}}#profile-aliases) these
keys go inside the alias's `profile` block, which lets several assistants with
different tools share one loaded model.
#### Agent Configuration (`agent`)
- **`max_iterations`**: Maximum number of MCP tool execution loop iterations (default: 10). Each iteration allows the model to call tools and receive results before generating the next response.
+88 -2
View File
@@ -22,7 +22,8 @@ alias: my-llama-3
```
That is the whole config: a `name` (the alias clients call) and an `alias` key
(the target that actually serves the request).
(the target that actually serves the request). An alias may add an
optional [profile](#profile-aliases) block.
## Rules and behavior
@@ -41,6 +42,90 @@ That is the whole config: a `name` (the alias clients call) and an `alias` key
To serve a name from several models with automatic fallback, use a [failover
chain]({{%relref "features/model-failover" %}}).
## Profile aliases
An alias can also act as a **profile** of its target: a different assistant on
the same loaded model. Add an optional `profile` block; its fields are applied on
top of the target's config for requests to the alias:
| Field | Effect |
|-------|--------|
| `profile.system_prompt` | Replaces the target's system prompt. A system message sent by the client still takes precedence, as for any model. |
| `profile.mcp` (`remote`, `stdio`) | Replaces the target's MCP servers. |
| `profile.mcp.auto` | Enables the MCP servers on `/v1/chat/completions` without `metadata.mcp_servers`, when MCP is enabled and the caller has the `mcp` permission. See [MCP]({{%relref "features/mcp" %}}). |
| `profile.client_tools: drop` | Discards tools and `tool_choice` sent by the client. |
| `profile.evidence_check.numbers` | Marks answers that contain numbers without evidence in the request. See [Evidence check](#evidence-check). |
| `profile.evidence_check.note` | Text appended by the evidence check. Defaults to an English note. |
```yaml
name: researcher
alias: my-llama-3
profile:
system_prompt: You are a careful research assistant. Cite your sources.
client_tools: drop
mcp:
auto: true
remote: |
{"mcpServers": {"search": {"url": "http://search.internal/mcp"}}}
```
Requests to `researcher` run on the **same backend process** as `my-llama-3`:
the model is not loaded a second time, however many profiles point at it. A
separate model config with its own `backend` and model file would load the
weights again. Fields the profile does not set are taken from the target.
The block is opt-in. Only fields inside `profile` are applied: top-level fields
on an alias (`system_prompt`, `mcp`, sampling parameters, templates, ...) are
still ignored, so an alias without a `profile` block behaves exactly as before.
A `profile` block on a model that is not an alias is rejected.
MCP sessions of a profile with its own servers are kept apart from the target's,
so profiles with different tool sets do not interfere with each other.
`GET /v1/mcp/servers/<alias>` (and the `prompts` and `resources` endpoints) report
the servers a chat request for the alias would use, so a profile's tool set can
be checked without calling the model.
### Evidence check
A profile with server-side MCP tools returns only the final answer to the
client; the tool calls and their results stay on the server. When the client
sends the conversation back on the next turn, that tool trail is missing, and a
model may answer a follow-up question with a number it never looked up instead
of calling the tool again. The evidence check marks such answers at the output,
independently of the prompt:
```yaml
name: bookkeeper
alias: my-llama-3
profile:
mcp:
auto: true
remote: |
{"mcpServers": {"ledger": {"url": "http://ledger.internal/mcp"}}}
evidence_check:
numbers: true
# optional, defaults to the note below
note: "_Note: this answer contains numbers that no tool returned in this turn — not verified._"
```
The rule is deterministic and applies to each request on its own:
1. If an MCP tool call succeeded on the server during this request, the answer
is left unchanged. A call that failed (the tool could not be called or
returned an error to LocalAI) delivers no evidence and does not count.
2. Otherwise every number in the final answer (a digit sequence, optionally with
one `.` or `,` decimal part) is looked up among the numbers of the request as
received: the profile's system prompt and the content of every system, user,
assistant and tool message. `3,5` and `3.5` count as the same number.
3. If at least one number of the answer is not found, the note is appended after
a blank line. In a streamed response it arrives as one additional content
chunk before the final chunk.
Answers that hand tool calls to the client are not checked. The check is off
unless `numbers: true` is set, and `note` without `numbers: true` is rejected.
It flags, it does not prove: a number the client happened to mention is
accepted even if the answer uses it wrongly.
## Managing aliases
You can create, swap, and remove aliases from any of the management surfaces.
@@ -96,6 +181,7 @@ time. See [Scheduling a model alias]({{%relref "features/distributed-mode" %}}#s
## Limits
Aliases are a static 1:1 redirect. For classifier-based or load-balanced
Aliases are a static 1:1 redirect; a profile block changes only the fields listed
above, not the model or its loading options. For classifier-based or load-balanced
selection across several downstream models, use the intelligent router in the
[Middleware]({{%relref "operations/middleware" %}}) feature instead.