feat(distributed): make MCP execution and discovery a selection

mcp.tools.execute and mcp.discovery were the only NATS subjects that
combined a queue group with a reply, and no carrier in this design
provides both. They never needed one: a queue group is a way of choosing
a subscriber, and choosing is a query.

The frontend now lists the approved, non-draining agent nodes, asks the
node_connections table in one joined statement which of those tunnels a
live replica holds, prefers one this replica holds so the call skips the
relay hop, and issues an ordinary control RPC on the path task 4 already
mounted. A peer-held tunnel is reached through the relay. That is a
choice a broker's hidden balancing could not make.

The selection reads presence and nothing else. It is filtered only on
node type and on the two statuses an operator controls, never on a health
verdict written on another clock, because refusing a worker that is
connected and answering is the same defect as picking one that is gone.
An empty fleet answers ErrNoAgentWorker, which is deliberately neither
ErrWorkerUnroutable nor anything cluster.IsWorkerAnswer accepts: nothing
was asked of any worker, so no reap guard may act on it.

A reply carrying an Error is the worker's own answer and is returned
unchanged; it is never offered to a second worker, which would turn "this
MCP server rejected your arguments" into "the fleet is broken" and could
run a tool twice. A call that never reached a worker is retried against a
different pick, at most three times, and whatever error is finally
returned is returned unwrapped so its identity survives the loop.

MCP prompts and resources now answer 501 in distributed mode instead of
an empty 200. They are served only from sessions the frontend holds, and
in distributed mode it holds none. That gap predates the removal of the
bus and is not closed by it; this only stops it being silent.

Agent workers keep every other subject, including nodes.<id>.backend.stop.
Their minted JWT loses the two MCP subjects and keeps a non-empty allow
list, because NATS reads an empty one as no restriction at all.

Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

pkg/natsauth/permissions.go

@@ -29,6 +29,12 @@ func WorkerPermissions(nodeID, nodeType string) (pubAllow, subAllow []string) {
 	case "agent":
 		// Agent workers consume queue workloads; they must not handle backend.install.
 		// Keep this list in sync with the subscriptions in core/cli/agent_worker.go.
+		//
+		// MCP tool execution and discovery are NOT here any more: they are
+		// control RPCs on the tunnel the worker holds, chosen by the frontend
+		// rather than by a queue group. Removing them narrowed this list; it
+		// must never be narrowed to nothing, because NATS reads an EMPTY allow
+		// list as no restriction at all.
 		subAllow = []string{
 			"agent.execute",
 			"agent.*.cancel",
@@ -37,9 +43,7 @@ func WorkerPermissions(nodeID, nodeType string) (pubAllow, subAllow []string) {
 			"jobs.*.cancel",
 			"jobs.*.progress",
 			"jobs.*.result",
-			"jobs.mcp-ci.new", // MCP CI jobs dispatched to agent workers
-			"mcp.tools.execute",
-			"mcp.discovery",
+			"jobs.mcp-ci.new",        // MCP CI jobs dispatched to agent workers
 			prefix + ".backend.stop", // stop events drive MCP session cleanup
 			"staging.*.progress",
 			"_INBOX.>",

pkg/natsauth/permissions_coverage_test.go

@@ -99,15 +99,33 @@ var _ = Describe("WorkerPermissions subject coverage", func() {
 	Context("agent worker", func() {
 		// node_type "agent"; subjects from core/cli/agent_worker.go.
 		pub, sub := natsauth.WorkerPermissions(nodeID, "agent")
-		_ = pub
 
 		subscribed := []string{
 			messaging.SubjectAgentExecute,            // dispatcher (default --agent-subject)
-			messaging.SubjectMCPToolExecute,          // QueueSubscribeReply
-			messaging.SubjectMCPDiscovery,            // QueueSubscribeReply
 			messaging.SubjectMCPCIJobsNew,            // QueueSubscribe — jobs.mcp-ci.new
 			messaging.SubjectNodeBackendStop(nodeID), // Subscribe — MCP session cleanup
 		}
+
+		// The half that catches a narrowing going too far. NATS reads an EMPTY
+		// allow list as NO restriction, so a branch trimmed to nothing does not
+		// lock an agent worker down, it opens the whole account to it.
+		It("keeps the agent worker's allow lists non-empty", func() {
+			Expect(sub).ToNot(BeEmpty(),
+				"an empty allow list is unrestricted in NATS, not restrictive")
+			Expect(pub).ToNot(BeEmpty(),
+				"an empty allow list is unrestricted in NATS, not restrictive")
+		})
+
+		// MCP execution and discovery are control RPCs on the worker's tunnel
+		// now, chosen by the frontend rather than by a queue group. An agent
+		// worker subscribes to neither subject, so a JWT that still granted
+		// them would be granting a subscription nothing serves.
+		for _, subject := range []string{"mcp.tools.execute", "mcp.discovery"} {
+			It("no longer grants an agent worker "+subject, func() {
+				Expect(anyAllows(sub, subject)).To(BeFalse(),
+					"agent JWT sub allow-list %v still covers %s", sub, subject)
+			})
+		}
 		for _, subject := range subscribed {
 			It("allows subscribing to "+subject, func() {
 				Expect(anyAllows(sub, subject)).To(BeTrue(),