feat(pii): apply PII middleware to the Open Responses API (#12708)

* feat(pii): apply PII middleware to the Open Responses API

The per-model PII filter ran on /v1/chat/completions, /v1/completions,
/v1/messages and the Ollama routes, but not on POST /v1/responses. A
model whose policy blocks or masks a secret on the chat route served the
same content unfiltered when it was sent through the Responses API.

Add a piiadapter.OpenResponses adapter and wire pii.RequestMiddleware
into the POST /v1/responses and /responses chain, after the model config
is resolved and merged and before the handler converts input items into
prompt messages. The adapter covers instructions, input as a string, and
in an input item array the text of message and reasoning items (string
content or input_text/output_text parts, including typeless
{role, content} items), function_call arguments and function_call_output
output; structured tool output is scanned as the JSON the handler
forwards. Image, file, audio and video parts are left untouched.

The Responses WebSocket mode (GET /v1/responses) is not HTTP middleware
and cannot filter, so a response.create for a model with PII filtering
enabled is now refused with pii_unsupported_transport instead of being
served unfiltered. The docs list the new route and this limit.

Tests: adapter specs for every covered field, each with a clean-text
counter-check; a route spec driving the real middleware chain with an
in-process pattern detector (block on string input, content part,
instructions and tool output; mask; clean input passes unchanged); and
WebSocket specs proving the refusal and that models without a PII
policy are still served.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

* fix(pii): scan complete Responses tool output arrays

The Responses handler forwards non-string tool output as JSON. Scan
that full encoding so string arrays and extra fields cannot bypass PII
filtering. Cover array scans, redaction, clean values and route blocking.

Document that stored history is resolved after request filtering.

Assisted-by: Codex:gpt-6
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

* feat(pii): report unscanned media on the Open Responses route

Extend block_unscanned_media to /v1/responses: input_image, input_file
and input_audio parts inside message items, and the same types as bare
input items, are reported as unscanned content. Tool output stays
allowed, as it is scanned as JSON text.

Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>

---------

Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>

core/http/endpoints/openresponses/websocket.go

@@ -16,6 +16,7 @@ import (
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/core/http/middleware"
 	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/pii"
 	"github.com/mudler/LocalAI/core/templates"
 	"github.com/mudler/LocalAI/pkg/functions"
 	"github.com/mudler/LocalAI/pkg/model"
@@ -72,6 +73,7 @@ func WebSocketEndpoint(application *application.Application) echo.HandlerFunc {
 	ml := application.ModelLoader()
 	evaluator := application.TemplatesEvaluator()
 	appConfig := application.ApplicationConfig()
+	piiPolicy := application.PIIPolicyResolver()
 
 	return func(c echo.Context) error {
 		owner := ownerFromContext(c)
@@ -96,13 +98,13 @@ func WebSocketEndpoint(application *application.Application) echo.HandlerFunc {
 
 		xlog.Debug("WebSocket Responses connection established", "address", ws.RemoteAddr().String())
 
-		handleWebSocketConnection(connCtx, conn, owner, cl, ml, evaluator, appConfig)
+		handleWebSocketConnection(connCtx, conn, owner, cl, ml, evaluator, appConfig, piiPolicy)
 		return nil
 	}
 }
 
 // handleWebSocketConnection runs the read loop for a single WebSocket connection.
-func handleWebSocketConnection(connCtx context.Context, conn *lockedConn, owner string, cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, appConfig *config.ApplicationConfig) {
+func handleWebSocketConnection(connCtx context.Context, conn *lockedConn, owner string, cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, appConfig *config.ApplicationConfig, piiPolicy pii.PolicyResolver) {
 	// Responses created with store=false remain available only to this WebSocket
 	// connection, matching the Responses WebSocket continuation contract without
 	// leaking zero-data-retention state into the process-wide response store.
@@ -172,7 +174,7 @@ func handleWebSocketConnection(connCtx context.Context, conn *lockedConn, owner
 				releaseOnce.Do(inflight.Unlock)
 			}
 			defer release()
-			handleWSResponseCreate(connCtx, conn, connectionStore, release, owner, wsMsg.Generate, &wsMsg.OpenResponsesRequest, cl, ml, evaluator, appConfig)
+			handleWSResponseCreate(connCtx, conn, connectionStore, release, owner, wsMsg.Generate, &wsMsg.OpenResponsesRequest, cl, ml, evaluator, appConfig, piiPolicy)
 		}()
 	}
 }
@@ -181,7 +183,7 @@ func handleWebSocketConnection(connCtx context.Context, conn *lockedConn, owner
 // It reuses the existing background stream infrastructure: the request is processed via
 // handleBackgroundStream which buffers events into the store, and a forwarder goroutine
 // reads those events and sends them over the WebSocket.
-func handleWSResponseCreate(connCtx context.Context, conn *lockedConn, connectionStore *ResponseStore, release func(), owner string, generate *bool, input *schema.OpenResponsesRequest, cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, appConfig *config.ApplicationConfig) {
+func handleWSResponseCreate(connCtx context.Context, conn *lockedConn, connectionStore *ResponseStore, release func(), owner string, generate *bool, input *schema.OpenResponsesRequest, cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, appConfig *config.ApplicationConfig, piiPolicy pii.PolicyResolver) {
 	createdAt := time.Now().Unix()
 	responseID := fmt.Sprintf("resp_%s", uuid.New().String())
 	fail := func(errType, message, param string) {
@@ -213,6 +215,19 @@ func handleWSResponseCreate(connCtx context.Context, conn *lockedConn, connectio
 		return
 	}
 
+	// The PII request middleware is HTTP middleware and never sees messages
+	// arriving over this socket. Rather than serve a PII-enabled model
+	// unfiltered (the same content would be blocked or masked on
+	// POST /v1/responses), refuse it here: fail closed, as the middleware
+	// does when its detector is unavailable.
+	if piiPolicy != nil {
+		if enabled, detectors := piiPolicy(cfg); enabled && len(detectors) > 0 {
+			xlog.Warn("WebSocket Responses: refusing PII-enabled model, PII filtering is not applied on the WebSocket transport", "model", input.Model)
+			fail("pii_unsupported_transport", "PII filtering is enabled for this model and is not available over the Responses WebSocket; use POST /v1/responses", "model")
+			return
+		}
+	}
+
 	// Set up context with cancellation tied to connection lifetime
 	reqCtx, reqCancel := context.WithCancel(connCtx)
 	defer reqCancel()

core/http/endpoints/openresponses/websocket_pii_test.go

@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: MIT
+package openresponses
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"time"
+
+	"github.com/gorilla/websocket"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/pii"
+	"github.com/mudler/LocalAI/pkg/system"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// The PII request middleware is HTTP middleware, so messages arriving over
+// the Responses WebSocket never pass through it. These specs drive the real
+// socket loop and prove a PII-enabled model is refused there instead of
+// being served unfiltered.
+var _ = Describe("WebSocket Responses PII guard", func() {
+	// dial runs the real connection loop behind a test server with the given
+	// policy and returns the first event answering one prewarm request
+	// (generate=false, so no backend is needed).
+	dial := func(policy pii.PolicyResolver) schema.ORStreamEvent {
+		GinkgoHelper()
+		dir := GinkgoT().TempDir()
+		appCfg := config.NewApplicationConfig()
+		appCfg.SystemState = &system.SystemState{Model: system.Model{ModelsPath: dir}}
+		cl := config.NewModelConfigLoader(dir)
+		m := config.ModelConfig{Name: "m", Backend: "test"}
+		m.Model = "m.gguf"
+		cl.ReplaceModelConfigs([]config.ModelConfig{m})
+
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			ws, err := wsUpgrader.Upgrade(w, r, nil)
+			if err != nil {
+				return
+			}
+			defer ws.Close()
+			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+			defer cancel()
+			handleWebSocketConnection(ctx, &lockedConn{Conn: ws}, "owner", cl, nil, nil, appCfg, policy)
+		}))
+		DeferCleanup(srv.Close)
+
+		conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(srv.URL, "http"), nil)
+		Expect(err).ToNot(HaveOccurred())
+		DeferCleanup(conn.Close)
+		Expect(conn.SetReadDeadline(time.Now().Add(10 * time.Second))).To(Succeed())
+		Expect(conn.WriteMessage(websocket.TextMessage,
+			[]byte(`{"type":"response.create","generate":false,"store":false,"model":"m","input":"my key is sk-secret"}`))).To(Succeed())
+
+		var ev schema.ORStreamEvent
+		Expect(conn.ReadJSON(&ev)).To(Succeed())
+		return ev
+	}
+
+	It("refuses a model whose PII policy is enabled", func() {
+		ev := dial(func(any) (bool, []string) { return true, []string{"det"} })
+		Expect(ev.Type).To(Equal("error"))
+		Expect(ev.Error).ToNot(BeNil())
+		Expect(ev.Error.Type).To(Equal("pii_unsupported_transport"))
+		Expect(ev.Error.Message).ToNot(ContainSubstring("sk-secret"), "the refusal must not echo the request content")
+	})
+
+	It("serves a model without a PII policy", func() {
+		ev := dial(func(any) (bool, []string) { return false, nil })
+		Expect(ev.Type).To(Equal("response.created"))
+	})
+
+	It("serves a PII-enabled model with no detectors, as the HTTP middleware does", func() {
+		ev := dial(func(any) (bool, []string) { return true, nil })
+		Expect(ev.Type).To(Equal("response.created"))
+	})
+})

core/http/routes/openresponses.go

@@ -9,6 +9,8 @@ import (
 	"github.com/mudler/LocalAI/core/http/endpoints/openresponses"
 	"github.com/mudler/LocalAI/core/http/middleware"
 	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/pii"
+	"github.com/mudler/LocalAI/core/services/routing/piiadapter"
 	"github.com/mudler/xlog"
 )
 
@@ -41,16 +43,7 @@ func RegisterOpenResponsesRoutes(app *echo.Echo,
 		agentControl,
 	)
 
-	responsesMiddleware := []echo.MiddlewareFunc{
-		// Intercept requests where the model name matches an agent — route directly
-		// to the agent pool without going through the model config resolution pipeline.
-		localai.AgentResponsesInterceptor(application),
-		middleware.UsageMiddleware(application.StatsRecorder(), application.FallbackUser()),
-		middleware.TraceMiddleware(application),
-		re.BuildFilteredFirstAvailableDefaultModel(config.BuildUsecaseFilterFn(config.FLAG_CHAT)),
-		re.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.OpenResponsesRequest) }),
-		setOpenResponsesRequestContext(re),
-	}
+	responsesMiddleware := openResponsesMiddleware(re, application)
 
 	// Main Open Responses endpoint
 	app.POST("/v1/responses", responsesHandler, responsesMiddleware...)
@@ -74,6 +67,28 @@ func RegisterOpenResponsesRoutes(app *echo.Echo,
 	app.POST("/responses/:id/cancel", cancelResponseHandler, middleware.TraceMiddleware(application))
 }
 
+// openResponsesMiddleware is the middleware chain of POST /v1/responses and
+// POST /responses. It is a separate function so tests can drive the real
+// chain in front of a stub handler.
+func openResponsesMiddleware(re *middleware.RequestExtractor, application *application.Application) []echo.MiddlewareFunc {
+	return []echo.MiddlewareFunc{
+		// Intercept requests where the model name matches an agent — route directly
+		// to the agent pool without going through the model config resolution pipeline.
+		localai.AgentResponsesInterceptor(application),
+		middleware.UsageMiddleware(application.StatsRecorder(), application.FallbackUser()),
+		middleware.TraceMiddleware(application),
+		re.BuildFilteredFirstAvailableDefaultModel(config.BuildUsecaseFilterFn(config.FLAG_CHAT)),
+		re.SetModelAndConfig(func() schema.LocalAIRequest { return new(schema.OpenResponsesRequest) }),
+		setOpenResponsesRequestContext(re),
+		// PII filtering runs once the model config is resolved and merged
+		// (it gates on the served model's pii block), and before the handler
+		// turns the input items into prompt messages. Without it, content
+		// that a model's PII policy blocks or masks on /v1/chat/completions
+		// would reach the same model unfiltered through this endpoint.
+		pii.RequestMiddleware(application.PIIRedactor(), application.PIIEvents(), piiadapter.OpenResponses(), application.FallbackUser(), pii.WithNERResolver(application.PIINERResolver()), pii.WithPolicyResolver(application.PIIPolicyResolver())),
+	}
+}
+
 // setOpenResponsesRequestContext sets up the context and cancel function for Open Responses requests
 func setOpenResponsesRequestContext(re *middleware.RequestExtractor) echo.MiddlewareFunc {
 	return func(next echo.HandlerFunc) echo.HandlerFunc {

core/http/routes/openresponses_pii_test.go

@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: MIT
+package routes
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/application"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/middleware"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/pkg/system"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// Drives the real POST /v1/responses middleware chain (openResponsesMiddleware)
+// in front of a stub handler, with an in-process pattern detector, to prove
+// the PII policy that guards /v1/chat/completions also guards this endpoint.
+var _ = Describe("Open Responses PII filtering", func() {
+	var (
+		e        *echo.Echo
+		cancel   context.CancelFunc
+		reached  bool
+		received *schema.OpenResponsesRequest
+	)
+
+	BeforeEach(func() {
+		tmp := GinkgoT().TempDir()
+		modelsDir := filepath.Join(tmp, "models")
+		Expect(os.MkdirAll(modelsDir, 0o755)).To(Succeed())
+		st, err := system.GetSystemState(
+			system.WithModelPath(modelsDir),
+			system.WithBackendPath(filepath.Join(tmp, "backends")),
+		)
+		Expect(err).ToNot(HaveOccurred())
+
+		var ctx context.Context
+		ctx, cancel = context.WithCancel(context.Background())
+		app, err := application.New(config.WithContext(ctx), config.WithSystemState(st))
+		Expect(err).ToNot(HaveOccurred())
+
+		// The pattern tier runs in-process; no backend is loaded.
+		files := map[string]string{
+			"secret-filter.yaml": `name: secret-filter
+backend: pattern
+pii_detection:
+  default_action: mask
+  patterns:
+    - name: SECRET
+      match: "sk-test-[A-Za-z0-9]+"
+      action: block
+    - name: TOKEN
+      match: "tok-[A-Za-z0-9]+"
+      action: mask
+`,
+			"guarded.yaml": `name: guarded
+backend: test
+pii:
+  enabled: true
+  detectors: [secret-filter]
+`,
+		}
+		for name, body := range files {
+			p := filepath.Join(modelsDir, name)
+			Expect(os.WriteFile(p, []byte(body), 0o644)).To(Succeed())
+			Expect(app.ModelConfigLoader().ReadModelConfig(p)).To(Succeed())
+		}
+
+		re := middleware.NewRequestExtractor(app.ModelConfigLoader(), app.ModelLoader(), app.ApplicationConfig())
+		reached, received = false, nil
+		e = echo.New()
+		e.POST("/v1/responses", func(c echo.Context) error {
+			reached = true
+			received = c.Get(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST).(*schema.OpenResponsesRequest)
+			return c.NoContent(http.StatusNoContent)
+		}, openResponsesMiddleware(re, app)...)
+	})
+
+	AfterEach(func() { cancel() })
+
+	post := func(body string) *httptest.ResponseRecorder {
+		req := httptest.NewRequest(http.MethodPost, "/v1/responses", strings.NewReader(body))
+		req.Header.Set("Content-Type", "application/json")
+		rec := httptest.NewRecorder()
+		e.ServeHTTP(rec, req)
+		return rec
+	}
+
+	It("blocks a secret in a string input with pii_blocked", func() {
+		rec := post(`{"model":"guarded","input":"my key is sk-test-ABC123"}`)
+		Expect(rec.Code).To(Equal(http.StatusBadRequest))
+		Expect(rec.Body.String()).To(ContainSubstring("pii_blocked"))
+		Expect(rec.Body.String()).ToNot(ContainSubstring("sk-test-ABC123"), "the block response must not echo the secret")
+		Expect(reached).To(BeFalse(), "a blocked request must not reach the handler")
+	})
+
+	It("blocks a secret hidden in a content part, instructions or tool output", func() {
+		for _, body := range []string{
+			`{"model":"guarded","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"key sk-test-ABC123"}]}]}`,
+			`{"model":"guarded","instructions":"use sk-test-ABC123","input":"hi"}`,
+			`{"model":"guarded","input":[{"type":"function_call_output","call_id":"c1","output":"sk-test-ABC123"}]}`,
+			`{"model":"guarded","input":[{"type":"function_call_output","call_id":"c1","output":["sk-test-ABC123"]}]}`,
+			`{"model":"guarded","input":[{"type":"function_call_output","call_id":"c1","output":[{"token":"sk-test-ABC123"}]}]}`,
+		} {
+			reached = false
+			rec := post(body)
+			Expect(rec.Code).To(Equal(http.StatusBadRequest), body)
+			Expect(rec.Body.String()).To(ContainSubstring("pii_blocked"), body)
+			Expect(reached).To(BeFalse(), body)
+		}
+	})
+
+	It("masks a maskable value before the handler sees it", func() {
+		rec := post(`{"model":"guarded","input":[{"type":"message","role":"user","content":"token tok-XYZ789 please"}]}`)
+		Expect(rec.Code).To(Equal(http.StatusNoContent))
+		Expect(reached).To(BeTrue())
+		content := received.Input.([]any)[0].(map[string]any)["content"].(string)
+		Expect(content).ToNot(ContainSubstring("tok-XYZ789"))
+		Expect(content).To(HavePrefix("token "))
+		Expect(content).To(HaveSuffix(" please"))
+	})
+
+	It("passes clean input through unchanged", func() {
+		rec := post(`{"model":"guarded","instructions":"be brief","input":"what is the capital of France?"}`)
+		Expect(rec.Code).To(Equal(http.StatusNoContent))
+		Expect(reached).To(BeTrue())
+		Expect(received.Input).To(Equal("what is the capital of France?"))
+		Expect(received.Instructions).To(Equal("be brief"))
+	})
+})

core/services/routing/piiadapter/openresponses.go

@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: MIT
+package piiadapter
+
+import (
+	"encoding/json"
+
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/services/routing/pii"
+	"github.com/mudler/xlog"
+)
+
+// Top-level positions in an Open Responses request. Real input-item
+// indices are >= 0, so the negative values cannot collide with them.
+const (
+	orInstructions = -2
+	orInputString  = -1
+)
+
+// encOR packs (item, part) into one ScannedText.Index. item is an index
+// into the input item array or one of the orInstructions / orInputString
+// sentinels; part=-1 means "the item's whole text field", part>=0 indexes
+// into a content-part array. Both are stored offset so the sentinels map
+// to non-negative bit fields.
+func encOR(item, part int) int     { return ((item + 2) << 24) | (part + 1) }
+func decOR(p int) (item, part int) { return (p >> 24) - 2, (p & 0xFFFFFF) - 1 }
+
+// orItemTextField names the field of an input item that carries the text
+// the Responses handler forwards to the model, or "" for items whose text
+// is resolved later by the handler (item_reference is not scanned here).
+// Scan and Apply both resolve the field through here so they can never
+// disagree on where a span lives.
+func orItemTextField(item map[string]any) string {
+	itemType, _ := item["type"].(string)
+	if itemType == "" {
+		// Bare {role, content} objects are message items; the handler
+		// (convertORInputToMessages) treats them the same way.
+		if _, hasRole := item["role"].(string); hasRole {
+			itemType = "message"
+		}
+	}
+	switch itemType {
+	case "message", "reasoning":
+		return "content"
+	case "function_call_output":
+		return "output"
+	case "function_call":
+		return "arguments"
+	}
+	return ""
+}
+
+// isORTextPart reports whether a content part carries plain text. Image,
+// file, audio and video parts are left alone, as in the other adapters.
+// orTextParts are the content part types that carry only text.
+var orTextParts = map[string]bool{"input_text": true, "output_text": true, "refusal": true, "summary_text": true}
+
+// orMediaItems are input item types that are media on their own.
+var orMediaItems = map[string]bool{"input_image": true, "input_file": true, "input_audio": true}
+
+func isORTextPart(part map[string]any) bool {
+	t, _ := part["type"].(string)
+	return t == "input_text" || t == "output_text"
+}
+
+// OpenResponses returns a pii.Adapter for *schema.OpenResponsesRequest
+// (POST /v1/responses). It scans every place the handler takes prompt text
+// from: Instructions, Input as a plain string, and in an Input item array
+// the text of message and reasoning items (string content or input_text /
+// output_text parts), function_call arguments and function_call_output
+// output. Without it the same content that is blocked on
+// /v1/chat/completions could be sent unfiltered through /v1/responses.
+//
+// A function_call_output whose output is not a string (including arrays)
+// is forwarded by the handler as its JSON encoding, so that encoding is
+// scanned as one span; a redaction replaces the value with the redacted
+// JSON string, which the handler then forwards verbatim.
+func OpenResponses() pii.Adapter {
+	return pii.Adapter{
+		Scan: func(parsed any) []pii.ScannedText {
+			req, ok := parsed.(*schema.OpenResponsesRequest)
+			if !ok || req == nil {
+				return nil
+			}
+			var out []pii.ScannedText
+			if req.Instructions != "" {
+				out = append(out, pii.ScannedText{Index: encOR(orInstructions, -1), Text: req.Instructions})
+			}
+			switch in := req.Input.(type) {
+			case string:
+				if in != "" {
+					out = append(out, pii.ScannedText{Index: encOR(orInputString, -1), Text: in})
+				}
+			case []any:
+				for i, raw := range in {
+					item, ok := raw.(map[string]any)
+					if !ok {
+						continue
+					}
+					field := orItemTextField(item)
+					if field == "" {
+						continue
+					}
+					// The handler serializes all non-string tool output, including
+					// arrays. Scan that whole JSON, not just typed text parts.
+					if _, isString := item[field].(string); field == "output" && !isString {
+						b, err := json.Marshal(item[field])
+						if err != nil {
+							xlog.Warn("pii: cannot encode function_call_output for scanning", "error", err)
+							continue
+						}
+						out = append(out, pii.ScannedText{Index: encOR(i, -1), Text: string(b)})
+						continue
+					}
+					switch v := item[field].(type) {
+					case nil:
+						// Missing field: nothing reaches the model.
+					case string:
+						if v != "" {
+							out = append(out, pii.ScannedText{Index: encOR(i, -1), Text: v})
+						}
+					case []any:
+						for j, p := range v {
+							part, ok := p.(map[string]any)
+							if !ok || !isORTextPart(part) {
+								continue
+							}
+							if text, ok := part["text"].(string); ok && text != "" {
+								out = append(out, pii.ScannedText{Index: encOR(i, j), Text: text})
+							}
+						}
+					}
+				}
+			}
+			return out
+		},
+		Unscanned: func(parsed any) []string {
+			req, ok := parsed.(*schema.OpenResponsesRequest)
+			if !ok || req == nil {
+				return nil
+			}
+			items, ok := req.Input.([]any)
+			if !ok {
+				return nil
+			}
+			var out []string
+			seen := map[string]bool{}
+			for _, raw := range items {
+				item, ok := raw.(map[string]any)
+				if !ok {
+					continue
+				}
+				if t, _ := item["type"].(string); orMediaItems[t] && !seen[t] {
+					// A bare media item outside a message.
+					seen[t] = true
+					out = append(out, t)
+					continue
+				}
+				if orItemTextField(item) != "content" {
+					continue
+				}
+				if parts, ok := item["content"].([]any); ok {
+					out = blockKinds(parts, orTextParts, seen, out)
+				}
+			}
+			return out
+		},
+		Apply: func(parsed any, updates []pii.ScannedText) {
+			req, ok := parsed.(*schema.OpenResponsesRequest)
+			if !ok || req == nil {
+				return
+			}
+			for _, u := range updates {
+				itemIdx, partIdx := decOR(u.Index)
+				switch itemIdx {
+				case orInstructions:
+					req.Instructions = u.Text
+					continue
+				case orInputString:
+					if _, ok := req.Input.(string); ok {
+						req.Input = u.Text
+					}
+					continue
+				}
+				items, ok := req.Input.([]any)
+				if !ok || itemIdx < 0 || itemIdx >= len(items) {
+					continue
+				}
+				item, ok := items[itemIdx].(map[string]any)
+				if !ok {
+					continue
+				}
+				field := orItemTextField(item)
+				if field == "" {
+					continue
+				}
+				if partIdx < 0 {
+					item[field] = u.Text
+					continue
+				}
+				parts, ok := item[field].([]any)
+				if !ok || partIdx >= len(parts) {
+					continue
+				}
+				if part, ok := parts[partIdx].(map[string]any); ok {
+					part["text"] = u.Text
+				}
+			}
+		},
+	}
+}

core/services/routing/piiadapter/openresponses_test.go

@@ -0,0 +1,195 @@
+// SPDX-License-Identifier: MIT
+package piiadapter
+
+import (
+	"encoding/json"
+	"strings"
+
+	"github.com/mudler/LocalAI/core/schema"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// parseOR decodes a request body the way the HTTP layer does, so Input
+// carries the same []any / map[string]any shapes the adapter sees live.
+func parseOR(body string) *schema.OpenResponsesRequest {
+	GinkgoHelper()
+	req := &schema.OpenResponsesRequest{}
+	Expect(json.Unmarshal([]byte(body), req)).To(Succeed())
+	return req
+}
+
+// redactSecret mimics a detector: only the secret is rewritten, clean
+// text passes through unchanged.
+func redactSecret(s string) string { return strings.ReplaceAll(s, "sk-secret", "[REDACTED]") }
+
+func orItems(req *schema.OpenResponsesRequest) []any {
+	GinkgoHelper()
+	items, ok := req.Input.([]any)
+	Expect(ok).To(BeTrue())
+	return items
+}
+
+var _ = Describe("OpenResponses adapter", func() {
+	a := OpenResponses()
+
+	It("scans and redacts a plain string input", func() {
+		req := parseOR(`{"model":"m","input":"my key is sk-secret"}`)
+		got := a.Scan(req)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Text).To(Equal("my key is sk-secret"))
+		applyAll(a, req, redactSecret)
+		Expect(req.Input).To(Equal("my key is [REDACTED]"))
+	})
+
+	It("leaves a clean string input byte-identical", func() {
+		req := parseOR(`{"model":"m","input":"nothing to hide here"}`)
+		applyAll(a, req, redactSecret)
+		Expect(req.Input).To(Equal("nothing to hide here"))
+	})
+
+	It("scans and redacts instructions without touching clean input", func() {
+		req := parseOR(`{"model":"m","instructions":"use sk-secret for the API","input":"hello"}`)
+		Expect(a.Scan(req)).To(HaveLen(2))
+		applyAll(a, req, redactSecret)
+		Expect(req.Instructions).To(Equal("use [REDACTED] for the API"))
+		Expect(req.Input).To(Equal("hello"))
+	})
+
+	It("redacts message items with string content, including typeless {role, content}", func() {
+		req := parseOR(`{"model":"m","input":[
+			{"type":"message","role":"user","content":"first sk-secret"},
+			{"role":"user","content":"second sk-secret"},
+			{"type":"message","role":"user","content":"clean text"}
+		]}`)
+		Expect(a.Scan(req)).To(HaveLen(3))
+		applyAll(a, req, redactSecret)
+		items := orItems(req)
+		Expect(items[0].(map[string]any)["content"]).To(Equal("first [REDACTED]"))
+		Expect(items[1].(map[string]any)["content"]).To(Equal("second [REDACTED]"))
+		Expect(items[2].(map[string]any)["content"]).To(Equal("clean text"))
+	})
+
+	It("redacts text content parts and leaves non-text parts untouched", func() {
+		req := parseOR(`{"model":"m","input":[{"type":"message","role":"user","content":[
+			{"type":"input_text","text":"look at sk-secret"},
+			{"type":"input_image","image_url":"data:image/png;base64,sk-secret"},
+			{"type":"input_file","file_data":"sk-secret"},
+			{"type":"input_text","text":"clean part"}
+		]}]}`)
+		got := a.Scan(req)
+		Expect(got).To(HaveLen(2), "only the two input_text parts are scanned")
+		applyAll(a, req, redactSecret)
+		parts := orItems(req)[0].(map[string]any)["content"].([]any)
+		Expect(parts[0].(map[string]any)["text"]).To(Equal("look at [REDACTED]"))
+		Expect(parts[1].(map[string]any)["image_url"]).To(Equal("data:image/png;base64,sk-secret"))
+		Expect(parts[2].(map[string]any)["file_data"]).To(Equal("sk-secret"))
+		Expect(parts[3].(map[string]any)["text"]).To(Equal("clean part"))
+	})
+
+	It("redacts tool output given as a string", func() {
+		req := parseOR(`{"model":"m","input":[
+			{"type":"function_call","call_id":"c1","name":"lookup","arguments":"{\"q\":\"sk-secret\"}"},
+			{"type":"function_call_output","call_id":"c1","output":"result sk-secret"},
+			{"type":"function_call_output","call_id":"c2","output":"clean result"}
+		]}`)
+		Expect(a.Scan(req)).To(HaveLen(3))
+		applyAll(a, req, redactSecret)
+		items := orItems(req)
+		Expect(items[0].(map[string]any)["arguments"]).To(Equal(`{"q":"[REDACTED]"}`))
+		Expect(items[1].(map[string]any)["output"]).To(Equal("result [REDACTED]"))
+		Expect(items[2].(map[string]any)["output"]).To(Equal("clean result"))
+	})
+
+	It("scans structured tool output as its JSON encoding (what the handler forwards)", func() {
+		req := parseOR(`{"model":"m","input":[{"type":"function_call_output","call_id":"c1","output":{"token":"sk-secret"}}]}`)
+		got := a.Scan(req)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Text).To(Equal(`{"token":"sk-secret"}`))
+		applyAll(a, req, redactSecret)
+		Expect(orItems(req)[0].(map[string]any)["output"]).To(Equal(`{"token":"[REDACTED]"}`))
+	})
+
+	DescribeTable("scans and redacts the complete JSON of array tool output", func(output, redacted string) {
+		req := parseOR(`{"model":"m","input":[{"type":"function_call_output","call_id":"c1","output":` + output + `}]}`)
+		got := a.Scan(req)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Text).To(Equal(output))
+		applyAll(a, req, redactSecret)
+		Expect(orItems(req)[0].(map[string]any)["output"]).To(Equal(redacted))
+	},
+		Entry("string elements", `["sk-secret"]`, `["[REDACTED]"]`),
+		Entry("object elements", `[{"token":"sk-secret"}]`, `[{"token":"[REDACTED]"}]`),
+		Entry("text parts with additional fields", `[{"extra":"sk-secret","text":"clean","type":"input_text"}]`, `[{"extra":"[REDACTED]","text":"clean","type":"input_text"}]`),
+	)
+
+	It("preserves clean array tool output when there are no updates", func() {
+		req := parseOR(`{"model":"m","input":[{"type":"function_call_output","call_id":"c1","output":["clean",{"ok":true}]}]}`)
+		before, err := json.Marshal(req)
+		Expect(err).ToNot(HaveOccurred())
+		got := a.Scan(req)
+		Expect(got).To(HaveLen(1))
+		Expect(got[0].Text).To(Equal(`["clean",{"ok":true}]`))
+		a.Apply(req, nil)
+		after, err := json.Marshal(req)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(after).To(Equal(before))
+	})
+
+	It("leaves structured tool output unchanged when nothing is redacted", func() {
+		req := parseOR(`{"model":"m","input":[{"type":"function_call_output","call_id":"c1","output":{"ok":true}}]}`)
+		Expect(a.Scan(req)).To(HaveLen(1))
+		// The middleware only applies spans that were actually redacted;
+		// with no updates the original object must survive as-is.
+		a.Apply(req, nil)
+		Expect(orItems(req)[0].(map[string]any)["output"]).To(Equal(map[string]any{"ok": true}))
+	})
+
+	It("redacts reasoning content parts and skips item references", func() {
+		req := parseOR(`{"model":"m","input":[
+			{"type":"reasoning","content":[{"type":"output_text","text":"user said sk-secret"}]},
+			{"type":"item_reference","id":"msg_sk-secret"}
+		]}`)
+		Expect(a.Scan(req)).To(HaveLen(1))
+		applyAll(a, req, redactSecret)
+		items := orItems(req)
+		parts := items[0].(map[string]any)["content"].([]any)
+		Expect(parts[0].(map[string]any)["text"]).To(Equal("user said [REDACTED]"))
+		Expect(items[1].(map[string]any)["id"]).To(Equal("msg_sk-secret"))
+	})
+
+	It("writes each update back only to its own position", func() {
+		req := parseOR(`{"model":"m","instructions":"inst","input":[
+			{"type":"message","role":"user","content":[{"type":"input_text","text":"a"},{"type":"input_text","text":"b"}]},
+			{"type":"message","role":"user","content":"c"}
+		]}`)
+		scans := a.Scan(req)
+		Expect(scans).To(HaveLen(4))
+		// Redact only the second part of the first item.
+		scans[2].Text = "B"
+		a.Apply(req, scans[2:3])
+		Expect(req.Instructions).To(Equal("inst"))
+		items := orItems(req)
+		parts := items[0].(map[string]any)["content"].([]any)
+		Expect(parts[0].(map[string]any)["text"]).To(Equal("a"))
+		Expect(parts[1].(map[string]any)["text"]).To(Equal("B"))
+		Expect(items[1].(map[string]any)["content"]).To(Equal("c"))
+	})
+
+	It("returns nothing for empty or foreign requests", func() {
+		Expect(a.Scan(&schema.OpenResponsesRequest{})).To(BeEmpty())
+		Expect(a.Scan(&schema.OpenAIRequest{})).To(BeNil())
+		Expect(a.Scan(nil)).To(BeNil())
+	})
+
+	It("round-trips the index encoding including the sentinels", func() {
+		for _, c := range []struct{ item, part int }{
+			{orInstructions, -1}, {orInputString, -1}, {0, -1}, {0, 0}, {5, 3}, {100, 70},
+		} {
+			i, p := decOR(encOR(c.item, c.part))
+			Expect(i).To(Equal(c.item))
+			Expect(p).To(Equal(c.part))
+		}
+	})
+})

core/services/routing/piiadapter/unscanned_test.go

@@ -56,3 +56,25 @@ var _ = Describe("Unscanned content", func() {
 		Expect(OllamaChat().Unscanned(textOnly)).To(BeEmpty())
 	})
 })
+
+var _ = Describe("Unscanned content (Open Responses)", func() {
+	It("lists media parts inside messages and bare media items", func() {
+		req := &schema.OpenResponsesRequest{Input: []any{
+			map[string]any{"role": "user", "content": []any{
+				map[string]any{"type": "input_text", "text": "what is this?"},
+				map[string]any{"type": "input_image", "image_url": "data:image/png;base64,xyz"},
+			}},
+			map[string]any{"type": "input_file", "file_data": "JVBERi0="},
+		}}
+		Expect(OpenResponses().Unscanned(req)).To(Equal([]string{"input_image", "input_file"}))
+	})
+
+	It("reports nothing for text input, tool output or a plain string", func() {
+		req := &schema.OpenResponsesRequest{Instructions: "be brief", Input: []any{
+			map[string]any{"type": "message", "role": "user", "content": []any{map[string]any{"type": "input_text", "text": "hi"}}},
+			map[string]any{"type": "function_call_output", "call_id": "1", "output": []any{map[string]any{"type": "input_image"}}},
+		}}
+		Expect(OpenResponses().Unscanned(req)).To(BeEmpty())
+		Expect(OpenResponses().Unscanned(&schema.OpenResponsesRequest{Input: "hi"})).To(BeEmpty())
+	})
+})

docs/content/operations/middleware.md

@@ -53,11 +53,27 @@ backend default.
 
 Filtering runs on every text-accepting endpoint that has an adapter wired:
 `/v1/chat/completions` and `/v1/messages` (chat), `/v1/completions`,
-`/v1/embeddings`, `/v1/edits`, and the Ollama `/api/chat`, `/api/generate`
-and `/api/embed` endpoints, plus the [MITM proxy]({{< relref "mitm-proxy.md" >}})
-request body. Image, audio (TTS/STT), video, rerank, and the realtime
-WebSocket are not filtered yet (different prompt-PII semantics; realtime is
-not HTTP middleware).
+`/v1/embeddings`, `/v1/edits`, the Open Responses `POST /v1/responses`
+(`instructions`, string `input`, and in input items the message and
+reasoning text, `function_call` arguments and `function_call_output` output),
+and the Ollama `/api/chat`, `/api/generate` and `/api/embed` endpoints, plus
+the [MITM proxy]({{< relref "mitm-proxy.md" >}}) request body. Image, audio
+(TTS/STT), video, rerank, and the realtime WebSocket are not filtered yet
+(different prompt-PII semantics; realtime is not HTTP middleware).
+
+The Responses WebSocket mode (`GET /v1/responses`) is not HTTP middleware
+either, so it cannot filter: a `response.create` for a model with PII
+filtering enabled is refused with the error type `pii_unsupported_transport`
+instead of being served unfiltered. Use `POST /v1/responses` for those models.
+Requests whose `model` names an agent are handed to the agent pool before
+model resolution and are not filtered by the per-model policy.
+
+Stored history loaded through `previous_response_id` or `item_reference`
+is resolved after filtering and is not scanned against the current model's
+policy. That history can contain unfiltered text from another model or
+model-generated output. Tool output supplied directly in the request is
+scanned as text, or as its complete JSON encoding for non-string values,
+including arrays.
 
 A request's messages are scanned **as one document** (joined in order), so
 the NER detector keeps conversational context: whether `4421` is a PIN or
@@ -277,8 +293,9 @@ they reach the backend unchecked. With the option set, a request that carries
 any such content is rejected with HTTP 400 (`error.type=pii_blocked`, audit
 event pattern `__unscanned_media__`) before a detector runs. Use it for egress
 policies where "cannot be inspected" must mean "not sent", e.g. on
-`cloud-proxy` models. It applies to the OpenAI chat, Anthropic messages and
-Ollama chat routes, and only while PII is enabled for the model.
+`cloud-proxy` models. It applies to the OpenAI chat, Open Responses,
+Anthropic messages and Ollama chat routes, and only while PII is enabled for
+the model.
 
 Multiple detectors **union** their detections; overlapping spans resolve to
 the strongest action (`block` > `mask` > `allow`). A configured detector