Merge branch 'master' into fix/9658-responses-streaming-reasoning

In the /v1/responses streaming handler a reasoning model's thinking
monologue was streamed to the client as normal message text (a msg_
output item with output_text.delta) and only reclassified into a
reasoning item after the stream completed. Subsequent output_text.delta
events also kept referencing the old msg_ item id instead of the
reasoning_ id.

Root causes:

1. The live reasoning item was gated on extractor.Reasoning(), which is
   only updated by the Go-side raw-tag parser (ProcessToken). When the
   C++ autoparser drives reasoning through reasoning_content ChatDeltas,
   the reasoning delta is computed via ProcessChatDeltaReasoning into a
   separate accumulator, so extractor.Reasoning() stays empty and the
   gate never fired. The reasoning item was thus only reconstructed at
   end-of-stream.

2. The non-tool-call path created the message/msg_ output item eagerly
   before any token, forcing reasoning to a higher output index and
   making mis-split <think> text land on the pre-existing message item.

3. Neither path carried the sticky preferAutoparser flag, so a
   content-only autoparser (the non-jinja pure-content fallback, #9985)
   could leak <think>...</think> tokens into content.

Extract the per-token reasoning-vs-message classification into a pure,
unit-tested streamReasoningRouter (mirroring chooseDeferredReasoning and
processStream in the chat streaming worker): it gates the reasoning item
on the reasoning delta, opens the message item lazily on the first
content delta, and keeps a sticky preferAutoparser fallback. Both
streaming paths now route reasoning deltas to the reasoning_ id and order
the reasoning item ahead of the message at completion.

Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.agents/ci-caching.md

@@ -122,7 +122,7 @@ The per-backend prefix match only sees files under a backend's own directory, so
 
 | Changed path | Rebuilds |
 |---|---|
-| `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
+| `backend/backend.proto` | everything (all languages compile or copy it) |
 | `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
 | `backend/python/common/` | Python, Linux + Darwin |
 | `scripts/build/package-gpu-libs.sh` | Python, Linux only |
@@ -132,17 +132,6 @@ The per-backend prefix match only sees files under a backend's own directory, so
 
 Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
 
-#### `backend/backend.proto` is content-filtered, not path-filtered
-
-Every language consumes the proto, so a path rule for it can only ever say "rebuild all 473 images". It changes in ~1.3% of commits, and that was enough to make it the single largest CI cost driver in the repo: on 2026-07-29 four runs totalling 935 queued jobs traced to nothing but a proto edit, one of which (#11158) was a six-line diff adding `bool cache_prompt = 8;`.
-
-An additive proto edit cannot change how a backend that never references the new symbol behaves, so `filterMatrix()` suppresses the rule for one. `changed-backends.js` fetches `backend/backend.proto` at the base revision (same contents-API pattern as `.github/backend-matrix.yml`) and hands both texts to `protoChangeIsAdditive()`, which compares them structurally rather than textually:
-
-- **Additive, rebuilds nothing**: a new field with an unused number, a new message, a new enum value, a new RPC. Comment, whitespace and ordering changes also land here.
-- **Breaking, rebuilds everything**: a removed, renumbered, retyped or renamed field, a dropped RPC, a changed `option` or `package`. So does an unresolvable base revision, matching the run-all posture used for a truncated diff.
-
-Checked against every proto commit in the preceding six months, all nine resolvable ones classify as additive. Note the tradeoff this accepts: generated stubs do change for an additive edit, so image bytes would differ on a rebuild even though behavior does not. That is the same standard already applied when the filter declines to rebuild on unrelated `pkg/` changes, and the weekly cron remains the backstop.
-
 The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
 
 ## The `DEPS_REFRESH` cache-buster (Python backends)

core/http/endpoints/openai/chat.go

@@ -696,9 +696,6 @@ func ChatEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator
 					template = predInput
 				}
 				thinkingStartToken := reason.DetectThinkingStartToken(template, &config.ReasoningConfig)
-				if config.TemplateConfig.UseTokenizerTemplate {
-					thinkingStartToken = reason.DetectThinkingStartTokenInTemplate(template, &config.ReasoningConfig)
-				}
 
 				xlog.Debug("Thinking start token", "thinkingStartToken", thinkingStartToken, "template", template)
 

core/http/endpoints/openai/chat_stream_workers.go

@@ -150,9 +150,6 @@ func processStream(
 		template = s
 	}
 	thinkingStartToken := reason.DetectThinkingStartToken(template, &cfg.ReasoningConfig)
-	if cfg.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reason.DetectThinkingStartTokenInTemplate(template, &cfg.ReasoningConfig)
-	}
 	extractor := reason.NewReasoningExtractor(thinkingStartToken, cfg.ReasoningConfig)
 
 	// preferAutoparser is sticky: once the C++ autoparser has ever classified
@@ -251,9 +248,6 @@ func processStreamWithTools(
 		template = prompt
 	}
 	thinkingStartToken := reason.DetectThinkingStartToken(template, &cfg.ReasoningConfig)
-	if cfg.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reason.DetectThinkingStartTokenInTemplate(template, &cfg.ReasoningConfig)
-	}
 	extractor := reason.NewReasoningExtractor(thinkingStartToken, cfg.ReasoningConfig)
 
 	result := ""

core/http/endpoints/openai/realtime.go

@@ -2480,9 +2480,6 @@ func triggerResponseAtTurn(ctx context.Context, session *Session, conv *Conversa
 		template = config.TemplateConfig.Chat
 	}
 	thinkingStartToken := reasoning.DetectThinkingStartToken(template, &config.ReasoningConfig)
-	if config.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reasoning.DetectThinkingStartTokenInTemplate(template, &config.ReasoningConfig)
-	}
 
 	// When the C++ autoparser emitted ChatDeltas with actionable data,
 	// prefer them — the backend clears Reply.Message in that path and

core/http/endpoints/openai/realtime_stream.go

@@ -145,9 +145,6 @@ func streamLLMResponse(ctx context.Context, session *Session, conv *Conversation
 		template = llmCfg.TemplateConfig.Chat
 	}
 	thinkingStartToken := reasoning.DetectThinkingStartToken(template, &llmCfg.ReasoningConfig)
-	if llmCfg.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reasoning.DetectThinkingStartTokenInTemplate(template, &llmCfg.ReasoningConfig)
-	}
 
 	// The autoparser (tokenizer-template path) already delivers reasoning-free
 	// content. Prefilling the thinking start token here would re-tag that clean

core/http/endpoints/openresponses/responses.go

@@ -1359,9 +1359,6 @@ func handleOpenResponsesNonStream(c echo.Context, responseID string, createdAt i
 		template = predInput
 	}
 	thinkingStartToken := reason.DetectThinkingStartToken(template, &cfg.ReasoningConfig)
-	if cfg.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reason.DetectThinkingStartTokenInTemplate(template, &cfg.ReasoningConfig)
-	}
 
 	// Extract reasoning from result before cleaning
 	reasoningContent, cleanedResult := reason.ExtractReasoningComplete(result, thinkingStartToken, cfg.ReasoningConfig)
@@ -1643,9 +1640,6 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 		template = predInput
 	}
 	thinkingStartToken := reason.DetectThinkingStartToken(template, &cfg.ReasoningConfig)
-	if cfg.TemplateConfig.UseTokenizerTemplate {
-		thinkingStartToken = reason.DetectThinkingStartTokenInTemplate(template, &cfg.ReasoningConfig)
-	}
 
 	// Track state for streaming
 	var currentMessageID string
@@ -1660,6 +1654,12 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 	var currentReasoningContentIndex int
 	var reasoningTokens int
 	extractor := reason.NewReasoningExtractor(thinkingStartToken, cfg.ReasoningConfig)
+	// router classifies each streamed token into reasoning vs message deltas
+	// and decides which output item they target. It encapsulates the
+	// sticky-preferAutoparser fallback and the reasoningDelta-based gate that
+	// fix issue #9658 (live reasoning was mis-routed onto the msg_ item and
+	// only re-classified as a reasoning item after the stream completed).
+	router := newStreamReasoningRouter(extractor)
 
 	// Collect all output items for storage
 	var collectedOutputItems []schema.ORItemField
@@ -1683,7 +1683,7 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 				// Reset reasoning and tool-call state for re-inference so reasoning
 				// extraction runs again on subsequent iterations
 				inToolCallMode = false
-				extractor.Reset()
+				router.resetForIteration()
 				currentMessageID = ""
 				lastEmittedToolCallCount = 0
 				currentReasoningID = ""
@@ -1844,110 +1844,101 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 
 				// If no tool calls detected yet, handle reasoning and text
 				if !inToolCallMode {
-					var reasoningDelta, contentDelta string
-					goReasoning, goContent := extractor.ProcessToken(token)
+					routing := router.route(token, tokenUsage)
 
-					if tokenUsage.HasChatDeltaContent() {
-						rawReasoning, cd := tokenUsage.ChatDeltaReasoningAndContent()
-						contentDelta = cd
-						reasoningDelta = extractor.ProcessChatDeltaReasoning(rawReasoning)
-					} else {
-						reasoningDelta = goReasoning
-						contentDelta = goContent
+					// Handle reasoning item. The reasoning item is opened lazily
+					// on the first reasoning delta - gating on routing, not
+					// extractor.Reasoning() (issue #9658): when the C++
+					// autoparser drives reasoning via reasoning_content,
+					// extractor.Reasoning() stays empty and the old gate dropped
+					// the live reasoning item.
+					if routing.OpenReasoningItem {
+						outputIndex++
+						currentReasoningID = fmt.Sprintf("reasoning_%s", uuid.New().String())
+						reasoningItem := &schema.ORItemField{
+							Type:   "reasoning",
+							ID:     currentReasoningID,
+							Status: "in_progress",
+						}
+						sendSSEEvent(c, &schema.ORStreamEvent{
+							Type:           "response.output_item.added",
+							SequenceNumber: sequenceNumber,
+							OutputIndex:    &outputIndex,
+							Item:           reasoningItem,
+						})
+						sequenceNumber++
+
+						// Emit content_part.added for reasoning
+						currentReasoningContentIndex = 0
+						emptyPart := makeOutputTextPart("")
+						sendSSEEvent(c, &schema.ORStreamEvent{
+							Type:           "response.content_part.added",
+							SequenceNumber: sequenceNumber,
+							ItemID:         currentReasoningID,
+							OutputIndex:    &outputIndex,
+							ContentIndex:   &currentReasoningContentIndex,
+							Part:           &emptyPart,
+						})
+						sequenceNumber++
 					}
 
-					// Handle reasoning item
-					if extractor.Reasoning() != "" {
-						// Check if we need to create reasoning item
-						if currentReasoningID == "" {
-							outputIndex++
-							currentReasoningID = fmt.Sprintf("reasoning_%s", uuid.New().String())
-							reasoningItem := &schema.ORItemField{
-								Type:   "reasoning",
-								ID:     currentReasoningID,
-								Status: "in_progress",
-							}
-							sendSSEEvent(c, &schema.ORStreamEvent{
-								Type:           "response.output_item.added",
-								SequenceNumber: sequenceNumber,
-								OutputIndex:    &outputIndex,
-								Item:           reasoningItem,
-							})
-							sequenceNumber++
-
-							// Emit content_part.added for reasoning
-							currentReasoningContentIndex = 0
-							emptyPart := makeOutputTextPart("")
-							sendSSEEvent(c, &schema.ORStreamEvent{
-								Type:           "response.content_part.added",
-								SequenceNumber: sequenceNumber,
-								ItemID:         currentReasoningID,
-								OutputIndex:    &outputIndex,
-								ContentIndex:   &currentReasoningContentIndex,
-								Part:           &emptyPart,
-							})
-							sequenceNumber++
-						}
-
-						// Emit reasoning delta if there's new content
-						if reasoningDelta != "" {
-							sendSSEEvent(c, &schema.ORStreamEvent{
-								Type:           "response.output_text.delta",
-								SequenceNumber: sequenceNumber,
-								ItemID:         currentReasoningID,
-								OutputIndex:    &outputIndex,
-								ContentIndex:   &currentReasoningContentIndex,
-								Delta:          strPtr(reasoningDelta),
-								Logprobs:       emptyLogprobs(),
-							})
-							sequenceNumber++
-							c.Response().Flush()
-						}
+					// Emit reasoning delta against the reasoning_ item id.
+					if routing.ReasoningDelta != "" {
+						sendSSEEvent(c, &schema.ORStreamEvent{
+							Type:           "response.output_text.delta",
+							SequenceNumber: sequenceNumber,
+							ItemID:         currentReasoningID,
+							OutputIndex:    &outputIndex,
+							ContentIndex:   &currentReasoningContentIndex,
+							Delta:          strPtr(routing.ReasoningDelta),
+							Logprobs:       emptyLogprobs(),
+						})
+						sequenceNumber++
+						c.Response().Flush()
 					}
 
-					// Only emit message content if there's actual content (not just reasoning)
-					if contentDelta != "" {
-						if currentMessageID == "" {
-							// Emit output_item.added for message
-							outputIndex++
-							currentMessageID = fmt.Sprintf("msg_%s", uuid.New().String())
-							messageItem := &schema.ORItemField{
-								Type:    "message",
-								ID:      currentMessageID,
-								Status:  "in_progress",
-								Role:    "assistant",
-								Content: []schema.ORContentPart{},
-							}
-							sendSSEEvent(c, &schema.ORStreamEvent{
-								Type:           "response.output_item.added",
-								SequenceNumber: sequenceNumber,
-								OutputIndex:    &outputIndex,
-								Item:           messageItem,
-							})
-							sequenceNumber++
-
-							// Emit content_part.added
-							currentContentIndex = 0
-							emptyPart := makeOutputTextPart("")
-							sendSSEEvent(c, &schema.ORStreamEvent{
-								Type:           "response.content_part.added",
-								SequenceNumber: sequenceNumber,
-								ItemID:         currentMessageID,
-								OutputIndex:    &outputIndex,
-								ContentIndex:   &currentContentIndex,
-								Part:           &emptyPart,
-							})
-							sequenceNumber++
+					// Open the message item lazily on the first content delta.
+					if routing.OpenMessageItem {
+						outputIndex++
+						currentMessageID = fmt.Sprintf("msg_%s", uuid.New().String())
+						messageItem := &schema.ORItemField{
+							Type:    "message",
+							ID:      currentMessageID,
+							Status:  "in_progress",
+							Role:    "assistant",
+							Content: []schema.ORContentPart{},
 						}
+						sendSSEEvent(c, &schema.ORStreamEvent{
+							Type:           "response.output_item.added",
+							SequenceNumber: sequenceNumber,
+							OutputIndex:    &outputIndex,
+							Item:           messageItem,
+						})
+						sequenceNumber++
 
-						// Emit text delta
+						// Emit content_part.added
+						currentContentIndex = 0
+						emptyPart := makeOutputTextPart("")
+						sendSSEEvent(c, &schema.ORStreamEvent{
+							Type:           "response.content_part.added",
+							SequenceNumber: sequenceNumber,
+							ItemID:         currentMessageID,
+							OutputIndex:    &outputIndex,
+							ContentIndex:   &currentContentIndex,
+							Part:           &emptyPart,
+						})
+						sequenceNumber++
+					}
+
+					// Emit text delta against the msg_ item id.
+					if routing.ContentDelta != "" {
 						sendSSEEvent(c, &schema.ORStreamEvent{
 							Type:           "response.output_text.delta",
 							SequenceNumber: sequenceNumber,
 							ItemID:         currentMessageID,
 							OutputIndex:    &outputIndex,
 							ContentIndex:   &currentContentIndex,
-							Delta:          strPtr(contentDelta),
+							Delta:          strPtr(routing.ContentDelta),
 							Logprobs:       emptyLogprobs(),
 						})
 						sequenceNumber++
@@ -2344,112 +2335,109 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 		return nil
 	}
 
-	// Non-tool-call streaming path
-	// Emit output_item.added for message
-	currentMessageID = fmt.Sprintf("msg_%s", uuid.New().String())
-	messageItem := &schema.ORItemField{
-		Type:    "message",
-		ID:      currentMessageID,
-		Status:  "in_progress",
-		Role:    "assistant",
-		Content: []schema.ORContentPart{},
-	}
-	sendSSEEvent(c, &schema.ORStreamEvent{
-		Type:           "response.output_item.added",
-		SequenceNumber: sequenceNumber,
-		OutputIndex:    &outputIndex,
-		Item:           messageItem,
-	})
-	sequenceNumber++
-
-	// Emit content_part.added
-	currentContentIndex = 0
-	emptyTextPart := makeOutputTextPart("")
-	sendSSEEvent(c, &schema.ORStreamEvent{
-		Type:           "response.content_part.added",
-		SequenceNumber: sequenceNumber,
-		ItemID:         currentMessageID,
-		OutputIndex:    &outputIndex,
-		ContentIndex:   &currentContentIndex,
-		Part:           &emptyTextPart,
-	})
-	sequenceNumber++
+	// Non-tool-call streaming path.
+	//
+	// The message output item is created LAZILY on the first content delta
+	// (mirroring the tool-call path), not eagerly before the first token.
+	// Issue #9658: an eager msg_ item forced reasoning to a higher output
+	// index and made mis-split <think> text land on the pre-existing message,
+	// so the thinking monologue streamed as message text instead of reasoning.
+	var messageItem *schema.ORItemField
 
 	// Stream text deltas with reasoning extraction
 	tokenCallback := func(token string, tokenUsage backend.TokenUsage) bool {
 		accumulatedText += token
 
-		var reasoningDelta, contentDelta string
-		goReasoning, goContent := extractor.ProcessToken(token)
+		routing := router.route(token, tokenUsage)
 
-		if tokenUsage.HasChatDeltaContent() {
-			rawReasoning, cd := tokenUsage.ChatDeltaReasoningAndContent()
-			contentDelta = cd
-			reasoningDelta = extractor.ProcessChatDeltaReasoning(rawReasoning)
-		} else {
-			reasoningDelta = goReasoning
-			contentDelta = goContent
+		// Open the reasoning item lazily on the first reasoning delta.
+		if routing.OpenReasoningItem {
+			outputIndex++
+			currentReasoningID = fmt.Sprintf("reasoning_%s", uuid.New().String())
+			reasoningItem := &schema.ORItemField{
+				Type:   "reasoning",
+				ID:     currentReasoningID,
+				Status: "in_progress",
+			}
+			sendSSEEvent(c, &schema.ORStreamEvent{
+				Type:           "response.output_item.added",
+				SequenceNumber: sequenceNumber,
+				OutputIndex:    &outputIndex,
+				Item:           reasoningItem,
+			})
+			sequenceNumber++
+
+			// Emit content_part.added for reasoning
+			currentReasoningContentIndex = 0
+			emptyPart := makeOutputTextPart("")
+			sendSSEEvent(c, &schema.ORStreamEvent{
+				Type:           "response.content_part.added",
+				SequenceNumber: sequenceNumber,
+				ItemID:         currentReasoningID,
+				OutputIndex:    &outputIndex,
+				ContentIndex:   &currentReasoningContentIndex,
+				Part:           &emptyPart,
+			})
+			sequenceNumber++
 		}
 
-		// Handle reasoning item
-		if extractor.Reasoning() != "" {
-			// Check if we need to create reasoning item
-			if currentReasoningID == "" {
-				outputIndex++
-				currentReasoningID = fmt.Sprintf("reasoning_%s", uuid.New().String())
-				reasoningItem := &schema.ORItemField{
-					Type:   "reasoning",
-					ID:     currentReasoningID,
-					Status: "in_progress",
-				}
-				sendSSEEvent(c, &schema.ORStreamEvent{
-					Type:           "response.output_item.added",
-					SequenceNumber: sequenceNumber,
-					OutputIndex:    &outputIndex,
-					Item:           reasoningItem,
-				})
-				sequenceNumber++
-
-				// Emit content_part.added for reasoning
-				currentReasoningContentIndex = 0
-				emptyPart := makeOutputTextPart("")
-				sendSSEEvent(c, &schema.ORStreamEvent{
-					Type:           "response.content_part.added",
-					SequenceNumber: sequenceNumber,
-					ItemID:         currentReasoningID,
-					OutputIndex:    &outputIndex,
-					ContentIndex:   &currentReasoningContentIndex,
-					Part:           &emptyPart,
-				})
-				sequenceNumber++
-			}
-
-			// Emit reasoning delta if there's new content
-			if reasoningDelta != "" {
-				sendSSEEvent(c, &schema.ORStreamEvent{
-					Type:           "response.output_text.delta",
-					SequenceNumber: sequenceNumber,
-					ItemID:         currentReasoningID,
-					OutputIndex:    &outputIndex,
-					ContentIndex:   &currentReasoningContentIndex,
-					Delta:          strPtr(reasoningDelta),
-					Logprobs:       emptyLogprobs(),
-				})
-				sequenceNumber++
-				c.Response().Flush()
-			}
+		// Emit reasoning delta against the reasoning_ item id.
+		if routing.ReasoningDelta != "" {
+			sendSSEEvent(c, &schema.ORStreamEvent{
+				Type:           "response.output_text.delta",
+				SequenceNumber: sequenceNumber,
+				ItemID:         currentReasoningID,
+				OutputIndex:    &outputIndex,
+				ContentIndex:   &currentReasoningContentIndex,
+				Delta:          strPtr(routing.ReasoningDelta),
+				Logprobs:       emptyLogprobs(),
+			})
+			sequenceNumber++
+			c.Response().Flush()
 		}
 
-		// Only emit message content if there's actual content (not just reasoning)
-		if contentDelta != "" {
-			// Emit text delta
+		// Open the message item lazily on the first content delta.
+		if routing.OpenMessageItem {
+			outputIndex++
+			currentMessageID = fmt.Sprintf("msg_%s", uuid.New().String())
+			messageItem = &schema.ORItemField{
+				Type:    "message",
+				ID:      currentMessageID,
+				Status:  "in_progress",
+				Role:    "assistant",
+				Content: []schema.ORContentPart{},
+			}
+			sendSSEEvent(c, &schema.ORStreamEvent{
+				Type:           "response.output_item.added",
+				SequenceNumber: sequenceNumber,
+				OutputIndex:    &outputIndex,
+				Item:           messageItem,
+			})
+			sequenceNumber++
+
+			// Emit content_part.added
+			currentContentIndex = 0
+			emptyTextPart := makeOutputTextPart("")
+			sendSSEEvent(c, &schema.ORStreamEvent{
+				Type:           "response.content_part.added",
+				SequenceNumber: sequenceNumber,
+				ItemID:         currentMessageID,
+				OutputIndex:    &outputIndex,
+				ContentIndex:   &currentContentIndex,
+				Part:           &emptyTextPart,
+			})
+			sequenceNumber++
+		}
+
+		// Emit text delta against the msg_ item id.
+		if routing.ContentDelta != "" {
 			sendSSEEvent(c, &schema.ORStreamEvent{
 				Type:           "response.output_text.delta",
 				SequenceNumber: sequenceNumber,
 				ItemID:         currentMessageID,
 				OutputIndex:    &outputIndex,
 				ContentIndex:   &currentContentIndex,
-				Delta:          strPtr(contentDelta),
+				Delta:          strPtr(routing.ContentDelta),
 				Logprobs:       emptyLogprobs(),
 			})
 			sequenceNumber++
@@ -2574,40 +2562,78 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 	// Convert logprobs for streaming events
 	mcpStreamLogprobs := convertLogprobsForStreaming(noToolLogprobs)
 
-	// Emit output_text.done
-	sendSSEEvent(c, &schema.ORStreamEvent{
-		Type:           "response.output_text.done",
-		SequenceNumber: sequenceNumber,
-		ItemID:         currentMessageID,
-		OutputIndex:    &outputIndex,
-		ContentIndex:   &currentContentIndex,
-		Text:           strPtr(result),
-		Logprobs:       logprobsPtr(mcpStreamLogprobs),
-	})
-	sequenceNumber++
+	// The message item is created lazily on the first content delta (issue
+	// #9658). If no content streamed but final extraction produced text (e.g.
+	// the autoparser delivered everything at once), open the message item now
+	// so the closing events below are valid. A pure-reasoning turn (no content
+	// at all) leaves messageItem nil and emits no message item.
+	if messageItem == nil && result != "" {
+		outputIndex++
+		currentMessageID = fmt.Sprintf("msg_%s", uuid.New().String())
+		messageItem = &schema.ORItemField{
+			Type:    "message",
+			ID:      currentMessageID,
+			Status:  "in_progress",
+			Role:    "assistant",
+			Content: []schema.ORContentPart{},
+		}
+		sendSSEEvent(c, &schema.ORStreamEvent{
+			Type:           "response.output_item.added",
+			SequenceNumber: sequenceNumber,
+			OutputIndex:    &outputIndex,
+			Item:           messageItem,
+		})
+		sequenceNumber++
 
-	// Emit content_part.done (with actual logprobs)
-	resultPart := makeOutputTextPartWithLogprobs(result, noToolLogprobs)
-	sendSSEEvent(c, &schema.ORStreamEvent{
-		Type:           "response.content_part.done",
-		SequenceNumber: sequenceNumber,
-		ItemID:         currentMessageID,
-		OutputIndex:    &outputIndex,
-		ContentIndex:   &currentContentIndex,
-		Part:           &resultPart,
-	})
-	sequenceNumber++
+		currentContentIndex = 0
+		emptyTextPart := makeOutputTextPart("")
+		sendSSEEvent(c, &schema.ORStreamEvent{
+			Type:           "response.content_part.added",
+			SequenceNumber: sequenceNumber,
+			ItemID:         currentMessageID,
+			OutputIndex:    &outputIndex,
+			ContentIndex:   &currentContentIndex,
+			Part:           &emptyTextPart,
+		})
+		sequenceNumber++
+	}
 
-	// Emit output_item.done (with actual logprobs)
-	messageItem.Status = "completed"
-	messageItem.Content = []schema.ORContentPart{makeOutputTextPartWithLogprobs(result, noToolLogprobs)}
-	sendSSEEvent(c, &schema.ORStreamEvent{
-		Type:           "response.output_item.done",
-		SequenceNumber: sequenceNumber,
-		OutputIndex:    &outputIndex,
-		Item:           messageItem,
-	})
-	sequenceNumber++
+	if messageItem != nil {
+		// Emit output_text.done
+		sendSSEEvent(c, &schema.ORStreamEvent{
+			Type:           "response.output_text.done",
+			SequenceNumber: sequenceNumber,
+			ItemID:         currentMessageID,
+			OutputIndex:    &outputIndex,
+			ContentIndex:   &currentContentIndex,
+			Text:           strPtr(result),
+			Logprobs:       logprobsPtr(mcpStreamLogprobs),
+		})
+		sequenceNumber++
+
+		// Emit content_part.done (with actual logprobs)
+		resultPart := makeOutputTextPartWithLogprobs(result, noToolLogprobs)
+		sendSSEEvent(c, &schema.ORStreamEvent{
+			Type:           "response.content_part.done",
+			SequenceNumber: sequenceNumber,
+			ItemID:         currentMessageID,
+			OutputIndex:    &outputIndex,
+			ContentIndex:   &currentContentIndex,
+			Part:           &resultPart,
+		})
+		sequenceNumber++
+
+		// Emit output_item.done (with actual logprobs)
+		messageItem.Status = "completed"
+		messageItem.Content = []schema.ORContentPart{makeOutputTextPartWithLogprobs(result, noToolLogprobs)}
+		sendSSEEvent(c, &schema.ORStreamEvent{
+			Type:           "response.output_item.done",
+			SequenceNumber: sequenceNumber,
+			OutputIndex:    &outputIndex,
+			Item:           messageItem,
+		})
+		sequenceNumber++
+	}
 
 	// Emit function_call items from automatic tool parsing fallback
 	for _, fc := range streamFallbackToolCalls {
@@ -2644,10 +2670,13 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 	// Emit response.completed
 	now := time.Now().Unix()
 
-	// Collect final output items (reasoning first, then messages, then tool calls)
+	// Collect final output items, ordered reasoning -> message -> tool calls.
+	// Issue #9658: reasoning is emitted as its own item ahead of the message,
+	// matching the streamed order (reasoning item is opened before the message
+	// item when the model thinks first).
 	var finalOutputItems []schema.ORItemField
-	// Add reasoning item if it exists
-	if currentReasoningID != "" && finalReasoning != "" {
+	// Add reasoning item if one was streamed.
+	if router.ReasoningStreamed() && finalReasoning != "" {
 		finalOutputItems = append(finalOutputItems, schema.ORItemField{
 			Type:    "reasoning",
 			ID:      currentReasoningID,
@@ -2655,18 +2684,12 @@ func handleOpenResponsesStream(c echo.Context, responseID string, createdAt int6
 			Content: []schema.ORContentPart{makeOutputTextPart(finalReasoning)},
 		})
 	}
-	// Add message item
-	if len(collectedOutputItems) > 0 {
-		// Use collected items (may include reasoning already)
-		for _, item := range collectedOutputItems {
-			if item.Type == "message" {
-				finalOutputItems = append(finalOutputItems, item)
-			}
-		}
-	} else {
+	// Add the message item if one was produced (created lazily, so it may be
+	// nil for a pure-reasoning turn).
+	if messageItem != nil {
 		finalOutputItems = append(finalOutputItems, *messageItem)
 	}
-	// Add function_call items from fallback
+	// Add function_call items from fallback parsing.
 	for _, item := range collectedOutputItems {
 		if item.Type == "function_call" {
 			finalOutputItems = append(finalOutputItems, item)

core/http/endpoints/openresponses/responses_stream_reasoning.go

@@ -0,0 +1,114 @@
+package openresponses
+
+import (
+	"github.com/mudler/LocalAI/core/backend"
+	reason "github.com/mudler/LocalAI/pkg/reasoning"
+)
+
+// streamTokenRouting describes how a single streamed token's deltas should be
+// routed to Open Responses output items: the reasoning/content split and
+// whether a new reasoning or message output item must be opened before the
+// corresponding delta can be emitted.
+type streamTokenRouting struct {
+	ReasoningDelta string
+	ContentDelta   string
+	// OpenReasoningItem is true when a reasoning output item must be created
+	// before emitting ReasoningDelta (the first reasoning delta of the stream).
+	OpenReasoningItem bool
+	// OpenMessageItem is true when a message output item must be created before
+	// emitting ContentDelta (the first content delta of the stream).
+	OpenMessageItem bool
+}
+
+// streamReasoningRouter classifies streamed tokens into reasoning vs message
+// deltas and tracks which output items have been opened, so the SSE-emitting
+// code in handleOpenResponsesStream becomes a thin shell over a unit-testable
+// decision.
+//
+// It mirrors the sticky-preferAutoparser logic in the OpenAI chat streaming
+// worker (core/http/endpoints/openai/chat_stream_workers.go, processStream):
+// once the C++ autoparser has surfaced reasoning_content, we trust its
+// classification for the rest of the stream; until then we fall back to the
+// Go-side reasoning extractor so a pure-content autoparser (the non-jinja PEG
+// fallback, issue #9985) does not leak <think>...</think> tokens into content.
+//
+// Crucially, the decision to open and target a reasoning item keys off the
+// per-token reasoningDelta, NOT extractor.Reasoning(): the autoparser path
+// computes reasoning through ProcessChatDeltaReasoning, which updates a
+// separate accumulator that extractor.Reasoning() never exposes. Gating on
+// extractor.Reasoning() (issue #9658) dropped live reasoning whenever the
+// autoparser drove it via reasoning_content, surfacing it only after the
+// stream completed and mis-routing earlier deltas onto the msg_ item.
+type streamReasoningRouter struct {
+	extractor        *reason.ReasoningExtractor
+	preferAutoparser bool
+	reasoningOpened  bool
+	messageOpened    bool
+}
+
+func newStreamReasoningRouter(extractor *reason.ReasoningExtractor) *streamReasoningRouter {
+	return &streamReasoningRouter{extractor: extractor}
+}
+
+// classify splits a token into reasoning/content deltas using the sticky
+// preferAutoparser preference. Once the C++ autoparser has surfaced
+// reasoning_content we trust it for the rest of the stream; until then we fall
+// back to the Go-side extractor so a pure-content autoparser (zero
+// reasoning_content, issue #9985) does not leak <think>...</think> tokens into
+// content.
+func (r *streamReasoningRouter) classify(token string, usage backend.TokenUsage) (reasoningDelta, contentDelta string) {
+	goReasoning, goContent := r.extractor.ProcessToken(token)
+	if usage.HasChatDeltaContent() {
+		rawReasoning, cd := usage.ChatDeltaReasoningAndContent()
+		if rawReasoning != "" {
+			r.preferAutoparser = true
+		}
+		if r.preferAutoparser {
+			contentDelta = cd
+			reasoningDelta = r.extractor.ProcessChatDeltaReasoning(rawReasoning)
+		} else {
+			reasoningDelta = goReasoning
+			contentDelta = goContent
+		}
+	} else {
+		reasoningDelta = goReasoning
+		contentDelta = goContent
+	}
+	return reasoningDelta, contentDelta
+}
+
+// route classifies a token and decides which output items its deltas target,
+// flipping the opened-flags as items are created.
+//
+// The reasoning gate keys off reasoningDelta, NOT extractor.Reasoning(): the
+// autoparser path computes reasoning via ProcessChatDeltaReasoning into a
+// separate accumulator that extractor.Reasoning() never reflects (issue #9658).
+func (r *streamReasoningRouter) route(token string, usage backend.TokenUsage) streamTokenRouting {
+	reasoningDelta, contentDelta := r.classify(token, usage)
+	out := streamTokenRouting{ReasoningDelta: reasoningDelta, ContentDelta: contentDelta}
+	if reasoningDelta != "" && !r.reasoningOpened {
+		out.OpenReasoningItem = true
+		r.reasoningOpened = true
+	}
+	if contentDelta != "" && !r.messageOpened {
+		out.OpenMessageItem = true
+		r.messageOpened = true
+	}
+	return out
+}
+
+// resetForIteration clears the per-stream routing state for an MCP re-inference
+// iteration, mirroring extractor.Reset() on the underlying extractor.
+func (r *streamReasoningRouter) resetForIteration() {
+	r.preferAutoparser = false
+	r.reasoningOpened = false
+	r.messageOpened = false
+	r.extractor.Reset()
+}
+
+// ReasoningStreamed reports whether a reasoning output item was opened during
+// the stream. The end-of-stream closing blocks key off this rather than a
+// reasoning-id string so the ordering (reasoning before message) is explicit.
+func (r *streamReasoningRouter) ReasoningStreamed() bool {
+	return r.reasoningOpened
+}

core/http/endpoints/openresponses/responses_stream_reasoning_test.go

@@ -0,0 +1,101 @@
+package openresponses
+
+import (
+	"github.com/mudler/LocalAI/core/backend"
+	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
+	reason "github.com/mudler/LocalAI/pkg/reasoning"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// usageWithChatDeltas builds a TokenUsage carrying a single C++ autoparser
+// ChatDelta with the given content / reasoning_content split.
+func usageWithChatDeltas(content, reasoningContent string) backend.TokenUsage {
+	return backend.TokenUsage{
+		ChatDeltas: []*pb.ChatDelta{
+			{Content: content, ReasoningContent: reasoningContent},
+		},
+	}
+}
+
+// Regression tests for issue #9658: in the /v1/responses streaming handler the
+// thinking monologue from a reasoning model was streamed to the client as a
+// normal message (msg_ item, output_text.delta) instead of as a reasoning
+// item, and was only re-classified into a reasoning item AFTER the stream
+// completed.
+//
+// Root cause: the live reasoning item was gated on extractor.Reasoning(),
+// which is only updated by the Go-side raw-tag parser (ProcessToken). When the
+// C++ autoparser drives reasoning through reasoning_content ChatDeltas, the
+// reasoning is computed via ProcessChatDeltaReasoning into a SEPARATE
+// accumulator, so extractor.Reasoning() stays empty and the gate never fires.
+var _ = Describe("streamReasoningRouter", func() {
+	Context("autoparser drives reasoning via reasoning_content (issue #9658)", func() {
+		It("opens a reasoning item during streaming and targets it (not the message)", func() {
+			extractor := reason.NewReasoningExtractor("", reason.Config{})
+			router := newStreamReasoningRouter(extractor)
+
+			// The raw token is empty: the autoparser carries the reasoning in
+			// ChatDelta.ReasoningContent, so the Go-side extractor's
+			// Reasoning() stays "" — exactly the state in which the buggy
+			// extractor.Reasoning() gate failed to open a reasoning item.
+			routing := router.route("", usageWithChatDeltas("", "Let me think about this"))
+
+			Expect(routing.ReasoningDelta).To(Equal("Let me think about this"),
+				"the autoparser's reasoning_content must surface as a reasoning delta during streaming")
+			Expect(routing.OpenReasoningItem).To(BeTrue(),
+				"a reasoning output item must be opened live, not deferred to end-of-stream (#9658)")
+			Expect(routing.ContentDelta).To(BeEmpty())
+			Expect(routing.OpenMessageItem).To(BeFalse(),
+				"reasoning deltas must target the reasoning_ item, never open/route to a msg_ item")
+		})
+
+		It("does not re-open the reasoning item on subsequent reasoning deltas", func() {
+			extractor := reason.NewReasoningExtractor("", reason.Config{})
+			router := newStreamReasoningRouter(extractor)
+
+			_ = router.route("", usageWithChatDeltas("", "first "))
+			routing := router.route("", usageWithChatDeltas("", "second"))
+
+			Expect(routing.ReasoningDelta).To(Equal("second"))
+			Expect(routing.OpenReasoningItem).To(BeFalse())
+		})
+	})
+
+	Context("pure content stream", func() {
+		It("never opens a reasoning item", func() {
+			extractor := reason.NewReasoningExtractor("", reason.Config{})
+			router := newStreamReasoningRouter(extractor)
+
+			// Content-only with no reasoning_content: the autoparser is in its
+			// pure-content mode, so the router stays on the Go-side extractor,
+			// which sees the content via the raw token.
+			routing := router.route("hello world", usageWithChatDeltas("hello world", ""))
+
+			Expect(routing.ContentDelta).To(Equal("hello world"))
+			Expect(routing.OpenMessageItem).To(BeTrue())
+			Expect(routing.OpenReasoningItem).To(BeFalse(),
+				"a content-only stream must never open a reasoning item")
+			Expect(router.ReasoningStreamed()).To(BeFalse())
+		})
+	})
+
+	Context("content-only autoparser with embedded <think> (issue #9985 fallback)", func() {
+		It("falls back to Go-side extraction instead of leaking <think> into content", func() {
+			extractor := reason.NewReasoningExtractor("", reason.Config{})
+			router := newStreamReasoningRouter(extractor)
+
+			// The autoparser is in its non-jinja pure-content fallback: it
+			// surfaces the whole string as Content with zero reasoning_content,
+			// tags and all. The router must NOT trust it (preferAutoparser must
+			// stay false) and instead use the Go-side split.
+			routing := router.route("<think>reasoning here</think>answer",
+				usageWithChatDeltas("<think>reasoning here</think>answer", ""))
+
+			Expect(routing.ContentDelta).To(Equal("answer"),
+				"content must be the cleaned answer, not the raw <think>...</think> string")
+			Expect(routing.ReasoningDelta).To(Equal("reasoning here"))
+			Expect(routing.OpenReasoningItem).To(BeTrue())
+		})
+	})
+})

gallery/gemma.yaml

@@ -10,5 +10,33 @@ config_file: |
         - <end_of_turn>
         - <start_of_turn>
     template:
-        use_tokenizer_template: true
+        chat: |
+            {{.Input }}
+            <start_of_turn>model
+        chat_message: |-
+            <start_of_turn>{{if eq .RoleName "assistant" }}model{{else}}{{ .RoleName }}{{end}}
+            {{ if .FunctionCall -}}
+            {{ else if eq .RoleName "tool" -}}
+            {{ end -}}
+            {{ if .Content -}}
+            {{.Content -}}
+            {{ end -}}
+            {{ if .FunctionCall -}}
+            {{toJson .FunctionCall}}
+            {{ end -}}<end_of_turn>
+        completion: |
+            {{.Input}}
+        function: |
+            <start_of_turn>system
+            You have access to functions. If you decide to invoke any of the function(s),
+            you MUST put it in the format of
+            {"name": function name, "parameters": dictionary of argument name and its value}
+
+            You SHOULD NOT include any other text in the response if you call a function
+            {{range .Functions}}
+            {'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
+            {{end}}
+            <end_of_turn>
+            {{.Input -}}
+            <start_of_turn>model
 name: gemma

gallery/index.yaml

@@ -1,230 +1,4 @@
 ---
-- &kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev"
-  variants:
-    - model: kat-coder-v2.5-dev-q8
-    - model: kat-coder-v2.5-dev-apex-i-quality
-    - model: kat-coder-v2.5-dev-apex-i-balanced
-    - model: kat-coder-v2.5-dev-apex-i-compact
-    - model: kat-coder-v2.5-dev-apex-i-mini
-    - model: kat-coder-v2.5-dev-apex-quality
-    - model: kat-coder-v2.5-dev-apex-balanced
-    - model: kat-coder-v2.5-dev-apex-compact
-  url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
-  urls:
-    - https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev
-    - https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF
-    - https://huggingface.co/mudler/KAT-Coder-V2.5-Dev-APEX-GGUF
-  description: |
-    KAT-Coder-V2.5-Dev is an Apache-2.0 agentic coding model from Kwaipilot,
-    post-trained from Qwen3.6-35B-A3B. It has 35 billion total parameters with
-    3 billion activated per token, a 262K-token context window, and text-only
-    weights tuned for repository-level coding and tool use.
-
-    This entry offers standard Q4_K_M and Q8_0 GGUF quantizations alongside
-    APEX mixed-precision variants with quality, balanced, compact, and mini
-    profiles. The APEX I-profiles use importance-matrix calibration.
-  license: "apache-2.0"
-  icon: https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/resolve/main/kat_logo_hd.png
-  tags:
-    - llm
-    - gguf
-    - qwen
-    - coding
-  last_checked: "2026-07-29"
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf
-      sha256: 4221c26e5663502d1c96fc901c9967d0e70ce2dcfaa5a9fb9280a46bd19e3c07
-      uri: huggingface://bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-q8"
-  variants: []
-  description: |
-    KAT-Coder-V2.5-Dev is an Apache-2.0 agentic coding model from Kwaipilot,
-    post-trained from Qwen3.6-35B-A3B. It has 35 billion total parameters with
-    3 billion activated per token, a 262K-token context window, and text-only
-    weights tuned for repository-level coding and tool use.
-
-    This entry uses the higher-quality Q8_0 GGUF quantization.
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
-      sha256: 5fa510f44779b0e3d38a6678985f417a1c65e3000405ca5d6dcf7fd065e47a15
-      uri: huggingface://bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-i-quality"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Quality.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Quality.gguf
-      sha256: e1cf7f33e13ee787a8557effee41eef5261b24696e283ef9d320830ba39f6784
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Quality.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-i-balanced"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Balanced.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Balanced.gguf
-      sha256: ee6e0ec15964c42ba91831d13e9709239f1b74da3dc49dd3edec4ad6aed8029f
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Balanced.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-i-compact"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf
-      sha256: 5235ac39e7989d9fcf08078acfa755611d42a1f80060d226e4aba04a3595d0d8
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Compact.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-i-mini"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Mini.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Mini.gguf
-      sha256: 9d901bf1c44946840e15e6f73a66782f56ae2f1a41d6508995ddcd976e9af878
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-I-Mini.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-quality"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Quality.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Quality.gguf
-      sha256: 849737baf59183ed518d34f2460f0dcd86190953ccc3213072bdcb1d7f8d2882
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Quality.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-balanced"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Balanced.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Balanced.gguf
-      sha256: 9a5d31b110a95bc085d9851c420dd7a1ffeaec7c9263073c8850d8fb81c4f8ba
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Balanced.gguf
-- !!merge <<: *kat-coder-v2-5-dev
-  name: "kat-coder-v2.5-dev-apex-compact"
-  variants: []
-  overrides:
-    backend: llama-cpp
-    function:
-      automatic_tool_parsing_fallback: true
-      grammar:
-        disable: true
-    known_usecases:
-      - chat
-    options:
-      - use_jinja:true
-    parameters:
-      model: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Compact.gguf
-    template:
-      use_tokenizer_template: true
-  files:
-    - filename: llama-cpp/models/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Compact.gguf
-      sha256: 98dfee53102bbf01e67a768066543a707b40f365ff71e35132ac432c58ad99fd
-      uri: huggingface://mudler/KAT-Coder-V2.5-Dev-APEX-GGUF/KAT-Coder-V2.5-Dev-APEX-Compact.gguf
 - name: "inkling"
   url: "github:mudler/LocalAI/gallery/virtual.yaml@master"
   urls:

pkg/reasoning/reasoning.go

@@ -21,17 +21,6 @@ import (
 // - [THINK]                 (Magistral models)
 // Custom tokens from config are checked first, then default tokens.
 func DetectThinkingStartToken(prompt string, config *Config) string {
-	return detectThinkingStartToken(prompt, config, true)
-}
-
-// DetectThinkingStartTokenInTemplate detects a possible prefill in an
-// unrendered tokenizer template. Marker ordering cannot reveal which Jinja
-// branch will render, so matching closing markers are intentionally ignored.
-func DetectThinkingStartTokenInTemplate(template string, config *Config) string {
-	return detectThinkingStartToken(template, config, false)
-}
-
-func detectThinkingStartToken(prompt string, config *Config, honorClosingToken bool) string {
 	// Common thinking start tokens (in order of specificity - longer first)
 	// Based on llama.cpp's chat-parser.cpp implementations
 	defaultTokens := []string{
@@ -55,8 +44,7 @@ func detectThinkingStartToken(prompt string, config *Config, honorClosingToken b
 	// Check if prompt ends with any of these tokens (allowing for trailing whitespace/newlines)
 	trimmedPrompt := strings.TrimRight(prompt, " \t\n\r")
 	for _, token := range thinkingStartTokens {
-		if strings.Contains(trimmedPrompt, token) &&
-			(!honorClosingToken || !thinkingTokenClosedAfterLastStart(trimmedPrompt, token, config)) {
+		if strings.Contains(trimmedPrompt, token) {
 			return token
 		}
 	}
@@ -79,15 +67,6 @@ func detectThinkingStartToken(prompt string, config *Config, honorClosingToken b
 	return ""
 }
 
-func thinkingTokenClosedAfterLastStart(prompt, startToken string, config *Config) bool {
-	endToken := ClosingTokenForStart(startToken, config)
-	if endToken == "" {
-		return false
-	}
-
-	return strings.LastIndex(prompt, endToken) > strings.LastIndex(prompt, startToken)
-}
-
 // ExtractReasoningWithConfig extracts reasoning from content with the given config.
 // If reasoning is disabled, it returns the original content.
 // If thinking start token prefill is enabled, it prepends the thinking start token to the content.

pkg/reasoning/reasoning_test.go

@@ -410,29 +410,6 @@ var _ = Describe("DetectThinkingStartToken", func() {
 			token := DetectThinkingStartToken(prompt, nil)
 			Expect(token).To(Equal("<think>"))
 		})
-
-		It("should ignore a Gemma thinking token that is already closed in the prompt", func() {
-			prompt := "<|turn>model\n<|channel>thought\n<channel|>\n"
-			token := DetectThinkingStartToken(prompt, nil)
-			Expect(token).To(BeEmpty())
-
-			extractor := NewReasoningExtractor(token, Config{})
-			reasoningDelta, contentDelta := extractor.ProcessToken("READY.")
-			Expect(reasoningDelta).To(BeEmpty())
-			Expect(contentDelta).To(Equal("READY."))
-		})
-
-		It("should preserve prefill detection for unrendered conditional templates", func() {
-			template := "{% if enable_thinking %}<think>{% else %}<think></think>{% endif %}"
-			token := DetectThinkingStartTokenInTemplate(template, nil)
-			Expect(token).To(Equal("<think>"))
-		})
-
-		It("should ignore user Jinja text before a preclosed Gemma prompt suffix", func() {
-			prompt := "Explain {{ variable }}\n<|turn>model\n<|channel>thought\n<channel|>\n"
-			token := DetectThinkingStartToken(prompt, nil)
-			Expect(token).To(BeEmpty())
-		})
 	})
 
 	Context("when prompt does not contain thinking tokens", func() {

scripts/changed-backends.js

@@ -6,7 +6,6 @@ import {
   getAllBackendPaths,
   filterMatrix,
   BACKEND_MATRIX_FILE,
-  BACKEND_PROTO_FILE,
 } from "./lib/backend-filter.mjs";
 
 // Matrix data lives in a small data-only YAML so both backend.yml (master push)
@@ -117,42 +116,6 @@ async function getPreviousMatrix(event) {
   }
 }
 
-// backend.proto at the base revision plus the checked-out copy, so filterMatrix
-// can tell an additive edit (a new field, message or RPC, which invalidates no
-// existing image) from a breaking one. Returning null means "rebuild
-// everything", the same posture as an unresolvable matrix diff.
-//
-// Only called when the changed-file list names the proto, so the common path
-// costs no extra API request.
-async function getProtoRevisions(event) {
-  const ref = event.pull_request ? event.pull_request.base.sha : event.before;
-  if (!ref || /^0+$/.test(ref)) return null;
-  const owner = event.repository.owner.login;
-  const repo = event.repository.name;
-  try {
-    const res = await octokit.request('GET /repos/{owner}/{repo}/contents/{path}', {
-      owner,
-      repo,
-      path: BACKEND_PROTO_FILE,
-      ref,
-      mediaType: { format: 'raw' },
-    });
-    const previous = typeof res.data === 'string'
-      ? res.data
-      : Buffer.from(res.data.content, 'base64').toString('utf8');
-    return {
-      previous,
-      current: fs.readFileSync(BACKEND_PROTO_FILE, "utf8"),
-    };
-  } catch (err) {
-    console.log(
-      `could not read ${BACKEND_PROTO_FILE} at ${ref}, falling back to run-all:`,
-      err.message
-    );
-    return null;
-  }
-}
-
 // Group matrix entries by tag-suffix and emit a merge-matrix entry per group.
 // Both multi-leg groups (per-arch fan-out) and singletons get one entry each:
 // the build job pushes by digest only with no tags applied, so every backend
@@ -277,7 +240,7 @@ function emitFullMatrix() {
   }
 }
 
-function emitFilteredMatrix(changedFiles, previousMatrix, protoRevisions) {
+function emitFilteredMatrix(changedFiles, previousMatrix) {
   console.log("Changed files:", changedFiles);
 
   const { filtered, filteredDarwin, changedBackends } = filterMatrix({
@@ -285,7 +248,6 @@ function emitFilteredMatrix(changedFiles, previousMatrix, protoRevisions) {
     includesDarwin,
     changedFiles,
     previousMatrix,
-    protoRevisions,
   });
 
   console.log("Filtered files:", filtered);
@@ -344,9 +306,5 @@ function emitFilteredMatrix(changedFiles, previousMatrix, protoRevisions) {
     ? await getPreviousMatrix(event)
     : null;
 
-  const protoRevisions = changedFiles.includes(BACKEND_PROTO_FILE)
-    ? await getProtoRevisions(event)
-    : null;
-
-  emitFilteredMatrix(changedFiles, previousMatrix, protoRevisions);
+  emitFilteredMatrix(changedFiles, previousMatrix);
 })();

scripts/lib/backend-filter.mjs

@@ -178,126 +178,6 @@ const GO_BACKEND_PKG_PREFIXES = [
   "pkg/utils/",
 ];
 
-export const BACKEND_PROTO_FILE = "backend/backend.proto";
-const PROTO_RULE_ID = "backend-proto";
-
-// Split a .proto into a map of symbol -> fingerprint so two revisions can be
-// compared structurally instead of textually. A comment reflow, a reindent or a
-// reordered field must not read as a change, and a renumbered or retyped field
-// must.
-//
-// The scanner is deliberately syntax-light: it tracks brace depth to build a
-// container path and records one entry per declaration (`message`, `enum`,
-// `service`, `oneof`, `rpc`) and one per statement (fields, enum values,
-// `option`, `reserved`). It never needs to understand types, only to notice
-// when the text describing one stops being identical.
-function protoSymbols(text) {
-  const symbols = new Map();
-  const stack = [];
-  const norm = s => s.trim().replace(/\s+/g, " ");
-
-  // A declaration is identified by its kind and name, so that changing its body
-  // shows up as changed members rather than as a wholesale replacement.
-  const declKey = header => {
-    const rpc = header.match(/^rpc\s+([A-Za-z_]\w*)/);
-    if (rpc) return `rpc ${rpc[1]}`;
-    const decl = header.match(/^(message|enum|service|oneof|extend)\s+([A-Za-z_]\w*)/);
-    if (decl) return `${decl[1]} ${decl[2]}`;
-    return header;
-  };
-
-  // `bool cache_prompt = 8` is identified by `cache_prompt`, so renumbering or
-  // retyping it changes the fingerprint under a stable key, while renaming it
-  // reads as a removal plus an addition. Statements with no `=` (`reserved 4;`)
-  // are their own identity.
-  const stmtKey = stmt => {
-    const eq = stmt.indexOf("=");
-    if (eq === -1) return stmt;
-    const lhs = norm(stmt.slice(0, eq)).split(" ");
-    return lhs[lhs.length - 1] || stmt;
-  };
-
-  let buf = "";
-  let i = 0;
-  while (i < text.length) {
-    const c = text[i];
-
-    // String literals first: `option go_package = "github.com/..."` contains a
-    // `//` that is not a comment.
-    if (c === '"' || c === "'") {
-      buf += c;
-      i++;
-      while (i < text.length) {
-        if (text[i] === "\\") {
-          buf += text.slice(i, i + 2);
-          i += 2;
-          continue;
-        }
-        buf += text[i];
-        i++;
-        if (text[i - 1] === c) break;
-      }
-      continue;
-    }
-    if (c === "/" && text[i + 1] === "/") {
-      while (i < text.length && text[i] !== "\n") i++;
-      continue;
-    }
-    if (c === "/" && text[i + 1] === "*") {
-      i += 2;
-      while (i < text.length && !(text[i] === "*" && text[i + 1] === "/")) i++;
-      i += 2;
-      continue;
-    }
-    if (c === "{") {
-      const header = norm(buf);
-      buf = "";
-      i++;
-      const key = header ? declKey(header) : "";
-      if (header) symbols.set(`${stack.join("/")}|${key}`, header);
-      stack.push(key);
-      continue;
-    }
-    if (c === "}") {
-      stack.pop();
-      buf = "";
-      i++;
-      continue;
-    }
-    if (c === ";") {
-      const stmt = norm(buf);
-      buf = "";
-      i++;
-      if (stmt) symbols.set(`${stack.join("/")}|${stmtKey(stmt)}`, stmt);
-      continue;
-    }
-    buf += c;
-    i++;
-  }
-  return symbols;
-}
-
-// True when every symbol the previous revision declared survives unchanged into
-// the current one. New symbols are free: a backend that never references a new
-// field, message or RPC produces the same behavior with or without it, so its
-// image does not need rebuilding.
-//
-// Anything else (a removed, renumbered, retyped or renamed field, a dropped
-// RPC, a changed option) is treated as breaking and rebuilds the full matrix.
-// Either text being unresolvable is breaking too, matching the run-all posture
-// changed-backends.js takes for a diff it cannot compute.
-export function protoChangeIsAdditive(previousText, currentText) {
-  if (typeof previousText !== "string" || typeof currentText !== "string") {
-    return false;
-  }
-  const before = protoSymbols(previousText);
-  const after = protoSymbols(currentText);
-  for (const [key, fingerprint] of before) {
-    if (after.get(key) !== fingerprint) return false;
-  }
-  return true;
-}
-
 // Shared build inputs: files that end up in, or decide the contents of, images
 // belonging to backends whose own directory they do not live under. The
 // per-backend prefix match in filterMatrix() structurally cannot see these, so
@@ -320,14 +200,7 @@ export const SHARED_BUILD_INPUTS = [
     // backends regenerate their stubs from it via `make protogen-go`, the C++
     // CMakeLists compile backend.pb.cc from it, and the Rust crate Makefile
     // copies it in before build.rs runs.
-    //
-    // Which is why this rule is always/always, and why it is the single most
-    // expensive entry in the table: 417 Linux plus 56 Darwin builds. It fires
-    // on 1.3% of commits (10 of 767 over six months), and every one of those
-    // ten was purely additive. filterMatrix() suppresses this rule for an
-    // additive-only diff, so `id` exists to identify it there.
-    id: PROTO_RULE_ID,
-    matches: file => file === BACKEND_PROTO_FILE,
+    matches: file => file === "backend/backend.proto",
     linux: always,
     darwin: always,
   },
@@ -489,19 +362,8 @@ export function filterMatrix({
   includesDarwin,
   changedFiles,
   previousMatrix,
-  protoRevisions,
 }) {
-  // An additive-only backend.proto edit invalidates no existing image, so drop
-  // its always/always rule. Every other matched rule still applies: a PR that
-  // touches the proto and scripts/build/ is still a full rebuild.
-  const protoAdditiveOnly =
-    changedFiles.includes(BACKEND_PROTO_FILE) &&
-    !!protoRevisions &&
-    protoChangeIsAdditive(protoRevisions.previous, protoRevisions.current);
-
-  const sharedRules = matchedSharedRules(changedFiles).filter(
-    rule => !(rule.id === PROTO_RULE_ID && protoAdditiveOnly)
-  );
+  const sharedRules = matchedSharedRules(changedFiles);
 
   const matrixFileChanged = changedFiles.includes(BACKEND_MATRIX_FILE);
   // The matrix file changed but we could not resolve what it used to say (API

scripts/lib/backend-filter_test.mjs

@@ -385,146 +385,3 @@ test("an unavailable previous matrix conservatively rebuilds everything", () =>
   assert.equal(filtered.length, includes.length);
   assert.equal(filteredDarwin.length, includesDarwin.length);
 });
-
-// --- backend.proto: additive changes must not rebuild the world -------------
-//
-// backend/backend.proto is consumed by every language, so the SHARED_BUILD_INPUTS
-// rule for it is always/always: a full 417-entry Linux matrix plus all 56 Darwin
-// entries. It fires on 1.3% of commits, and in practice every one of those has
-// been purely additive (a new field with an unused number, a new message, a new
-// RPC). Adding `bool cache_prompt = 8;` (PR #11158) cannot change the behavior of
-// a backend that never reads it, yet it rebuilt all 473 images.
-//
-// So the rule becomes content-aware rather than path-aware, using the same shape
-// as previousMatrix above: changed-backends.js resolves the base revision and
-// hands the two texts in, and everything here stays pure.
-
-const protoWith = body => `
-syntax = "proto3";
-
-package backend;
-
-service Backend {
-  rpc Health(HealthMessage) returns (Reply) {}
-  rpc Predict(PredictOptions) returns (Reply) {}
-}
-
-message HealthMessage {}
-
-message PredictOptions {
-${body}
-}
-`;
-
-const BASE_FIELDS = `  string Prompt = 1;
-  int32 Tokens = 2;
-  bool UseTokenizerTemplate = 3;`;
-
-const runProto = (previous, current) =>
-  filterMatrix({
-    includes,
-    includesDarwin,
-    changedFiles: ["backend/backend.proto"],
-    protoRevisions: { previous, current },
-  });
-
-test("an added proto field rebuilds nothing", () => {
-  // The real PR #11158 diff: six added lines, one new field, unused number.
-  const { filtered, filteredDarwin, changedBackends } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`${BASE_FIELDS}\n  bool cache_prompt = 8;`)
-  );
-
-  assert.deepEqual(filtered, []);
-  assert.deepEqual(filteredDarwin, []);
-  assert.equal(changedBackends.size, 0);
-});
-
-test("an added proto message and RPC rebuild nothing", () => {
-  const current = protoWith(BASE_FIELDS).replace(
-    "message HealthMessage {}",
-    "message HealthMessage {}\n\nmessage ScoreRequest {\n  string Text = 1;\n}"
-  ).replace(
-    "  rpc Predict(PredictOptions) returns (Reply) {}",
-    "  rpc Predict(PredictOptions) returns (Reply) {}\n  rpc Score(ScoreRequest) returns (Reply) {}"
-  );
-
-  const { filtered, filteredDarwin } = runProto(protoWith(BASE_FIELDS), current);
-
-  assert.deepEqual(filtered, []);
-  assert.deepEqual(filteredDarwin, []);
-});
-
-test("a removed proto field rebuilds every backend on every OS", () => {
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`  string Prompt = 1;\n  bool UseTokenizerTemplate = 3;`)
-  );
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});
-
-test("a renumbered proto field rebuilds every backend on every OS", () => {
-  // Wire-incompatible: an old backend reading field 2 gets nothing.
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`  string Prompt = 1;\n  int32 Tokens = 9;\n  bool UseTokenizerTemplate = 3;`)
-  );
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});
-
-test("a retyped proto field rebuilds every backend on every OS", () => {
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`  string Prompt = 1;\n  int64 Tokens = 2;\n  bool UseTokenizerTemplate = 3;`)
-  );
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});
-
-test("a renamed proto field rebuilds every backend on every OS", () => {
-  // Same number and type, but every generated accessor changes name.
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`  string Prompt = 1;\n  int32 MaxTokens = 2;\n  bool UseTokenizerTemplate = 3;`)
-  );
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});
-
-test("a removed proto RPC rebuilds every backend on every OS", () => {
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(BASE_FIELDS).replace(
-      "  rpc Predict(PredictOptions) returns (Reply) {}\n",
-      ""
-    )
-  );
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});
-
-test("a comment-only proto change rebuilds nothing", () => {
-  const { filtered, filteredDarwin } = runProto(
-    protoWith(BASE_FIELDS),
-    protoWith(`  string Prompt = 1;\n  // how many tokens to emit\n  int32 Tokens = 2;\n  bool UseTokenizerTemplate = 3;`)
-  );
-
-  assert.deepEqual(filtered, []);
-  assert.deepEqual(filteredDarwin, []);
-});
-
-test("unresolvable proto revisions conservatively rebuild everything", () => {
-  // Same posture as the previousMatrix fallback: if we cannot resolve what the
-  // proto used to say, we must not claim the change was additive.
-  const { filtered, filteredDarwin } = runProto(null, protoWith(BASE_FIELDS));
-
-  assert.equal(filtered.length, includes.length);
-  assert.equal(filteredDarwin.length, includesDarwin.length);
-});