In distributed mode the Open Responses store is process-local: a
sync.OnceValue over a map behind an RWMutex. With several frontend
replicas behind a round-robin load balancer, every request that lands on
a replica other than the creator misses.
Measured on a live 2-replica cluster (#10993): the same response id
returns 200 on the creating replica and 404 on its peer, and a cancel on
the peer returns 404 without ever invoking CancelFunc, so generation runs
to completion on the other replica while the caller is told the response
does not exist. previous_response_id chaining fails through the same
lookup.
Split the state by what can actually cross a process boundary:
- Replicated: response metadata (request, response resource, owner,
expiry, stream/background flags) via syncstate.SyncedMap, the same
component finetune, quantization and agent tasks already use. A local
miss in Get/FindItem now falls back to it and returns a read-only
remote view, so polling and chaining resolve on any replica.
- Delegated: cancellation. context.CancelFunc is a function pointer and
exists only in the creating process, so a cancel that lands elsewhere
is broadcast on responses.<id>.cancel and applied by whichever replica
holds the function. The broadcast is fire-and-forget rather than
request/reply: if the owner crashed or was scaled down nobody answers,
and the handler must not block on a reply that will never come. The
replicated status moves to cancelled either way, which is truthful,
since a dead owner's generation died with its process.
- Refused: streaming resume. The resume buffer is a byte log plus a live
notification channel and cannot be replicated without shipping every
token over the bus. A resume that reaches the wrong replica now returns
HTTP 409 naming the owning replica via the new ErrResponseNotLocal,
instead of an empty event list that looks like a finished stream. It is
deliberately distinct from ErrOffsetLost, which means the owner's
buffer evicted the requested events.
Standalone deployments never call EnableDistributed and keep exactly the
previous process-local behaviour.
Fixes#10993
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
