The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
PR #10454 added a `cancellable bool` parameter to GalleryStore.UpdateProgress
but missed two callers under tests/e2e/distributed, breaking the build on
master (golangci-lint and tests-e2e-backend both failed to compile with
"not enough arguments in call to ... UpdateProgress").
Pass cancellable=true (both ops are downloading installs, which are
cancellable) and assert the flag is persisted, exercising the new behavior.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat: add distributed mode (experimental)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix data races, mutexes, transactions
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix events and tool stream in agent chat
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* use ginkgo
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(cron): compute correctly time boundaries avoiding re-triggering
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not flood of healthy checks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* do not list obvious backends as text backends
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* tests fixups
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactoring and consolidation
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* Drop redundant healthcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* enhancements, refactorings
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto