syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto